在Keras中處理分詞任務通常需要使用Tokenizer類,該類用于將文本數據轉換為整數序列。以下是處理分詞任務的主要步驟:
from keras.preprocessing.text import Tokenizer
tokenizer = Tokenizer()
tokenizer.fit_on_texts(train_texts)
train_sequences = tokenizer.texts_to_sequences(train_texts)
test_sequences = tokenizer.texts_to_sequences(test_texts)
from keras.preprocessing.sequence import pad_sequences
max_len = 100
train_sequences_padded = pad_sequences(train_sequences, maxlen=max_len)
test_sequences_padded = pad_sequences(test_sequences, maxlen=max_len)
from keras.models import Sequential
from keras.layers import Embedding, LSTM, Dense
model = Sequential()
model.add(Embedding(input_dim=num_words, output_dim=embedding_dim, input_length=max_len))
model.add(LSTM(units=64))
model.add(Dense(units=num_classes, activation='softmax'))
model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])
model.fit(train_sequences_padded, train_labels, epochs=10, batch_size=32)
predictions = model.predict(test_sequences_padded)
這些是處理分詞任務的基本步驟,你可以根據具體的需求和數據集進行調整和擴展。