from sklearn.datasets import fetch_20newsgroups
twenty_train = fetch_20newsgroups(subset='train', shuffle=True)
twenty_train.keys()
The return value from this function is a dictionary and these are the keys within the dictionary. The data key is what contains our training data.
print(twenty_train.data[0])
twenty_train.target_names
twenty_train.target
from sklearn.feature_extraction.text import CountVectorizer
count_vect = CountVectorizer()
X_train_counts = count_vect.fit_transform(twenty_train.data)
X_train_counts.shape
The output of the count vectorizer is a sparse matrix. Every word is identified uniquely using its document ID and its unique word ID and the frequency of the word in that document is specified. Here you can see the document ID, the word ID and the associated frequency.
print(X_train_counts[0])
This is different from TfidfVectorizer:
from sklearn.feature_extraction.text import TfidfTransformer
tfidf_transformer = TfidfTransformer()
X_train_tfidf = tfidf_transformer.fit_transform(X_train_counts)
X_train_tfidf.shape
Here you can see a mapping of document ID word ID and the corresponding tfidf score.
print(X_train_tfidf[0])
from sklearn.svm import LinearSVC
clf_svc = LinearSVC(penalty="l2", dual=False, tol=1e-3)
clf_svc.fit(X_train_tfidf, twenty_train.target)
from sklearn.pipeline import Pipeline
clf_svc_pipeline = Pipeline([
('vect', CountVectorizer()),
('tfidf', TfidfTransformer()),
('clf',LinearSVC(penalty="l2", dual=False, tol=0.001))
])
In our example:
clf_svc_pipeline.fit(twenty_train.data, twenty_train.target)
twenty_test = fetch_20newsgroups(subset='test', shuffle=True)
predicted = clf_svc_pipeline.predict(twenty_test.data)
Remember, there are 20 categories, so wild guesses will result in an accuracy of about 0.05
from sklearn.metrics import accuracy_score
acc_svm = accuracy_score(twenty_test.target, predicted)
acc_svm
clf_svc_pipeline = Pipeline([
('vect', CountVectorizer()),
('clf',LinearSVC(penalty="l2", dual=False, tol=0.001))
])
clf_svc_pipeline.fit(twenty_train.data, twenty_train.target)
predicted = clf_svc_pipeline.predict(twenty_test.data)
acc_svm = accuracy_score(twenty_test.target, predicted)
acc_svm