In order to preserve frequency information of individual words in text, you can use frequency-based embedding.
from sklearn.feature_extraction.text import CountVectorizer
corpus = ['This is the first document.',
'This is the second document.',
'Third document. Document number three',
'Number four. To repeat, number four']
The bag of words is a sparse 4 by 12 matrix, four documents and a total vocabulary of 12 words.
vectorizer = CountVectorizer()
bag_of_words = vectorizer.fit_transform(corpus)
bag_of_words
print(bag_of_words)
You can access the ID that corresponds to a particular word by calling vectorizer.vocabulary.get method on that word. The word document corresponds to ID zero.
vectorizer.vocabulary_.get('document')
vectorizer.vocabulary_
import pandas as pd
print(pd.__version__)
pd.DataFrame(bag_of_words.toarray(), columns=vectorizer.get_feature_names())
Every word in every document is associated with a score. Every document has a unique ID. Every word has a unique ID as well, and a document ID word ID combination is associated with a score.
from sklearn.feature_extraction.text import TfidfVectorizer
vectorizer = TfidfVectorizer()
bag_of_words = vectorizer.fit_transform(corpus)
print(bag_of_words)
vectorizer.vocabulary_.get('document')
pd.DataFrame(bag_of_words.toarray(), columns=vectorizer.get_feature_names())
vectorizer.vocabulary_
from sklearn.feature_extraction.text import HashingVectorizer
vectorizer = HashingVectorizer(n_features=8)
feature_vector = vectorizer.fit_transform(corpus)
print(feature_vector)