Categorical variables which take on discrete values may need special treatment and preprocessing before you can feed them into a machine learning module. This is because machine learning modules can only accept numeric data.
There are many ways in which categorical data can be encoded as numbers. Simply mapping categories to numbers is one of them, but another common way is one-hot encoding. Consider the days of the week, Sunday through Saturday. One-hot encoding will assume that each of these days represent one position in an array, with Sunday at position zero and Saturday at position six. If you want to have a numeric vector which represents Monday, it will have the number one at the position Monday. The
import pandas as pd
print(pd.__version__)
Download link: http://roycekimmons.com/system/generate_data.php?dataset=exams&n=100
exam_data = pd.read_csv('../data/exams.csv', quotechar='"')
exam_data.head(10)
math_average = exam_data['math score'].mean()
reading_average = exam_data['reading score'].mean()
writing_average = average = exam_data['writing score'].mean()
print('Math Avg: ', math_average)
print('Reading Avg: ', reading_average)
print('Writing Avg: ', writing_average)
Apply scaling on the test scores to express them in terms of z-score
Z-score is the expression of a value in terms of the number of standard deviations from the mean
The effect is to give a score which is relative to the the distribution of values for that column
from sklearn import preprocessing
exam_data[['math score']] = preprocessing.scale(exam_data[['math score']])
exam_data[['reading score']] = preprocessing.scale(exam_data[['reading score']])
exam_data[['writing score']] = preprocessing.scale(exam_data[['writing score']])
exam_data.head(10)
math_average = exam_data['math score'].mean()
reading_average = exam_data['reading score'].mean()
writing_average = average = exam_data['writing score'].mean()
print('Math Avg: ', math_average)
print('Reading Avg: ', reading_average)
print('Writing Avg: ', writing_average)
Convert text values to numbers. These can be used in the following situations:
le = preprocessing.LabelEncoder()
exam_data['gender'] = le.fit_transform(exam_data['gender'].astype(str))
type(exam_data['gender'][0])
exam_data.head(10)
le.classes_
pd.get_dummies(exam_data['race/ethnicity'])
exam_data = pd.get_dummies(exam_data, columns=['race/ethnicity'])
exam_data.head(10)
exam_data = pd.get_dummies(exam_data, columns=['parental level of education',
'lunch',
'test preparation course'])
exam_data.head(10)