Let's address a question we posed with this cancer data earlier in the lesson - does the size of a tumor affect its malignancy? We can use descriptive statistics and visualizations to help us.
import pandas as pd
df = pd.read_csv('cancer_data_edited.csv')
df.head()
In order to do this analysis, we'd ideally compare sizes of tumors that are benign and malignant. We can use masks to select all rows in the dataframe that were diagnosed as malignant.
# Create new dataframe with only malignant tumors
df_m = df[df['diagnosis'] == 'M']
df_m.head()
Let's break down how we got df_m
.
df['diagnosis'] == 'M'
returns a Pandas Series of booleans indicating whether the value in the diagnosis
columns is equal to M
.
mask = df['diagnosis'] == 'M'
print(mask)
And indexing the dataframe with this mask will return all rows where the value in mask
is True (ie. where diagnosis == 'M'
).
df_m = df[mask]
df_m
Now that we have all the malignant tumors together in a dataframe, let's see summary statistics about the area
feature, which offers a good metric for size.
# Display summary statistics for area of malignant tumors
df_m['area'].describe()
Let's do the same for all the benign tumors.
# Create new dataframe with only benign tumors
df_b = df[df['diagnosis'] == 'B']
# Display summary statistics for area of benign tumors
df_b['area'].describe()
print('The mean area of malignant tumors is {0:.4f} while that of benign \
tumors is {1:.4f}.'.format(df_m['area'].mean(), df_b['area'].mean()))
Although summary statistics like the mean are helpful, it would be nice to be able to compare the distributions of the areas of malignant and benign tumors visually. Let's see a simple example of using matplotlib to create histograms for both distributions on the same plot.
(We'll learn how to use matplotlib in the next lesson.)
import matplotlib.pyplot as plt
%matplotlib inline
# Plot histogram of benign and malignant tumor areas on the same axes
fig, ax = plt.subplots(figsize=(8, 6))
ax.hist(df_b['area'], alpha=0.5, label='benign')
ax.hist(df_m['area'], alpha=0.5, label='malignant')
ax.set_title('Distributions of Benign and Malignant Tumor Areas')
ax.set_xlabel('Area')
ax.set_ylabel('Count')
ax.legend(loc='upper right')
plt.show()
The visual above suggests that there is a difference between the distribution of areas for benign and malignant tumors. We don't yet have the tools to conclude that these distributions are different or whether the size definitely affects a tumor's malignancy. However, we can observe from summary statistics and these histograms that malignant tumors are generally larger in size than benign tumors.