Drawing Conclusions Example

Let's address a question we posed with this cancer data earlier in the lesson - does the size of a tumor affect its malignancy? We can use descriptive statistics and visualizations to help us.

In [1]:
import pandas as pd

df = pd.read_csv('cancer_data_edited.csv')
df.head()
Out[1]:
id diagnosis radius texture perimeter area smoothness compactness concavity concave_points symmetry fractal_dimension
0 842302 M 17.99 19.293431 122.80 1001.0 0.118400 0.27760 0.3001 0.14710 0.2419 0.07871
1 842517 M 20.57 17.770000 132.90 1326.0 0.084740 0.07864 0.0869 0.07017 0.1812 0.05667
2 84300903 M 19.69 21.250000 130.00 1203.0 0.109600 0.15990 0.1974 0.12790 0.2069 0.05999
3 84348301 M 11.42 20.380000 77.58 386.1 0.096087 0.28390 0.2414 0.10520 0.2597 0.09744
4 84358402 M 20.29 14.340000 135.10 1297.0 0.100300 0.13280 0.1980 0.10430 0.1809 0.05883

Selecting Data with Masks

In order to do this analysis, we'd ideally compare sizes of tumors that are benign and malignant. We can use masks to select all rows in the dataframe that were diagnosed as malignant.

In [2]:
# Create new dataframe with only malignant tumors
df_m = df[df['diagnosis'] == 'M']
df_m.head()
Out[2]:
id diagnosis radius texture perimeter area smoothness compactness concavity concave_points symmetry fractal_dimension
0 842302 M 17.99 19.293431 122.80 1001.0 0.118400 0.27760 0.3001 0.14710 0.2419 0.07871
1 842517 M 20.57 17.770000 132.90 1326.0 0.084740 0.07864 0.0869 0.07017 0.1812 0.05667
2 84300903 M 19.69 21.250000 130.00 1203.0 0.109600 0.15990 0.1974 0.12790 0.2069 0.05999
3 84348301 M 11.42 20.380000 77.58 386.1 0.096087 0.28390 0.2414 0.10520 0.2597 0.09744
4 84358402 M 20.29 14.340000 135.10 1297.0 0.100300 0.13280 0.1980 0.10430 0.1809 0.05883

Let's break down how we got df_m.

df['diagnosis'] == 'M' returns a Pandas Series of booleans indicating whether the value in the diagnosis columns is equal to M.

In [3]:
mask = df['diagnosis'] == 'M'
print(mask)
0       True
1       True
2       True
3       True
4       True
5       True
6       True
7       True
8       True
9       True
10      True
11      True
12      True
13      True
14      True
15      True
16      True
17      True
18      True
19     False
20     False
21     False
22      True
23      True
24      True
25      True
26      True
27      True
28      True
29      True
       ...  
534    False
535    False
536    False
537    False
538    False
539    False
540    False
541    False
542    False
543    False
544    False
545    False
546    False
547    False
548    False
549    False
550    False
551    False
552    False
553    False
554    False
555    False
556    False
557     True
558     True
559     True
560     True
561     True
562     True
563    False
Name: diagnosis, Length: 564, dtype: bool

And indexing the dataframe with this mask will return all rows where the value in mask is True (ie. where diagnosis == 'M').

In [4]:
df_m = df[mask]
df_m
Out[4]:
id diagnosis radius texture perimeter area smoothness compactness concavity concave_points symmetry fractal_dimension
0 842302 M 17.99 19.293431 122.80 1001.0 0.118400 0.27760 0.30010 0.14710 0.241900 0.07871
1 842517 M 20.57 17.770000 132.90 1326.0 0.084740 0.07864 0.08690 0.07017 0.181200 0.05667
2 84300903 M 19.69 21.250000 130.00 1203.0 0.109600 0.15990 0.19740 0.12790 0.206900 0.05999
3 84348301 M 11.42 20.380000 77.58 386.1 0.096087 0.28390 0.24140 0.10520 0.259700 0.09744
4 84358402 M 20.29 14.340000 135.10 1297.0 0.100300 0.13280 0.19800 0.10430 0.180900 0.05883
5 843786 M 12.45 15.700000 82.57 477.1 0.127800 0.17000 0.15780 0.08089 0.208700 0.07613
6 844359 M 18.25 19.980000 119.60 1040.0 0.094630 0.10900 0.11270 0.07400 0.181091 0.05742
7 84458202 M 13.71 20.830000 90.20 577.9 0.118900 0.16450 0.09366 0.05985 0.219600 0.07451
8 844981 M 13.00 21.820000 87.50 519.8 0.127300 0.19320 0.18590 0.09353 0.235000 0.07389
9 84501001 M 12.46 24.040000 83.97 475.9 0.118600 0.23960 0.22730 0.08543 0.203000 0.08243
10 845636 M 16.02 23.240000 102.70 797.8 0.082060 0.06669 0.03299 0.03323 0.152800 0.05697
11 84610002 M 15.78 17.890000 103.60 781.0 0.097100 0.12920 0.09954 0.06606 0.184200 0.06082
12 846226 M 19.17 24.800000 132.40 1123.0 0.097400 0.24580 0.20650 0.11180 0.239700 0.07800
13 846381 M 15.85 19.293431 103.70 782.7 0.084010 0.10020 0.09938 0.05364 0.184700 0.05338
14 84667401 M 13.73 22.610000 93.60 578.3 0.113100 0.22930 0.21280 0.08025 0.206900 0.07682
15 84799002 M 14.54 27.540000 96.73 658.8 0.113900 0.15950 0.16390 0.07364 0.181091 0.07077
16 848406 M 14.68 20.130000 94.74 684.5 0.098670 0.07200 0.07395 0.05259 0.158600 0.05922
17 84862001 M 16.13 19.293431 108.10 798.8 0.117000 0.20220 0.17220 0.10280 0.216400 0.07356
18 849014 M 19.81 22.150000 130.00 1260.0 0.098310 0.10270 0.14790 0.09498 0.158200 0.05395
22 8511133 M 15.34 14.260000 102.50 704.4 0.107300 0.21350 0.20770 0.09756 0.252100 0.07032
23 851509 M 21.16 23.040000 137.20 1404.0 0.094280 0.10220 0.10970 0.08632 0.176900 0.05278
24 852552 M 16.65 21.380000 110.00 904.6 0.112100 0.14570 0.15250 0.09170 0.199500 0.06330
25 852631 M 17.14 16.400000 116.00 912.7 0.096087 0.22760 0.22290 0.14010 0.304000 0.07413
26 852763 M 14.58 21.530000 97.41 644.8 0.105400 0.18680 0.14250 0.08783 0.225200 0.06924
27 852781 M 18.61 20.250000 122.10 1094.0 0.094400 0.10660 0.14900 0.07731 0.169700 0.05699
28 852973 M 15.30 25.270000 102.40 732.4 0.108200 0.16970 0.16830 0.08751 0.192600 0.06540
29 853201 M 17.57 15.050000 115.00 955.1 0.096087 0.11570 0.09875 0.07953 0.173900 0.06149
30 853401 M 18.63 25.110000 124.80 1088.0 0.096087 0.18870 0.23190 0.12440 0.218300 0.06197
31 853612 M 11.84 18.700000 77.93 440.6 0.110900 0.15160 0.12180 0.05182 0.230100 0.07799
32 85382601 M 17.02 19.293431 112.80 899.3 0.119700 0.14960 0.24170 0.12030 0.224800 0.06382
... ... ... ... ... ... ... ... ... ... ... ... ...
438 909445 M 17.27 25.420000 112.40 928.8 0.083310 0.11090 0.12040 0.05736 0.146700 0.05407
441 9110127 M 18.03 16.850000 117.50 990.0 0.089470 0.12320 0.10900 0.06254 0.172000 0.05780
443 9110732 M 17.75 28.030000 117.30 981.6 0.099970 0.13140 0.16980 0.08293 0.171300 0.05916
446 911157302 M 21.10 20.520000 138.10 1384.0 0.096840 0.11750 0.15720 0.11550 0.155400 0.05661
448 9111805 M 19.59 25.000000 127.70 1191.0 0.103200 0.09871 0.16550 0.09063 0.166300 0.05391
457 911296201 M 17.08 27.150000 111.20 930.9 0.098980 0.11100 0.10070 0.06431 0.179300 0.06281
458 911296202 M 27.42 26.270000 186.90 2501.0 0.108400 0.19880 0.36350 0.16890 0.206100 0.05623
465 9113538 M 17.60 23.330000 119.00 980.5 0.092890 0.20040 0.21360 0.10020 0.169600 0.07369
476 911916 M 16.25 19.510000 109.80 815.8 0.102600 0.18930 0.22360 0.09194 0.215100 0.06578
484 913505 M 19.44 18.820000 128.10 1167.0 0.108900 0.14480 0.22560 0.11940 0.182300 0.06115
488 914062 M 18.01 20.560000 118.40 1007.0 0.100100 0.12890 0.11700 0.07762 0.211600 0.06077
494 914769 M 18.49 17.520000 121.30 1068.0 0.101200 0.13170 0.14910 0.09183 0.183200 0.06697
495 91485 M 20.59 21.240000 137.80 1320.0 0.096087 0.16440 0.21880 0.11210 0.184800 0.06222
497 91504 M 13.82 24.490000 92.33 595.9 0.116200 0.16810 0.13570 0.06759 0.227500 0.07237
499 915143 M 23.09 19.830000 152.10 1682.0 0.093420 0.12750 0.16760 0.10030 0.150500 0.05484
505 915460 M 15.46 23.950000 103.80 731.3 0.118300 0.18700 0.20300 0.08520 0.180700 0.07083
508 915691 M 13.40 20.520000 88.64 556.7 0.110600 0.14690 0.14450 0.08172 0.211600 0.07325
510 91594602 M 15.05 19.070000 97.26 701.9 0.092150 0.08597 0.07486 0.04335 0.156100 0.05915
512 916799 M 18.31 20.580000 120.80 1052.0 0.106800 0.12480 0.15690 0.09451 0.186000 0.05941
513 916838 M 19.89 20.260000 130.50 1214.0 0.103700 0.13100 0.14110 0.09431 0.180200 0.06188
517 91762702 M 24.63 21.600000 165.50 1841.0 0.103000 0.21060 0.23100 0.14710 0.199100 0.06739
529 91930402 M 20.47 20.670000 134.70 1299.0 0.096087 0.13130 0.15230 0.10150 0.216600 0.05419
531 919555 M 20.55 20.860000 137.80 1308.0 0.104600 0.17390 0.20850 0.13220 0.212700 0.06251
532 91979701 M 14.27 22.550000 93.77 629.8 0.103800 0.11540 0.14630 0.06139 0.192600 0.05982
557 925622 M 15.22 30.620000 103.40 716.9 0.104800 0.20870 0.25500 0.09429 0.181091 0.07152
558 926125 M 20.92 25.090000 143.00 1347.0 0.109900 0.22360 0.31740 0.14740 0.214900 0.06879
559 926424 M 21.56 22.390000 142.00 1479.0 0.111000 0.11590 0.24390 0.13890 0.172600 0.05623
560 926682 M 20.13 28.250000 131.20 1261.0 0.097800 0.10340 0.14400 0.09791 0.175200 0.05533
561 926954 M 16.60 28.080000 108.30 858.1 0.084550 0.10230 0.09251 0.05302 0.159000 0.05648
562 927241 M 20.60 29.330000 140.10 1265.0 0.117800 0.27700 0.35140 0.15200 0.239700 0.07016

210 rows × 12 columns

Now that we have all the malignant tumors together in a dataframe, let's see summary statistics about the area feature, which offers a good metric for size.

In [5]:
# Display summary statistics for area of malignant tumors
df_m['area'].describe()
Out[5]:
count     210.000000
mean      976.582857
std       365.494289
min       361.600000
25%       706.850000
50%       932.000000
75%      1200.750000
max      2501.000000
Name: area, dtype: float64

Let's do the same for all the benign tumors.

In [6]:
# Create new dataframe with only benign tumors
df_b = df[df['diagnosis'] == 'B']

# Display summary statistics for area of benign tumors
df_b['area'].describe()
Out[6]:
count    354.000000
mean     462.712429
std      134.769158
min      143.500000
25%      374.975000
50%      458.150000
75%      551.550000
max      992.100000
Name: area, dtype: float64
In [7]:
print('The mean area of malignant tumors is {0:.4f} while that of benign \
tumors is {1:.4f}.'.format(df_m['area'].mean(), df_b['area'].mean()))
The mean area of malignant tumors is 976.5829 while that of benign tumors is 462.7124.

Although summary statistics like the mean are helpful, it would be nice to be able to compare the distributions of the areas of malignant and benign tumors visually. Let's see a simple example of using matplotlib to create histograms for both distributions on the same plot.

(We'll learn how to use matplotlib in the next lesson.)

In [9]:
import matplotlib.pyplot as plt
%matplotlib inline

# Plot histogram of benign and malignant tumor areas on the same axes
fig, ax = plt.subplots(figsize=(8, 6))
ax.hist(df_b['area'], alpha=0.5, label='benign')
ax.hist(df_m['area'], alpha=0.5, label='malignant')
ax.set_title('Distributions of Benign and Malignant Tumor Areas')
ax.set_xlabel('Area')
ax.set_ylabel('Count')
ax.legend(loc='upper right')
plt.show()

The visual above suggests that there is a difference between the distribution of areas for benign and malignant tumors. We don't yet have the tools to conclude that these distributions are different or whether the size definitely affects a tumor's malignancy. However, we can observe from summary statistics and these histograms that malignant tumors are generally larger in size than benign tumors.