I've selected TMDb movie data for my analysis.
Apart from entertainment to audience - making a movie involves lot of decision to be taken in order to make it successful, when we say successful we mean making profit - popular movies in terms of vote_average can be related to being successful as well.
Below are the questions we will be answering by exploring the dataset:
import numpy as np
import pandas as pd
from datetime import datetime
import matplotlib.pyplot as plt
import seaborn as sns
%matplotlib inline
# using pandas to load csv file
df = pd.read_csv('tmdb-movies.csv')
df.head()
df.shape
We have 10,866 dataset meaning data of 10,866 movies and each movie have 21 features.
df.info()
Looking at the data and info, we can conclude that certain features doesn't really influence the questions that we are trying to answer. We can go ahead and drop those columns.
Afte performing some operation on data, we shall check the data by calling df.head(1)
and confirm it is working.
del_col = ['id', 'popularity', 'homepage', 'tagline', 'keywords', 'overview', 'vote_count', 'budget_adj', 'revenue_adj']
df.drop(del_col, axis=1, inplace=True)
# check for NaN
df.isnull().sum()
Removing movies which have missing genres
df.dropna(subset=['genres'], inplace=True)
Removing movies which doesn't have imdb_id
later we will splitting movie to different rows based on genres, this allows us to groupby using imdb_id and perform more analysis.
df.dropna(subset=['imdb_id'], inplace=True)
# check for NaN
df.isnull().sum()
Now the data looks good.
But we still need to extract more information from existing feature which might help us analyse data better.
Calculating profit
df['profit'] = df.apply(lambda x: x['revenue']-x['budget'], axis=1)
df.head(1)
Extracting month
# converting date to right format
df['release_date'] = pd.to_datetime(df['release_date'])
df['release_month'] = df.apply(lambda x: x.release_date.month, axis=1)
df.head(1)
Unique values for each feature in entire dataset
df.nunique()
Who directed the most movies
df['director'].value_counts()
Which production company made most movies
df['production_companies'].value_counts()
Let's see the finance of movies
df['budget'].describe()
df['revenue'].describe()
plt.title('financials of movies over the years')
# plt.legend(loc='upper left')
df.groupby(['release_year'])['budget'].mean().plot(figsize=(15,6), color='red', legend='Budget')
df.groupby(['release_year'])['revenue'].mean().plot(figsize=(15,6), color='blue', legend='Revenue')
df.groupby(['release_year'])['profit'].mean().plot(figsize=(15,6),color='green', legend='Profit');
len(df.query('revenue == 0 and budget == 0')), len(df.query('revenue != 0 or budget != 0'))
len(df.query('revenue != 0 and budget != 0')), len(df.query('revenue == 0 or budget == 0'))
We have substantial number of movie not having budget and revenue.
We can still take average of existing data of these feature and update them but the problem is that would affect our analysis because we are looking at data from 1960 to 2015 which has seen budget and revenue changed over years.
df.hist(figsize=(10,8));
df.query('revenue != 0 and budget != 0').hist(figsize=(10,8));
Let's fill in missing values with average and then check the budget, revenue and profit plot
Since fillna only works for NaN values, first replace all 0 budget and 0 revenue with Nan and then apply fillna
df['budget'] = df['budget'].apply(lambda x: np.NaN if x==0 else x)
df['revenue'] = df['revenue'].apply(lambda x: np.NaN if x==0 else x)
df.isnull().sum()
now we can see there are 5667 movies with NaN budget and 5985 movies with NaN revenue
df['budget'].fillna(df['budget'].mean(), inplace = True)
df['revenue'].fillna(df['revenue'].mean(), inplace = True)
Calculating profit again since we have updated budget and revenue with average value
df['profit'] = df.apply(lambda x: x['revenue']-x['budget'], axis=1)
plt.title('financials of movies over the years')
# plt.legend(loc='upper left')
df.groupby(['release_year'])['budget'].mean().plot(figsize=(15,6), color='red', legend='Budget')
df.groupby(['release_year'])['revenue'].mean().plot(figsize=(15,6), color='blue', legend='Revenue')
df.groupby(['release_year'])['profit'].mean().plot(figsize=(15,6),color='green', legend='Profit');
# Check distribution of ratings
df['vote_average'].describe()
Unique Genres
df.genres.value_counts()
Problem with genres is that we can't do any analysis involving it unless we split genres and create more rows for each movies having more genres seperated by pipe character
spread_df = pd.DataFrame()
for index in range(df.shape[0]):
original = df.iloc[index]
splited = original['genres'].split('|')
for s in splited:
temp_df = original.copy()
temp_df['genres'] = s
spread_df = spread_df.append(temp_df, ignore_index = True)
spread_df.shape
spread_df.head(1)
we ended up with only 20 genres as a result of above operation
spread_df['genres'].nunique()
check whether spreading of genres worked by grouping them and compare with original shape
spread_df.groupby(['imdb_id'])['genres'].sum()
df.shape
above shape comparison tells us that our spread worked.
Now we can check how many movies are released under each genre
spread_df.genres.value_counts()
Plot the genre for visual understanding
spread_df.genres.value_counts().plot(kind='bar', figsize=(10,7));
# Few other variants
# spread_df.genres.value_counts().plot(kind='bar', figsize=(10,7)).invert_xaxis();
# spread_df.genres.value_counts().plot(kind='bar', figsize=(10,7)).invert_yaxis();
vote average among genres
spread_df.groupby(['genres'])['vote_average'].mean().plot(kind='bar');
The above plot doesn't take into account for years, lets do the same analysis over years
What movies people rate high?
# spread_df.groupby(['release_year', 'genres'])['vote_average'].mean()
# plt.figure()
spread_df.groupby(['release_year', 'genres'])['vote_average'].mean().unstack().plot(figsize=(15,8));
let's break down above plot by just comparing two genres
spread_df.query('genres == "Adventure" or genres == "Horror"').groupby(['release_year', 'genres'])['vote_average'].mean().unstack().plot(figsize=(15,7));
looks like people started rating low for horror movies after 1987 and while adventure movies almost maintained their rating.
Let's see how movies are invested for same genre movies over the years
spread_df.query('genres == "Adventure" or genres == "Horror"').groupby(['release_year', 'genres'])['budget'].mean().unstack().plot(figsize=(15,5));
budget for Adventure movies increased after 1997 when compared to Horror
df.groupby(['release_month'])['profit'].mean().plot(kind='bar', figsize=(15,7));
movies released in June made more profit
Comparing revenue for same genre movies over the years
spread_df.query('genres == "Adventure" or genres == "Horror"').groupby(['release_year', 'genres'])['revenue'].mean().unstack().plot(figsize=(15,6));
looks like Adventure movies revenue is higher as well
Budget and Revenue among genres
spread_df.query('release_year == 2015').groupby(['genres'])['budget', 'revenue'].mean()
spread_df.query('release_year == 2015').groupby(['genres'])['budget'].mean().plot(kind='bar', alpha=0.5, color='red', figsize=(15,7))
spread_df.query('release_year == 2015').groupby(['genres'])['revenue'].mean().plot(kind='bar', alpha=0.5, color='blue', figsize=(15,7))
plt.legend();
spread_df.query('release_year == 2015').groupby(['genres'])['profit'].mean().plot(kind='bar', figsize=(15,7));
Adventure, Action, Animation, Fantasy, Science Fiction, Family movies are making good profit
df.query('release_year == 2015').groupby(['release_month'])['profit'].mean().plot(kind='bar', figsize=(15,7));
in 2015, December released movies made more profit just ahead of June which has overall lead when compared over the years.
df.runtime.describe()
have to remove outliers
Better way to drop is by applying condition and create new dataframe
runtime_df = df[df.runtime < 350]
plt.title('Runtime Distribution of all movies')
plt.xlabel('minutes')
runtime_df['runtime'].plot(kind='hist', figsize=(10, 6), bins=30);
most movie runtime is around 90 to 110 minutes.
df['release_month'].value_counts()
Plotting the above
plt.figure()
df['release_month'].value_counts().sort_index().plot(kind='bar', figsize=(12, 6));
Months = ['January', 'February', 'March', 'April', 'May', 'June', 'July', 'August', 'September', 'October', 'November', 'December']
plt.title('# of movies released by month')
plt.xlabel(Months, fontsize=12)
plt.show()
So looking at all the investigation done so far, if one were to make a movie I can recommend following: