Scraping NFL Historical Vegas Lines 2014 - Forward

This notebook shows how to scrape Pro Football Reference historical Vegas lines. Comments start with # and describe each step. Feel free to email any questions/comments to Adam@unintelligible.xyz or Tweet me @Gitzalytics. Thanks!

In [1]:
#Import Packages
import requests
from bs4 import BeautifulSoup
import pandas as pd
In [2]:
#Define the function to take a team and a year then parse the team's season's Vegas Line data
def processTeamYear(team, year):
    #Grab Team/Year Page
    r=requests.get("https://www.pro-football-reference.com/{0}/{1}_lines.htm".format(team, str(year)))
    
    #Put Team/Year Page in BeautifulSoup, Pythons html parser
    soup = BeautifulSoup(r.text, 'html.parser')
    
    #Start an empty list where you'll put all the data from this page
    teamYearList = []

    #Start a weeknum count at 0
    weeknum = 0
    
    #Go through each row(week) in the table
    for i in soup.find_all('tr'):
        #Start an empty dictionary to put the info from the row
        teamWeekDict ={}

        #Set the games metadata up
        teamWeekDict['team'] = team
        teamWeekDict['gameNum'] = weeknum
        teamWeekDict['year'] =year
        weeknum = weeknum +1

        #For each column in the row put an entry into the row's dictionary like {label:value}
        for cell in i.find_all('td'):
            teamWeekDict[cell['data-stat']] = cell.text
        #This is because I hacked this together in one night, probably could change the logic so it doesn't need this
        if  teamWeekDict['gameNum'] == 0:
            pass
        
        #Put the week's data in the team year list
        else:    
            teamYearList.append(teamWeekDict)
    
    #Send the team year list back as the output of the function
    return teamYearList
In [3]:
#This is where the action happens

#Get the PFR Team page
r = requests.get("https://www.pro-football-reference.com/teams/")

#Throw it in the html parser
soup = BeautifulSoup(r.text, 'html.parser')

#Initiate an empty list to take all game data
masterList = []

#For each row in the table of teams
for i in soup.find_all('tr'):
    #Find the header cell
    for cell in i.find_all('th'):
        #For each team find where the link to the team is 
        for link in cell.find_all('a'):
            #For each year from 2014 to today for each team
            for year in range(2014,2019):
                #Go through each game in the teamyear list provided by the function and place it in the master list
                for game in processTeamYear(link.get('href'), year):
                    masterList.append(game)
In [4]:
#Put the master list in python's data tool, Pandas
df  = pd.DataFrame(masterList)
In [5]:
#Check it worked
df.head()
Out[5]:
cover gameNum game_result opp ou_result over_under team vegas_line year
0 Lost 1 W, 18-17 SDG Under 45.0 /teams/crd/ -3.0 2014
1 Won 2 W, 25-14 @NYG Under 42.5 /teams/crd/ -2.0 2014
2 Won 3 W, 23-14 SFO Under 42.0 /teams/crd/ 3.0 2014
3 Lost 4 L, 20-41 @DEN Over 48.0 /teams/crd/ 7.5 2014
4 Won 5 W, 30-20 WAS Over 47.0 /teams/crd/ -5.5 2014
In [6]:
#Check how many games we've got
df[df.year ==2018].shape
Out[6]:
(96, 9)
In [7]:
#Describe the data set
df.describe(include='all')
Out[7]:
cover gameNum game_result opp ou_result over_under team vegas_line year
count 2232 2232.000000 2232 2232 2232 2232 2232 2232 2232.000000
unique 3 NaN 885 68 3 45 32 65 NaN
top Won NaN W, 27-24 @NWE Under 44.0 /teams/nwe/ -3.0 NaN
freq 1090 NaN 20 40 1120 138 78 182 NaN
mean NaN 8.578853 NaN NaN NaN NaN NaN NaN 2015.607527
std NaN 4.962618 NaN NaN NaN NaN NaN NaN 1.205879
min NaN 1.000000 NaN NaN NaN NaN NaN NaN 2014.000000
25% NaN 4.000000 NaN NaN NaN NaN NaN NaN 2015.000000
50% NaN 8.000000 NaN NaN NaN NaN NaN NaN 2016.000000
75% NaN 13.000000 NaN NaN NaN NaN NaN NaN 2017.000000
max NaN 19.000000 NaN NaN NaN NaN NaN NaN 2018.000000
In [8]:
#Send it to CSV
df.to_csv('nflLines.csv')