LABS
by
Casey Fitzpatrick
America's Next Top (Statistical) Model - 2020¶
US presidential elections come but once every 4 years, and this one's a big one. The new president will help shape policies on the pandemic response, healthcare, the environment, the economy, and more. There are lots of people trying to predict what will happen. Can you top them?
In our newest competition, you are asked to predict the fraction of each state that will vote for each major candidate. You can use any data that is freely available to the public. Come election night (or election week... or election month), we'll see who's model had the most accurate vision for the country!
This competition allows the use of any publically available data, so in this post we'll walk through downloading some polling data, cleaning it up a bit, and using it to generate predictions for the 2016 and 2020 elections.
%matplotlib inline
from pathlib import Path
import re
import numpy as np
import pandas as pd
import us
pd.set_option("display.max_columns", 50)
pd.set_option("display.max_rows", 50)
What do the pollsters say?¶
First things first, we need some data. The bread and butter of election forecasting is polling data, though some believe that is changing rapidly. The forecasting website FiveThirtyEight is a great place to get polling data. Thanks to them, we can load the thousands of polls used in their 2016 and 2020 forecasts directly into pandas in just couple of lines. Scroll around to check out the columns!
polls_2016_url = "http://projects.fivethirtyeight.com/general-model/president_general_polls_2016.csv"
polls_2020_url = "https://projects.fivethirtyeight.com/polls-page/president_polls.csv"
polls_2016 = pd.read_csv(polls_2016_url)
polls_2020 = pd.read_csv(polls_2020_url)
polls_2016.head()
polls_2020.head()
The data schema has changed a bit between 2016 and 2020, so let's see what columns these polls have in common.
polls_2016.columns.intersection(polls_2020.columns)
We can use the state column to map into our submission format. But the FiveThirtyEight data uses the full state name for the state, and our submission format uses the state abbreviation. We'll want to
- Create a map between states and abbreviations using the
uslibrary - Subset the polling data to only those states in our submission format (removing any national polls or territories)
First, let's load our submission formats
DATA_DIR = Path("../data/processed/public/")
index_col = "state_abbreviation"
submission_format_2016 = pd.read_csv(
(DATA_DIR / "submission-format-2016.csv"),
index_col=index_col
)
submission_format_2020 = pd.read_csv(
(DATA_DIR / "submission-format-2020.csv"),
index_col=index_col
)
# indices are the same for 2016 and 2020
assert (submission_format_2016.index == submission_format_2020.index).all()
print(f"There are {len(submission_format_2016)} abbreviations") # includes D.C.
submission_format_2016.head()
submission_format_2020.head()
It will be our job to replace those 0.33 with some machine-learned predictions of our own!
First, since the FiveThirtyEight data uses the full state name in state, we'll create a map between full state names and abbreviations. This is easy using a fun little package called us.
states = {
state.name: state.abbr for state in (us.STATES + [us.states.DC])
if state.abbr in submission_format_2016.index # ok to check only 2016 since 2020 same
}
assert len(states) == len(submission_format_2016) == len(submission_format_2020)
states
Let's look at how many polls each state has. Some polls have multiple rows (in 2020) so we'll use the unique poll_id to count.
poll_counts = pd.DataFrame(
[
polls_2016.groupby("state").poll_id.count().rename("# 2016 Polls"),
polls_2020.groupby("state").poll_id.count().rename("# 2020 Polls")
]
).T.sort_index(ascending=False)
poll_counts["in_states_map"] = poll_counts.index.isin(states.keys())
poll_counts[poll_counts.in_states_map].plot.barh(figsize=(10, 20), title="Number of polls conducted")
It's interesting to note that Michigan and Wisconsin, two of the states that are said to have delievered Trump the presidency by about 80,000 votes in 2016, are being polled substantially more in 2020.
Now let's look at the polling data that doesn't appear in our submission formats
poll_counts[~poll_counts.in_states_map]
A couple of things stand out
- Nebraska and Maine poll both at the state and congressional district levels (
CD). This is because they award electoral votes based on statewide and district returns. Since we are ultimately concerned with statewide vote-shares, we will drop these polls for this benchmark. - National polls are grouped in here, apparently as
U.S.in 2016 andNaNin 2020. We'll drop those as well.
Then we will have a one-to-one mapping between our polling data and submission formats, since every poll will have a key in our state dictionary.
polls_2016 = polls_2016[polls_2016.state.isin(states.keys())]
polls_2020 = polls_2020[polls_2020.state.isin(states.keys())]
Now any state in our polling data is a state in our submission format.
assert polls_2016.state.isin(states.keys()).all()
assert polls_2020.state.isin(states.keys()).all()
Also note that we're only looking at presidental polling.
# different coding between 2016 and 2020
assert (polls_2016.branch == "President").all()
assert (polls_2020.office_type == "U.S. President").all()
Turning polls into features¶
In order to predict the vote percentage for a candidate in each state, we need state-level features. For each state, we have a different number of polls, conducted using a different methodology and with varying recency (polls are less often conducted in states that aren't up for grabs). We'll take a very simple approach to start and create the following features:
- Number of days to the election: This helps the model understand how recent the poll is (therefore, how useful its measures will be).
- FiveThirtyEight Pollster Grade: FiveThirtyEight puts a lot of work into rating pollsters they use to generate forecasts. This could be a useful feature in our models. (It certainly is in theirs!)
- Democrat percentage: The percentage of responds that say they will vote for the democratic candidate (For 2016, this is Clinton; for 2020, this is Biden).
- Republican percentage: The percentage of responds that say they will vote for the republican candidate (Trump).
- Other percentage: 1 - (republican percentage + democratic percentage). While 3rd party candidates had a important impact in 2016, they are playing less of a role in 2020 so we lump them all together for the 2020 model.
Prepare the FiveThirtyEight data¶
In order to create the above features, we'll need to make the 2016 and 2020 data consistent
- Make column names consistent
- Convert dates to
datetimedata types - Remove extraneous data
rename_2016_to_2020 = {
"enddate": "end_date",
"timestamp": "election_date",
"grade": "fte_grade",
"startdate": "start_date"
}
polls_2016 = polls_2016.rename(rename_2016_to_2020, axis=1)
# convert to datetime
for col in ["start_date", "end_date", "election_date"]:
polls_2016[col] = pd.to_datetime(polls_2016[col]).dt.date
polls_2020[col] = pd.to_datetime(polls_2020[col]).dt.date
# check that election dates are correct
assert (polls_2016.election_date == pd.to_datetime("2016-11-08")).all()
assert (polls_2020.election_date == pd.to_datetime("2020-11-03")).all()
For the 2016 election, FiveThirtyEight ran three different forecast models called
- Polls-only
- Polls-plus
- Now-cast
which we can see using the type column
polls_2016.type.value_counts(dropna=False)
In 2020, they are only running the Polls-plus model, as described in the model field described on their official data page on Github.
For consistency, we'll reduce the 2016 data to "type" == "polls-plus".
polls_2016 = polls_2016[polls_2016.type == "polls-plus"]
assert polls_2016.poll_id.nunique() == len(polls_2016)
We'll also want to use the 538 grades as a feature. The grading scheme is a bit different between 2016 and 2020 but we will map the union of both to ordinal values for use in all of our models.
all_grades = (
pd.concat([polls_2016.fte_grade, polls_2020.fte_grade])
.sort_values(ascending=False, na_position="first")
.unique()
)
grade_map = {grade: i for i, grade in enumerate(all_grades)}
grade_map
def prepare_polls(polls, earliest_poll_date, is_2020, sort_by="days_to_election", grade_map=grade_map):
polls["days_to_election"] = (polls.election_date - polls.end_date).dt.days
poll_start_mask = polls.end_date > pd.to_datetime(earliest_poll_date)
rows = []
for poll_id, subdf in polls[poll_start_mask].groupby("poll_id"):
assert len(subdf.days_to_election.unique()) == 1
assert len(subdf.state.unique()) == 1
assert len(subdf.fte_grade.unique()) == 1
days_to_election = list(subdf.days_to_election.unique()).pop()
state = list(subdf.state.unique()).pop()
fte_grade_num = grade_map[list(subdf.fte_grade.unique()).pop()]
if is_2020:
dem_frac = subdf.loc[subdf.candidate_party == "DEM", "pct"].mean() / 100
rep_frac = subdf.loc[subdf.candidate_party == "REP", "pct"].mean() / 100
else:
dem_frac = subdf.adjpoll_clinton.mean() / 100
rep_frac = subdf.adjpoll_trump.mean() / 100
other_frac = (100 - (dem_frac + rep_frac)) / 100
rows.append(
{
"state": state,
"days_to_election": days_to_election,
"dem_frac": dem_frac,
"rep_frac": rep_frac,
"other_frac": other_frac,
"fte_grade_num": fte_grade_num,
"poll_id": poll_id,
}
)
return pd.DataFrame(rows).sort_values(by=sort_by, ascending=True)
prepared_2020 = prepare_polls(
polls=polls_2020,
earliest_poll_date="2020-08-01",
is_2020=True
)
prepared_2020.head()
prepared_2016 = prepare_polls(
polls_2016,
earliest_poll_date="2016-08-01",
is_2020=False
)
prepared_2016.head()
Build model features using the last N polls¶
For each state, use the last N prepared polling data as the state features. Each row of our prepared data now corresponds to a unique poll.
assert prepared_2016.poll_id.nunique() == len(prepared_2016)
assert prepared_2020.poll_id.nunique() == len(prepared_2020)
But states can have many polls
assert not prepared_2016.state.nunique() == len(prepared_2016)
assert not prepared_2020.state.nunique() == len(prepared_2020)
We only want one prediction for a state, so for our first pass model features, each row will be a state and there will be N * len(feature_cols) columns, where N is the number of polls we choose to consider for model features, and feature_cols is a list of column names representing the features we'll consider in each poll. In cases where a state doesn't have enough polling data, we'll simply fillna.
While we're at it, we'll use the state mapping created above to rename states to their abbreviation, in line with our submission formats.
feature_cols = ["days_to_election", "dem_frac", "rep_frac", "other_frac", "fte_grade_num"]
def build_features_from_last_n_polls(polls, num_polls, feature_cols=feature_cols, state_map=states):
all_states_rows = {}
for state, subdf in polls.groupby("state"):
state_row = {}
subdf = subdf.sort_values(by="days_to_election").iloc[:min(len(subdf), num_polls)]
state_row.update(
{
f"poll_{i}_{col}": poll[col] for col in feature_cols
for i, (_, poll) in enumerate(subdf.iterrows())
}
)
all_states_rows[state_map[state]] = state_row
return (
pd.DataFrame.from_dict(all_states_rows, orient="index")
.fillna(0)
.rename_axis("state_abbreviation", axis="index")
.sort_index()
)
NUM_POLLS = 5
features_2020 = build_features_from_last_n_polls(
polls=prepared_2020,
num_polls=NUM_POLLS
)
features_2020.head()
features_2016 = build_features_from_last_n_polls(
polls=prepared_2016,
num_polls=NUM_POLLS
)
features_2016.head()
Our features are ready! Finally, before we model we'll load up our labels for 2016 (obviously, we do not yet have labels for 2020) and confirm thaat index is ordered as we expect.
results_2016 = pd.read_csv("../data/processed/public/returns-2016.csv", index_col="state_abbreviation")
assert (results_2016.index == features_2016.index).all()
assert (results_2016.index == submission_format_2016.index).all()
results_2016.head()
Time to do some forecasting!¶
Now it's time to make some predictions. We'll start off with a battle-tested out-of-the-box SciKit Learn regression model––the treeb-based GradientBoostingRegressor. These models are ensembles of classification and regression (CART) trees that are trained using gradient descent (of #deeplearning fame). We note that these are non-linear models, which extends their expressive power. Given the differences between 2016 and 2020 polling trends, a non-linear model is likely to be useful.
We'll also fit this model using the GridSearchCV, which will compare the cross-validated scores for different hyperparameter values on the model and choose the best one. Initially, we will vary the learning rate, the number of estimators used in the ensemble, and the number of features used to split nodes on the tree.
from sklearn.ensemble import GradientBoostingRegressor
from sklearn.model_selection import GridSearchCV
from sklearn.multioutput import MultiOutputRegressor
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import MinMaxScaler
clf = make_pipeline(
MinMaxScaler(),
GridSearchCV(
MultiOutputRegressor(GradientBoostingRegressor(random_state=0)),
dict(
estimator__learning_rate=[0.1, 0.01, 0.001],
estimator__n_estimators=[10, 100, 1000],
estimator__max_features=["auto", "sqrt", "log2"],
),
scoring="neg_mean_squared_error"
)
)
clf.fit(features_2016, results_2016)
Take note of the best parameters found during grid search.
clf["gridsearchcv"].best_params_
Listen to the voice of the people¶
Now that we have the model, we can predict data for 2016 and evaluate how good the fit of that model is. To do that, we'll predict the percentages that each candidate will receive in each state.
# predict vote percentages with our trained model
preds = clf.predict(features_2016)
# get submission format
preds_2016 = submission_format_2016.copy()
# fill in our predicted values and write to csv
preds_2016.iloc[:, :] = preds
preds_2016.to_csv("gbr-model-2016.csv")
preds_2016.head()
If we submit on DrivenData, we can see our score against the 2016 election:
fec_electoral_college_url = "https://www.fec.gov/documents/1890/federalelections2016.xlsx"
electoral_by_candidate = (
pd.read_excel(
fec_electoral_college_url,
sheet_name="Table 2. Electoral & Pop Vote",
skiprows=[0, 1]
)[["STATE", "ELECTORAL VOTE", "ELECTORAL VOTE.1"]]
.set_index("STATE")
.loc["AL": "WY"]
.fillna(0)
)
for col in electoral_by_candidate.columns:
electoral_by_candidate[col] = electoral_by_candidate[col].map(
lambda val: int((str(val).replace("*", ""))) # ignore the ** vals in data
)
electoral_college = (
electoral_by_candidate.sum(axis=1)
.to_frame()
.rename({0: "electoral_votes"}, axis=1)
)
electoral_college.head()
How did we do? Recall, it takes 270 electoral votes to win.
electoral_college.sort_values(by='STATE', inplace=True)
preds_2016.sort_index(inplace=True)
preds_2016['Dems'] = np.where(
preds_2016.Clinton > preds_2016.Trump,
electoral_college.electoral_votes,
0
)
preds_2016['Reps'] = np.where(
preds_2016.Clinton < preds_2016.Trump,
electoral_college.electoral_votes,
0
)
print("===== PREDICTED ELECTORAL VOTES FOR EACH PARTY 2016 =======\n")
print(preds_2016[['Dems', 'Reps']].sum())
print("\n===== ACTUAL ELECTORAL VOTES FOR EACH PARTY 2016 =======\n")
print("Dems", 227)
print("Reps", 304)
That's pretty good, but we may be overfitting and when it comes to predicting elections using polling data, as we all learned in 2016, past performance is not necessarily an indicator of future results!
2020¶
Feel the will of the people¶
Finally, it's time to see what we think about Trump v. Biden. We've got a model now to predict presidential election outcomes based poll results. We can use this model to predict the current presidential election.
# predict vote percentages with our trained model
preds = clf.predict(features_2020)
# get submission format
preds_2020 = submission_format_2020.copy()
# fill in our predicted values and write to csv
preds_2020.iloc[:, :] = preds
preds_2020.to_csv("gbr-model-2020.csv")
preds_2020.head()
We can see we now have predictions for every major candidate, for every state. We can submit this to DrivenData and mark it as our "Evaluation Submission" to indicate that these are our predictions for 2020:
But, one important question remains: who will win the election? As we saw above, that is subject to the rules of the Electoral College. The winner needs at least 270 electoral votes to become president. We can calculate that using data about how many electoral votes each state gets:
electoral_college.sort_values(by='STATE', inplace=True)
preds_2020.sort_index(inplace=True)
preds_2020['Dems'] = np.where(
preds_2020.Biden > preds_2020.Trump,
electoral_college.electoral_votes,
0
)
preds_2020['Reps'] = np.where(
preds_2020.Biden < preds_2020.Trump,
electoral_college.electoral_votes,
0
)
print("===== PREDICTED ELECTORAL VOTES FOR EACH PARTY 2020 =======\n")
print(preds_2020[['Dems', 'Reps']].sum())
print("\n===== ACTUAL ELECTORAL VOTES FOR EACH PARTY 2020 =======\n")
print("We don't know yet...")
Ok, this first-pass model is complete. We have probably done some overfitting, and could certainly do some more feature engineering, but this is a start. It's your job to take it to the next level! And there are many next levels, see the FiveThirtyEight 2020 methodology write-up for proof. All we can do now is stay calm and model on... Just kidding, we can also
👉👉 VOTE 👈👈
