How Our Polling Averages Work
Methodology explainer
Politics are all about public opinion, and there’s no better way to capture opinion than a well-designed survey. Unfortunately, polls are prone to error, whether it’s due to a small sample size or poor methodology. Averaging a large number of poll can minimize both sources of error by combining larger samples and diverse methodologies.
Collecting Polls
We try to include as many polls as possible by referencing different sources. This involves doing online searches, following the social media pages of pollsters, and checking our lists against those compiled by RealClearPolitics, the New York Times, Silver Bulletin, and DecisionDeskHQ.
Adjustments
A few factors tell us a poll is “biased”—not in the sense of being wrong, since we can’t know its accuracy ahead of time—but in terms of consistently deviating from other pollsters. This can cause artificial shifts in polling averages depending on which pollsters were most active. To combat this, we adjust for two types of systemic differences between pollsters:
Pollster-specific partisan lean: based on pollster house effects data from Nate Silver/538. These adjustments are quite minor—for a typical pollster, margins are shifted left or right by around 2%.
For presidential approval, where partisan lean is less applicable, we make a stronger adjustment for pollster biases based on how they deviate from an average that doesn’t adjust for pollster lean.
Population polled: whether polls are of adults or registered/likely voters. For presidential approval, the starting assumption is that different screens don’t impact results. For elections, we assume that screens will follow historic patterns, adjusting for the partisan alignment of college-educated voters (who tend to be more likely voters) and the type of election (the president’s party tends to do worse in likely voter polls during midterms).
We revisit these adjustments as more polls are released. When polls do come out, we check if there are separate results under different screens. We then aggregate these differences, initially by taking an average. If there are enough polls taken over a long timeframe, we use polynomial regression to construct a trend line.
Weighting
Polls can influence the average to varying degrees based on a few indicators of quality.
Recency: as polls age, their weight decays exponentially. For election poll averages, weighting is less aggressive far from the election, especially for state or district polls (polling is very sparse, making averages more susceptible to noise).
Sample size: larger samples are favored in all of our averages. We use the same system as the now defunct 538 averages, weighting polls by the square root of their sample size, capped at 5000 to avoid low-quality mass panels.
Population polled: elections are determined by those most likely to vote, so averages weight likely voter polls most heavily (weight of 1), followed by registered voter (0.9), and adult polls (0.7). If the average is measuring the views of all Americans (like the presidential approval tracker), we weight in the opposite direction: 1 for polls of adults, 0.9 for registered voters, and 0.7 for likely voters.
Pollster accuracy: pollsters with a track record of accuracy are favored. All of our current and recent averages are weighted using Silver Bulletin ratings, while historic averages use the older version of those ratings by 538. We deduct 5% from a pollster’s weight for each Silver Bulletin/538 grade drop (A+ weighted 1, A weighted 0.95, and so on). Ungraded pollsters get the median weight of 0.65.
Flood penalty: to ensure that a diverse range of methodologies are represented in our averages, pollsters that release more than one survey every three weeks have their weights reduced. For example, releasing two polls in three weeks will result in each poll having its weight cut by half.
All five weights are then multiplied together to determine a poll's overall weight. You can check out some of our polling averages below:
