Photo: Graeme Sloan/Bloomberg/Getty Images
With the “reset” of the 2024 presidential election via the substitution of Kamala Harris for Joe Biden, it’s looking very much like the very close election we originally envisioned for November has returned. And why shouldn’t it have? There has been exactly one comfortably decided election in this century (Barack Obama’s 2008 victory over John McCain), the two major parties are in equipoise, and three-time Republican nominee Donald Trump has polarized American politics to an almost incredible degree.
Now, an increasingly attentive public is paying a lot of attention to the presidential election, some out of excitement and others out of fear. It’s a good time to review some of the mistakes people tend to make in seeking to follow and interpret the polls.
When a poll favorable to one candidate or the other comes out, that “team” is very likely to hype the numbers as absolutely true and predictive of a great landslide to come (that’s particularly true of Trump’s MAGA fans; Democrats have been burned by poll-driven irrational exuberance too many times). Some pollsters are prone (deliberately or not) to partisan bias, but any one survey by any pollster can, for statistical reasons, turn out to be an outlier.
There are two simple ways to avoid the temptation to overreact to individual polls: (1) utilize polling averages, which tend to greatly reduce the importance of outliers, and (2) look at trends in the results found by specific pollsters over time.
This year, there are a host of polling averages available. Some (notably RealClearPolitics and, to a lesser extent, Decision Desk HQ) use simple arithmetical averaging without any adjustments or weighting of results, while others (FiveThirtyEight, the New York Times, the Washington Post, and FiveThirtyEight founder and now independent analyst Nate Silver) have sophisticated methodologies that seek to place a premium on higher-quality and more recent data. I personally prefer FiveThirtyEight’s averages, which are easy to navigate and aren’t quite as loaded with junk polls as RCP’s. But any of the above averages are a lot better than reliance on any one pollster or any one result.
In close races where small swings in polling results can seem huge (particularly those in which the lead, however small, changes hands), it’s easy to forget that every reputable poll is accompanied by a “margin of error” reflecting the size of the sample and thus the likely range of possible underlying numbers (which, in turn, is modified slightly by a “confidence interval,” which is typically 95 percent). A recent national poll from Emerson College showing Harris leading Trump by a 50-46 percent margin had a margin of error of 3 percent, which means the results for either or both candidates could be off by that percentage. Thus a relatively robust Harris lead is actually “within the margin of error” (amounting to 6 percent in terms of the difference between the candidates) and could be misleading. To put it another way, no really close lead is safe and could represent an illusion.
The margin of error can become really large in subsamples of particular parts of the electorate (e.g., voters under 30, voters with or without a college education), which some pollsters compensate for with “oversamples” of particular groups of interest. When you see a poll with a finding that seems really odd (such as Trump leading among young voters or ringing up 30 percent of Black voters), always look for the size of the sample and the margin of error. This is why large-sample polls (all else being equal) are generally more reliable and why state polls are typically less accurate than national polls.
Until fairly recently, there was a very strong preference among polling experts for surveys based on live telephone interviews, until (a) cell phones began replacing landlines in households and (b) the unwillingness of Americans to respond to phone-poll solicitations began making it very difficult (and expensive) for old-school pollsters to get a representative sample. Now there remain “gold standard” pollsters (e.g., New York Times–Siena or Ann Selzer’s Iowa poll) that rely almost entirely on live-caller surveys but that now pay extra attention to the design and weighting of samples (e.g., by comparing them to verified voter files from the most recent election). There are also perfectly reputable polls that utilize refined, online voting “panels” and other methods. Pew found after the 2022 midterm elections (when pollsters had an excellent record) that “17% of national pollsters used at least three different methods to sample or interview people (sometimes in the same survey), up from 2% in 2016.”
FiveThirtyEight’s database of pollster ratings remains an essential tool for separating good from bad polls, based not just on accuracy but on transparency (pollsters who won’t tell you how they reach their results should not be trusted). But in general, you should beware of small-sample, one-day polls that are clearly designed to grab headlines.
For varying reasons, pollsters (or, more often, the media outlets that pay for and sponsor polls) don’t always release polling data the minute it’s collected. So it’s possible a “new” poll will represent old data. For example, some media folk jumped on a Fairleigh Dickinson poll of the Harris-Trump race that was released the day after the Democratic National Convention ended and that showed a “post-convention bounce,” even though much of the polling was conducted before the DNC began. Keeping in mind the gap between the surveying and the reporting of results is important any time people look for a “bounce” from some significant event (particularly a candidate debate). Indeed, it’s wise to wait a few days after such an event to look for polling data since much of the impact is likely to come from secondary coverage rather than live viewership.
It obviously matters a lot whether the people polled and reported as favoring one candidate or another actually turn out to vote. But it’s not always easy to separate the participating sheep from the nonparticipating goats until fairly close to Election Day. This is why most pollsters stick with samples based on registered voters until they conduct a “switchover” to likely-voter surveys shortly before early voting begins (others, like Times-Siena, offer both registered-voter and likely-voter results much earlier).
There are different forms of “likely-voter screens” with different strengths and weaknesses. Some focus on stated voter intentions, which can overestimate turnout because people don’t like to admit they might find something better to do on Election Day than fulfilling their civic obligations. Others emphasize past voting behavior, but that obviously doesn’t work for newly eligible voters and may miss surges in turnout among voters who did not participate earlier (this has been one factor frequently cited as contributing to the underpolling of Trump voters in 2016 and especially in 2020). Likely-voter screens are especially important in non-presidential elections, when turnout is often low and variable.
Much higher percentages of registered voters participate in presidential elections, making assessments of likelihood to vote somewhat less essential. In the past, the application of likely-voter screens has often produced improved numbers for Republican candidates since they were disproportionately drawn from segments of the electorate most likely to vote (e.g., older voters). That may be less true in the Trump era, in which Democrats have improved their performance among both highly educated and older voters, while Republicans are doing better among non-college-educated voters who aren’t quite as likely to vote.
Finally, it should be noted that some polls (typically “issue polls” that don’t measure candidate preferences and some job-approval or favorability polls) don’t even screen for voter-registration status but use samples of “adults.” These results should be taken with a few grains of salt.
One by-product of this era of close elections and partisan balance is that polls can get the outcome “wrong” even if they are reasonably accurate. It’s also important to note that national presidential polls estimate the national popular vote, not the results in the Electoral College (both George W. Bush in 2000 and Trump in 2016 won the latter while losing the former, and in 2020 Trump came within a whisker of winning while pretty decisively losing the popular vote). So, for example, in 2016 the final polling averages at RealClearPolitics showed Hillary Clinton leading Trump by 3.2 percent. She won the national popular vote by 2.1 percent. That’s a pretty small error. But sparse state polling gave no hint that Trump was going to win the historically Democratic states of Michigan and Wisconsin — and thus the election. So when Trump did win by the equivalent of an inside straight, a lot of shocked observers felt betrayed by the polls, and some concluded they were worthless. They weren’t — at all — but they were, of course, not flawless.
Polling error was actually more evident in 2020. RCP’s final averages showed Biden leading Trump by 7.2 percent; he actually won the popular vote by 4.5 percent, a margin small enough to get Trump within reach of another inside straight in the Electoral College. Postmortems of this relatively poor showing didn’t reach any clear conclusions, but explanations often focused either on the pandemic conditions that greatly affected both polling and voting or on a continued problem pollsters were encountering in identifying Trump voters. Either explanation was consistent with the excellent record of the polls in 2022, when the pandemic had subsided and Trump wasn’t on the ballot. So there’s no reason to assume the polls will be right or wrong in 2024. But Harris supporters will pray that she is far enough ahead as voters vote that she can win in the Electoral College. And a big win might also reduce the very high odds that Trump and his supporters will again fight against certification of a defeat.
Source link
