Photo: Intelligencer; Photo:Getty Images
When Nate Cohn releases a poll, people pay attention. Cohn, who created the New York Times/Siena College poll, is perhaps alone among peers in his capability to drive a narrative with proprietary data. For example, in November 2023, NYT/Siena polls that found President Biden trailing Donald Trump in several swing states set off a (familiar) cycle of Democratic anxiety and recriminations about Biden’s prospects that continues to the present moment. Cohn is highly regarded in the field — his sophisticated methods and high transparency levels merit 538’s highest rating, and his 2022 polls were uncannily accurate. But Cohn’s methods are hardly infallible: Among other things, The NYT/Siena surveys in 2016 and 2020, like many others, significantly underestimated Donald Trump’s strength in swing states. Last week, I spoke with Cohn about the unusual elements of the Biden vs. Trump 2024 rematch, why we still don’t know what went wrong with 2020 polling, and whether the New York Times covers their own polls appropriately. (Our conversation took place hours before Donald Trump was found guilty on 34 charges, which, Cohn later wrote, could provide a boost for Biden among key voters he’s been struggling with.)
Most polls, including yours, have Trump currently leading in the important swing states. I want to start with the question of how predictive polls are at this point in the election cycle, because I’ve seen various theories on that. And I’m wondering if they’re more predictive this far out than they used to be, since people’s political views are so entrenched now. We don’t see huge swings like we did in the Dukakis v. Bush days, for example.
The short answer is that polls at this stage are not typically extraordinarily predictive of the final outcome, but as you mentioned, they’ve become more accurate in recent decades. American politics is more polarized, so there are fewer swing voters. Voters know a lot more about the candidates in recent cycles than they have in the past. We’ve had a lot of well-known candidates like Hillary Clinton or Donald Trump, who aren’t unknown like Michael Dukakis was in 1988.
There are other things that we see in the data nowadays that suggest that the polls are less volatile than they used to be. For instance, Michael Dukakis built that tremendous lead in the summer of 1988 after the Democratic convention that year. It was pretty common for candidates to come out of a convention enjoying a huge double-digit bounce, and we just don’t see things like that happen anymore. The catch is that our elections are much closer than they used to be. So even though the polls at this stage may be more quote-on-quote predictive than they were 20 or 30 years ago, because the range of election outcomes has also narrowed as well, it’s not like now we know who’s going to win. And that’s especially true in this race, where the candidates are, as you know, fairly close in Michigan, Pennsylvania, and Wisconsin.
Plus, there weren’t real primaries this time around, and it doesn’t really feel like there’s been the usual election slog yet, just a creeping sense of dread among Americans that it’s going to be the same two guys.
Yeah. This is pretty subjective, but I don’t really feel like this campaign is underway yet.
I agree. You were early to mark one of the major polling storylines of this election, which is the split that’s showing up in polling between likely voters and registered voters. Democrats are doing quite well among the first group, made up of enthusiastic voters who cast ballots regularly, and less well among the second group, who vote irregularly. Every poll seems to show this split — how confident are you at this point that it’s actually happening?
When we’re talking about polling, saying something like, “Actually happening” is always hard. Polls are fuzzy things, and they are inherently imperfect, so even when all the polls agree on something, it’s hard to ever be 100 percent sure it’s real. But in this particular case, as far as I’m aware, all of the evidence suggests that something like this story is real. We can debate about the degrees, we can debate about whether it’ll last until November, but I don’t think that there’s very much room to disagree with the notion that the Democrats are showing a lot of strength among the most regular and highly engaged voters, and Joe Biden in particular, is extremely weak among less engaged voters, and in particular less engaged Democrats.
Do you have any theories as to why it’s playing out this way? That seems to be the big question right now.
I think there are two broad theories. One is that it’s a matter of engagement, that these people aren’t paying close attention, that if they did — or when they do — that they will end up behaving like more highly engaged voters of the same demographic profile. So in this theory, a Black voter who’s paying attention is for Biden and one who’s not paying attention is ambivalent. And once they pay attention, they’ll support Biden. You can come up with evidence to support that theory.
You could also look at their evaluations of the economy. Less engaged Democrats are much more likely to say the economy is poor than highly engaged Democrats. That may reflect actual factual knowledge about the unemployment rate, or how the stock market is doing and so on. You can posit that some of these voters, as they tune in, and as the campaign heats up, will glean the same information the highly engaged voters already have, and they’ll swing Biden’s way.
The second theory is that it’s not just about engagement, it’s that these are fundamentally different kinds of voters, who will never be as engaged as someone who watches MSNBC every night. And as a consequence of their relative lack of knowledge of American politics, of policy and economics, they’re much less likely to be voting on the issues. They’re less likely to think about abortion or democracy, and instead vote on character, and which candidate looks strong or weak. They’re more likely to want a candidate who supports change, just because they’re not happy with the status quo, and they’re not going to make the rational calculation that, “Oh, I’m not happy, but I think Donald Trump is a threat to democracy, and therefore, I’ll vote for Joe Biden, even though I don’t like him.” Those just aren’t the kinds of calculations that less engaged voters make. And it’s worth noting that we know that the people who didn’t vote in recent elections, almost by definition, are not the ones who were motivated to stop Donald Trump last time. If they were, they would have voted. So we know that this is a group of people that just can’t be that upset about Donald Trump. And if that’s true, then Joe Biden’s problem is going to be really deep.
That group of voters, who aren’t paying attention to the news and are voting on things like character, is always enormous. The difference for Democrats, though, is that there’s this gap between likely and not-so-likely voters that I don’t think existed even four years ago. So what changed?
Donald Trump has never been popular. He’s still not popular. His unfavorability rating is just as high today as it was heading into Election Day 2020. This group of disengaged voters doesn’t like Donald Trump, and never did. What’s changed, to my mind, is that Joe Biden went from being a broadly appealing person — they didn’t necessarily love him, they didn’t necessarily even like him, but he was acceptable — to someone who many voters do not find acceptable anymore.
They think his stewardship of the economy is terrible, and they think he’s simply too old to be president and that he no longer passes some sort of basic visual gut check for millions of Americans. And Biden is trying to still win those voters on a pretty sophisticated argument, frankly, about Donald Trump as a threat to democracy and about his opposition to abortion rights. It’s an issue-based appeal that I think very naturally would resonate among those who pay close attention to politics, who have well-informed opinions on the issues, and would naturally not resonate so well with those who don’t.
And then for good measure, we have the economy looming over this whole picture. Many less engaged voters are less likely to have college degrees, and they are less likely to make $100,000 a year. Donald Trump in 2020 and 2016, was seen as the best candidate on the economy, and now that issue matters much more for voters, because they say the economy is bad.
It doesn’t sound like you think that much of Biden’s current strategy in going after these voters. And his team’s general attitude toward negative polls, at least publicly, is “They’re not right.” Do you think their outreach strategy reflects a denial of the reality they’re up against?
I’m not sure whether it reflects reality denial or not. I think that it may reflect the hand they have to play. Voters don’t think Joe Biden has done a good job, and they don’t think the economy’s especially good. And many of the arguments that Democrats would traditionally make against the Republican, like “Oh, they’re going to help the rich and not the working class” are harder to make against Donald Trump than they were against Mitt Romney in 2012. The arguments that the Democrats have at their disposal that are strongest, and that might well win them the election in the end, are about democracy and abortion. There just aren’t very many other places where they can fight.
In other words, trying to convince people that the economy is strong, which it is on paper, is a losing battle.
Well, I’m not saying they shouldn’t try and do more in that space, and I can’t prescribe what the optimal economic message is for Biden. I don’t know if it’s to say that the economy is good, and try and convince people that it’s not as bad as they think: “Look at these great stats.” I don’t know if it’s to acknowledge that things are bad, but say things are getting better. I don’t know if it’s to say that Donald Trump would make it worse. I’m not going to speculate about what would do best on the economy. I’m just saying that, in terms of how Joe Biden can actually try to win this election, as opposed to playing defense, clearly abortion and democracy are two powerful arguments, and they may be their best arguments. If they have another argument that has the potential to resonate a little more broadly with voters, in particular those concerned about pocketbook issues, I haven’t seen anyone floating it.
Abortion and democracy have worked very well for Democratic candidates in the last two to four years. But we’re seeing that state Democratic candidates are outrunning Biden right now, which also ties into the theory that is really just about the president, and less about Trump.
Yeah. The thing that the Democratic Senate numbers and other numbers down ballot remind us of is that Biden’s weakness right now is among voters who usually vote for Democratic candidates, and maybe even people who don’t usually need to think too hard about who they’re going to vote for. Maybe they’re registered as Democrats, and they come from demographic groups that typically vote Democratic. And there are lots of reasons why people are Democrats. For some, it’s because of their views on a host of policy issues. Others are from identity groups that bind them in different ways towards the Democrats, like as a union worker or an African-American. And other people support the Democrats out of perceived self-interest — they might think that Democrats are going to protect their Medicare and Social Security, or that they’re good for working people, and so on.
And I think that latter group, that group of people who have traditionally supported the Democrats on pocketbook issue grounds, might well be the heart of Biden’s problem here. It explains, to my satisfaction at least, why the numbers are so unique at the presidential level. Because Democrats aren’t being weighed down by Biden’s economic stewardship, it seems, and similarly, other Republicans don’t have Trump’s unique brand name on economic management.
He’s a businessman, as voters like to remind us. Biden has a problem with Black and Hispanic voters in particular right now, as you alluded to. How much does that overlap with the disengaged-voter problem? Is it basically the same problem, or are these two distinct issues?
That’s a good question. They’re certainly related, but it is worth noting that Biden is holding up among well-engaged, high-frequency Black voters. He’s also holding up among Hispanic midterm voters in our poll. We’re talking about small groups, and now we’re looking at the subset of small groups that don’t regularly vote. So there’s an extra layer, I think, of uncertainty around exactly where Biden’s weakness among Black and Hispanic voters lies. But it’s conceivably consistent that most of his weakness among Black and Latino voters is among relatively disengaged Black and Latino voters. And if that’s true, it carries some interesting possibilities. If it’s a low-turnout election, maybe you end up with the Black and Latino share of the electorate declining, but then Biden does better than the polls show among Black and Hispanic voters. Which wouldn’t necessarily net out any differently in the top-line poll result, because many of these Black and Latino voters staying home are still, on-net, Biden voters. They may not be as good for Democrats as in the past, but they still lean Democratic. And so you could get the sort of scenario where the final results demographically look considerably more normal, but the turnout of Black and Latino voters is quite low. I think that’s at least imaginable.
Which is one reason Democrats may want lower turnout for once — another strange thing about this election.
The funny thing about turnout is it’s not just the level — it’s who is voting and staying at home. If you get low turnout and the people who show up are the people who showed up in the midterms, that may be fine. If you get high turnout and the increase is coming out of Black and Latino voters, that could be good for Biden, but if it’s coming out of white working-class voters, it wouldn’t be. And so on.
Pivoting a bit to polling more generally: The New York Times/Siena polls were dead-on in 2022, and also extremely accurate in 2018. They were considerably less accurate in 2016 and 2020, underestimating Trump support in swing states in those years. I remember, after 2020, you were trying to understand what had happened. Four years later, what is your thesis about why your surveys were so off 2020, yet so good two years later?
The truth is that there is not an excellent explanation for what went wrong in 2020. And I should note that I didn’t feel like this was true after 2016. After 2016, there was no shortage of plausible explanations for why the polls had underestimated Donald Trump, which were backed by hard evidence. We knew that the polls in that cycle were probably underestimating the number of people with a college degree. We thought that undecided voters were prone to switching, and there was real evidence that you could look at after the election that suggested that this could explain why most of why the polls underestimated Donald Trump, and therefore, why we could be pretty confident that we could do well in the future, provided that we represented people without a college degree in the numbers that we wish we had.
As a consequence, we went out and we did 100 polls in 2018. That was a reflection of the level of confidence that we had, that despite what we saw in 2016, that there were real credible paths to being accurate in the future, and that we didn’t think it represented some sort of lasting issue. And I can’t say the same thing after 2020. There are, at the end of the day, no really strong theories backed by hard evidence that explain why the polls were wrong in 2020 and offered us a path to doing something different, or alternately, to be sure that it wouldn’t happen next time.
In polling, when you can’t come up with a strong theory to explain what went wrong, you usually fall back on this diagnosis of exclusion — non-response bias. There’s this idea that if everything we did looks right, but we didn’t get the answer we wish we did, there must be some reason why the people who backed Donald Trump were still underrepresented in our surveys, and it was probably just that they’re less likely to respond than demographically similar Biden voters. So even if, say, you had the right number of white working-class voters, you didn’t have the right number of white working-class Trump voters.
I don’t think that’s what happened in 2016. I think in 2016, if you had the right number of white voters without a degree, you would’ve come way closer to the result than you would’ve in 2020. So there’s something extra going on in 2020 that’s still not fully explained. One possibility, and I think this is the most optimistic one for the survey industry, is that it’s about COVID, that all these liberals were staying home and taking polls, while Republicans were out living their lives. We know that the coronavirus had different effects on the behavior of Democrats and Republicans — we can see that in the vaccine numbers. Why couldn’t it be the case that we were just systematically getting more people who were concerned about COVID, who were staying indoors, and consequently, that our polls had too many Biden voters controlling for all the demographic traits we try to adjust for?
Yeah, you’ve said that your numbers were more accurate in 2019.
The 2019 polls were great, and our 2022 polls were as well. But it’s also worth noting that there are more pessimistic possibilities as well. Perhaps Trump voters are just so distrustful of the media and institutions that they’re just systematically less likely to respond than Biden voters among the same demographic group — that’s a real possibility too.
Not only is it that Trump supporters are less likely to respond to surveys than Biden supporters, but they’re overwhelmingly concentrated among the kinds of people who don’t show up in midterm elections. That’s another theory that I think you could argue that would sort of square all of the various data points. I think that’s conceivable. One interesting thing I could note in this respect is that our polls right now show Donald Trump doing really well among disengaged voters. That was the whole start of this conversation. That wasn’t true in 2020. Is it possible that Trump was similarly strong among these voters in 2020, and that’s who we were missing? I don’t know. I don’t know that that’s not true. The 2020 polling was so bad that it’s hard to rule out anything about it. But it’s just an example of how the introduction of this uncertainty about the way polls are handling these presidential year voters makes it harder to be confident in what we’re measuring sometimes.
I appreciate the transparency and lack of confidence. It goes to this whole larger discussion about polling that makes people crazy, especially in the internet era. I feel like every four years, there’s this ongoing and kind of tiresome debate, where someone asserts that polling is fundamentally broken, and then the pollsters respond, “Actually, the polling errors aren’t as bad 40 years ago.” Are you tired of that conversation? And are people expecting too much here? This is an unclear question.
It is an unclear question. I sympathize with all of the various emotions that went into it, though. In some respects, it’s a miracle how good polling is. It is so hard to talk to people, and the response rates of telephone surveys can be 1 percent. And I know that I just spent all this time talking about how bad the 2020 polls were, and they’re bad by the standards that we need in our era of fairly close and competitive elections. But it is true that they weren’t horrible for many purposes. The polls overestimated Joe Biden nationwide, by about four points on the margin, which is to say the difference between Trump and Biden. So Biden was at plus eight and a half, as opposed to Biden plus four and a half. But that’s two points off on the percentage for the two candidates: Biden at 52 percent, as opposed to 54 percent. A two point difference on 95 percent of questions that we ask is entirely tolerable.
Does it make a difference whether 72 or 74 percent of Americans think Biden is just too old to be an effective president? That’s substantively the same. So polls are actually still pretty darn good for lots of things, but they are inherently imprecise. And we live in this era of close elections, where the level of precision that polls offer may not be sufficient to tell, at least to readers, what they really want to know, which is just: Who’s going to win? And in 1988, and in 1984, and in 1996, the polls could tell you who was going to win, because Reagan and Bush were up double digits in the final month of the race. And frankly, if Trump is up double digits in the final month of this race, he’s probably going to win. But the odds are that our polls will not show him with such a large lead — they don’t do so now — and consequently, we’ll go into election day knowing full well that either candidate will probably win.
The other difference now is that there’s the internet, and every piece of data is digested and spit out so quickly. The panic that has greeted your recent polls from Democrats, pundits and otherwise — and I include myself here — is quite something. Does it always surprise you how exercised people get?
Yeah. I remember our November 2019 polls, which showed Biden narrowly ahead and Elizabeth Warren losing.
That one helped tank Warren’s campaign, I think, because she was doing so poorly compared to everyone else.
I did not think that Elizabeth Warren was on a path to win personally, but I certainly did not help her. Of all the polls we’ve ever done, I feel like that’s the one where the response was just completely beyond anything that I had expected, and maybe that was naive of me. But it is extraordinary how much attention the polls get. In some sense, it’s merited. If you look at how our polls have done compared to others, it really is pretty remarkable how well we’ve done in a way that doesn’t seem sustainable. It’s hard to believe, over the long run, that if you average our poll against all other polls that we’re going to win, but it’s true up to this point.
But I think that it really reflects much more people’s interest in who’s going to win the election, not some considered view of just how much better our poll is or something. And I’m not sure that is healthy. As we said at the beginning, A, we’re still six months out, and B, even at the end of the race, it’ll probably be so close that either candidate can win. People are obsessed with our polls, and not just our polls, but all the polls, in hopes that we will be able to answer a question that, in all likelihood, we can’t. And maybe even worse than that, people sometimes do think we’re answering that question even when we’re not, and we can’t.
I also just wonder what the alternative is. There’s no better method of analyzing what’s happening in the election, I think.
There’s no real substitute.
I guess we could have fewer polls, or none at all.
If we didn’t have polls, I think we know what we would get. We would get information based on partisan polls, filtered through the campaigns and campaign committees and party committees and so on, and they would tell us a similar story. Or it would be based on on-the-ground reporting. I actually think that economics coverage is probably the best analogy to what the world without polls would look like. Like politics, economics is fundamentally a social science problem. Economics coverage solves this problem through survey research. The jobs report is a poll.
I didn’t quite realize that, actually.
The jobs report is a great poll, it’s a census-funded poll, but it’s a poll. We don’t have the equivalent of that in politics. There is no federally funded political survey. And so we basically are stepping in as the New York Times, and other media organizations do the same, to provide what the government provides in economics. It’s hard to imagine what it would be like to talk about the economy without having a jobs report.And similarly, it’s very hard to understand how we would talk about the electorate without surveys.
It would probably be the old cliché about yard signs — anecdotal stuff.
Yeah. And you can imagine the same thing for the economy perhaps, that people would go and talk to the person running some shop on Main Street in suburban New York somewhere, and then someone would have a different conflicting report from someone running a different kind of shop in New Mexico, and it would be a total mess. We wouldn’t have any idea.
This all plays into something I’m also curious about, which is the New York Times’s treatment of your polls. You started the poll in 2016. And since then, the paper has made them into more and more of an event, where the readers get a write-up of the actual findings with the topline numbers, which gets sent out as a breaking news alert. And then there’s all these follow-up articles for five, seven days afterward reporting on the poll’s sub-findings, some of them written by you, some by others. And then the numbers from the poll are cited again and again in more articles by Times reporters. I’ve always been impressed about how careful you are in your write-ups to say, “Hey, here are all the caveats. Here’s what I found. Here’s how we did it. And it might be wrong.” But I sometimes wonder about the Times’s presentation of all this stuff, as if it’s like, “Here’s the news.” And it’s actually polls.
Yeah. There’s a real tension between the thing I just told you, which is that polling is this inherently imprecise instrument — probably better than anything else, but still imperfect — and an organization like the New York Times, which is accustomed to writing up its articles as their objective truth.
Exactly.
And a New York Times/Siena is not an objective truth, it’s not the census. And by the way, the census in the jobs report isn’t objective truth either — they revise them every month.
But it’s not in that tier of survey either. That’s a really hard needle to thread. And we’ve tried a lot of different ways over the years to try and thread the needle of “How do we both relay what we think is the best information that probably exists on the electorate, and also make it clear to people that that it could be wrong, that it’s subject to all these sources of error?”
And there are lots of things that we’ve done in this space that I think were somewhat promising. We have tried to show the margins of error. Like even this most recent poll, you could see that there was a little band around each of the numbers that was supposed to indicate that there’s this fuzziness around our estimates. We’ve done something called “live polling” before, where we showed people every call we made and every response. As a consequence, people can see that most people weren’t responding, which in some way, communicated the imperfection of it, even in the moment.
We try to report different turnout scenarios. All of these things — they don’t solve the ultimate problem, which is that we still have to figure out how to write up a New York Times survey result for the public, even though we know it’s not perfect. That’s a really hard problem that I don’t think we’ve had luck in solving. At the same time, we do have to write it up. These polls are really expensive, they do contain a wealth of information, and very little of what we’ve found we couldn’t print. I’m not saying that every article is perfect, and I know sometimes it’s a week-long event, but that seventh article that comes out usually still contains useful information that we couldn’t have realistically gotten after the first story.
No, I’m not saying it’s padded out in that way — more what you pointed out at the top, that there’s just fundamental tension between the Times as a news organ and them putting out the poll in the first place, even though it may be the best one.
Yeah. For informational purposes, the findings are essentially true. When we blare that voters think Biden is too old to be president, that’s a finding that’s so far outside of any margin of error that we can be reasonably confident in it. I don’t know whether, if we have an absolutely perfect poll, like literal truth for a moment and we could basically conduct an election today, or simulate an election — I don’t know Donald Trump would be ahead or behind in the three Midwestern battleground sites, Pennsylvania, Michigan, Wisconsin, that are reasonably close in our polling. It’s possible that with such a perfect poll that truly simulated the election, with the exact right people going into the voting room tomorrow, that Biden could take the lead.
I do have reasonable confidence that such a perfect poll, however, would contain a whole lot of very bad news for Joe Biden in terms of the public’s assessment of the economy, in terms of what they think about how he’s handling his job as president, favorability ratings, his strength among Black and Latino, and young voters.
You can imagine that there could be a case where we put out a poll that says “X result,” and it’s a news alert, it’s on the front page, it leads the home page, there’s a Daily episode about it. And then the very next day, another reputable poll says the exact opposite, and then we then go and try and poll again in a few weeks, and we no longer find that either. That hasn’t happened this cycle. And that reflects, I think, that the way our articles are framed appreciates that we have to rely most heavily on the findings that are robust and resilient to survey error. There’s some element that we can’t escape, like who’s up or down in our poll, and that’s really hard. We really could do a poll again, and now find Biden had a lead in Pennsylvania. That’s conceivable.
And then the news cycle would be totally different, It’d be like, “Oh, he’s back.”
Exactly. And one reason we do these six state battleground polls is because they offer extra layers of robustness to the overall finding. One state poll can bounce one way, but all six of them are a lot less likely to move around.
This interview has been edited for length and clarity.
Source link
