Alarm bells went up again among the hyper-informed class as the latest NBC News poll showed former President Trump winning the 18-34 age group by 4 points over President Biden. That result is within the margin of error for the group—meaning we can’t be certain of that Trump lead—but it’s one of several polls lately showing a relatively weak Biden performance among that age group.
What we don’t hear much about is just how difficult it is to poll young adults; media coverage certainly doesn’t address it in the race to create headlines. We’re assured in methodology statements that data are “representative” and that they are “weighted to x, y, z according to Census estimates,” but rarely do we hear about what it means that some groups are much more difficult to reach than others.
Older people are generally more willing to take surveys than younger people. Young people don’t want to answer questionnaires—even when it’s the Census. This is not a new problem, but technology has exacerbated it.
There are two ways to solve the problem if you’re a pollster. One is to simply reach whomever you can and then weight the data to match the overall demographics of the population. For example, if 25 percent of your sample should be aged 18-34, and you only have 15 percent, you would force that 15 percent upward to be 25 percent and adjust the older proportion down. You’ve made it look right—but you have to hope those 15 percent are accurately reflecting the 25 percent you should have polled.
The other solution is to keep making calls or sending emails or texting until you reach the right proportion. That means you would also stop talking to older people once you have enough of them. This approach is costly in terms of time and resources. It costs significant money to keep searching for those younger respondents, and it also takes longer to complete the poll.
Online panels are a bit different in that respondents have signed up specifically to take surveys, but it is harder to get young people to do just that. Nonetheless, it is easier to get young voters when you know who they are and can target them appropriately.
There’s one more complication, which affects even the most carefully constructed and fielded poll: nonrandom nonresponse, which means the people who don’t answer the poll systematically differ from those who do respond. If nonresponse was random, there would be no important differences between poll takers and rejecters, and the poll would accurately reflect the group’s views. Unfortunately, nonresponse is rarely random.
Of particular concern for horse-race and other political polls is nonresponse based on partisanship or views of a party. The theory goes that if Republicans are having a rough news cycle, Republicans are less likely to answer a political poll because they don’t want to talk about their team doing badly. The same holds for Democrats. This phenomenon got considerable attention and study in the wake of the 2016 election, but it hasn’t come up a ton since then.
In the last month or so, there are clear reasons to think that young Democrats would be turned off from answering polls, or at minimum, responding unfavorably toward Biden—we know there is a considerable intra-party generational divide on the Israel-Gaza war, and young people are not aligned with Biden’s stance.
Let’s be clear: That probably doesn’t mean a Trump bump from the group a year from now. Trump has done little to affirmatively win the votes of younger cohorts. But it does mean that young liberals are likely shifting their views—or not answering polls at all—which means we might have a more right-leaning sample of young people that skews toward Trump.
Unfortunately, public polls almost never give enough information to assess these issues. We rarely see pollsters report their unweighted sample sizes, or anything else that would reveal any response issues. We also usually don’t get information in the crosstabs that might help assess any differential nonresponse issues—for example, partisanship within age groups. Additionally, these are small groups within a poll that come with large margins of error.
The coverage these horse-race numbers get, especially at the subgroup level, is wildly disproportionate to the fuzzy nature of the polls’ estimates. Is Biden facing some headwinds? Absolutely. But don’t buy into the alarmism; unless you are advising a campaign, there’s no reason to place too much importance on any of this a year out.
Weekly column written by Vice President Natalie Jackson for National Journal.
This column was originally published on November 21, 2023, on nationaljournal.com and is owned by and licensed from National Journal Group LLC.