We are all flawed thinking machines

A case study in Alex Berenson's statistical trickery

I know I said I would write about market efficiency this week. I am working on it, and it’ll be a doozie, trust me. But in the meantime, please consider this interesting scenario¹ and the accompanying question:

Linda is 31 years old, single, outspoken, and very bright. She majored in philosophy. As a student, she was deeply concerned with issues of discrimination and social justice, and also participated in anti-nuclear demonstrations.

Which is more probable?

a. Linda is a bank teller.
b. Linda is a bank teller and is active in the feminist movement.

Since you subscribe to my newsletter, you are clearly a smart, numerate person. You answered correctly (a).

But, the majority of people answer incorrectly (b).

The reason is that we confuse vividness for probability. It means that—even as we add qualifiers that make something less likely (Linda is not merely a bank teller, she is also active in the feminist movement)—we perceive it as more likely, because the image it conjures in our minds is more vivid.

Humans don’t think naturally in statistical terms. We think in stories.

When we encounter statistics that are in tension with our stories, we get a headache, and most of us move onto something less taxing. On the other hand, when statistics support our stories, we are likely to share them. That’s how twits like Alex Berenson become famous.

Alex Berenson’s stupid brilliance

In case you’re fortunate enough to be unfamiliar with him, Alex Berenson is a Yale-educated former New York Times reporter turned serial misinformer of the general public. He has been critical of the government’s response to the pandemic from the start, arguing early in 2020 that we were overestimating the virus, and that Covid was a cover for government overreach. Fox News was set to give him a show called COVID Contrarian, until cases surged and they quietly dropped the idea. More recently, Berenson has been braying about the risks of Covid vaccines.

His primary platform was Twitter, but he was banned last August for repeated violation of the platform’s Covid misinformation rules. Berenson has turned to Substack to peddle his nonsense, and his thousands of followers now do the tweeting for him. This is one of his latest posts:

That headline—”Vaccinated English adults under 60 are dying at twice the rate of unvaccinated people the same age”—likely goes against everything you understand about the pandemic and vaccines. But to be clear: nothing in his post is false. Those are indeed the numbers released by the UK’s Office of National Statistics. But the implied message—that vaccines are dangerous—is entirely wrong.

What’s going on here?

Here’s the thing: Berenson isn’t dumb. In fact he is brilliant at finding statistics that coincide with his followers narratives about the world—especially the idea, held by the 60% of Republicans who remain unvaccinated², that vaccines are harmful. Whether you think they lead to severe illness, or suppress our immune systems, or that their side effects are worse than COVID itself—Berenson has a number to back you up.

His post about vaccinated vs unvaccinated death rates is a tour de force of statistical trickery.

Simpson’s Paradox

In the fall of 1973 the dean of the graduate school at the University of California, Berkeley was worried about the school being sued. In the prior year, the school had admitted about 44% of male applicants, but had only let in 35% of female applicants—a statistically significant difference.

Was Berkeley biased against women? The dean asked one of the school’s statistics professors—Peter J. Bickel—to investigate.

When Mr. Bickel and his colleagues scrutinized the data, they found little evidence of gender bias. Instead, they discovered that more women had applied to departments that admitted a small percentage of applicants, like English, than to departments that admitted a large percentage of applicants, like mechanical engineering.

Wall Street Journal, When Combined Data Reveal the Flaw of Averages

In other words, the gender breakdown in admissions reflected an association between gender and a propensity to apply to competitive departments—and not the school’s attitude towards gender itself³. In fact, on a department-by-department basis, Berkeley admitted a higher percentage of women applicants than men.

This phenomenon—where a trend that appears in groups of data disappears, or reverses, when the groups are combined—is called Simpson’s paradox, after the statistician who first wrote about it in 1951.

Why Berenson’s chart is true and false at the same time

In his post, Berenson cleverly exploits Simpson’s paradox to tell a compelling anti-vax story. After all, the numbers do show a large group of vaccinated individuals dying more than a matched group of unvaccinated people.

But the truth is that the higher number of deaths amongst the vaccinated compared to the unvaccinated has little to do with vaccines and a lot to do with fact that the population of vaccinated people skews much older than the population of unvaccinated people (not least because kids were not eligible for vaccinations until recently).

Older people are at greater risk of dying from COVID, whether they are vaccinated or not.

So while Berenson’s chart is technically accurate, it says nothing about the vaccine, merely that older people are both more likely to be vaccinated and more likely to die of COVID.

And just as the truth about UC Berkeley’s admissions became clear when the school looked at admissions by department, breaking down the COVID data reveals the truth about vaccines.

Simply put: Berenson’s chart mixes the age effect (older people die more from COVID) and the vaccine effect (unvaccinated people die more from COVID). When we disaggregate the data, we can see these factors separately, and the truth is revealed: vaccines substantially reduce deaths from COVID across all age groups.

Oh, wow, you too

It’s hard not to pile onto the mockery of Berenson’s followers. It’s tempting to write them off as wing-nuts—as people we have nothing in common with.

But I wonder if we can scrounge up some sympathy for them. Statistics are tricky. At first glance it’s not immediately clear why Berenson’s chart—and similar ones casting doubt on the merits of vaccines—is misleading. And his followers are especially unlikely to scrutinize it because it’s consistent with their most vivid narratives. Linda is not just a bank teller, she is also a feminist.

Vulnerability to misinformation isn't partisan. As the novelist George Saunders puts so well, we are all ultimately “flawed thinking machines”:

I mean, imagine if somebody saw in all the wrong colors and all the shapes that he saw were incorrect. And all of his understandings were messed up. That person would be wise to be a little humble, because the data’s coming in, and he’s messing it up. And essentially, I think that’s what human beings are doing in our little, sweet, pathetic way.

So then, if you are in that kind of flawed thinking machine, and you see another flawed thinking machine, it would seem almost crazy and irrational to start judging and fighting that person. You might more reasonably say, oh, wow, you too.

This isn’t to say we should resign ourselves to a world of Kellyanne Conway’s “alternative facts”. Intentions matter: people like Alex Berenson seek to exploit—for their personal gain—our vulnerability to misinformation. He is putting people at risk, and should be called out.

But Berenson’s followers are another matter. Like everyone else, they’re just flawed thinking machines trying to navigate a complicated world. Even as we disagree with them over statistics and stories, we might give their intentions the benefit of the doubt.

1. The “Linda scenario” was popularized by behavioral economists Daniel Kahneman and Amos Tversky. Kahneman went on to win a Nobel prize for his work, much of which focused on examining our heuristic biases.

2. Compared to 17% of Democrats, reports the Kaiser Foundation’s vaccine monitor

3. Though, it is entirely possible that gender discrimination may have played a role in the funding—and capacity to admit students—of various departments: English at Berkeley might be really competitive because the school has historically given more resources to departments that are of more interest to men, like Engineering.