Fact Check: Israel Statistics Do NOT Show Vaccines Are Ineffective -- They Illustrate A Common Weakness With Overly-Simple Analysis

Fact Check

  • by: Dean Miller
Fact Check: Israel Statistics Do NOT Show Vaccines Are Ineffective -- They Illustrate A Common Weakness With Overly-Simple Analysis Stats Can Fib

Do Israel's health statistics show COVID-19 vaccines are ineffective and have not changed health outcomes of Israelis? No, that's not true. The data demonstrate Simpson's Paradox, the name for a common statistical problem: a trend apparent in several groups of data disappears or reverses when the data are combined because you aren't considering an extra, and consequential, variable. Looked at all together, more than 60% of Israel's hospitalized COVID patients are indeed vaccinated, which suggests vaccines have failed. But, broken down by age, you see that almost all of Israel's elderly are vaccinated, while the young are much less likely to be vaccinated. What confounds the simple conclusion is that the elderly are both much more likely to be vaccinated and much more vulnerable to serious respiratory illness. While the vaccine is not perfectly effective, an age breakdown of the data shows it does reduce serious illness, death and infection.

The claims about Israel's COVID data have circulated widely in social media posts such as this October 27, 2021 video Ben Swann video posted to his "Sovren" platform titled "Study May Prove This Is a Pandemic of the VACCINATED " (archived here) which opened:

New data out of Israel where a very large portion of the population is vaccinated now shows that the rate of hospitalization and death is largely unchanged.

Users on social media only saw this title, description and thumbnail:

Study May Prove This Is a Pandemic of the VACCINATED

New data out of Israel where a very large portion of the population is vaccinated now shows that the rate of hospitalization and death is largely unchanged.

At about the 3:55 mark, host Ben Swann summarizes his analysis of Israel's public health data as follows:

The reality is is that the numbers for COVID deaths, the numbers for COVID hospitalizations, the number of sick people the transmission rate, none of that has changed as a result of these vaccines.

The simplest explanation for the stubbornly high percentage of vaccinated persons in the COVID statistics is known in statistics as the base rate trap. In a population like the UK, where many eligible people are vaccinated and elderly persons are both more vaccinated and more vulnerable to respiratory illness, it makes logical sense that at this point in the pandemic there will be breakthrough infections sending elderly people to the hospital.

What data out of Israel show, Swann says at 3:05, is that while the number of hospitalizations in the country is (in October, 2021) lower than it was during a spring of 2021 spike, the number of hospitalizations in September of 2021 is actually higher than it was in the summer and fall of 2020 before there was a vaccine. "And it gets worse than that," Swann says at 3:25. "Because if we're talking about deaths, the numbers haven't really changed either. And once again, it shows that the number of people dying in hospitals from COVID is higher if not statistically unchanged...when you take a macro look when you look at a country like Israel and say, let's look ... over time, have the numbers really changed? They haven't."

As he says at the 4:15 mark: "...none of the health issues surrounding COVID seem to be altered by the vaccine and certainly not in a positive way."

That's a misleading analysis, says professor Jeffrey Morris, the Ph.D. who is head of the biostatistics division of the University of Pennsylvania's Perelman School of Medicine. In a critique of misuses of the Israel health statistics Morris did not quarrel with the raw figures, writing:

... while these numbers are true, to quote them as evidence for low vaccine effectiveness is wrong and misleading. Sometimes, with observational data there is confounding of multiple factors that can make it easy to misinterpret simple percentages like this, and the current vaccination situation in Israel brings a perfect storm of confounding factors that lead to confusion if not thought through carefully.

Morris describes how the progress of a vaccine campaign can change the statistic sketch of vaccine effectiveness. "When vaccination rates are low, use of raw counts can exaggerate the vaccine effectiveness," he writes. "When vaccination rates are high, use of raw counts like this can attenuate the vaccine effectiveness, making it seem lower than it in fact is."

Morris said the vaccination program in Israel there are big age differences in vaccination rates. By the time late 2021's breakthrough infections hit, the patients needing hospital care tended to be older and more vulnerable:

In particular, the key factors here that contribute to this confusion are:

  • High vaccination rates in the country (nearly 80% of all residents >12yr)
  • Age disparity in vaccinations, including
  • Nearly all older people being vaccinated (>90% of residents >50yr) and
  • The vast majority of unvaccinated being younger people (>85% of unvaccinated <50yr)
  • Older people are orders of magnitude more likely to be hospitalized with a respiratory virus than young people (residents >50yr are >20x more likely to have hospitalized serious infections than residents <50yr, and residents 90+ are >1,600x more likely to have hospitalized serious infections than residents 12-15yr)

Slicing Israel's data by age and vaccination status, Morris said it's clear the vaccines are working well, even against the highly infectious delta variant. Severe cases, he finds are 3.1x higher in unvaccinated than fully vaccinated individuals.

Splitting the population by under 50 and older than 50, Israel's data show 90% of the elderly are vaccinated, while just 73% of younger Israelis are vaccinated. Looking at those age groups, a very different picture emerges:

Morris chart.JPG

(Source: covid-datascience.com screenshot taken Mon Nov 1, 23:05:03 2021 UTC)

Morris said there are two systemic factors that confound a simple analysis of vaccine efficacy, making the overall number 67%, while in each of the broad age groups, it is markedly higher. "These effectiveness measures are quite high and suggests the vaccines are doing a very good job of preventing severe disease in both older and young cohorts," he wrote. To explain, he turned to data science:

This is an example of Simpson's Paradox, a well-known phenomenon in which misleading results can sometimes be obtained from observational data in the presence of confounding factors. ...The reason for this paradoxical result that is that the both dosage and recovery probability were systematically higher in one group (younger) and lower in the other group (older). This creates a specific type of confounding that can produce such a paradox ... if we do not stratify by the confounding factor (age), then the overall analysis gives a blatantly misleading result.

Want to inform others about the accuracy of this story?

See who is sharing it (it might even be your friends...) and leave the link in the comments.:

Lead Stories is working with the CoronaVirusFacts/DatosCoronaVirus Alliance, a coalition of more than 100 fact-checkers who are fighting misinformation related to the COVID-19 pandemic. Learn more about the alliance here.

  Dean Miller

Lead Stories Managing Editor Dean Miller has edited daily and weekly newspapers, worked as a reporter for more than a decade and is co-author of two non-fiction books. After a Harvard Nieman Fellowship, he served as Director of Stony Brook University's Center for News Literacy for six years, then as Senior Vice President/Content at Connecticut Public Broadcasting. Most recently, he wrote the twice-weekly "Save the Free Press" column for The Seattle Times. 

Read more about or contact Dean Miller

About us

International Fact-Checking Organization

Lead Stories is a fact checking website that is always looking for the latest false, deceptive or inaccurate stories (or media) making the rounds on the internet.
Spotted something? Let us know!.

Lead Stories is a:

Follow us on social media

Most Read

Most Recent

Share your opinion