Do Israel's health statistics show COVID-19 vaccines are ineffective and have not changed health outcomes of Israelis? No, that's not true. The data demonstrate Simpson's Paradox, the name for a common statistical problem: a trend apparent in several groups of data disappears or reverses when the data are combined because you aren't considering an extra, and consequential, variable. Looked at all together, more than 60% of Israel's hospitalized COVID patients are indeed vaccinated, which suggests vaccines have failed. But, broken down by age, you see that almost all of Israel's elderly are vaccinated, while the young are much less likely to be vaccinated. What confounds the simple conclusion is that the elderly are both much more likely to be vaccinated and much more vulnerable to serious respiratory illness. While the vaccine is not perfectly effective, an age breakdown of the data shows it does reduce serious illness, death and infection.
The claims about Israel's COVID data have circulated widely in social media posts such as this October 27, 2021 video Ben Swann video posted to his "Sovren" platform titled "Study May Prove This Is a Pandemic of the VACCINATED " (archived here) which opened:
New data out of Israel where a very large portion of the population is vaccinated now shows that the rate of hospitalization and death is largely unchanged.
Users on social media only saw this title, description and thumbnail:
Study May Prove This Is a Pandemic of the VACCINATED
New data out of Israel where a very large portion of the population is vaccinated now shows that the rate of hospitalization and death is largely unchanged.
At about the 3:55 mark, host Ben Swann summarizes his analysis of Israel's public health data as follows:
The reality is is that the numbers for COVID deaths, the numbers for COVID hospitalizations, the number of sick people the transmission rate, none of that has changed as a result of these vaccines.
The simplest explanation for the stubbornly high percentage of vaccinated persons in the COVID statistics is known in statistics as the base rate trap. In a population like the UK, where many eligible people are vaccinated and elderly persons are both more vaccinated and more vulnerable to respiratory illness, it makes logical sense that at this point in the pandemic there will be breakthrough infections sending elderly people to the hospital.
What data out of Israel show, Swann says at 3:05, is that while the number of hospitalizations in the country is (in October, 2021) lower than it was during a spring of 2021 spike, the number of hospitalizations in September of 2021 is actually higher than it was in the summer and fall of 2020 before there was a vaccine. "And it gets worse than that," Swann says at 3:25. "Because if we're talking about deaths, the numbers haven't really changed either. And once again, it shows that the number of people dying in hospitals from COVID is higher if not statistically unchanged...when you take a macro look when you look at a country like Israel and say, let's look ... over time, have the numbers really changed? They haven't."
As he says at the 4:15 mark: "...none of the health issues surrounding COVID seem to be altered by the vaccine and certainly not in a positive way."
That's a misleading analysis, says professor Jeffrey Morris, the Ph.D. who is head of the biostatistics division of the University of Pennsylvania's Perelman School of Medicine. In a critique of misuses of the Israel health statistics Morris did not quarrel with the raw figures, writing:
... while these numbers are true, to quote them as evidence for low vaccine effectiveness is wrong and misleading. Sometimes, with observational data there is confounding of multiple factors that can make it easy to misinterpret simple percentages like this, and the current vaccination situation in Israel brings a perfect storm of confounding factors that lead to confusion if not thought through carefully.
Morris describes how the progress of a vaccine campaign can change the statistic sketch of vaccine effectiveness. "When vaccination rates are low, use of raw counts can exaggerate the vaccine effectiveness," he writes. "When vaccination rates are high, use of raw counts like this can attenuate the vaccine effectiveness, making it seem lower than it in fact is."
Morris said the vaccination program in Israel there are big age differences in vaccination rates. By the time late 2021's breakthrough infections hit, the patients needing hospital care tended to be older and more vulnerable:
In particular, the key factors here that contribute to this confusion are:
- High vaccination rates in the country (nearly 80% of all residents >12yr)
- Age disparity in vaccinations, including
- Nearly all older people being vaccinated (>90% of residents >50yr) and
- The vast majority of unvaccinated being younger people (>85% of unvaccinated <50yr)
- Older people are orders of magnitude more likely to be hospitalized with a respiratory virus than young people (residents >50yr are >20x more likely to have hospitalized serious infections than residents <50yr, and residents 90+ are >1,600x more likely to have hospitalized serious infections than residents 12-15yr)
Slicing Israel's data by age and vaccination status, Morris said it's clear the vaccines are working well, even against the highly infectious delta variant. Severe cases, he finds are 3.1x higher in unvaccinated than fully vaccinated individuals.
Splitting the population by under 50 and older than 50, Israel's data show 90% of the elderly are vaccinated, while just 73% of younger Israelis are vaccinated. Looking at those age groups, a very different picture emerges:
(Source: covid-datascience.com screenshot taken Mon Nov 1, 23:05:03 2021 UTC)
Morris said there are two systemic factors that confound a simple analysis of vaccine efficacy, making the overall number 67%, while in each of the broad age groups, it is markedly higher. "These effectiveness measures are quite high and suggests the vaccines are doing a very good job of preventing severe disease in both older and young cohorts," he wrote. To explain, he turned to data science:
This is an example of Simpson's Paradox, a well-known phenomenon in which misleading results can sometimes be obtained from observational data in the presence of confounding factors. ...The reason for this paradoxical result that is that the both dosage and recovery probability were systematically higher in one group (younger) and lower in the other group (older). This creates a specific type of confounding that can produce such a paradox ... if we do not stratify by the confounding factor (age), then the overall analysis gives a blatantly misleading result.
