STORY UPDATED: check for updates below.

No Excess Risk

Are Pfizer and Moderna mRNA COVID-19 vaccines associated with an increased risk of serious adverse events of special interest? No, that's not true: The U.S. Food and Drug Administration (FDA) said in an email to Lead Stories that the vaccines meet the agency's rigorous standards for safety, effectiveness, and manufacturing quality. In addition, the authors of this study created their own subset of serious adverse events of special interest (AESI).

The claim appeared in a peer-reviewed article published in the September 22, 2022, edition of the journal Vaccine (archived here), titled "Serious adverse events of special interest following mRNA COVID-19 vaccination in randomized trials in adults." It opened:

Introduction

In 2020, prior to COVID-19 vaccine rollout, the Brighton Collaboration created a priority list, endorsed by the World Health Organization, of potential adverse events relevant to COVID-19 vaccines. We adapted the Brighton Collaboration list to evaluate serious adverse events of special interest observed in mRNA COVID-19 vaccine trials.

This is what the post looked like in Vaccine at the time of writing:

(Source: Vaccine screenshot taken on Mon Sep 12 18:37:29 2022 UTC)

A nearly identical essay by the same authors with essentially the same headline appeared in a preprint article published by SSRN (archived here) on June 23, 2022, titled "Serious Adverse Events of Special Interest Following mRNA Vaccination in Randomized Trials." It opened:

Introduction: In 2020, prior to COVID-19 vaccine rollout, the Coalition for Epidemic Preparedness Innovations and Brighton Collaboration created a priority list, endorsed by the World Health Organization, of potential adverse events relevant to COVID-19 vaccines. We leveraged the Brighton Collaboration list to evaluate serious adverse events of special interest observed in phase III randomized trials of mRNA COVID-19 vaccines.

Both articles look at data that was included in the FDA clinical review memo, statistical safety review memo and other public documents used in the authorization of the Moderna and Pfizer (Pfizer partnered with BioNTech) vaccines. The "harm-benefit analyses" called for in the study have already been performed by the FDA. This type of analysis can't provide new information about product safety beyond what is already documented.

Additionally, the original clinical trials by Pfizer and Moderna are now nearly three years old and COVID variants like delta and omicron have changed the playing field. New trials would have to be conducted to determine if there's a cause and effect between the vaccines and the serious AESI, which are pre-specified potential side effects.

In its discussion section, the peer-reviewed version of the article comes to this conclusion:

The excess risk of serious adverse events found in our study points to the need for formal harm-benefit analyses, particularly those that are stratified according to risk of serious COVID-19 outcomes. These analyses will require public release of participant level datasets.

In Dr. David Gorski's September 5, 2022, article "Peer review fail: Vaccine publishes antivax propaganda disguised as 'reanalyses' of Pfizer and Moderna COVID-19 vaccine clinical trial data," he challenged the conclusion the study's authors came to, singling out Peter Doshi, the senior editor at the BMJ and a faculty member at the University of Maryland School of Pharmacy:

If you really want to know how safe and effective the vaccines are now (and have been over time), the original clinical trials are far from the best dataset to use, and this 'reanalysis' failed to find even a signal that there might have been a severe design defect or fraud in the original clinical trials. I'm all for data transparency and have no objections to releasing patient level data, but Doshi knows that this is not a trivial thing to do, as all the data have to be carefully de-identified and curated.

In a July 1, 2022, email to Lead Stories, Abram L. Wagner with the department of epidemiology at the University of Michigan provided his interpretation of the preprint version of the study, which he said in a September 12, 2022, email still holds with the peer-reviewed report:

The pre-print compares the number of adverse events in those vaccinated compared to those not vaccinated. Even taking the results at face value, there needs to be a cautious interpretation of them. The data they analyzed included almost 40,000 individuals enrolled in clinical trials, and yet their results include wide confidence (or 'compatibility') intervals. This speaks to the very low number of adverse events as well as uncertainty in their analysis.

I have the following issues with the study. For one, they do not have access to individual-level data, and so have to compute statistics based on aggregate data. This limits a comparison of their data with what vaccine companies previously published. So in this pre-print, one individual reporting multiple adverse events over time is counted as multiple events, whereas in previous reports, this would just be one case of an adverse event. Their analysis therefore could result in double counting of events if they derive from a single adverse reaction to a vaccine.

Additionally, they modified the adverse events included in the original Bright Collaboration list. They do include the original list as a sensitivity analysis, and this sensitivity analysis shows a much broader, and therefore higher level of uncertainty in their data.

The authors of the study effectively created their own special list of AESI. Adverse events can be either serious or non-serious. The study only looks at what it calls the serious ones. It's a category defined by the authors of the study and isn't used by government agencies when putting vaccines through the authorization process.

In his September 12, 2022, email, Wagner said he welcomed the release of additional vaccine trial data as the study authors have called for but also backed the value and efficacy of the shots while challenging the methods of the peer-reviewed article:

Release of participant-level data is important and would be welcome for efforts to better understand the functioning of COVID vaccines. ... COVID-19 is a serious disease and the vaccines are highly effective. Adults who get vaccinated protect themselves and those around them. Further harm-benefit analyses could be useful but should consider a population and not just individual - level approach.

Wagner's emails point to issues known in statistical analysis known as p-hacking/selective reporting and data dredging/data fishing. Basically, it's a way of manipulating a data pool. Doing so can unearth something that got lost in the broader context, but it can also be used to show results that really aren't backed by the data.

In a June 24, 2022, email to Lead Stories, the FDA said it found the preprint article lacking:

FDA disagrees with the conclusions in the paper from Doshi et.al. Based on the agency's thorough evaluation of the safety and effectiveness data for the mRNA COVID-19 vaccines, as well as the ongoing safety surveillance of the vaccines, we continue to find their benefits far outweigh their risks in preventing COVID-19, including its most serious outcomes of hospitalization and death.

Lead Stories has asked the FDA for a response to the peer-reviewed version of the article and will add it when it becomes available.

In a July 1, 2022, email to Lead Stories, the University of Maryland, Baltimore responded to a request for comment on the study:

The paper you referenced was co-authored by a faculty member in our school of pharmacy, Peter Doshi. Faculty at the University of Maryland, Baltimore are free to research and publish, and to engage in public discourse as they will, but in so doing they do not represent the opinions or views of the university.

In its email to Lead Stories, the FDA said it takes adverse events seriously and investigates them:

It's important to note that reports of adverse events to the Vaccine Adverse Event Reporting System (VAERS) following vaccination, including deaths, do not necessarily mean that a vaccine caused a health problem. In fact, reviews by FDA and CDC have determined that the vast majority of the deaths reported are not directly attributable to the vaccines. FDA requires healthcare providers to report any death after COVID-19 vaccination to VAERS, even if it's unclear whether the vaccine was the cause. Reports of adverse events to VAERS following vaccination, including deaths, do not necessarily mean that a vaccine caused a health problem.

VAERS data

The data on adverse events comes from the Vaccine Adverse Event Reporting System (VAERS). It is co-managed by the FDA and the Centers for Disease Control and Prevention as a crude early warning system and not as a database for the quantification of specific outcomes.

Anyone with internet access can add a report to the VAERS list of reports. The public access link to it expressly warns against unwarranted conclusions based on VAERS material because the list only provides a tally of unverified notes about any health event people experience after they are vaccinated.

The list itself cannot be used to prove or quantify, since all it shows is a chronological correlation, not the causal link that would be more difficult to establish. It's the equivalent of a police precinct's running "blotter" reports that may serve as a starting point for police work, not an endpoint.

VAERS website warnings include:

When evaluating data from VAERS, it is important to note that for any reported event, no cause-and-effect relationship has been established. Reports of all possible associations between vaccines and adverse events (possible side effects) are filed in VAERS. Therefore, VAERS collects data on any adverse event following vaccination, be it coincidental or truly caused by a vaccine. The report of an adverse event to VAERS is not documentation that a vaccine caused the event.

Vaccination deaths rare

The FDA continued in its email, saying COVID shots are safe:

Reports of death after COVID-19 vaccination are rare. As reported on CDC's website, More than 592 million doses of COVID-19 vaccines were administered in the United States from December 14, 2020, through June 21, 2022. During this time, VAERS received 15,193 preliminary reports of death (0.0026%) among people who received a COVID-19 vaccine. CDC and FDA clinicians review reports of death to VAERS including death certificates, autopsy, and medical records.

To put into perspective the approximate number of doses of vaccine administered during a specific time frame after authorization of COVID-19 vaccines we suggest that you reference CDC's data at https://covid.cdc.gov/covid-data-tracker/#vaccinations_vacc-total-admin-rate-total. CDC clinicians review reports of death to VAERS including death certificates, autopsy, and medical records.

FDA continues to find the three authorized and approved COVID-19 vaccinations meet the agency's rigorous standards for safety, effectiveness, and manufacturing quality. These vaccines have proven to be an important tool for fighting the COVID-19 pandemic.