The Results Should Make Sense (All of Them!)

I used to teach my students that a critical review of an observational study should start by searching for incoherent results. If found, the rest of the results should be ignored. They cannot be trusted.

If a student had asked why, I would draw an analogy to a courtroom. If a witness is caught lying about one part of their testimony, nobody is expected to believe the rest. We simply reject the entire testimony. At least most jurors will.

That’s an analogy, of course, not a claim that intentional wrongdoing was at play by the authors.

A recently posted article on Covid vaccines and mortality in Norway provides an opportunity to demonstrate that approach.

Using a retrospective cohort design, the authors reported a detailed analysis of all-cause mortality in Norway between 2021 and 2023 in relation to Covid vaccination status. There are lots of numbers in the paper, but I will examine data only from the elderly population (65+) among whom almost 90% of the deaths have occurred.

No waning of effectiveness?

I constructed the figure below from parts of the original figure in the article. Anything in red is mine.

That Covid vaccine effectiveness (VE), if any, attenuates over several months is uncontroversial. The authors also state so: “Vaccine effectiveness studies estimate that the protective effect of mRNA covid-19 vaccines against severe outcomes is significantly reduced six months after administration.” And elsewhere: “We expect the differences in mortality between the vaccination groups to wane over time, as the protective effect of the vaccine itself is reduced alongside the establishment of natural immunity in the general population.”

In the authors’ data, however, the opposite is observed (figure above). Comparing those who received only 1–2 doses to the unvaccinated, the adjusted rate ratio of death monotonically changed from 0.82 (VE=18%) in 2021 to 0.70 (VE=30%) in 2023. Since almost all the population received 1–2 doses in 2021, those who contributed 42,000 person-years at risk in 2023 (figure above) had been vaccinated 1–2 years earlier. There is no way that the effectiveness of the initial 1–2 doses has increased over time. That’s enough to discard the article, but there is more.

No healthy vaccinee bias?

Data from various countries point to the universal nature of the healthy vaccinee phenomenon. On average, people who are vaccinated have better background health than people who are not, and therefore, they are less likely to die from “anything.” This phenomenon does not disappear after stratification on age and is not fully eliminated by stringent methods of deconfounding, such as multivariable modeling and various forms of matching.

The authors are aware of the bias. They write: “There will always be a healthy vaccinee bias present in observational studies of vaccination outcomes.”

So, was the bias found in their data?

The average health status of each vaccination group was estimated by a binary variable called “medical risk group,” which was based on a complex algorithm. Then, risk group status was modeled as a covariate to account for confounding by baseline health status — essentially, to remove the healthy vaccinee bias.

Well, that variable did not perform as expected.

My table below was extracted from supplemental table 1. As you can see, the percentage of people who belonged to the risk group was higher, not lower, in the vaccinated than in their unvaccinated counterparts. They were sicker, not healthier. That’s the opposite of the healthy vaccinee phenomenon.

Of course, the authors acknowledge the finding:

“There was also a higher proportion of individuals with a risk condition among those vaccinated compared to those left unvaccinated through the study period…”

As a result, adjustment has moved the estimated effect away from the null rather than toward the null. They write:

“Due to the higher proportion of individuals with a risk condition among those that received a vaccination compared to those that were unvaccinated, these differences in rates of death became more pronounced in the adjusted models.”

That’s exactly what we see when adjusted rate ratios are compared to crude rate ratios (my tables, based on their data).

Was Norway an exception to the healthy vaccinee bias?

Not according to a previous publication from Norway. Like many others, however, the authors of that 2022 article (some of whom are also authors of the present article) dismiss the bias as a short-term phenomenon. That’s wishful thinking. The bias diminishes over time but does not disappear for many months. It’s not an on-off switch.

That we observe the opposite of the healthy vaccinee bias is a red flag, an incoherent result. I suspect that the authors did not expect this result, and their attempt to explain it is unconvincing.

Did the healthy vaccinee bias appear only when the first booster was administered?

The healthy vaccinee bias is not restricted to vaccinated versus unvaccinated. Those who continue to take the next dose are healthier than those who do not. There is a hint of the phenomenon in the data but only in 2021. In that year, recipients of 3+ doses (essentially, the third dose) appear to have been healthier than recipients of only 1–2 doses. There is no material difference in 2022, whereas the opposite is observed in 2023.

So, what happened to the rate ratio (3+ doses versus 1–2 doses) after adjustment?

My computation is shown below. I estimated the adjusted rate ratio in each year by dividing the adjusted rate ratios versus the unvaccinated.

Although the models included more than the risk group variable, the results follow the expectation in 2022 (no difference between crude and adjusted rate ratio) and in 2023 (some accentuation).

Not so in 2021. Since recipients of the third dose were healthier, adjustment should have attenuated the association, which was near null anyway. We observe the opposite: 0.98→0.76. We can add another incoherent result to the list.

It is impossible to tell exactly what went wrong in the analysis of Norway data. Accounting for time trends, waves, and variants is a difficult task, but one factor is clear. Their made-up risk group variable did not deliver the goods. Not only did it not capture the healthy vaccinee bias, but it somehow created bias by itself. In addition, collapsing the vaccination categories (1 dose or 2 doses; 3 or 4 or 5 doses) might have played a role. It was certainly unjustified.

Like Denmark and Finland, Norway succumbed to SARS-CoV-2 on a delayed timeline (graph below). It is impossible to reconcile highly effective Covid mRNA vaccines with the death toll of the virus in Norway in 2022. Correcting for a bias factor of 2 is enough to move the rate ratio (0.43) in 2022 close to the null.