Høeg et al. were right: the Covid booster had zero effectiveness

This is a (long) post on two competing methods to remove the healthy vaccinee bias, the fourth in a sequence. One method is telling us that the first booster had zero effectiveness against Covid death; the other — over 50%. Not a minor difference, considering the fact that much of the population was revaccinated (or perhaps just injected and suffered adverse effects and fatalities).

We should keep in mind the following methodological principles:

· Every method that is used to estimate an effect must be connected to background causal assumptions.

· Causal assumptions should be displayed in a causal diagram.

· Running statistical models alone is never sufficient.

That’s about all I wanted to share with most readers besides the conclusion in the title. I don’t expect a casual reader to understand the rest of the post. It is technical and complicated. I could have tried to write it up for a professional journal, but I no longer play on that field. By the way, you can sometimes find there peer-reviewed real nonsense on Covid vaccines.

I have a few discoveries to share.

First, I discovered small, relevant methodological literature that was missed by me and by other parties to the discussion on the healthy vaccinee bias. It was missed because the method was called “prior event rate ratio (PERR) adjustment,” an uninformative descriptor. Proposed long ago, the PERR computation accounts for all confounders simultaneously, measured and unmeasured, known and unknown. The healthy vaccinee bias is not fully removed because of unknown confounders.

Second, I discovered that my method (proposed earlier last year by another group) for removing the bias can be found in that literature. The context is different, and the phrase “healthy vaccinee bias” is not mentioned. Nonetheless, the underlying rationale and the trivial computation are identical.

Third, I (re-)discovered that causal diagrams clarify foggy notions of biases and “adjustments.” More importantly, results derived from causal diagrams agree with counterfactual reasoning about the removal of the healthy vaccinee bias. That’s reassuring.

It all boils down to this post on X (twitter) in response to correspondence in The New England Journal of Medicine on a study from Israel about the effect of a Covid booster dose. Høeg et al. successfully exposed the healthy vaccinee bias in that study. I was less successful back in 2021 (see my second example).

The approach delineated above was based on what was done in a study from Hungary. I will call it the Hungarian method.

There is a problem here, however. What is the theoretical basis for the method? How does the computation remove the healthy vaccinee bias? I explained my method, which was model-free. They did not. The numbers 0.10 and 0.23 were taken from multivariable models. Why did they use such models? For sure, it is not self explanatory, and hand-waving is not good enough.

My first task is to help followers of the Hungarian method understand exactly what they tried to achieve. I doubt they knew it. It is a long trail to follow.

Then there is another problem. The typical correction factor for the healthy vaccinee bias is between 2 and 3. According to my computation from the data exposed by Høeg et al., the bias factor was 20 (table below). Why was it so large? We usually fail to remove biases completely, not to correct them “too much.” Did my computation add more bias than was removed, setting the estimate exactly on the null?

That’s peculiar.

At the end, I will discuss another source of bias, which my method successfully removed. I will explain why the Hungarian method failed to do so. That’s why vaccine effectiveness was mistakenly estimated to be 57% instead of zero.

Let’s start with a simple case, a causal diagram that shows a basic confounding structure. Z is a binary confounder, a shared cause of VAX and Covid death.

On the left, we see the usual method to remove confounding: conditioning on Z (denoted by a box), which breaks the paths on both sides. We block the confounding path by using methods such as restriction to one value of Z, stratification on Z and a weighted average, or adding Z to a regression model.

On the right, we see a different approach. The explanation is somewhat lengthy, but not too complicated.

Consider two binary variables that are associated through an open path, whether a causal path or not. We may quantify the association by the probability ratio (PR) of the value of one variable in relation to the other, with no claim of causality. For example, we may compute the PR of death (vaccinated versus unvaccinated) or the PR of having been vaccinated (dead versus alive) or the PR of lung cancer (yellow stained fingers versus not). The number we compute in each case indicates a statistical association — one variable informs us about another. We don’t claim anything more. No causal language is invoked.

The right diagram above shows two paths between VAX and Covid death, each creating a path-specific association, which may be quantified by a probability ratio as I just explained. To simplify, I called them both risk ratios (RR) even though only one denotes an effect, RR causal path (VAX →Covid death).

The total (overall) association between VAX and Covid death is the product of the two risk ratios. For example, if the association along each path is RR=0.5, the overall association is RR=0.5x0.5=0.25. If it is 2 along one path and 0.5 along the other, the overall association is null (RR=1) and so on.

My long preamble explains the two equations below the graph:

Notice that we compute the true effect of VAX by dividing the overall association (which is not a measure of effect) by the association through the confounding path (which is not a measure of effect either). Interestingly, we may also phrase it as follows:

We compute the true RR (RR causal path) by dividing a biased RR (RR total) by another biased RR (RR confounding path). Remember: the last two do not quantify risks. They remain probability ratios in a statistical sense (i.e., measures of association).

For example, if RR total = 0.5 and RR confounding path = 0.8, then RR causal path = 0.5/0.8 = 0.62. The bias by Z was removed.

I have never seen this computation before, which is not surprising. How could we separate RR confounding path from RR total, when RR causal path is unknown and that’s what we want to estimate in the first place? We also need to estimate this associational path for many confounders. It sounds like an impossible task.

It turns out, however, that we can estimate this RR— from non-Covid deaths.

The bottom diagram is similar to the top diagram, except for RR causal path = 1. We do not expect a vaccine against Covid to prevent death from other causes (and let’s ignore vaccine-related death).

Therefore, we get a sequence of (almost) equalities.

RR total (non-Covid) = RR confounding path (non-Covid) ≈ RR confounding path (Covid).

We can now estimate the effect:

RR causal path (Covid) = RR total (Covid) / RR total (non-Covid)

Perfect?

No. No such thing in methodology. I have an approximation sign. Why?

The approximation arises from the right segment of the confounding path, which is not necessarily identical (light-blue arrow versus dark-blue arrow). The effect of Z on non-Covid death might differ from its effect on Covid death. For instance, the effect of age on death from stroke might differ from its effect on death from Covid.

Formally, the error here is called information bias. We substitute non-Covid death for Covid death because that’s the available information for the computation. There was no other choice. As a result of information bias, we might not completely remove confounding bias. That’s a key point, which is well known in methodology.

A short diversion:

The ideal substitute for Covid death is the kind of death that shares not only all the confounders for VAX and Covid death but also the magnitude of their effects on the latter.

Along this line of thought, it was proposed to try different causes of death or other outcomes, called “negative controls.” The idea belongs to “sensitivity analysis,” but there are two kinds: 1) state the expected results from competing outcomes and check your hypotheses against data. 2) conduct a fishing expedition. In my view, the latter is not sharp science.

Back to the main road.

Extending to all Z-type variables, we can remove the bias caused by all confounding paths.

RR causal path (Covid) ≈ RR total (Covid) / RR all confounding paths (non-Covid).

The denominator is the product of the probability ratio of non-Covid death, arising from every Z-type path. As we know, it is also RR total (non-Covid).

Let’s assume, next, that only three variables, X, Y, and Z, create confounding paths between VAX and Covid death. However, we only measured Z about which we knew.

Since we measured Z, we can reduce information bias — and thereby reduce confounding bias — by eliminating the need to estimate the bias due to the Z-path from non-Covid death.

We compute RR causal path from RR total (Covid) after conditioning on Z, divided by RR total (non-Covid) after conditioning on Z. If we knew and measured Y too, we would have conditioned on both. The more confounders in the models, the smaller the bias and the better the estimate of RR causal path. Of course, no matter how many variables we add, we will never add them all. An unknown number of confounders will always remain, at least theoretically.

By now, you might have recognized that that was the logic of the Hungarian method. I hope I clearly explained to them the rationale of dividing two hazard ratios from multivariable models. It was neither self explanatory nor simple.

How much we gain by this method is an empirical question. Perhaps a lot, perhaps not. We also pay a price in some models in the form of increased variance and run the risk of mistakenly including colliders in the model.

There is another pitfall. As the diagram shows, we should condition on the same set of confounders in the two models (Covid death, non-Covid death). The X-post’s suggestion to compute the booster effect from a model of non-Covid death that included hospitalization as a confounder (0.10/0.27) in not valid. The variable was not included in the model of Covid death. Likewise, some of the models in the Hungarian study were not valid for the same reason. Asthma status was strongly associated with mortality in the epidemic period (HR>3) but was not always computable in the non-epidemic period (Tables S6, S7).

My method was based on the same principle of partitioning an association, but without trying to reduce information bias:

RR total (Covid) / RR all confounding paths (Covid)

which is estimated by

RR total (Covid) / RR total (non-Covid)

The core idea of my method can be found in a paper from 2009, which refers to the original source from about the same time. As you can see from the quote, the topic is not the healthy vaccinee bias:

“We also used a prior event rate ratio (PERR) approach to adjust the Cox hazard ratio, as described recently.28 29 This analysis requires that neither the exposed nor unexposed patients are treated with the study drug before the start of the study. It assumes that the hazard ratio of the exposed to unexposed for a specific outcome before the start of the study reflects the combined effect of all confounders (both measured and unmeasured) independent of any influence of treatment. To apply the PERR adjustment method, we divided the unadjusted hazard ratio of exposed versus unexposed groups during the study by the unadjusted hazard ratio of exposed versus unexposed ‘before’ the study.”

Translation and extension: Divide RR total by RR all confounding paths. Estimate the latter from an earlier period (prior event) or from another event (non-Covid death) that meets the following criterion:

RR total (in an earlier period or for another event) = RR all confounding paths

Back to the Hungarian method.

How much improvement was achieved in the booster study from Israel? How different were the light-blue arrows from the dark-blue arrows for those paths that were blocked by the multivariable models? How big was the error in my method that simply included those confounders among all confounders rather than conditioning on them?

We can get a rough idea from two tables that summarize the results from multivariable models: Table 2 (article) and Table 1 (correspondence). I will consider hazard ratios as an approximation of probability ratios.

The left table shows “independent associations” of confounders with Covid death (dark-blue arrow). The product of 13 estimates (excluding the booster) is 1.10 x 2.49 x 0.98 x…x 1.10 = 77. This is the number that we try to estimate by modeling non-Covid death as the outcome.

The right table shows “independent associations” of confounders with non-Covid death (light-blue arrow). The product of 13 estimates is 1.27 x 1.11 x 1.00 x…x 0.90 = 60.

Which means that blocking 13 confounding paths (the Hungarian method) improved the estimation of RR causal path by a factor of 60/77. That’s not a substantial improvement. Stated differently, the estimate of Høeg et al. should have been higher than the estimate in the X-post by a factor of 77/60=1.27. Instead of RR causal = 0.435 (see again the first post), Høeg et al. should have found RR causal = 0.55.

But the estimate of Høeg et al. was 1, as I explicitly showed in my table above. No effect of the booster. What has moved their expected result from roughly RR=0.55 all the way to RR=1? What was the source of that additional factor of 2? Remember, we typically fail to remove biases completely, not to overcorrect bias erroneously.

I think I have an answer.

There was another kind of information bias: differential misclassification of the cause of death. The unvaccinated were much more likely to be classified as Covid death than the vaccinated, or vice versa.

The variable that was analyzed was not true Covid death; it was death that was classified as such (notation: Covid death*), sometimes correctly and sometimes incorrectly. What is the causal structure of this kind of information bias? What are the causes of Covid death* and non-Covid death*?

The diagram shows three: 1) the truth, of course; 2) vaccination status; 3) some Z-type variables. I omitted another group (X →Covid death*, non-Covid death*).

Under counterfactual reasoning, the bias factor of 20 in the study from Israel removed not only the healthy vaccinee bias, but also differential misclassification bias. It provided an answer to the following question:

What would have been the probability of being classified as Covid death in the vaccinated — if they were as unhealthy as the unvaccinated and they were also classified similarly?

Answer: 20 times the observed probability.

Elsewhere, I showed evidence of substantial misclassification of non-Covid deaths as Covid deaths in Israel. In another essay, I showed evidence that it was differential in the first vaccination campaign in early 2021.

Here is one piece of evidence from the first campaign, where officials encourage differential PCR testing.

This bias did not disappear six months later, when the booster was administered concurrent to another Covid wave. As a result of the policy (below), death of a vaccinated patient was less likely to be attributed to Covid than death of an unvaccinated patient.

How exactly did the misclassification happen? Was it influenced by vaccination status alone? Unlikely. The causal process was probably complicated. Besides differential PCR testing, a whole set of clinical circumstances influenced the mindset of the physician who filled out a death certificate, including Z-type variables.

The narrative above is called effect modification between vaccination status and some Z-type variables. (Statisticians like to call it interaction.) The effect of vaccination on Covid death* depended on the value of other variables. For instance, it might have been different for the young and the old or for men and women.

RR total (classified death) included paths of confounding and paths of differential misclassification. My computation accounted for both biases. Most importantly, I did not impose any statistical model on the causal structure.

That was not the case for the Hungarian method where multivariable models were fit to the data. It is difficult to tell exactly how this method failed, but the essence is this:

Adjusting for confounders that were effect modifiers distorted the picture. It was no longer just reducing one kind of information bias. The so-called multivariable adjustment altered 13 confounding paths where some variables played the role of modifiers of differential misclassification by vaccination status. The biases were not properly removed.

Formally, it is called model misspecification. Informally, it is a computational mess.

Høeg et al. were right. The booster had zero effectiveness against Covid death. As I wrote at the beginning: Running statistical models alone is never sufficient.

Acknowledgment: Tomas Fürst for comments on an early draft.

p.s., I was surprised to find out that the PERR method for deconfounding has been missing from classical epidemiology textbooks and key publications on methodology. I suspect that the reason is sociological/political: the authors did not hold a membership card in the exclusive club of “certified methodologists.”