A year ago I wrote a column about the paradoxical arithmetic of mass testing for a rare disease, showing that even very accurate tests are wrong more often than we think. It was a fun intellectual exercise thrown together as a last-minute replacement for a column that fell through, but when COVID-19 arrived, it suddenly became relevant.
I’ll reprint an updated version of the column below, but here’s the synopsis: When you’re testing people to see if they have a rare disease, simple arithmetic shows that positive results can be false more often than true even if the test is 95% accurate. Very weird.
However, this isn’t much of an issue with COVID-19 testing because (alas) the disease is not rare.
I bring this up again because news of potential vaccines with efficacy of around 95% has led people to ask me whether this paradox applies – that is, whether they won’t protect us as much as it sounds like they will.
The short answer is no, because the situation with a vaccine, which tries to prevent a disease, differs from a test, which tries to see if that disease exists, The results of vaccine trials are quite obvious: Participants either get sick or they don’t. No interpretation is involved – although statistical analysis is needed to make sure the result is not due to random fluctuations.
Anyway, even though the medical-testing paradox doesn’t really apply to vaccines, here’s an edited version of the two earlier columns I’ve written about it:
Let’s say we test 1 million New Hampshire residents for a novel virus, which is roughly everybody over age 18. And let’s say 4% of people have the disease, and we use a test that is 95% accurate.
In this scenario 40,000 people have COVID-19 (that’s 4% of 1 million). The test accurately finds 38,000 of them (95% of 40,000) and misses 2,000 (5% of 40,000).
On the other hand, 960,000 people (96% of 1 million) don’t have the disease. The test accurately tells 912,000 (95% of 960,000) that they are free of the disease but, and this is the key point, it inaccurately tells 5% of those people that they do have the disease. In other words, it gives 48,000 false positives.
So 48,000 people will get a false positive results and 38,000 a true positive. Even though each individual test is 95% likely to be right, overall there’s a 56% chance that a positive result is wrong.
It’s very counter-intuitive, even mind-blowing, but you can’t argue with arithmetic.
Even a test that is 99% accurate it would give false positive one out of every 12 times in this scenario. And if the disease is less widespread, the problem gets worse.
The problem goes away with more common diseases. If you don’t believe that, crunch the numbers from my example above but assume that 20% of the population is infected. Then do it again with 40% and 60%. You’ll see that the ratio of false positives declines to almost zero.
But wait, there’s more: Specificity and sensitivity. These are confusingly similar terms that refer to how accurate a test is when giving a positive result, and how accurate when giving a negative result.
It’s common for a test to be more accurate when saying “yea” than when saying “nay,” or vice-versa. It comes into play because society needs two types of tests to move on post-COVID.
One, currently performed by the tens of thousands daily, can spot the disease. It needs high sensitivity (meaning its negative results must be very accurate) but its specificity is less important.
Why? If a COVID-19 test gives me a false positive, telling me I have the disease when I don’t, then I will get worried but my life won’t be in danger. On the other hand, if a COVID-19 test gives a false negative, saying I’m free of virus when I actually have it, then I could walk around spreading it to other people. Which would be bad.
So when developing these tests – deciding how to collect mucus so you’re sure it contains virus when it’s present, how to transport and store it, how to expand samples with the process known as PCR without tainting the result, how to quickly and accurately interpret what you see – what really matters is that the negative results can be counted on. Being super-accurate with positive results is less important.
The opposite is the case for antibody tests, which don’t try to determine if you have COVID-19 right now but whether you have had COVID-19 in the past. That involves looking in your blood to find antibodies, the virus-fighting mechanism created by our immune system during the earlier infection.
Antibody tests must be highly specific, providing very few false positives. When they give a positive result, you must be very sure that the result is accurate.
Why? If it gives a false positive, I will think I’m protected by antibodies when I’m not. I’ll go out into the world unprotected and possibly get COVID-19, then maybe even die. Once again, that would be bad.
If an antibody test gives me a false negative, on the other hand, I will be inconvenienced but not endangered because I am protected. I’ll stay at home and wear a mask unnecessarily, but nothing more.
This difference between the need for specificity and sensitivity is one reason why developing tests is so hard. It’s not just that they have to be accurate, they have to be accurate in certain ways.
And then we decide how and when to use them, to decide whether the harm from inevitable false positives and false negatives outweighs the benefit of the true results.
So, yes, we should test for the disease and for antibodies, but we also need keep in mind what I said when I ended the first column last October: Public health is complicated. In some ways, I think it’s the most complicated thing that society does, since it has to factor in biology, economics, individual preferences, group behavior and politics.
And statistics. Whatever you do, don’t forget statistics.
Return to the Concord Monitor