Those who suspend judgment – ephektikoi

“Ancient skepticism: The Greek word skepsis means investigation. By calling themselves skeptics, the ancient skeptics thus describe themselves as investigators. They also call themselves ‘those who suspend’ (ephektikoi), thereby signaling that their investigations lead them to suspension of judgment. They do not put forward theories, and they do not deny that knowledge can be found. At its core, ancient skepticism is a way of life devoted to inquiry. Also, it is as much concerned with belief as with knowledge. As long as knowledge has not been attained, the skeptics aim not to affirm anything. This gives rise to their most controversial ambition: a life without belief.”

Stanford Encyclopedia of Philosophy

First published Wed Feb 24, 2010; substantive revision Fri Jul 20, 2018

https://plato.stanford.edu/entries/skepticism-ancient/

They can not all be correct, again

There are more opinions, sometimes camps, on anything you could possibly debate, and some views you would think no intelligent person could debate. Almost any assertion can be found to be made by someone with some presumed expertise in a relevant field. Almost any assertion can be backed up by some interpretation of published studies – probably cherry picked and selectively interpreted. The problem is, since these various assertions and studies contradict each other, they cannot all be true. It does not follow that any are correct. I still stick by my analysis made in my article on experts: https://www.opednews.com/articles/Trusting-the-experts-by-Mike-Zimmer-Epistemology_Expertise_Ideas–Philosophy_Philosophy-200720-688.html

Testing: simplest explanation of false alarms to hits

 

We have a test for some condition, say it is the dreaded Lurgi, and are testing some sample of the population, say 10,000 folks.

We have reason to believe that the prevalence of Lurgi is 1% of our population. That means that 99% do not have the condition.

We have an estimate that this test correctly identifies 99% of the people with Lurgi where the condition exists, missing 1%. This former percentage is called the sensitivity of the test.

We also have an estimate that it correctly identifies 99% of the people without Lurgi where the condition does not exist, giving false alarms for 1%.  This former percentage is called the specificity or selectivity of the test.

Higher sensitivity or selectivity or both reduce the rate of errors. We can call this the power of the test.

If we test the 1% of the people with Lurgi, we can expect to identify 99% of that 1% or 0.99% of the sample correctly as having the condition, missing 1% of that 1% or 0.01% of the sample.

This also means that if we test the 99% of the people without Lurgi, we can expect to identify 99% of that 99% or 98.01% of the sample correctly as not having the condition, with false alarms of  1% of that 99% or 0.99% of the sample.

Since our sample is 10,000 people, we get the following results:

Prevalence With Condition (1%) Without Condition (99%) Totals
Tests Positive Hits = 99

(10,000 X .99 X .01)

False Alarms = 99

(10,000 X .01 X .99)

198
Tests Negative Misses = 1

(10,000 X .01 X .01)

Correct Rejections = 9,801

(10,000 X .99 X .99)

9,802
Totals 100 9,900 10,000

You can see that for this population and test power, the number of false alarms is as great as the number of hits. That is, the number of hits is overstated by a factor of two, since we do not know from the test which ones are the false alarms, only their expected frequency.

The misses are not a problem. If you reverse the prevalence numbers, so that 99% have the condition, the false alarm problem goes away, and the miss problem becomes significant. For prevalence figures in the middle, the problems with errors become smaller.

With poor tests, less power, we get more errors. With even lower prevalence, we get more false alarms.

In medicine and statistics in general:

  1. hits are called true positives
  2. misses are called false negatives
  3. correct rejections are called true negatives
  4. false alarms are called false positives

You may encounter these terms. The language I have used is both more intuitively understood, and is also the language of the formal methods underlying this type of analysis, Signal Detection Theory. You can relate it to smoke detectors, fire alarms, or many common situations, once you understand it.

 

Medical Testing in Brief

Overview

With respect to medical testing in general, the prevalence of a condition in the population has a huge effect on the number of false alarms and misses. Depending on the prevalence of the condition, and the performance of the test, the number of false alarms can exceed the number of hits. It is simple arithmetic, but a bit confusing.

If there is 0% infection in the population, any report of infection logically has to be a false alarm; there can be no hits. If there is 100% infection in the population, every report of infection logically has to be a hit; there can be no false alarms. The number of estimated false alarms drops as the percent infection rises.

If there is 0% infection in the population, logically there can be no misses. If there is 100% infection in the population, any report of no infection logically must be a miss. The number of estimated misses rises as the percent infection rises.

This is true even with a test of high sensitivity (detects infections if infection exists, with not too many misses) and high selectivity/specificity (detection of no infection if no infection exists, with not too many false alarms).

As sensitivity increases the number of hits increases and number of misses decreases. With a sensitivity of 100%, which does not happen, there would be no misses. See the graph below for a visual explanation of this.

As selectivity/specificity increases the number of correct rejections increases and the number of false alarms decreases. With a selectivity/specificity of 100%, which does not happen, there would be no false alarms. See the graph below for a visual explanation of this.

If selectivity/specificity is low, you get more false alarms. Combine this with a low prevalence, and the false alarms outshine the hits. This overstates the prevalence of the infection.

If the sensitivity is low, you get more misses. Combine this with a high prevalence, and the misses outshine the correct rejections. This understates the prevalence of the infection.

An Example

Use the calculations:

  1. Hits = Sensitivity X Test Cases X Prevalence
  2. False Alarms = (1-Selectivity) X Test Cases X  (1- Prevalence)
  3. Misses = (1 – Sensitivity) X Test Cases X Prevalence
  4. Correct Rejections = Selectivity X Test Cases X (1 – Prevalence)
Sensitivity Selectivity/

specificity

Test Cases Prevalence Hits False Alarms Misses Correct Rejections
95% 85% 10000 0.00% 0 1500 0 8500
95% 85% 10000 10.00% 950 1350 50 7650
95% 85% 10000 20.00% 1900 1200 100 6800
95% 85% 10000 30.00% 2850 1050 150 5950
95% 85% 10000 40.00% 3800 900 200 5100
95% 85% 10000 50.00% 4750 750 250 4250
95% 85% 10000 60.00% 5700 600 300 3400
95% 85% 10000 70.00% 6650 450 350 2550
95% 85% 10000 80.00% 7600 300 400 1700
95% 85% 10000 90.00% 8550 150 450 850
95% 85% 10000 100.00% 9500 0 500 0

Note the changes in 1) false alarms to hits, and 2) misses to correct rejections, as the prevalence increases.

Another Example Showing Ratios of Incorrect to Correct

For a selectivity/specificity of 99%, a sensitivity of 99%, and a prevalence of 1%, 50% of positives will be false alarms. It gets better as the prevalence increases (table below).

For a selectivity/specificity of 99%, a sensitivity of 99%, and a prevalence of 99%, 50% of negatives will be misses. It gets better as the prevalence decreases (table below).

 Prevalence

0.00% 1.00% 2.00% 50.00% 98.00% 99.00% 100.00%
False Positive to True Positive (FP/TP) Infinite 1.00 0.49 0.01 0.00 0.00 0.00
False Negative to True Negative Ratio (FN/TN) 0.00 0.00 0.00 0.01 0.49 1.00 Infinite
False Positive to All Positive (FP/(TP + FP)) 100.00% 50.00% 33.11% 1.00% 0.02% 0.01% 0.00%
False Negative to All Negative (FN/(TN + FN)) 0.00% 0.01% 0.02% 1.00% 33.11% 50.00% 100.00%

Measurement

In medical testing, we need to establish some test measure, some quantification of the condition of interest. We then need some method of assessment for determining this measurement, the values for that condition of interest. We need a consistent and reliable protocol for administering and scoring the test.

Discriminating Signal from Noise

We can look at the measure as the signal. We can look at spurious signals as noise, extraneous information which will make it hard to detect the signal.

We need some method to discriminate the signal from the noise. Different tests have different abilities to differentiate the signal from random noise. That is to say not all tests give the same level of performance.

The Right and the Wrong of It

In signal detection theory, there are two ways to be wrong: false alarms and misses, and there are two ways to be right: hits and correct rejections.

  1. Hits are a measure of how many with the condition are correctly identified as having the condition. This is also called a true positive (TP).
  2. False alarms are a measure of how many without the condition are incorrectly identified as having the condition. This is also called a false positive (FP).
  3. Misses are a measure of how many with the condition are incorrectly identified as not having the condition. This is also called a false negative (FN).
  4. Correct rejections are a measure of how many without the condition are correctly identified as not having the condition. This is also called a true negative (TN).

Measuring Performance

Sensitivity, given as a fraction or a percent, is the ability of a test to correctly identify those who have the condition. It can only be assessed against the percentage of those who have the condition, the prevalence. It gives a rate for hits, and when the one’s complement is taken, a rate for misses.

Selectivity/specificity, given as a fraction or a percent, is the ability of a test to correctly identify those who do not have the condition. It can only be assessed against the percentage of those who do not have the condition, the infrequency. It gives a rate of correct rejections, and when the one’s complement is taken, a rate for false alarms.

Setting a Threshold

For a given test method we establish a threshold, some cut-off value, for our measurement. We use this to determine if we are getting a signal, or just noise. Above the threshold a measure will be deemed to be a detected signal, below the threshold will be no detected signal. We can set the threshold to bias the detection one way or another. The resulting differences in type of error will be dependent upon the threshold setting.

The ratio of hits to misses depends on the threshold, as does the ratio of correct rejections to false alarms. A decreased threshold shifts the bias towards more hits and fewer misses. At the same time, it shifts the bias towards more false alarms and fewer correct rejections. So hits and false alarms rise and fall in the same direction according to the bias.

Improving the Detector

A better test gives better discrimination of correct versus incorrect results, that is, better accuracy. This can be accomplished by changing the test, or changing the test protocols.

Looking at Counts

We may have statistics on expected performance for our test, but we also want to calculate estimated statistics for some given number of tests. We will want to count the number of independent tests performed and use those numbers in our calculations.

Prevalence and Infrequency

Prevalence is the estimated measure of the percent of the total population who have the condition.

Infrequency is the complement of prevalence, and is the estimated measure of the percent of the total population who do not have the condition.

Note that I use the word infrequency, as an antonym to frequency, which itself is a synonym for prevalence. There may be another term in common use, but I did not discover such.

Accuracy, Discrimination and Errors

We can compute a simple measure of accuracy by taking the total errors and dividing by the total of correct plus erroneous observations. With a better test, more discriminatory power, more accuracy, the error rate decreases.

With decreasing prevalence, the number of false alarms increases, and the number of misses decreases.

With an increasing prevalence, the number of false alarms decreases, and the number of misses increases.

Ratio of Errors to Correct

The ratio of false alarms to hits is found by taking the infrequency rate multiplied by the expected false alarm percentage and then dividing this quantity by the prevalence multiplied by the expected hit percentage.

The ratio of misses to correct rejections is found by taking the prevalence rate multiplied by the expected miss rate and then dividing this quantity by the infrequency rate multiplied by the expected correct rejection percentage.

Working through an example

Let me work through an example:

  1. We have an outbreak of Lurgi in Upper Middlemarsh. It is a terrible disease. 1
  2. We have reason to believe that 2% of the population is infected. This is the prevalence. So 98% will not be infected. This is the infrequency.
  3. We have a test for Lurgi, validated so that it identifies 90% of those infected and misses 10%. This is the sensitivity of the test.
  4. The test also identifies, gives a correct rejection, of 90% of those not infected, and gives a false alarm for 10% of those who are not infected. This is the selectivity/specificity of the test.
  5. We test 1000 people once. This is the test count. Of those 1000 people, 2% or 20 will be infected. It follows that 98% or 980 will not be infected.
  6. If we run our test on the 2% infected, it will identify correctly 90% or 18. These are the hits.
  7. It follows that 10% or 2 of those infected will be missed. These are the misses.
  8. If we run our test on that 98% not infected, it will identify correctly 90% or 882. These are the correct rejections.
  9. It follows that 10% or 98 of those not infected will be identified as infected. These are the false alarms.
  10. We have 18 hits versus 98 false alarms.
  11. We have 2 misses verses 882 not infected, correct rejections.
  12. It can be concluded that the false alarm rate to hit rate of 98/18 = 544% is very bad because of this mediocre level of test accuracy and the low disease prevalence.
  13. It can be concluded that the misses to correct rejection rate of 2/882 = 23% is not as bad because of the high disease infrequency.
  14. If we reverse this, changing the prevalence figure to 90% infected, the false alarms become small and the missed infections much larger.
  15. If we have a better test, we can reduce the error rate.
  16. We should try to get a better test.

I’m going to put these figures in the table below:

Does The Condition Exist?

Testing 1000 for Lurgi

with Sensitivity of 90% and Selectivity/specificity of  90%

Condition Exists

2% Estimated Prevalence

= 20

Condition Is Absent

98% Estimated Infrequency = 980

Was the Effect Observed? Effect Observed 90% x 20 = 18 Hits 10% x 980 = 98 False Alarms
Effect Not Observed 10% x 20 = 2 Misses 90% x 980 = 882 Correct Rejections
Estimated Counts

Based on Test Performance, Prevalence, and Number of Tests

1 https://gladbloke.wordpress.com/2009/06/04/the-dreaded-lurgi/

See also:

  1. https://ephektikoi.ca/blog/2020/10/11/selectivity-sensitivity-and-prevalence/
  2. https://ephektikoi.ca/blog/2020/10/13/bibliography-for-testing-and-uncertainty/
  3. https://ephektikoi.ca/blog/2020/10/12/true-and-false-key-aspects-of-medical-tests/

 

Bibliography for testing and uncertainty

  1. administrator, JHCHS website. “Global Progress on COVID-19 Serology-Based Testing.” Johns Hopkins Center for Health Security. Accessed September 29, 2020. https://www.centerforhealthsecurity.org/resources/COVID-19/serology/Serology-based-tests-for-COVID-19.html.
  2. Baratloo, Alireza, Mostafa Hosseini, Ahmed Negida, and Gehad El Ashal. “Part 1: Simple Definition and Calculation of Accuracy, Sensitivity and Specificity.” Emergency 3, no. 2 (2015): 48–49.
  3. Coursera. “Bayes’ Rule and Diagnostic Testing – The Basics of Bayesian Statistics.” Accessed September 29, 2020. https://www.coursera.org/lecture/bayesian/bayes-rule-and-diagnostic-testing-5crO7.
  4. Be Smart Wear A Mask. Accessed October 6, 2020. https://www.youtube.com/watch?v=KANNNty9V3o.
  5. HubPages. “Chief Science Officer for Pfizer Says ‘Second Wave’ Faked on False-Positive COVID Tests, ‘Pandemic Is Over.’” Accessed September 28, 2020. https://hubpages.com/politics/Pfizer-Chief-Science-Officer-Second-Wave-Based-on-Fake-Data-of-False-Positives-for-New-Cases-Pandemic-is-Over.
  6. Cohen, Andrew N, Bruce Kessel, and Michael G Milgroom. “Diagnosing COVID-19 Infection: The Danger of over-Reliance on Positive Test Results.” Preprint. Epidemiology, May 1, 2020. https://doi.org/10.1101/2020.04.26.20080911.
  7. Colombo, Marco, Joseph Mellor, Helen M. Colhoun, M. Gabriela M. Gomes, and Paul M. McKeigue. “Trajectory of COVID-19 Epidemic in Europe.” MedRxiv, September 28, 2020, 2020.09.26.20202267. https://doi.org/10.1101/2020.09.26.20202267.
  8. Global News. “COMMENTARY: Understanding False Positive and False Negative COVID-19 Tests.” Accessed October 2, 2020. https://globalnews.ca/news/7323725/what-do-false-positive-covid-19-tests-mean/.
  9. “Confusion Matrix – an Overview | ScienceDirect Topics.” Accessed September 30, 2020. https://www.sciencedirect.com/topics/computer-science/confusion-matrix.
  10. STAT. “Covid-19 Testing: Overcoming the Challenges,” March 31, 2020. https://www.statnews.com/2020/03/31/covid-19-overcoming-testing-challenges/.
  11. D, David M. Green PH, and John a Swets. Signal Detection Theory & Psychophysics. Los Altos Hills, Calif: Peninsula Pub, 1989.
  12. Diamond, George A. “The Wizard of Odds: Bayes’ Theorem and Diagnostic Testing.” Mayo Clinic Proceedings 74, no. 11 (November 1, 1999): 1179–82. https://doi.org/10.4065/74.11.1179.
  13. Diamond, George A. “The Wizard of Odds: Bayes’ Theorem and Diagnostic Testing,” n.d., 4.
  14. Doshi, Peter. “Covid-19: Do Many People Have Pre-Existing Immunity?” BMJ 370 (September 17, 2020). https://doi.org/10.1136/bmj.m3563.
  15. “FAQ: Testing for COVID-19 | MIT Medical.” Accessed September 29, 2020. https://medical.mit.edu/faqs/faq-testing-covid-19.
  16. ResearchGate. “Fig. 1.-Signal Detection Theory (SDT) Contingency Table, or “confusion…” Accessed September 30, 2020. https://www.researchgate.net/figure/Signal-detection-theory-SDT-contingency-table-or-confusion-matrix-TP-true_fig1_263812021.
  17. Floriano, Idevaldo, Antônio Silvinato, Wanderley M. Bernardo, João C. Reis, Gabriel Soledade, Idevaldo Floriano, Antônio Silvinato, Wanderley M. Bernardo, João C. Reis, and Gabriel Soledade. “Accuracy of the Polymerase Chain Reaction (PCR) Test in the Diagnosis of Acute Respiratory Syndrome Due to Coronavirus: A Systematic Review and Meta-Analysis.” Revista Da Associação Médica Brasileira 66, no. 7 (July 2020): 880–88. https://doi.org/10.1590/1806-9282.66.7.880.
  18. Gomes, M. Gabriela M., Rodrigo M. Corder, Jessica G. King, Kate E. Langwig, Caetano Souto-Maior, Jorge Carneiro, Guilherme Goncalves, Carlos Penha-Goncalves, Marcelo U. Ferreira, and Ricardo Aguas. “Individual Variation in Susceptibility or Exposure to SARS-CoV-2 Lowers the Herd Immunity Threshold.” MedRxiv, May 21, 2020, 2020.04.27.20081893. https://doi.org/10.1101/2020.04.27.20081893.
  19. Green, David Marvin, and John A Swets. Signal Detection Theory and Psychophysics. New York: Wiley, 1966.
  20. Grifoni, Alba, Daniela Weiskopf, Sydney I. Ramirez, Jose Mateus, Jennifer M. Dan, Carolyn Rydyznski Moderbacher, Stephen A. Rawlings, et al. “Targets of T Cell Responses to SARS-CoV-2 Coronavirus in Humans with COVID-19 Disease and Unexposed Individuals.” Cell 181, no. 7 (June 25, 2020): 1489-1501.e15. https://doi.org/10.1016/j.cell.2020.05.015.
  21. Harvey, Lew. “Detection Theory.” Psychology of Perception, 2014, 17.
  22. Heneghan, Prof Carl. “How Many Covid Diagnoses Are False Positives? | The Spectator.” Accessed October 6, 2020. https://www.spectator.co.uk/article/how-many-covid-diagnoses-are-false-positives-.
  23. Lockdown Sceptics. “How Likely Is a Second Wave?” Accessed September 28, 2020. https://lockdownsceptics.org/addressing-the-cv19-second-wave/.
  24. Lockdown Sceptics. “How Likely Is a Second Wave?” Accessed September 30, 2020. https://lockdownsceptics.org/addressing-the-cv19-second-wave/.
  25. Lockdown Sceptics. “How Likely Is a Second Wave?” Accessed October 6, 2020. https://lockdownsceptics.org/addressing-the-cv19-second-wave/.
  26. Jefferson, Prof Carl Heneghan & Tom. “Coronavirus Cases Are Mounting but Deaths Remain Stable. Why? | The Spectator.” Accessed October 6, 2020. https://www.spectator.co.uk/article/coronavirus-cases-are-mounting-but-deaths-remain-stable-why-.
  27. ———. “It’s a Mistake to Think All Positive Covid Tests Are the Same | The Spectator.” Accessed October 6, 2020. https://www.spectator.co.uk/article/it-s-a-mistake-to-think-all-positive-covid-tests-mean-the-same.
  28. Kahneman, Daniel, and Amos Tversky. “On the Psychology of Prediction.” Psychological Review 80, no. 4 (1973): 237–51. https://doi.org/10.1037/h0034747.
  29. learncuriously. “Confused by The Confusion Matrix Part 2: ‘Accuracy’ Is But One of Many Measures of Accuracy….” The Curious Learner (blog), October 27, 2018. https://learncuriously.wordpress.com/2018/10/28/confused-by-the-confusion-matrix-part-2/.
  30. ———. “Confused by The Confusion Matrix: What’s the Difference between Hit Rate, True Positive Rate, Sensitivity, Recall and Statistical Power?” The Curious Learner (blog), October 20, 2018. https://learncuriously.wordpress.com/2018/10/21/confused-by-the-confusion-matrix/.
  31. Lesaffre, Emmanuel, Niko Speybroeck, and Dirk Berkvens. “Bayes and Diagnostic Testing.” Veterinary Parasitology 148, no. 1 (August 2007): 58–61. https://doi.org/10.1016/j.vetpar.2007.05.010.
  32. Lockdown Sceptics. “Lies, Damned Lies and Health Statistics – the Deadly Danger of False Positives.” Accessed October 6, 2020. https://lockdownsceptics.org/lies-damned-lies-and-health-statistics-the-deadly-danger-of-false-positives/.
  33. “Likelihood Ratios and Diagnostic Tests (Bayes’ Theorem) – StatsDirect.” Accessed September 29, 2020. https://www.statsdirect.com/help/clinical_epidemiology/likelihood_ratios.htm.
  34. Lisboa Bastos, Mayara, Gamuchirai Tavaziva, Syed Kunal Abidi, Jonathon R Campbell, Louis-Patrick Haraoui, James C Johnston, Zhiyi Lan, et al. “Diagnostic Accuracy of Serological Tests for Covid-19: Systematic Review and Meta-Analysis.” The BMJ 370 (July 1, 2020). https://doi.org/10.1136/bmj.m2516.
  35. Lourenco, Jose, Francesco Pinotti, Craig Thompson, and Sunetra Gupta. “The Impact of Host Resistance on Cumulative Mortality and the Threshold of Herd Immunity for SARS-CoV-2.” MedRxiv, July 16, 2020, 2020.07.15.20154294. https://doi.org/10.1101/2020.07.15.20154294.
  36. Macmillan, Neil A. “Signal Detection Theory.” In Stevens’ Handbook of Experimental Psychology: Methodology in Experimental Psychology, Vol. 4, 3rd Ed, 43–90. Hoboken, NJ, US: John Wiley & Sons Inc, 2002.
  37. Macmillan, Neil A., and C. Douglas Creelman. Detection Theory: A User’s Guide. Mahwah, N.J: Psychology Press, 2004.
  38. Mandrekar, Jayawant N. “Simple Statistical Measures for Diagnostic Accuracy Assessment.” Journal of Thoracic Oncology 5, no. 6 (June 1, 2010): 763–64. https://doi.org/10.1097/JTO.0b013e3181dab122.
  39. Nevin, John A. “SIGNAL DETECTION THEORY AND OPERANT BEHAVIOR: A Review of David M. Green and John A. Swets’ Signal Detection Theory and Psychophysics.1.” Journal of the Experimental Analysis of Behavior 12, no. 3 (1969): 475–80. https://doi.org/10.1901/jeab.1969.12-475.
  40. ResearchGate. “(PDF) The Base Rate Fallacy in Probability Judgments.” Accessed September 28, 2020. https://www.researchgate.net/publication/223684493_The_Base_Rate_Fallacy_in_Probability_Judgments.
  41. Peak Prosperity. “Perspective on ‘Cases’ and PCR.” Accessed September 30, 2020. https://www.peakprosperity.com/forum-topic/perspective-on-cases-and-pcr/.
  42. Peak Prosperity. “Presumptive Positive PCR Test Definition.” Accessed September 30, 2020. https://www.peakprosperity.com/forum-topic/presumptive-positive-pcr-test-definition/.
  43. Ramdas, Kamalini, Ara Darzi, and Sanjay Jain. “‘Test, Re-Test, Re-Test’: Using Inaccurate Tests to Greatly Increase the Accuracy of COVID-19 Testing.” Nature Medicine 26, no. 6 (June 2020): 810–11. https://doi.org/10.1038/s41591-020-0891-7.
  44. Read “Intelligence Analysis: Behavioral and Social Scientific Foundations” at NAP.Edu. Accessed September 28, 2020. https://doi.org/10.17226/13062.
  45. “ROC Analysis: Web-Based Calculator for ROC Curves.” Accessed October 11, 2020. http://www.rad.jhmi.edu/jeng/javarad/roc/JROCFITi.html.
  46. “ROC Curves – What Are They and How Are They Used?” Accessed October 4, 2020. https://acutecaretesting.org/en/articles/roc-curves-what-are-they-and-how-are-they-used.
  47. Root, Matthew. “COVID-19 Laboratory Testing Q&As,” n.d., 6.
  48. “Signal Detection Theory.” Accessed September 30, 2020. https://www.cns.nyu.edu/~david/handouts/sdt/sdt.html.
  49. “Signal Detection Theory.” Accessed September 30, 2020. https://www.cns.nyu.edu/~david/courses/perception/lecturenotes/sdt/sdt.html.
  50. “Signal Detection Theory.” Accessed September 30, 2020. https://www.cns.nyu.edu/~david/courses/perception/lecturenotes/sdt/sdt.html.
  51. “Signal Detection Theory – an Overview | ScienceDirect Topics.” Accessed October 3, 2020. https://www.sciencedirect.com/topics/computer-science/signal-detection-theory.
  52. “Signal Detection Theory: Signals and Noise | American Association for the Advancement of Science.” Accessed September 30, 2020. https://www.aaas.org/signal-detection-theory-signals-and-noise.
  53. “Signal Detection Theory: The Receiver Operating Characteristic | American Association for the Advancement of Science.” Accessed September 30, 2020. https://www.aaas.org/signal-detection-theory-receiver-operating-characteristic.
  54. “Signals, Sensors, and Data | American Association for the Advancement of Science.” Accessed September 30, 2020. https://www.aaas.org/signals-sensors-and-data.
  55. Šimundić, Ana-Maria. “Measures of Diagnostic Accuracy: Basic Definitions.” EJIFCC 19, no. 4 (January 20, 2009): 203–11.
  56. Surkova, Elena, Vladyslav Nikolayevskyy, and Francis Drobniewski. “False-Positive COVID-19 Results: Hidden Problems and Costs.” The Lancet Respiratory Medicine 0, no. 0 (September 29, 2020). https://doi.org/10.1016/S2213-2600(20)30453-7.
  57. Swets, John A. “Detection Theory and Psychophysics: A Review.” Psychometrika 26, no. 1 (March 1, 1961): 49–63. https://doi.org/10.1007/BF02289684.
  58. “The Great Covid-19 Versus Flu Comparison Revisited.” Bloomberg.Com, August 6, 2020. https://www.bloomberg.com/opinion/articles/2020-08-06/revisiting-how-covid-19-ranks-with-seasonal-flu-and-1918-pandemic.
  59. Treisman, Michel. “Is Signal Detection Theory Fundamentally Flawed? A Response to Balakrishnan (1998a, 1998b, 1999).” Psychonomic Bulletin & Review 9, no. 4 (December 1, 2002): 845–57. https://doi.org/10.3758/BF03196344.
  60. Walsh, James D. “How Reliable Are COVID-19 Tests?” Intelligencer, June 10, 2020. https://nymag.com/intelligencer/2020/06/how-accurate-are-covid-19-tests.html.
  61. Watson, Jessica, Penny F Whiting, and John E Brush. “Interpreting a Covid-19 Test Result,” 2020, 7.
  62. ———. “Interpreting a Covid-19 Test Result,” 2020, 7.
  63. We Are Being Lied To! Here Is How…, 2020. https://www.youtube.com/watch?v=Ljxah4NrYKU.
  64. “What Is the Diagnostic Accuracy of Antibody Tests for the Detection of Infection with the COVID-19 Virus?” Accessed September 29, 2020. https://doi.org/10.1002/14651858.CD013652.
  65. Woloshin, Steven, Neeraj Patel, and Aaron S. Kesselheim. “False Negative Tests for SARS-CoV-2 Infection — Challenges and Implications.” New England Journal of Medicine 383, no. 6 (August 6, 2020): e38. https://doi.org/10.1056/NEJMp2015897.

True and False – Key Aspects of Medical Tests

“If 1 in 100 disease-free samples are wrongly coming up positive, the disease is not present, we call that a 1% false positive rate.

Because of the high false positive rate and the low prevalence, almost every positive test, a so-called case, identified by Pillar 2 since May of this year has been a FALSE POSITIVE. Not just a few percent. Not a quarter or even a half of the positives are FALSE, but around 90% of them. Put simply, the number of people Mr Hancock sombrely tells us about is an overestimate by a factor of about ten-fold. Earlier in the summer, it was an overestimate by about 20-fold.” — Dr Michael Yeadon, https://lockdownsceptics.org/lies-damned-lies-and-health-statistics-the-deadly-danger-of-false-positives/

Foreword

This is a write up on false positive, false negative, true positive, true negative thinking. It is nice to now have those words, “specificity” (true negative) and “sensitivity” (true positive) which I lacked before.

I realize this fits within the framework of signal detection theory. When I studied signal detection, I had no idea why it was considered important, particularly in Psychology, but over the years, I have often gone back to it, since it comes up routinely in a number of guises. It has to be statistical signal detection, which would be an obvious extension to binary, deterministic signal detection. This also fits within the framework of fuzzy logic, which I have looked at in years past.

Medical testing makes use of the ideas on signal detection and upon reflection, is no different than the consideration of any other sort of evidence – it has to be obtained, assessed for quality, and interpreted to understand the implications.

Anybody who tells you that such and such a test is any given percentage effective is misleading you, unintentionally or perhaps even deliberately.

There are four quadrants in the “gold standard” detection matrix: true positive, false positive, true negative, and false negative. There are at least two percentages to be considered, and they vary independently according to the bias in your test, the detection threshold decided upon. You can bias so that you find all the cases, with the consequence that you get a lot of false positives (false alarms). You can bias so that you get few false positives, and as a result, get a lot of false negatives (miss the fire). In addition, the baseline must be considered, with respect to a Bayesian statistical analysis. So, the prior information on general infection rates will make the percentages change. Low base rate of infection gives a high number of false positives. High base rate of infection gives a low number of false positives.

Bayesian reasoning: start with prior probabilities (assessed somehow) and see how probabilities change with new evidence.

If you do not have much noise masking the signal, results are easier to interpret. If you have large numbers and effects that are strong with respect to variability, the statistics should bear you out. Potentially confounding factors can be accounted for.

Introduction

It is routine for medical tests to be used to determine the health status of people. The simple view of a test is that it returns a true or false result, using some measure, some test instrument, and some testing protocol. Of course anyone reflecting on the issue even a little bit will realize that this is a much over-simplified view of things. For one thing, what threshold, what cut-off point, is being used to make the decision of true or false? What is the measure being used, and what is the measuring instrument? Most things we measure involve some sort of continuous scale. Are the measurements continuous or is it yes and no? What are the typical values that are used to make the judgement? What are the health implications? How are the numbers to be interpreted as a screening device or diagnostic tool? All of these considerations are important for understanding the test.

In this discussion I draw on ratios and proportions, odds, signal detection theory, statistics including Bayesian statistics, and simple arithmetic. I use these tools to examine the accuracy of testing.

Key Points to be Explained

Medical tests are not perfect; they give erroneous results along with correct results. We can estimate the accuracy of a test using scientific investigation. We can estimate how likely the test is to find that there is some condition (a hit). We can estimate how likely the test is to find there is not some condition (correct rejection). We can bias the test by changing the threshold, the cut-off value. We can increase hits and false alarms together, or reduce both together. In addition a low prevalence of the condition will give a lot of false alarms. A high prevalence of the condition will give a lot of misses. Also, a highly selective test will reduce the number of false alarms whereas a highly sensitive test will reduce the number of misses.

The Perfect Test

Here is a diagram which shows testing with no allowance for error.

Does the Condition Exist?
Is the Effect Observed? Effect Observed Condition exists
Effect Not Observed Condition is absent
True or false – assuming no errors

The above chart shows:

  1. Is the effect observed using the test?
  2. Does the condition exist in the person tested?

As a result, we have two cases for the test result:

  1. The condition is observed so is deemed to exist
  2. The condition is not observed so is deemed to not exist

The Imperfect, Real-world Test

With a bit of thought, the question of errors for the test will come up. Is the test perfect? This would seem highly unlikely.

In testing, there are two ways for the results to be true, and two ways for the results to be false.

Does The Condition Exist?
Condition Exists Condition Is Absent
Was the Effect Observed? Effect Observed Condition Is Correctly Considered To Exist

 

 

(HIT)

(TRUE POSITIVE)

Condition Is Falsely Considered To Exist

 

 

(FALSE ALARM)

(FALSE POSITIVE)

Effect Not Observed Condition Is Falsely Considered To Be Absent

 

 

(MISS)

(FALSE NEGATIVE)

Condition Is Correctly Consider To Be Absent

 

 

 

(TRUE REJECTION)

(TRUE NEGATIVE)

True or false – Assuming Errors
  1. Observing an effect when the effect exists is a Hit
  2. Not observing an effect when the effect exists is a Miss
  3. Observing an effect when no effect exists is a False Alarm
  4. Observing an effect when no effect exist is a Correct Rejection

Synonyms for these terms are:

  1. Hit – True Positive (TP), Sensitivity
  2. False Alarm – False Positive (FP), Type I Error
  3. Miss – False Negative (FN), Type II Error
  4. Correct Rejection – True Negative (TN), Specificity

I will use the abbreviations TP, FP, FN, TN in most of the discussion, although the meanings are probably not as easily grasped.

Proportions

The above matrix may be used to show more than one thing. It can be used to show proportions, expected percentages, the odds, for each cell of the matrix for some testing scenario. It may also be used to show the expected counts for each cell, given that we have an overall count for the number of tests.

In assigning proportions to these categories, these ratios can be expressed as fractions, decimal fractions or percentages. We have the following proportions of interest:

Overall Estimates of True Percentages:

  1. percent of people who are actually infected
  2. percent of people who are truly not infected

We will make this a binary split, not allowing for degrees of infection. That latter is important, but it is not important for this discussion.

Since you don’t know the percentage of infections, you must make an estimate. How this should be done is problematic in many cases. There may be little data, and the data may be suspect.

How we arrive at these estimated percentages is complex: scientific, statistical, and not without error. It should be done independently of the test being evaluated. We call these percentages prior odds, priors, or baseline values.

Test Performance

We also need to look at the performance of a given test for classifying the results.

  1. Of those people who are actually infected, what percentages test as true positive (TP)?
  2. We can get the false negative percentage (FN), the misses, by subtracting the true positive percentage from 100 percent. This is the arithmetic complement. Conversely, if you know the percentage of false negatives, you can take the complement of the number of cases to get the percentage of true positives.
  3. Of those people who are truly not infected, what percentage test as true negative (TN)?
  4. We can get the false positive percentage (FP), the false alarms, by subtracting the true negative percentage from 100 percent. This is the arithmetic complement. Conversely, if you know the percentage of false positives, you can take the complement of the number of cases to get the percentage of true negatives

At first glance, you might think that you can apply these percentages against the whole matrix, assuming that the matrix represents 100%, and each of the four cells has some fraction, all adding up to 100%. Things don’t work that way.

The test performance, the percentages, for separating false positives from true negatives only applies to those who are uninfected. Remember, this information on overall infection rates is obtained in some other manner, including other studies, some wild-assed guess, or a deity told you.

On the other side of the matrix, the test performance, the percentages, for separating false negatives from true positives only applies to those who are infected. Remember the sources of this information laid out above.

Here is a diagram adapted from a very good tutorial on this topic.  See “Confused by The Confusion Matrix: What’s the difference between Hit Rate, True Positive Rate, Sensitivity, Recall and Statistical Power?” by  The Curious Learner, https://learncuriously.wordpress.com/2018/10/21/confused-by-the-confusion-matrix/

Probabilities Based on Whether or Not the Effect Exists

Does the Effect Exist?
Effect Exists Effect Doesn’t Exist
Was the Effect Observed? Effect Observed ·         Hit Rate

·         True Positive Rate

·         Sensitivity

·         Statistical Power

·         (1 – Beta)

·         False Alarm Rate

·         False Positive Rate

·         Statistical Significance

·         Type I Error Rate (Alpha)

Effect Not Observed ·         Miss Rate

·         False Negative Rate

·         Type II Error Rate (Beta)

·         Correct Rejection Rate

·         True Negative Rate

Testing as Evidence

Tests provide evidence. Evidence must be:

  1. found or produced somehow.
  2. assessed for reliability, quality, internal validity.
  3. interpreted, examined for external validity, the implications made clear.

Tests can be given scores based on sensitivity (true positives) and selectivity (true negatives). As shown above, true positives and false negatives are complements of one and other, and also, true negatives and false positives are also complements of one and other.

Incorporating the Priors

Testing must take the priors into account when the calculations are done. It makes no sense to apply percentages for false positives and true negatives against the category of estimated infected. Likewise, it makes no sense to apply percentage for true positives and false negatives against the category of estimated uninfected. The false positives and true negative test percentages are based on the uninfected. The true positives and false negative test percentages are based on the infected.

Incorporating the Case Counts

We can work with percentages, but for analysis, we really want to see actual counts. We make use of the overall number of independent tests, the priors, and the test performance to make a two by two matrix of estimated test performance.

Threshold

You can set a threshold for a test score, setting the bias point. If you set the threshold, the sensitivity, to give more hits, you will get more false alarms and miss less often. If you set the threshold, the sensitivity, to give fewer hits, you will get fewer false alarms and miss more often.

False Versus True and Priors

With a low prior rate of infection, the number of false positives can be much greater than the number of true positives, even with an accurate test.

Test Performance and Receiver Operating Characteristic (ROC)

Estimated or actual values for a given test can be plotted, putting False Positives (X) against True Positives (Y) to give a curve. This plot is called the receiver operating characteristic (ROC) curve. Any point along the curve can be selected to give a cut-off point, a threshold. If this threshold is set to detect more cases, you also get more false positives. If this threshold is set to exclude more cases, you also get fewer false positives.

The area under the ROC curve also gives a measure of accuracy. The greater the area is, the more accurate the test. Since the axis both go from 0 to 1, the maximum area is 1 squared. A diagonal line for the ROC curve gives performance at chance levels.

Not all tests are equal. Some have much better accuracy overall. The more bowed the ROC curve is above the diagonal, the better the test.

Example Data for ROC Curve

See http://www.rad.jhmi.edu/jeng/javarad/roc/JROCFITi.html

False Positive Fraction (FPF)  True Positive Fraction (TPF) Lower Upper
0.0000 0.0000 0.0000 0.0000
0.0050 0.2301 0.0169 0.7407
0.0100 0.3135 0.0430 0.7718
0.0200 0.4168 0.0996 0.8061
0.0300 0.4860 0.1545 0.8282
0.0400 0.5384 0.2056 0.8449
0.0500 0.5807 0.2523 0.8587
0.0600 0.6159 0.2949 0.8705
0.0700 0.6461 0.3337 0.8808
0.0800 0.6723 0.3690 0.8901
0.0900 0.6955 0.4012 0.8985
0.1000 0.7161 0.4306 0.9062
0.1100 0.7347 0.4575 0.9132
0.1200 0.7515 0.4821 0.9198
0.1300 0.7668 0.5047 0.9258
0.1400 0.7809 0.5255 0.9314
0.1500 0.7938 0.5447 0.9366
0.2000 0.8454 0.6214 0.9577
0.2500 0.8822 0.6757 0.9723
0.3000 0.9096 0.7160 0.9824
0.4000 0.9466 0.7727 0.9934
0.5000 0.9691 0.8119 0.9978
0.6000 0.9832 0.8424 0.9994
0.7000 0.9918 0.8684 0.9999
0.8000 0.9967 0.8927 1.0000
0.9000 0.9992 0.9189 1.0000
0.9500 0.9998 0.9357 1.0000
1.0000 1.0000 1.0000 1.0000

Summary ROC Statistics

Number of Cases:   50

Number Correct:    42

Accuracy:          84%

Sensitivity:       88%

Specificity:       80%

Positive Cases Missed:  3

Negative Cases Missed:  5

(A rating of 3 or greater is considered positive.)

Fitted ROC Area:   0.905

Empiric ROC Area:  0.892

Plotting the ROC Curve

ROC Curve Type:   Fitted

Key for the ROC Plot

RED symbols and BLUE line:  Fitted ROC curve.
GRAY lines:  95% confidence interval of the fitted ROC curve.
BLACK symbols ± GREEN line:  Points making up the empirical ROC curve (does not apply to Format 5).

Testing Overall

The test can be viewed as the measure plus the measuring method. It can also include the procedures, the protocol for conducting the test. Differing protocols can change and confound the test results. Tests can be very accurate and still give a large number of false positives when the estimate of infection rates is low.

Testing Interpretation

The test results require interpretation by a skilled clinician. Sometimes, tests are used for screening, and sometimes for actual diagnosis. One test alone should not be relied upon. Tests should be repeated.

Testing Signal versus Noise

Test results for the same individual can vary because of “noise” masking the “signal.”  By noise we mean fluctuations in the measurement of interest that are based on other factors than the condition of interest, perhaps random factors.

Testing and Time Variance

Test results can vary for the same individual because the underlying conditions can change from one time to the next. Levels of any condition can fluctuate over time: hourly, daily, weekly, … . With heath conditions: you get infected, you get sick, you get better, you die, … .

The Calculations of the Estimates

False Positive Calculations

The simple calculations of false positive expected rates:

  1. Prior baseline estimates for not infected
  2. Test accuracy estimates for false positive
  3. Number of tests

Multiply them together to get the expected count of false positives. False positives are only evaluated against the uninfected cases, not all test cases.

False Negative Calculations

Calculate the false negative expected rates

  1. Prior baseline estimates for not infected
  2. Test accuracy estimates for false negatives
  3. Number of tests

Multiply them together to get the expected count of false negatives. False negatives are only evaluated against the infected cases, not all test cases.

True Positive Calculations

Calculate the true positive expected rates

  1. Prior baseline estimates for infected
  2. Test accuracy estimates for true positives
  3. Number of tests

Multiply them together to get the expected count of true positives. True positives are only evaluated against the infected cases, not all test cases.

True Negative Calculations

Calculate the true negatives expected rates

  1. Prior baseline estimates for not infected
  2. Test accuracy estimates for true negatives
  3. Number of tests

Multiply them together to get the expected count of true negatives. True negatives are only evaluated against the uninfected cases, not all test cases.

An Example

In the example below, I set the following parameters:

Number of Tests 1,000.00

 

Population Baseline Estimates
Prior Baseline Infection Rate Estimate 2 %
Baseline True Positives =
Prior Baseline X Number of Tests
20
Baseline True Negatives =
(1 – Prior Baseline) X Number of Tests
980

 

Testing Method Performance
Hit Rate (Sensitivity) 95%
Miss Rate =
One’s Complement of Hit Rate
5%
False Alarm Rate 10%
Correct Rejection Rate (Specificity) =
One’s Complement of False Alarm Rate
90%

 

Using these parameters, I calculate expected counts:

Expected Counts
True Positives (TP) =
Baseline True Positives X
Hit Rate
19
False Positives (FP) =
Baseline True Negatives X
False Alarm Rate
98
False Negatives (FN) =
Baseline True Positives X
Miss Rate
1
True Negatives (TN) =
Baseline True Negatives X
Correct Rejection Rate
882

Summaries

I summarize the calculated values in the matrix below. You can see that the number of false positives, under these assumptions, is 5 times the amount of true positives, i.e., very high. Also, the false negative rate is very low for this test and the prior infection rates. This is with a test selectivity of 90%, a test sensitivity of 95%, and an estimated infection rate of 2%.

Does The Condition Exist?
Condition Exists Condition Is Absent
Was the Effect Observed? Effect Observed TP = 19 FP = 98
Effect Not Observed FN = 1 TN = 882
Estimated Counts

Based on Test Performance, Priors, and Number of Tests

Measures of Test Performance

Below are various measures of test performance, test quality. They use the previous data from the previous example. The calculations presented here are simple. The interpretation takes more skill.

Core Set of Measures

Measures of Test Performance
Diagnostic Accuracy =
(TP + TN) / TP + TN + FP + FN
0.90
Sensitivity =
(TP) / (TP + FN)
0.95
Specificity =
(TN) / (TN +FP)
0.90

Predictive Values

Positive Predictive Value (PPV) =
(TP) / (TP + FP)
0.16
Negative Predictive Value (NPV) =
(TN) / (TN + FN)
1.00

The Positive Predictive Value (PPV) and the Negative Predictive Value (NPV) give the probabilities based on whether of not the effect was observed. This contrasts with the sensitivity and selectivity that give probabilities based on the estimated existence of the effect.

Probabilities Based on Whether or Not the Effect was Observed

Does the Effect Exist?
Effect Exists Effect Doesn’t Exist
Was the Effect Observed? Effect Observed True Discovery Rate

Positive Predictive Value

Precision

False Discovery Rate
Effect Not Observed False Omission Rate True Omission Rate

Negative Predictive Value

Adapted from “Confused by The Confusion Matrix: What’s the difference between Hit Rate, True Positive Rate, Sensitivity, Recall and Statistical Power?” by  The Curious Learner, https://learncuriously.wordpress.com/2018/10/21/confused-by-the-confusion-matrix/

Test Estimates with Differing Priors

Here are test estimates based upon five differing population baseline estimates, that is, differing estimates of priors. I vary the priors from 0.1 percent to 99.9 percent.

N.B. In order to avoid division by 0, I did not use 0.0 percent and 100 percent.

Number of Tests 1,000 1,000 1,000 1,000 1,000
Testing Method
Hit Rate (Sensitivity) 95% 95% 95% 95% 95%
Miss Rate =
Ones Complement of Hit Rate
5% 5% 5% 5% 5%
False Alarm Rate 10% 10% 10% 10% 10%
Correct Rejection Rate (Specificity) =
Ones Complement of False Alarm Rate
90% 90% 90% 90% 90%
Population Baseline Estimates
Prior Baseline 0.10% 2% 50% 98% 99.90%
Baseline True Positives =
Prior Baseline X Number of Tests
1 20 500 980 999
Baseline True Negatives =
(1 – Prior Baseline) X Number of Tests
999 980 500 20 1
Expected Counts
True Positives (TP) =
Baseline True Positives X
Hit Rate
0.95 19 475 931 949.05
False Positives (FP) =
Baseline True Negatives X
False Alarm Rate
99.9 98 50 2 0.1
False Negatives (FN) =
Baseline True Positives X
Miss Rate
0.05 1 25 49 49.95
True Negatives (TN) =
Baseline True Negatives X
Correct Rejection Rate
899.1 882 450 18 0.90
Quality Tests
Diagnostic Accuracy =
(TP + TN) / TP + TN + FP + FN
0.90 0.90 0.93 0.95 0.95
Sensitivity =
(TP) / (TP + FN)
0.95 0.95 0.95 0.95 0.95
Specificity =
(TN) / (TN +FP)
0.90 0.90 0.90 0.90 0.90
Positive Predictive Value (PPV) =
(TP) / (TP + FP)
0.01 0.16 0.90 1.00 1.00
Negative Predictive Value (NPV) =
(TN) / (TN + FN)
1.00 1.00 0.95 0.27 0.02
Positive Predictive Likelihood Ratios =
Sensitivity / (1 – Specificity)
9.50 9.50 9.50 9.50 9.50
Negative Predictive Likelihood Ratios =
(1 – Sensitivity) / Specificity
0.06 0.06 0.06 0.06 0.06
Youden’s Index =
(Sensitivity + Specificity) – 1
0.85 0.85 0.85 0.85 0.85
Diagnostic Odds Ratio (DOR) =
(TP / FN) / (FP / TN)
171.00 171.00 171.00 171.00 171.00

 

 

 

Bibliography

  1. Baratloo, Alireza, Mostafa Hosseini, Ahmed Negida, and Gehad El Ashal. “Part 1: Simple Definition and Calculation of Accuracy, Sensitivity and Specificity.” Emergency 3, no. 2 (2015): 48–49.
  2. Harvey, Lew. “Detection Theory.” Psychology of Perception, 2014, 17.
  3. learncuriously. “Confused by The Confusion Matrix Part 2: ‘Accuracy’ Is But One of Many Measures of Accuracy….” The Curious Learner (blog), October 27, 2018. https://learncuriously.wordpress.com/2018/10/28/confused-by-the-confusion-matrix-part-2/.
  4. ———. “Confused by The Confusion Matrix: What’s the Difference between Hit Rate, True Positive Rate, Sensitivity, Recall and Statistical Power?” The Curious Learner (blog), October 20, 2018. https://learncuriously.wordpress.com/2018/10/21/confused-by-the-confusion-matrix/.
  5. Lockdown Sceptics. “Lies, Damned Lies and Health Statistics – the Deadly Danger of False Positives.” Accessed October 6, 2020. https://lockdownsceptics.org/lies-damned-lies-and-health-statistics-the-deadly-danger-of-false-positives/.
  6. Read “Intelligence Analysis: Behavioral and Social Scientific Foundations” at NAP.Edu. Accessed September 28, 2020. https://doi.org/10.17226/13062.
  7.  “ROC Analysis: Web-Based Calculator for ROC Curves.” Accessed October 11, 2020. http://www.rad.jhmi.edu/jeng/javarad/roc/JROCFITi.html.
  8. “ROC Curves – What Are They and How Are They Used?” Accessed October 4, 2020. https://acutecaretesting.org/en/articles/roc-curves-what-are-they-and-how-are-they-used.
  9. Šimundić, Ana-Maria. “Measures of Diagnostic Accuracy: Basic Definitions.” EJIFCC 19, no. 4 (January 20, 2009): 203–11.

 

 

 

Commentary on “Be Smart Wear A Mask”

From a YouTube video, “Be Smart Wear a Mask”,  by Chris Martenson at https://www.youtube.com/watch?v=KANNNty9V3o

Most people who follow Chris are aware that he has consistently promoted the wearing of masks to reduce your viral load. This is not new. Here he references some studies which he feels support his point, that is, he provides evidence, and discusses the implications. All of a sudden, we have the invasion of the anti-maskers. It is pretty obvious that at the very least many of these ignored the central point of viral load reduction and disease severity. Chris is not making any of this up. Science is never settled – studies provide evidence. Numbers are nice to have to give more information. Studies may or may not be well done. There are always, always, confounding factors. Studies must be interpreted; the implications made clear. On the issue of masks, there are many studies. Some supporting masks, some not providing as much support. Some provide a meta-analysis. Given that, Chris makes his interpretations of the evidence, and gives his analysis. He may be correct, or not. However, all of a sudden we have this influx of people commenting who act as if they are unaware that Chris has been pretty consistent on this for many month. Where did they come from? These people threatening to unsubscribe – were they ever following him, or is that an invention? It almost seems like a coordinated attack. What was the origin of this? I have examined more studies than most. The evidence can be cherry picked to support various positions, and often is dishonestly or incompetently interpreted. See here: https://www.opednews.com/articles/Masks-and-Motivated-Reason-by-Mike-Zimmer-Belief_Bias_Covid-19_Empathy-200805-253.html and here http://ephektikoi.ca/blog/2020/10/07/836/

Error in Testing: Selectivity, Sensitivity and Prevalence

Preface

This is a write up on false positive, false negative, true positive, true negative thinking in testing for some condition, trying to detect if it is there or not. It is useful for discussing medical tests, but it applies to any sort of testing where decisions must be made.

I said I would write up testing, so I did. It always comes out seeming more complex than it seems at first glance, even though the calculations are simple arithmetic. I have another one in draft, hitting the material from a different angle, but unfortunately not making it simpler.

This fits within the framework of signal detection theory. When I studied signal detection, I had no idea why it was considered important, particularly in Psychology, but over the years, I have often gone back to it, since it comes up routinely in a number of guises. It has to be statistical signal detection, which would be an obvious extension to binary, deterministic signal detection. This also fits within the framework of fuzzy logic, which I have looked at in years past.

Medical testing makes use of the ideas on signal detection and upon reflection, is no different than the consideration of any other sort of evidence – it has to be obtained, assessed for quality, and interpreted to understand the implications.

The take away is that for low prevalence of a condition (e.g. infection), you can get a lot of false positives (false alarms), even with a fairly specific test.  For a high prevalence of a condition, you can get a lot of false negatives (misses). In addition, for a highly selective test, you get fewer false positives (false alarms), and for a highly sensitive test, you get fewer false negatives (misses).

In the example below, I show how false positives can be far greater than true positives, even for a fairly well performing test, when the prevalence is low:

Parameters
Complement of Parameters
Selectivity = .95 1 – selectivity = .05
Sensitivity = .99 1 – sensitivity = .01
Prevalence = .01 1 – prevalence = .99

We have this formula: Ratio of false positives (FP) to true positives (TP) = ( (1-selectivity) x (1-prevalence) ) / ((sensitivity) x (prevalence) ) .

False positives to true positives = ((.05) x (.99))/(.99) x (.01)) = 5

Using the formula, we get a 5 fold increase of false positives over true positives, for a test that has quite good performance. Below I will show how for any test with error rates, both low prevalence and high prevalence give more false results.

It is nice to now have those words, “specificity” (true negative) and “sensitivity” (true positive) for test performance which I lacked before. I give more explanation below.

Anybody who tells you that such and such a test is any given percentage effective is misleading you somewhat, unintentionally or perhaps even deliberately.

There are four quadrants in the “gold standard” detection matrix: true positive, false positive, true negative, and false negative. There are at least two percentages to be considered, and they vary independently according to the bias in your test, the detection threshold decided upon. You can bias so that you find all the cases, with the consequence that you get a lot of false positives (false alarms). You can bias so that you get few false positives, and as a result, get a lot of false negatives (miss the fire). In addition, the baseline must be considered, with respect to a Bayesian statistical analysis. So, the prior information on general infection rates will make the percentages change. Low base rate of infection gives a high number of false positives. High base rate of infection gives a low number of false positives.

Bayesian reasoning: start with prior probabilities (assessed somehow) and see how probabilities change with new evidence.

If you do not have much noise masking the signal, results are easier to interpret. If you have large numbers and effects that are strong with respect to variability, the statistics should bear you out. Potentially confounding factors can be accounted for.

See http://ephektikoi.ca/blog/2020/10/11/bibliography-on-testing-and-uncertainty/ for more information.

Introduction

It is routine for medical tests to be used to determine the health status of people. The simple view of a test is that it returns a true or false result, using some measure, some test instrument, and some testing protocol. Of course anyone reflecting on the issue even a little bit will realize that this is a much over-simplified view of things. For one thing, what threshold, what cut-off point, is being used to make the decision of true or false? What is the measure being used, and what is the measuring instrument? Most things we measure involve some sort of continuous scale. Are the measurements continuous or is it yes and no? What are the typical values that are used to make the judgement? What are the health implications? How are the numbers to be interpreted as a screening device or diagnostic tool? All of these considerations are important for understanding the test.

The Perfect Test

Here is a diagram of a test which does not consider that there might be some error.

Does the Condition Exist?
Is the Effect Observed? Observed Condition exists
Not observed Condition is absent
True or false – assuming no errors

The Imperfect, Real-world Test

With a bit of thought, the question of errors for the test will come up. Is the test perfect? This would seem highly unlikely.

In testing, there are two ways for the results to be true, and two ways for the results to be false.

Does The Condition Exist?
    Condition Exists Condition Is Absent
Is The Effect Observed? Observed Condition Is Correctly Considered To Exist

 

 

(HIT)

(TRUE POSITIVE)

Condition Is Falsely Considered To Exist

 

 

(FALSE ALARM)

(FALSE POSITIVE)

Not Observed Condition Is Falsely Considered To Be Absent

 

 

(MISS)

(FALSE NEGATIVE)

Condition Is Correctly Consider To Be Absent

 

 

 

(TRUE REJECTION)

(TRUE NEGATIVE)

True Or False – Assuming Errors

 

 

  1. Observing an effect when the effect exists is a Hit
  2. Not observing an effect when the effect exists is a Miss
  3. Observing an effect when no effect exists is a False Alarm
  4. Observing an effect when no effect exist is a Correct Rejection

Synonyms for these terms are:

 

  1. Hit – True Positive (TP), Sensitivity
  2. False Alarm – False Positive (FP), Type I Error
  3. Miss – False Negative (FN), Type II Error
  4. Correct Rejection – True Negative (TN), Specificity

I will use the abbreviations TP, FP, FN, TN in most of the discussion, although the meanings are probably not as easily grasped.

Three factors for calculations:

  1. Performance or discrimination ability of the test
  2. Prevalence of the condition, present or absent to some percentage
  3. Number of tests done independently

Performance of a test: selectivity and sensitivity

Sensitivity applies to a population that all truly have the condition. It is the ratio of hits to misses, usually as a percentage of hits to overall tests. Percentage is calculated (hits / (hits + misses) x 100). Sensitivity gives the percent hits. The one’s complement of sensitivity gives the percent misses. With poorer sensitivity tests, the number of misses gets larger.

Selectivity applies to a population were none have the condition. It is the ratio of correct rejections to false alarms, usually as a percentage of rejections to overall tests. Percentage is calculated as (rejections / (rejections + false alarms) x 100). Selectivity gives the percent correct rejections. The one’s complement of selectivity gives the false alarm percent. With poorer selectivity tests, the number of false alarms gets larger.

Prevalence gives the estimated number where the condition is present. The one’s complement of prevalence gives the estimated number where the condition is absent.

Lower prevalence gives more false positives or false alarms and also lower prevalence gives fewer false negatives or misses.

Higher prevalence gives more false negatives or misses and also higher prevalence gives fewer false positives or false alarms.

False negatives are a function of sensitivity and prevalence whereas false positives are a function of selectivity and prevalence.

The number of tests done independently gives the counts.

In calculating expected values for a condition and errors, use performance, prevalence and the number of tests

Hits False alarms
Misses Correct rejections
  1. Hits = sensitivity x prevalence x number of tests
  2. False alarms = (1- selectivity) x (1 – prevalence) x number of tests
  3. Misses = (1 – sensitivity) x prevalence x number of tests
  4. Correct rejections = selectivity x (1 – prevalence) x number of tests

Hits versus false alarms are a function of test performance and prevalence.

Considerations: It is a mistake to only use just performance and test count without taking prevalence into account – a big mistake

Given that you know the performance of a test, and the prevalence of the condition, you can calculate the ratios of:

  1. False positives to true positives
  2. False negatives to true negatives

Ratio of FP to TP ( (1-selectivity) x (1-prevalence) ) / ((sensitivity) x (prevalence) )

Ratio of FN to TN ( (1-sensitivity) x (prevalence) ) / ((selectivity) x (1 – prevalence) )

Example 1

Parameters Complement
Selectivity = .95 1 – selectivity = .05
Sensitivity = .99 1 – sensitivity = .01
Prevalence = .01 1 – prevalence = .99

False positives to true positives = ((.05) x (.99))/(.99) x (.01)) = 5

Using the formula, we get a 5 fold increase of false positives over true positives

Example 2

Parameters Complement
Selectivity = 1.00 1 – selectivity = 0.0
Sensitivity = 1.00 1 – sensitivity = 0.0
Prevalence = .50 1 – prevalence = .50

False positives to true positives = ((0) x (.5))/(1) x (.5)) = 0

Using the formula, we get a no increase in false positives over true positives; in fact, there are no false positives or false negatives.

Formulae Summarized

 Test Count = number of tests

 selectivity (fraction of correct rejections for those without condition)

complement of selectivity (fraction of false alarms for those without condition) = (1 – selectivity)

sensitivity (fraction of hits for those with condition)

complement of sensitivity (fraction of misses for those with condition) = (1 – sensitvity)

prevalence is actual incidence as percent of population with condition

complement of prevalance is percent of population without condition = (1 – prevalence)

Estimated True Positive (Hits, TP) = sensitivity x prevalence x case count

Estimated False Positive (False Alarms, FP) = (1 – selectivity) x (1 – prevalence) x (case count)

Estimated False Negatives (Misses, FN) = (1 – sensitivity) * (prevalence) x case count

Estimated True Negatives (Correct Rejections, TN) = (selectivity) x (1 – prevalence) x (case count)

Ratio of FP to TP ( (1-selectivity) x (1-prevalence) ) / ((sensitivity) x (prevalence) )

Ratio of FN to TN ( (1-sensitivity) x (prevalence) ) / ((selectivity) x (1 – prevalence) )

Worked Example For Different Test Parameters and Prevalence

The pattern for all is that the end regions of the curve get high inversely in proportion to the performance of the test. That is for low prevalence and high prevalence both, there are more errors. For low prevalence, the errors are false positives. For high prevalence, the errors are false negatives. The error rates for both types of errors are lower in the middle. High selectivity reduces false positive errors. High sensitivity reduces false negative errors. The ultimate high is the non-existent perfectly performancing test with selectivity of 100% and sensitivity of 100%.

Selectivity 100%, Sensitivity 100%
A Balanced Perfect Test Counts
Count Selectivity Sensitivity Prevalence Ratio FP to TP Ratio FN to TN TP FP FN TN
1000 1.00 1.00 0.00 0.00 0.00 0.00 0.00 0.00 1000.00
1000 1.00 1.00 0.02 0.00 0.00 20.00 0.00 0.00 980.00
1000 1.00 1.00 0.03 0.00 0.00 30.00 0.00 0.00 970.00
1000 1.00 1.00 0.04 0.00 0.00 40.00 0.00 0.00 960.00
1000 1.00 1.00 0.05 0.00 0.00 50.00 0.00 0.00 950.00
1000 1.00 1.00 0.10 0.00 0.00 100.00 0.00 0.00 900.00
1000 1.00 1.00 0.20 0.00 0.00 200.00 0.00 0.00 800.00
1000 1.00 1.00 0.30 0.00 0.00 300.00 0.00 0.00 700.00
1000 1.00 1.00 0.40 0.00 0.00 400.00 0.00 0.00 600.00
1000 1.00 1.00 0.50 0.00 0.00 500.00 0.00 0.00 500.00
1000 1.00 1.00 0.60 0.00 0.00 600.00 0.00 0.00 400.00
1000 1.00 1.00 0.70 0.00 0.00 700.00 0.00 0.00 300.00
1000 1.00 1.00 0.80 0.00 0.00 800.00 0.00 0.00 200.00
1000 1.00 1.00 0.90 0.00 0.00 900.00 0.00 0.00 100.00
1000 1.00 1.00 0.98 0.00 0.00 980.00 0.00 0.00 20.00
1000 1.00 1.00 1.00 0.00 0.00 1000.00 0.00 0.00 0.00

Selectivity 90%, Sensitivity 99%
An Unbalanced Test with Reasonable Performance Counts
Count Selectivity Sensitivity Prevalence Ratio FP to TP Ratio FN to TN TP FP FN TN
1000 0.90 0.99 0.01 10.00 0.00 9.90 99.00 0.10 891.00
1000 0.90 0.99 0.02 4.95 0.00 19.80 98.00 0.20 882.00
1000 0.90 0.99 0.03 3.27 0.00 29.70 97.00 0.30 873.00
1000 0.90 0.99 0.04 2.42 0.00 39.60 96.00 0.40 864.00
1000 0.90 0.99 0.05 1.92 0.00 49.50 95.00 0.50 855.00
1000 0.90 0.99 0.10 0.91 0.00 99.00 90.00 1.00 810.00
1000 0.90 0.99 0.20 0.40 0.00 198.00 80.00 2.00 720.00
1000 0.90 0.99 0.30 0.24 0.00 297.00 70.00 3.00 630.00
1000 0.90 0.99 0.40 0.15 0.01 396.00 60.00 4.00 540.00
1000 0.90 0.99 0.50 0.10 0.01 495.00 50.00 5.00 450.00
1000 0.90 0.99 0.60 0.07 0.02 594.00 40.00 6.00 360.00
1000 0.90 0.99 0.70 0.04 0.03 693.00 30.00 7.00 270.00
1000 0.90 0.99 0.80 0.03 0.04 792.00 20.00 8.00 180.00
1000 0.90 0.99 0.90 0.01 0.10 891.00 10.00 9.00 90.00
1000 0.90 0.99 0.98 0.00 0.54 970.20 2.00 9.80 18.00
1000 0.90 0.99 0.99 0.00 1.10 980.10 1.00 9.90 9.00

Selectivity 99%, Sensitivity 90%
An Unbalanced Test with Reasonable Performance
Counts
Count Selectivity Sensitivity Prevalence Ratio FP to TP Ratio FN to TN TP FP FN TN
1000 0.99 0.90 0.01 1.10 0.00 9.00 9.90 1.00 980.10
1000 0.99 0.90 0.02 0.54 0.00 18.00 9.80 2.00 970.20
1000 0.99 0.90 0.03 0.36 0.00 27.00 9.70 3.00 960.30
1000 0.99 0.90 0.04 0.27 0.00 36.00 9.60 4.00 950.40
1000 0.99 0.90 0.05 0.21 0.01 45.00 9.50 5.00 940.50
1000 0.99 0.90 0.10 0.10 0.01 90.00 9.00 10.00 891.00
1000 0.99 0.90 0.20 0.04 0.03 180.00 8.00 20.00 792.00
1000 0.99 0.90 0.30 0.03 0.04 270.00 7.00 30.00 693.00
1000 0.99 0.90 0.40 0.02 0.07 360.00 6.00 40.00 594.00
1000 0.99 0.90 0.50 0.01 0.10 450.00 5.00 50.00 495.00
1000 0.99 0.90 0.60 0.01 0.15 540.00 4.00 60.00 396.00
1000 0.99 0.90 0.70 0.00 0.24 630.00 3.00 70.00 297.00
1000 0.99 0.90 0.80 0.00 0.40 720.00 2.00 80.00 198.00
1000 0.99 0.90 0.90 0.00 0.91 810.00 1.00 90.00 99.00
1000 0.99 0.90 0.98 0.00 4.95 882.00 0.20 98.00 19.80
1000 0.99 0.90 0.99 0.00 10.00 891.00 0.10 99.00 9.90

Selectivity 99%, Sensitivity 99%
A Balanced Test, with Excellent Performance
Counts
Count Selectivity Sensitivity Prevalence Ratio FP to TP Ratio FN to TN TP FP FN TN
1000 0.99 0.99 0.01 1.00 0.00 9.90 9.90 0.10 980.10
1000 0.99 0.99 0.02 0.49 0.00 19.80 9.80 0.20 970.20
1000 0.99 0.99 0.03 0.33 0.00 29.70 9.70 0.30 960.30
1000 0.99 0.99 0.04 0.24 0.00 39.60 9.60 0.40 950.40
1000 0.99 0.99 0.05 0.19 0.00 49.50 9.50 0.50 940.50
1000 0.99 0.99 0.10 0.09 0.00 99.00 9.00 1.00 891.00
1000 0.99 0.99 0.20 0.04 0.00 198.00 8.00 2.00 792.00
1000 0.99 0.99 0.30 0.02 0.00 297.00 7.00 3.00 693.00
1000 0.99 0.99 0.40 0.02 0.01 396.00 6.00 4.00 594.00
1000 0.99 0.99 0.50 0.01 0.01 495.00 5.00 5.00 495.00
1000 0.99 0.99 0.60 0.01 0.02 594.00 4.00 6.00 396.00
1000 0.99 0.99 0.70 0.00 0.02 693.00 3.00 7.00 297.00
1000 0.99 0.99 0.80 0.00 0.04 792.00 2.00 8.00 198.00
1000 0.99 0.99 0.90 0.00 0.09 891.00 1.00 9.00 99.00
1000 0.99 0.99 0.98 0.00 0.49 970.20 0.20 9.80 19.80
1000 0.99 0.99 0.99 0.00 1.00 980.10 0.10 9.90 9.90

Selectivity 95%, Sensitivity 95%
A Balanced Test, with Decent Performance
Counts
Count Selectivity Sensitivity Prevalence Ratio FP to TP Ratio FN to TN TP FP FN TN
1000 0.95 0.95 0.01 5.21 0.00 9.50 49.50 0.50 940.50
1000 0.95 0.95 0.02 2.58 0.00 19.00 49.00 1.00 931.00
1000 0.95 0.95 0.03 1.70 0.00 28.50 48.50 1.50 921.50
1000 0.95 0.95 0.04 1.26 0.00 38.00 48.00 2.00 912.00
1000 0.95 0.95 0.05 1.00 0.00 47.50 47.50 2.50 902.50
1000 0.95 0.95 0.10 0.47 0.01 95.00 45.00 5.00 855.00
1000 0.95 0.95 0.20 0.21 0.01 190.00 40.00 10.00 760.00
1000 0.95 0.95 0.30 0.12 0.02 285.00 35.00 15.00 665.00
1000 0.95 0.95 0.40 0.08 0.04 380.00 30.00 20.00 570.00
1000 0.95 0.95 0.50 0.05 0.05 475.00 25.00 25.00 475.00
1000 0.95 0.95 0.60 0.04 0.08 570.00 20.00 30.00 380.00
1000 0.95 0.95 0.70 0.02 0.12 665.00 15.00 35.00 285.00
1000 0.95 0.95 0.80 0.01 0.21 760.00 10.00 40.00 190.00
1000 0.95 0.95 0.90 0.01 0.47 855.00 5.00 45.00 95.00
1000 0.95 0.95 0.98 0.00 2.58 931.00 1.00 49.00 19.00
1000 0.95 0.95 0.99 0.00 5.21 940.50 0.50 49.50 9.50

Selectivity 90%, Sensitivity 90%
A Balanced Test, with OK Performance
Counts
Count Selectivity Sensitivity Prevalence Ratio FP to TP Ratio FN to TN TP FP FN TN
1000 0.90 0.90 0.01 11.00 0.00 9.00 99.00 1.00 891.00
1000 0.90 0.90 0.02 5.44 0.00 18.00 98.00 2.00 882.00
1000 0.90 0.90 0.03 3.59 0.00 27.00 97.00 3.00 873.00
1000 0.90 0.90 0.04 2.67 0.00 36.00 96.00 4.00 864.00
1000 0.90 0.90 0.05 2.11 0.01 45.00 95.00 5.00 855.00
1000 0.90 0.90 0.10 1.00 0.01 90.00 90.00 10.00 810.00
1000 0.90 0.90 0.20 0.44 0.03 180.00 80.00 20.00 720.00
1000 0.90 0.90 0.30 0.26 0.05 270.00 70.00 30.00 630.00
1000 0.90 0.90 0.40 0.17 0.07 360.00 60.00 40.00 540.00
1000 0.90 0.90 0.50 0.11 0.11 450.00 50.00 50.00 450.00
1000 0.90 0.90 0.60 0.07 0.17 540.00 40.00 60.00 360.00
1000 0.90 0.90 0.70 0.05 0.26 630.00 30.00 70.00 270.00
1000 0.90 0.90 0.80 0.03 0.44 720.00 20.00 80.00 180.00
1000 0.90 0.90 0.90 0.01 1.00 810.00 10.00 90.00 90.00
1000 0.90 0.90 0.98 0.00 5.44 882.00 2.00 98.00 18.00
1000 0.90 0.90 0.99 0.00 11.00 891.00 1.00 99.00 9.00

Selectivity 50%, Sensitivity 50%
A Balanced Test, with Poor Performance
Counts
Count Selectivity Sensitivity Prevalence Ratio FP to TP Ratio FN to TN TP FP FN TN
1000 0.50 0.50 0.01 99.00 0.01 5.00 495.00 5.00 495.00
1000 0.50 0.50 0.02 49.00 0.02 10.00 490.00 10.00 490.00
1000 0.50 0.50 0.03 32.33 0.03 15.00 485.00 15.00 485.00
1000 0.50 0.50 0.04 24.00 0.04 20.00 480.00 20.00 480.00
1000 0.50 0.50 0.05 19.00 0.05 25.00 475.00 25.00 475.00
1000 0.50 0.50 0.10 9.00 0.11 50.00 450.00 50.00 450.00
1000 0.50 0.50 0.20 4.00 0.25 100.00 400.00 100.00 400.00
1000 0.50 0.50 0.30 2.33 0.43 150.00 350.00 150.00 350.00
1000 0.50 0.50 0.40 1.50 0.67 200.00 300.00 200.00 300.00
1000 0.50 0.50 0.50 1.00 1.00 250.00 250.00 250.00 250.00
1000 0.50 0.50 0.60 0.67 1.50 300.00 200.00 300.00 200.00
1000 0.50 0.50 0.70 0.43 2.33 350.00 150.00 350.00 150.00
1000 0.50 0.50 0.80 0.25 4.00 400.00 100.00 400.00 100.00
1000 0.50 0.50 0.90 0.11 9.00 450.00 50.00 450.00 50.00
1000 0.50 0.50 0.98 0.02 49.00 490.00 10.00 490.00 10.00
1000 0.50 0.50 0.99 0.01 99.00 495.00 5.00 495.00 5.00

 

 

Masks and motivated reasoning

Originally published on August 5, 2020 at https://www.opednews.com/articles/Masks-and-Motivated-Reason-by-Mike-Zimmer-Belief_Bias_Covid-19_Empathy-200805-253.html

 

The effectiveness of mask wearing is called into doubt by some people and by some studies. Although I believe the evidence shows that it is of value in controlling the spread of Covid-19, both infectivity and viral load, all evidence is subject to interpretation. All evidence has a possibility of being incorrect. All findings are underdetermined.

 

Surgical masks and goggles
Surgical masks and goggles
(Image by mich225)
  Details   DMCA

 

There is a great diversity of opinion about the wearing of masks for protection against Covid-19 infection. Sometimes, the opinions are allegedly supported by studies and expertise. In other cases, they seem to be little more than the result of a game of “telephone,” conducted over the Internet and social media. The discussion is much politicized, and brings in issues going far beyond the efficacy of masks in slowing the spread of a virus. People with claimed relevant expertise have weighed in on various sides of the topic, each with their arguments, each with information presented as fact. A lot of the views seem to be springing from bias – confirmation and disconfirmation – and in general motivated reasoning. It is clear that motivated reasoning, beliefs and values, determine how we evaluate evidence.

I also have my biases, and these are towards the wearing of masks as a reasonable and somewhat effective infection control measure and risk management measure. I cite a meta-analysis published in the Lancet which I find reasonably convincing, although the statistical techniques are far more advanced than any that I have ever studied. It can be found here at Physical distancing, face masks, and eye protection to prevent person-to-person transmission of SARS-cov-2 and COVID-19: a systematic review and meta-analysis.

I also refer you to a presentation of the UCSF School of Medicine by three specialists in disease control speaking on infection control. It can be found here at Covid-19: How the Virus Gets in and How to Block It: Aerosols, Droplets, Masks, Face Shields, & More.

You can undoubtedly locate contrary opinions with ease. These may or may not be sound. When opinions differ, and contradict one and other, at best one can be correct.

On the issue of masks for infection control and Covid-19 there are factors ranging far beyond the medical case. Cultural, social and emotional thinking colour the opinions that people hold.

Here are some of the issues that I feel might be important. There are undoubtedly other aspects to this that I have not covered but I think this is a good set for thinking about the issues.

Disagreement about the infectivity and virulence of Covid 19 are rampant in various communities. Your belief that the virus is mild or it is only those with co-morbidities who suffer will influence the way you think about masks. If you believe that it is a very mild infection and that you probably won’t be infected, you will certainly not see the wisdom of wearing masks.

The effectiveness of mask wearing is called into doubt by some people and by some studies. Although I believe the evidence shows that it is of value in controlling the spread of Covid-19, both infectivity and viral load, all evidence is subject to interpretation. All evidence has a possibility of being incorrect. All findings are underdetermined.

A person’s attitudes towards personal freedom and control by the state will affect their view of the wisdom and the appropriateness of wearing masks. This seems to trump any notion that the masks might be useful for controlling infection. I strongly suspect that those who are concerned with personal freedom are also prone to denigrate the infectivity and virulence of the virus and to also downplay the effectiveness of mask wearing. This is confirmation bias at work.

The degree of one’s concern for others as opposed to concern for the self may well play a role in attitudes towards the wearing of masks. If you’re someone with low empathy, someone who basically feels it’s every man for himself, then you may well be biased toward saying masks are ineffective or that masks should not be worn; its not the function of the state to mandate them.

The identification with political factions and shared viewpoints about the advisability of mask wearing will bias those holding certain views. In general people on the right tend to be more anti-mask than people in the left but it is not as clear-cut as all of that.

You may have a belief that masks have harmful social consequences because it is essential that we be able to see others face in order to function smoothly in society. Of course, there are no cultures where masks are routinely worn, are there?

You may have a belief that masks have harmful medical consequences. For instance you may believe that masks do not allow the adequate intake of oxygen and the exhalation of carbon dioxide. There are studies on both sides of this issue but in fact people do wear masks in society and in medicine for extended periods of time; there do not seem to routine and significant reported negative consequences.

In addition there is a school of thought that by wearing a mask you do not allow the virus to escape and it will recirculate in your own body and increase your viral load. I have to admit this makes very little sense to me.

It’s my thinking that if you are strongly libertarian, you may find reasons to downplay both the severity of the infection and the need to wear masks.

If you are deficient in concern about others, and believe that the mask just protects others, you will find reasons to disparage mask wearing.

Perhaps those who believe in wearing a mask think along these lines:

  1. The virus is a serious, infectious, virulent business
  2. I don’t want to be infected
  3. I don’t want to infect others
  4. Masks help protect the other, help protect me and collectively benefit us all
  5. Masks do not harm us, individually or collectively
  6. The state has a right to mandate in some areas of public safety and health
  7. My sources of information are better
  8. Those who dispute my views on mask wearing are idiots if not downright evil

Perhaps those who do not believe in wearing a mask think along these lines:

  1. The virus is a no worse than the seasonal flu
  2. I am not going to be badly infected
  3. Infection of others is not of concern to me
  4. Masks don’t protect anyone
  5. Masks harm us, individually and collectively
  6. The state has no right to mandate mask wearing
  7. My sources of information are better
  8. Those who dispute my views on mask wearing are idiots if not downright evil

So, if you believe that the seriousness of the infection is vastly overblown you will find reasons to downplay mask wearing is necessary. These opinions all interact to some unknown degree. They are not independent factors. Beliefs, values and biases are an aspect of motivated reasoning.

An article on “How to Disagree”

How to Disgree

Paul Graham

March 2008

The web is turning writing into a conversation. Twenty years ago, writers wrote and readers read. The web lets readers respond, and increasingly they do—in comment threads, on forums, and in their own blog posts.

Many who respond to something disagree with it. That’s to be expected. Agreeing tends to motivate people less than disagreeing. And when you agree there’s less to say. You could expand on something the author said, but he has probably already explored the most interesting implications. When you disagree you’re entering territory he may not have explored.

The result is there’s a lot more disagreeing going on, especially measured by the word. That doesn’t mean people are getting angrier. The structural change in the way we communicate is enough to account for it. But though it’s not anger that’s driving the increase in disagreement, there’s a danger that the increase in disagreement will make people angrier. Particularly online, where it’s easy to say things you’d never say face to face.

If we’re all going to be disagreeing more, we should be careful to do it well. What does it mean to disagree well? Most readers can tell the difference between mere name-calling and a carefully reasoned refutation, but I think it would help to put names on the intermediate stages. So here’s an attempt at a disagreement hierarchy: … ” more How to Disgree

Bibliography of Articles on Mask Wearing

A bibliography of articles – studies and opinion, on the wearing of masks for reducing infection spread and severity.

I consider it prudent to wear a mask when in close contact with people not in your personal “bubble.” I do not feel it makes sense to wear one when outside in the fresh air while maintaining a couple of meters of distance from those who are not in my “bubble.” This is risk management 101. I also think that claims of harm, physical and psychological from the wearing of masks are overblown to the point of absurdity. The other issue, that of social control, seems possibly overblown as well, although I am aghast at some of the draconian steps taken by authorities is some places. There are also a number of ill-informed interpretations of the research literature.

I am by most indicators some sort of socialist libertarian, so yes, a libertarian. I still am not sold on the idea of masks being some part of a plot to control us. I believe that a lot of conspiracist views are well founded, since there are clearly many deep state conspiracies. I am not so sure about this one.

See for instance: https://www.opednews.com/articles/Motivated-Reasoning-by-Mike-Zimmer-Belief_Bias_Ideas_Motivated-Reasoning-200804-599.html and https://www.opednews.com/articles/Masks-and-Motivated-Reason-by-Mike-Zimmer-Belief_Bias_Covid-19_Empathy-200805-253.html

 

  1. Psychology Today. “5 More Ways That COVID-19 Is Not Like the Flu.” Accessed September 10, 2020. https://www.psychologytoday.com/blog/beastly-behavior/202005/5-more-ways-covid-19-is-not-the-flu.
  2. Bored Panda. “30 People Shame Those Who Refuse To Wear A Mask Due To Their Own Stupid Reasons.” Accessed September 10, 2020. https://www.boredpanda.com/no-excuse-not-to-wear-a-mask-reasons/.
  3. Aj, Sant, and McMichael A. “Revealing the Role of CD4(+) T Cells in Viral Immunity.” The Journal of Experimental Medicine 209, no. 8 (July 1, 2012): 1391–95. https://doi.org/10.1084/jem.20121517.
  4. Twitter. “Alex Berenson on Twitter.” Accessed October 6, 2020. https://twitter.com/AlexBerenson/status/1308798932097658881.
  5. Twitter. “Alex Berenson on Twitter.” Accessed October 6, 2020. https://twitter.com/AlexBerenson/status/1308798932097658881.
  6. Anfinrud, Philip, Valentyn Stadnytskyi, Christina E. Bax, and Adriaan Bax. “Visualizing Speech-Generated Oral Fluid Droplets with Laser Light Scattering.” New England Journal of Medicine 382, no. 21 (May 21, 2020): 2061–63. https://doi.org/10.1056/NEJMc2007800.
  7. “Are Face Masks Effective? The Evidence. – Swiss Policy Research.” Accessed September 9, 2020. https://swprs.org/face-masks-evidence/.
  8. ResearchGate. “Assessment of Fabric Masks as Alternatives to Standard Surgical Masks in Terms of Particle Filtration Efficiency | Request PDF.” Accessed September 10, 2020. https://doi.org/10.1101/2020.04.17.20069567.
  9. “Association of Country-Wide Coronavirus Mortality with Demographics, Testing, Lockdowns, and Public Wearing of Masks. Update August 4, 2020. | MedRxiv.” Accessed September 9, 2020. https://www.medrxiv.org/content/10.1101/2020.05.22.20109231v5.
  10. Aug 06, 2020. “OSHA Addresses Inaccurate Claims That Face Coverings Cause Wearer Harm -.” Occupational Health & Safety. Accessed September 10, 2020. https://ohsonline.com/articles/2020/08/06/osha-addresses-inaccurate-claims-that-face-coverings-cause-wearer-harm.aspx.
  11. Author, Alternate. “Blaylock: Face Masks Pose Serious Risks To The Healthy.” Technocracy News. Accessed September 9, 2020. https://www.technocracy.news/blaylock-face-masks-pose-serious-risks-to-the-healthy/?fbclid=IwAR0EtvZsmXqKARtPaJNamsuFPsAOrrHq-CHcH0_wcPDwMihQfpgcCgMA0g8.
  12. Av, Mueller, Eden Mj, Oakes Jj, Bellini C, and Fernandez La. “Quantitative Method for Comparative Assessment of Particle Filtration Efficiency of Fabric Masks as Alternatives to Standard Surgical Masks for PPE,” April 22, 2020. https://doi.org/10.1101/2020.04.17.20069567.
  13. B, Chandrasekaran, and Fernandes S. “‘Exercise with Facemask; Are We Handling a Devil’s Sword?’ – A Physiological Hypothesis.” Medical hypotheses. Med Hypotheses, June 22, 2020. https://doi.org/10.1016/j.mehy.2020.110002.
  14. Bae, Seongman, Min-Chul Kim, Ji Yeun Kim, Hye-Hee Cha, Joon Seo Lim, Jiwon Jung, Min-Jae Kim, et al. “Effectiveness of Surgical and Cotton Masks in Blocking SARS–CoV-2: A Controlled Comparison in 4 Patients.” Annals of Internal Medicine 173, no. 1 (April 6, 2020): W22–23. https://doi.org/10.7326/M20-1342.
  15. Beder, A., U. Büyükkoçak, H. Sabuncuoğlu, Z. A. Keskil, and S. Keskil. “Preliminary Report on Surgical Mask Induced Deoxygenation during Major Surgery.” Neurocirugia (Asturias, Spain) 19, no. 2 (April 2008): 121–26. https://doi.org/10.1016/s1130-1473(08)70235-5.
  16. Psychology Today. “‘Beginner’s Mind’ Just Might Save Lives.” Accessed September 10, 2020. https://www.psychologytoday.com/blog/think-act-be/202004/beginners-mind-just-might-save-lives.
  17. News Break. “Both Big and Tiny Coronavirus Droplets Can Travel Far through Air.” Accessed September 10, 2020. https://www.newsbreak.com/news/1611636431970/both-big-and-tiny-coronavirus-droplets-can-travel-far-through-air.
  18. Bourouiba, Lydia. “Turbulent Gas Clouds and Respiratory Pathogen Emissions: Potential Implications for Reducing Transmission of COVID-19.” JAMA 323, no. 18 (May 12, 2020): 1837–38. https://doi.org/10.1001/jama.2020.4756.
  19. Brooks, John T., Jay C. Butler, and Robert R. Redfield. “Universal Masking to Prevent SARS-CoV-2 Transmission-The Time Is Now.” JAMA, July 14, 2020. https://doi.org/10.1001/jama.2020.13107.
  20. Brosseau, Lisa M., ScD, Margaret Sietsema, PhD | Apr 01, and 2020. “COMMENTARY: Masks-for-All for COVID-19 Not Based on Sound Data.” CIDRAP. Accessed September 8, 2020. https://www.cidrap.umn.edu/news-perspective/2020/04/commentary-masks-all-covid-19-not-based-sound-data.
  21. Caddy, Sarah L. “Coronavirus: Does the Amount of Virus You Are Exposed to Determine How Sick You’ll Get?” The Conversation. Accessed September 23, 2020. http://theconversation.com/coronavirus-does-the-amount-of-virus-you-are-exposed-to-determine-how-sick-youll-get-135119.
  22. Mayo Clinic. “Can Face Masks Protect against the Coronavirus?” Accessed September 9, 2020. https://www.mayoclinic.org/diseases-conditions/coronavirus/in-depth/coronavirus-mask/art-20485449.
  23. CDC. “Coronavirus Disease 2019 (COVID-19).” Centers for Disease Control and Prevention, February 11, 2020. https://www.cdc.gov/coronavirus/2019-ncov/hcp/planning-scenarios.html.
  24. Chaudhry, Rabail, George Dranitsaris, Talha Mubashir, Justyna Bartoszko, and Sheila Riazi. “A Country Level Analysis Measuring the Impact of Government Actions, Country Preparedness and Socioeconomic Factors on COVID-19 Mortality and Related Health Outcomes.” EClinicalMedicine 25 (August 1, 2020). https://doi.org/10.1016/j.eclinm.2020.100464.
  25. Chu, Derek K., Elie A. Akl, Stephanie Duda, Karla Solo, Sally Yaacoub, Holger J. Schünemann, Derek K. Chu, et al. “Physical Distancing, Face Masks, and Eye Protection to Prevent Person-to-Person Transmission of SARS-CoV-2 and COVID-19: A Systematic Review and Meta-Analysis.” The Lancet 395, no. 10242 (June 27, 2020): 1973–87. https://doi.org/10.1016/S0140-6736(20)31142-9.
  26. Chughtai, Abrar A., Holly Seale, and C. Raina Macintyre. “Early Release – Effectiveness of Cloth Masks for Protection Against Severe Acute Respiratory Syndrome Coronavirus 2 – Volume 26, Number 10—October 2020 – Emerging Infectious Diseases Journal – CDC.” Accessed September 10, 2020. https://doi.org/10.3201/eid2610.200948.
  27. Clase, Catherine M., Edouard L. Fu, Aurneen Ashur, Rupert CL. Beale, Imogen A. Clase, Myrna B. Dolovich, Meg J. Jardine, et al. “Forgotten Technology in the COVID-19 Pandemic. Filtration Properties of Cloth and Cloth Masks: A Narrative Review.” Mayo Clinic Proceedings, July 31, 2020. https://doi.org/10.1016/j.mayocp.2020.07.020.
  28. Clase, Catherine M., Edouard L. Fu, Meera Joseph, Rupert C.L. Beale, Myrna B. Dolovich, Meg Jardine, Johannes F.E. Mann, Roberto Pecoits-Filho, Wolfgang C. Winkelmayer, and Juan J. Carrero. “Cloth Masks May Prevent Transmission of COVID-19: An Evidence-Based, Risk-Based Approach.” Annals of Internal Medicine, May 22, 2020. https://doi.org/10.7326/M20-2567.
  29. “COVID-19: Considerations for Wearing Masks | CDC.” Accessed September 10, 2020. https://www.cdc.gov/coronavirus/2019-ncov/prevent-getting-sick/cloth-face-cover-guidance.html.
  30. “COVID19 PCR Tests Are Scientifically Meaningless – Bulgarian Pathology Association.” Accessed September 10, 2020. https://bpa-pathology.com/covid19-pcr-tests-are-scientifically-meaningless/.
  31. “COVID-19–Related Perceptions, Context and Attitudes of Adults with Chronic Conditions: Results from a Cross-Sectional Survey Nested in the ComPaRe e-Cohort.” Accessed September 10, 2020. https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0237296.
  32. “Dear Humans: Face Masks Don’t Work; the Study-Review Was Published by Your Very Own CDC « Jon Rappoport’s Blog.” Accessed September 10, 2020. https://blog.nomorefakenews.com/2020/07/20/face-masks-dont-work-study-published-by-your-very-own-cdc/.
  33. News Break. “Does Personality Predict Willingness to Wear a Mask?” Accessed September 10, 2020. https://www.newsbreak.com/news/1605745385073/does-personality-predict-willingness-to-wear-a-mask.
  34. “Does Wearing a Mask Cause Diagnostic Tests to Read False-Positive for COVID? « Jon Rappoport’s Blog.” Accessed September 10, 2020. https://blog.nomorefakenews.com/2020/07/23/wearing-mask-cause-diagnostic-tests-read-false-positive-covid/.
  35. “Does Wearing a Mask Pose Any Health Risks?” Accessed September 10, 2020. https://medicalxpress.com/news/2020-07-mask-pose-health.html.
  36. Healthline. “Does Wearing a Mask Prevent the Flu?,” July 29, 2020. https://www.healthline.com/health/cold-flu/mask.
  37. “Does Wearing a Mask Protect Me? Some Evidence Says Yes – The New York Times.” Accessed September 10, 2020. https://www.nytimes.com/2020/07/27/health/coronavirus-mask-protection.html?campaign_id=34&emc=edit_sc_20200728&instance_id=20709&nl=science-times&regi_id=4688638&segment_id=34547&te=1&user_id=87f07d5c82fd6eeab8a4f850cd830fa6.
  38. News Break. “Dutch Dr. Fauci Thinks Masks Won’t Work. Here’s Why.” Accessed September 10, 2020. https://www.newsbreak.com/news/1611980964379/dutch-dr-fauci-thinks-masks-wont-work-heres-why.
  39. EDT, Basit Mahmood On 7/23/20 at 1:29 PM. “Doctor Runs 22 Miles Wearing Face Mask to Show It Doesn’t Cut Oxygen Levels.” Newsweek, July 23, 2020. https://www.newsweek.com/masks-covid-19-doctor-oxygen-1520060.
  40. Esposito, Susanna, Nicola Principi, Chi Chi Leung, and Giovanni Battista Migliori. “Universal Use of Face Masks for Success against COVID-19: Evidence and Implications for Prevention Policies.” European Respiratory Journal 55, no. 6 (June 1, 2020). https://doi.org/10.1183/13993003.01260-2020.
  41. “Face Masks Considerably Reduce COVID-19 Cases in Germany: A Synthetic Control Method Approach.” Accessed September 10, 2020. https://www.iza.org/publications/dp/13319/face-masks-considerably-reduce-covid-19-cases-in-germany-a-synthetic-control-method-approach.
  42. MSN. “Face Masks Protect You More From COVID Than You Thought, Doctors Say.” Accessed September 10, 2020. https://www.msn.com/en-us/Health/medical/face-masks-protect-you-more-from-covid-than-you-thought-doctors-say/ar-BB16ylTh.
  43. “Facemasks and Similar Barriers to Prevent Respiratory Illness Such as COVID-19: A Rapid Systematic Review | MedRxiv.” Accessed September 10, 2020. https://www.medrxiv.org/content/10.1101/2020.04.01.20049528v1.
  44. “Formal Request for the Retraction of Zhang et al., 2020 | Meta-Research Innovation Center at Stanford.” Accessed September 10, 2020. https://metrics.stanford.edu/PNAS%20retraction%20request%20LoE%20061820.
  45. Frankel, Todd C. “The Outbreak That Didn’t Happen: Masks Credited with Preventing Coronavirus Spread inside Missouri Hair Salon.” Washington Post. Accessed September 10, 2020. https://www.washingtonpost.com/business/2020/06/17/masks-salons-missouri/.
  46. G, Monica, and hi. “Mask Wearers Are ‘Dramatically Less Likely’ to Get a Severe Case of Covid-19.” Inverse. Accessed September 9, 2020. https://www.inverse.com/mind-body/masks-breathing-in-less-coronavirus-means-you-get-less-sick.
  47. Gandhi, Monica. “Can People Spread the Coronavirus If They Don’t Have Symptoms? 5 Questions Answered about Asymptomatic COVID-19.” The Conversation. Accessed September 9, 2020. http://theconversation.com/can-people-spread-the-coronavirus-if-they-dont-have-symptoms-5-questions-answered-about-asymptomatic-covid-19-140531.
  48. Gandhi, Monica, Chris Beyrer, and Eric Goosby. “Masks Do More Than Protect Others During COVID-19: Reducing the Inoculum of SARS-CoV-2 to Protect the Wearer.” Journal of General Internal Medicine, July 31, 2020. https://doi.org/10.1007/s11606-020-06067-8.
  49. Gandhi, Monica, and Diane Havlir. “The Time for Universal Masking of the Public for Coronavirus Disease 2019 Is Now.” Open Forum Infectious Diseases 7, no. 4 (April 2020): ofaa131. https://doi.org/10.1093/ofid/ofaa131.
  50. Gandhi, Monica, and George W. Rutherford. “Facial Masking for Covid-19 — Potential for ‘Variolation’ as We Await a Vaccine.” New England Journal of Medicine 0, no. 0 (September 8, 2020): null. https://doi.org/10.1056/NEJMp2026913.
  51. Ghesquierre, Wayne. “Comment: Wear a Mask to Protect Yourself and Others. You Will Not Be Harmed.” Times Colonist. Accessed September 10, 2020. https://www.timescolonist.com/opinion/op-ed/comment-wear-a-mask-to-protect-yourself-and-others-you-will-not-be-harmed-1.24177281.
  52. Greenhalgh, Zeynep Tufekci, Jeremy Howard, Trisha. “The Real Reason to Wear a Mask.” The Atlantic, April 22, 2020. https://www.theatlantic.com/health/archive/2020/04/dont-wear-mask-yourself/610336/.
  53. Grinshpun, Sergey A., Hiroki Haruta, Robert M. Eninger, Tiina Reponen, Roy T. McKay, and Shu-An Lee. “Performance of an N95 Filtering Facepiece Particulate Respirator and a Surgical Mask During Human Breathing: Two Pathways for Particle Penetration.” Journal of Occupational and Environmental Hygiene 6, no. 10 (September 9, 2009): 593–603. https://doi.org/10.1080/15459620903120086.
  54. Hamzelou, Jessica. “Do Face Masks Work against the Coronavirus and Should You Wear One?” New Scientist. Accessed September 10, 2020. https://www.newscientist.com/article/2240288-do-face-masks-work-against-the-coronavirus-and-should-you-wear-one/.
  55. Hibino, Sawako, Kazutaka Hayashida, Andrew C Ahn, and Yasutaka Hayashida. “Dynamic Change of COVID-19 Seroprevalence among Asymptomatic Population in Tokyo during the Second Wave.” Preprint. Infectious Diseases (except HIV/AIDS), September 23, 2020. https://doi.org/10.1101/2020.09.21.20198796.
  56. Howard, Jeremy, Austin Huang, Zhiyuan Li, Zeynep Tufekci, Vladimir Zdimal, Helene-Mari van der Westhuizen, Arne von Delft, et al. “Face Masks Against COVID-19: An Evidence Review,” April 12, 2020. https://doi.org/10.20944/preprints202004.0203.v1.
  57. Newsweek. “‘I Ran 22 Miles In A Mask To Show They Are Safe,’” July 28, 2020. https://www.newsweek.com/trolled-online-22-mile-running-show-masks-safe-1520710.
  58. Ing, Alvin J., Christine Cocks, and Jeffery Peter Green. “COVID-19: In the Footsteps of Ernest Shackleton.” Thorax 75, no. 8 (August 1, 2020): 693–94. https://doi.org/10.1136/thoraxjnl-2020-215091.
  59. The Daily Wire. “Is Herd Immunity An Effective Strategy? Here’s What The Latest Studies Found.” Accessed September 10, 2020. https://www.dailywire.com/news/is-herd-immunity-an-effective-strategy-latest-studies.
  60. Jacobs, Joshua L., Sachiko Ohde, Osamu Takahashi, Yasuharu Tokuda, Fumio Omata, and Tsuguya Fukui. “Use of Surgical Face Masks to Reduce the Incidence of the Common Cold among Health Care Workers in Japan: A Randomized Controlled Trial.” American Journal of Infection Control 37, no. 5 (June 2009): 417–19. https://doi.org/10.1016/j.ajic.2008.11.002.
  61. Jefferson, Tom, Mark Jones, Lubna A. Al Ansari, Ghada Bawazeer, Elaine Beller, Justin Clark, John Conly, et al. “Physical Interventions to Interrupt or Reduce the Spread of Respiratory Viruses. Part 1 – Face Masks, Eye Protection and Person Distancing: Systematic Review and Meta-Analysis.” MedRxiv, April 7, 2020, 2020.03.30.20047217. https://doi.org/10.1101/2020.03.30.20047217.
  62. July 9, Charlie Duerr, and 2020. “The One Face Mask Hack You’re Not Doing But Should Be.” Best Life, July 9, 2020. https://bestlifeonline.com/face-mask-hack-tiktok/.
  63. Jun 24, Stephanie Soucheray | News Reporter | CIDRAP News |, and 2020. “Controversy on COVID-19 Mask Study Spotlights Messiness of Science during a Pandemic.” CIDRAP. Accessed September 10, 2020. https://www.cidrap.umn.edu/news-perspective/2020/06/controversy-covid-19-mask-study-spotlights-messiness-science-during.
  64. June 15, Colby Hall, and 2020. “172 Studies Agree You Need to Do This to Combat Coronavirus.” Best Life, June 15, 2020. https://bestlifeonline.com/masks-coronavirus/.
  65. Kai, De, Guy-Philippe Goldstein, Alexey Morgunov, Vishal Nangalia, and Anna Rotkirch. “Universal Masking Is Urgent in the COVID-19 Pandemic: SEIR and Agent Based Models, Empirical Validation, Policy Recommendations.” ArXiv:2004.13553 [Physics, q-Bio], April 22, 2020. http://arxiv.org/abs/2004.13553.
  66. “Keep Britain Free.” Accessed September 10, 2020. https://www.keepbritainfree.com/.
  67. Klompas, Michael, Charles A. Morris, Julia Sinclair, Madelyn Pearson, and Erica S. Shenoy. “Universal Masking in Hospitals in the Covid-19 Era.” New England Journal of Medicine 382, no. 21 (May 21, 2020): e63. https://doi.org/10.1056/NEJMp2006372.
  68. kristac@stanford.edu, <img src=’//sgec stanford edu/content/dam/sm-news/images/2015/10/conger-krista-90 jpg img 620 high png’ alt=’Krista Conger’> By Krista Conger Krista Conger is a science writer in the Office of Communications Email her at. “5 Questions: Stanford Scientists on COVID-19 Mask Guidelines.” News Center. Accessed September 10, 2020. http://med.stanford.edu/news/all-news/2020/06/stanford-scientists-contribute-to-who-mask-guidelines.html.
  69. kyle8425. “On Masks: Notes for Rebuttal.” Another Logical Take (blog), July 24, 2020. https://davidkylejohnson.wordpress.com/2020/07/24/on-masks-notes-for-rebuttal/.
  70. ———. “Resolved: Public Mask Mandates Assist in Curbing the Spread of Covid-19.” Another Logical Take (blog), July 24, 2020. https://davidkylejohnson.wordpress.com/2020/07/24/resolved-public-mask-mandates-assist-in-curbing-the-spread-of-covid-19/.
  71. “Lack of COVID-19 Transmission on an International Flight | CMAJ.” Accessed September 10, 2020. https://www.cmaj.ca/content/192/15/E410.
  72. “Letter to the Editor: Look at the Range of Science on Face Masks | TribLIVE.Com.” Accessed September 10, 2020. https://triblive.com/opinion/letter-to-the-editor-look-at-the-range-of-science-on-face-masks/.
  73. Leung, Nancy H. L., Daniel K. W. Chu, Eunice Y. C. Shiu, Kwok-Hung Chan, James J. McDevitt, Benien J. P. Hau, Hui-Ling Yen, et al. “Respiratory Virus Shedding in Exhaled Breath and Efficacy of Face Masks.” Nature Medicine 26, no. 5 (May 2020): 676–80. https://doi.org/10.1038/s41591-020-0843-2.
  74. Lopez, German. “Why You Should Wear a Mask to Fight Covid-19, Explained by Several New Studies.” Vox, July 15, 2020. https://www.vox.com/future-perfect/21299527/masks-coronavirus-covid-19-studies-research-evidence.
  75. “Low-Cost Measurement of Face Mask Efficacy for Filtering Expelled Droplets during Speech | Science Advances.” Accessed September 10, 2020. https://advances.sciencemag.org/content/6/36/eabd3083.
  76. Lu, Xiuhua, Terrence M. Tumpey, Timothy Morken, Sherif R. Zaki, Nancy J. Cox, and Jacqueline M. Katz. “A Mouse Model for the Evaluation of Pathogenesis and Immunity to Influenza A (H5N1) Viruses  Isolated from Humans.” Journal of Virology 73, no. 7 (July 1999): 5903–11.
  77. Lukashev, Dmitriy, Boris Klebanov, Hidefumi Kojima, Alex Grinberg, Akiko Ohta, Ludmilla Berenfeld, Roland H. Wenger, Akio Ohta, and Michail Sitkovsky. “Cutting Edge: Hypoxia-Inducible Factor 1α and Its Activation-Inducible Short Isoform I.1 Negatively Regulate Functions of CD4+ and CD8+ T Lymphocytes.” The Journal of Immunology 177, no. 8 (October 15, 2006): 4962–65. https://doi.org/10.4049/jimmunol.177.8.4962.
  78. Lyu, Wei, and George L. Wehby. “Community Use Of Face Masks And COVID-19: Evidence From A Natural Experiment Of State Mandates In The US.” Health Affairs 39, no. 8 (June 16, 2020): 1419–25. https://doi.org/10.1377/hlthaff.2020.00818.
  79. MacIntyre, C. Raina, Simon Cauchemez, Dominic E. Dwyer, Holly Seale, Pamela Cheung, Gary Browne, Michael Fasher, et al. “Face Mask Use and Control of Respiratory Virus Transmission in Households – Volume 15, Number 2—February 2009 – Emerging Infectious Diseases Journal – CDC.” Accessed September 10, 2020. https://doi.org/10.3201/eid1502.081167.
  80. MacIntyre, C. Raina, and Abrar Ahmad Chughtai. “A Rapid Systematic Review of the Efficacy of Face Masks and Respirators against Coronaviruses and Other Respiratory Transmissible Viruses for the Community, Healthcare Workers and Sick Patients.” International Journal of Nursing Studies 108 (August 1, 2020): 103629. https://doi.org/10.1016/j.ijnurstu.2020.103629.
  81. ———. “Facemasks for the Prevention of Infection in Healthcare and Community Settings.” BMJ 350 (April 9, 2015). https://doi.org/10.1136/bmj.h694.
  82. MacIntyre, C. Raina, Holly Seale, Tham Chi Dung, Nguyen Tran Hien, Phan Thi Nga, Abrar Ahmad Chughtai, Bayzidur Rahman, Dominic E. Dwyer, and Quanyi Wang. “A Cluster Randomised Trial of Cloth Masks Compared with Medical Masks in Healthcare Workers.” BMJ Open 5, no. 4 (April 1, 2015): e006577. https://doi.org/10.1136/bmjopen-2014-006577.
  83. OpEdNews. “Masks and Motivated Reasoning.” Accessed October 7, 2020. https://www.opednews.com/articles/Masks-and-Motivated-Reason-by-Mike-Zimmer-Belief_Bias_Covid-19_Empathy-200805-253.html.
  84. primarydoctor. “Masks Are Neither Effective nor Safe.” Accessed September 8, 2020. https://www.primarydoctor.org/masks-not-effect.
  85. ZDoggMD. “Masks May Do More Than We Think (w/Dr. Monica Gandhi),” September 14, 2020. https://zdoggmd.com/monica-gandhi/.
  86. Psychology Today. “Masks (Still) Work: Debunking (More) Pseudoscience.” Accessed September 10, 2020. https://www.psychologytoday.com/blog/logical-take/202007/masks-still-work-debunking-more-pseudoscience.
  87. McCabe, Caitlin. “Face Masks Really Do Matter. The Scientific Evidence Is Growing.” Wall Street Journal, August 13, 2020, sec. Life. https://www.wsj.com/articles/face-masks-really-do-matter-the-scientific-evidence-is-growing-11595083298.
  88. McKenney, Douglas G, Gael Kurath, and Andrew R Wargo. “Characterization of Infectious Dose and Lethal Dose of Two Strains of Infectious Hematopoietic Necrosis Virus (IHNV).” Virus Research 214 (March 2016): 80–89. https://doi.org/10.1016/j.virusres.2015.12.020.
  89. Mitchell, N. J., and S. Hunt. “Surgical Face Masks in Modern Operating Rooms—a Costly and Unnecessary Ritual?” Journal of Hospital Infection 18, no. 3 (July 1, 1991): 239–42. https://doi.org/10.1016/0195-6701(91)90148-2.
  90. Morawska, Lidia, and Donald K. Milton. “It Is Time to Address Airborne Transmission of Coronavirus Disease 2019 (COVID-19).” Clinical Infectious Diseases. Accessed September 23, 2020. https://doi.org/10.1093/cid/ciaa939.
  91. Twitter. “Nassim Nicholas Taleb on Twitter.” Accessed September 30, 2020. https://twitter.com/nntaleb/status/1249296844712218624.
  92. Institute for Health Metrics and Evaluation. “New IHME COVID-19 Model Projects Nearly 180,000 US Deaths,” June 24, 2020. http://www.healthdata.org/news-release/new-ihme-covid-19-model-projects-nearly-180000-us-deaths.
  93. Offeddu, Vittoria, Chee Fu Yung, Mabel Sheau Fong Low, and Clarence C. Tam. “Effectiveness of Masks and Respirators Against Respiratory Infections in Healthcare Workers: A Systematic Review and Meta-Analysis.” Clinical Infectious Diseases 65, no. 11 (November 13, 2017): 1934–42. https://doi.org/10.1093/cid/cix681.
  94. Oran, Daniel P., and Eric J. Topol. “Prevalence of Asymptomatic SARS-CoV-2 Infection.” Annals of Internal Medicine 173, no. 5 (June 3, 2020): 362–67. https://doi.org/10.7326/M20-3012.
  95. Safety News Alert. “OSHA: Cloth Masks Don’t Cause Harmful CO2 Levels,” August 12, 2020. https://www.safetynewsalert.com/articles/osha-cloth-masks-co2/.
  96. ResearchGate. “(PDF) A Complete Debunking of Denis Rancourt’s Mask Don’t Work.” Accessed September 9, 2020. https://doi.org/10.13140/RG.2.2.22021.37603.
  97. ResearchGate. “(PDF) The Efficacy of Medical Masks and Respirators against Respiratory Infection in Health Workers.” Accessed September 10, 2020. https://doi.org/10.1111/irv.12474.
  98. ResearchGate. “(PDF) Universal Masking Is Urgent in the COVID-19 Pandemic: SEIR and Agent Based Models, Empirical Validation, Policy Recommendations.” Accessed September 10, 2020. https://doi.org/10.13140/RG.2.2.21662.08001.
  99. Psychology Today. “People Who Don’t Wear Masks Misunderstand Coronavirus Spread.” Accessed September 10, 2020. https://www.psychologytoday.com/blog/shouldstorm/202007/people-who-dont-wear-masks-misunderstand-coronavirus-spread.
  100. Person, E., C. Lemercier, A. Royer, and G. Reychler. “[Effect of a surgical mask on six minute walking distance].” Revue Des Maladies Respiratoires 35, no. 3 (March 2018): 264–68. https://doi.org/10.1016/j.rmr.2017.01.010.
  101. Psychology Today. “Personality and Our Willingness to Wear Masks.” Accessed September 10, 2020. https://www.psychologytoday.com/blog/why-bad-looks-good/202007/personality-and-our-willingness-wear-masks.
  102. Pike, Lili. “Why 15 US States Suddenly Made Masks Mandatory.” Vox, May 29, 2020. https://www.vox.com/2020/5/29/21273625/coronavirus-masks-required-virginia-china-hong-kong.
  103. Presstv. “Masks Get Credit for Protecting a Conservative Community.” Video. PressTV. PressTV. Accessed September 10, 2020. https://www.presstv.com/Detail/2020/07/07/629062/Masks-get-credit-for-protecting-a-conservative-community.
  104. Radonovich, Lewis J., Michael S. Simberkoff, Mary T. Bessesen, Alexandria C. Brown, Derek A. T. Cummings, Charlotte A. Gaydos, Jenna G. Los, et al. “N95 Respirators vs Medical Masks for Preventing Influenza Among Health Care Personnel: A Randomized Clinical Trial.” JAMA 322, no. 9 (September 3, 2019): 824–33. https://doi.org/10.1001/jama.2019.11645.
  105. Rancourt, Denis G. “Masks Don’t Work: A Review of Science Relevant to Covid-19 Social Policy,” June 1, 2020. https://vixra.org/abs/2006.0044.
  106. Rengasamy, Samy, Benjamin Eimer, and Ronald E. Shaffer. “Simple Respiratory Protection—Evaluation of the Filtration Performance of Cloth Masks and Common Fabric Materials Against 20–1000 Nm Size Particles.” The Annals of Occupational Hygiene 54, no. 7 (October 1, 2010): 789–98. https://doi.org/10.1093/annhyg/meq044.
  107. Rouse, Barry T., and Sharvan Sehrawat. “Immunity and Immunopathology to Viruses: What Decides the Outcome?” Nature Reviews Immunology 10, no. 7 (July 2010): 514–26. https://doi.org/10.1038/nri2802.
  108. Ryan. “When It Comes to Masks, There Is No ‘Settled Science.’” Text. Mises Institute, July 23, 2020. https://mises.org/wire/when-it-comes-masks-there-no-settled-science.
  109. Shimasaki, Noriko, Akira Okaue, Ritsuko Kikuno, and Katsuaki Shinohara. “Comparison of the Filter Efficiency of Medical Nonwoven Fabrics against Three Different Microbe Aerosols.” Biocontrol Science 23, no. 2 (2018): 61–69. https://doi.org/10.4265/bio.23.61.
  110. “SIMPLE METHOD OF ESTIMATING FIFTY PER CENT ENDPOINTS12 | American Journal of Epidemiology | Oxford Academic.” Accessed September 9, 2020. https://academic.oup.com/aje/article-abstract/27/3/493/99616?redirectedFrom=fulltext.
  111. Smith, Adam. “Masks Are Important but No Cure-All: Prof. Tells of Way to Slow Covid Spread.” TheStreet. Accessed September 10, 2020. https://www.thestreet.com/latest-news/ben-cowling-of-the-university-of-hong-kong-discusses-whats-known-among-doctors-as-shortening-the-serial-interval-distribution.
  112. Smith, Jeffrey D., Colin C. MacDougall, Jennie Johnstone, Ray A. Copes, Brian Schwartz, and Gary E. Garber. “Effectiveness of N95 Respirators versus Surgical Masks in Protecting Health Care Workers from Acute Respiratory Infection: A Systematic Review and Meta-Analysis.” CMAJ 188, no. 8 (May 17, 2016): 567–74. https://doi.org/10.1503/cmaj.150835.
  113. News Break Boston, MA. “Some Hospitals Move To ‘Universal Mask’ Policy. Should Everyone Wear Masks In Public? | News Break.” Accessed September 10, 2020. https://www.newsbreak.com/massachusetts/boston/news/1533890844656/some-hospitals-move-to-universal-mask-policy-should-everyone-wear-masks-in-public.
  114. The World from PRX. “Some Public Figures Eschew Masks. Scientists Say Wear Them.” Accessed September 10, 2020. https://www.pri.org/stories/2020-06-22/some-public-figures-eschew-masks-scientists-say-wear-them.
  115. Spencer, Saranac Hale. “Video Misrepresents the Science Behind Face Masks.” FactCheck.Org (blog), July 24, 2020. https://www.factcheck.org/2020/07/video-misrepresents-the-science-behind-face-masks/.
  116. Staff, Reuters. “Dutch Government Will Not Advise Public to Wear Masks – Minister.” Reuters, July 29, 2020. https://www.reuters.com/article/us-health-coronavirus-netherlands-idUSKCN24U2UJ.
  117. Still Confused About Masks? Here’s the Science Behind How Face Masks Prevent Coronavirus | UC San Francisco. “Still Confused About Masks? Here’s the Science Behind How Face Masks Prevent Coronavirus.” Accessed September 10, 2020. https://www.ucsf.edu/news/2020/06/417906/still-confused-about-masks-heres-science-behind-how-face-masks-prevent.
  118. News Break. “Sweden’s Equivalent to Dr. Fauci Says ‘it Is Very Dangerous’ to Believe Face Masks Are a Cure-All.” Accessed September 10, 2020. https://www.newsbreak.com/news/2045077234116/swedens-equivalent-to-dr-fauci-says-it-is-very-dangerous-to-believe-face-masks-are-a-cure-all.
  119. Fortune. “Sweden’s Top Virologist Has a Message on How to Defeat Coronavirus: Open Schools and No Masks.” Accessed September 10, 2020. https://fortune.com/2020/08/05/sweden-anders-tegnell-face-masks-school-opening-coronavirus-covid-19-europe/.
  120. Taleb, Nassim Nicholas. “The Masks Masquerade.” Medium, June 14, 2020. https://medium.com/incerto/the-masks-masquerade-7de897b517b7.
  121. “Talebpandemic.Mp3.” Accessed September 30, 2020. http://files.libertyfund.org/econtalk/y2020/Talebpandemic.mp3.
  122. The Skeptical Cardiologist. “The Economic Impact of Wearing Face Masks During COVID-19,” July 7, 2020. https://theskepticalcardiologist.com/2020/07/07/the-economic-impact-of-wearing-face-masks-during-covid-19/.
  123. Psychology Today. “The Evidence on Face Masks.” Accessed September 10, 2020. https://www.psychologytoday.com/blog/evidence-based-living/202004/the-evidence-face-masks.
  124. “The Forgotten Science Behind Face Masks.” Accessed September 10, 2020. https://www.webmd.com/lung/news/20200826/the-forgotten-science-behind-face-masks.
  125. MSN. “This Face Mask Claim Is Going Viral—and It’s 100 Percent Wrong.” Accessed September 10, 2020. https://www.msn.com/en-us/Health/medical/this-face-mask-claim-is-going-viral-and-it-s-100-percent-wrong/ar-BB16tUzs.
  126. MSN. “This Is How Many People Need to Wear Masks to Stop the Coronavirus.” Accessed September 10, 2020. https://www.msn.com/en-my/health/wellness/this-is-how-many-people-need-to-wear-masks-to-stop-the-coronavirus/ar-BB13ZGBA.
  127. Tufekci, Zeynep. “How Hong Kong Did It.” The Atlantic, May 12, 2020. https://www.theatlantic.com/technology/archive/2020/05/how-hong-kong-beating-coronavirus/611524/.
  128. ———. “Keep the Parks Open.” The Atlantic, April 7, 2020. https://www.theatlantic.com/health/archive/2020/04/closing-parks-ineffective-pandemic-theater/609580/.
  129. Tunevall, Th. Göran. “Postoperative Wound Infections and Surgical Face Masks: A Controlled Study.” World Journal of Surgery 15, no. 3 (May 1, 1991): 383–87. https://doi.org/10.1007/BF01658736.
  130. “Tyson Foods, Inc. Releases Covid-19 Test Results At Northwest Arkansas Facilities.” Accessed September 9, 2020. https://www.tysonfoods.com/news/news-releases/2020/6/tyson-foods-inc-releases-covid-19-test-results-northwest-arkansas.
  131. Video Clip: Nassim Nicholas Taleb on Wearing Masks, 2020. https://www.youtube.com/watch?v=fJrjQ30R5Ag.
  132. Viola, I. M., B. Peterson, G. Pisetta, G. Pavar, H. Akhtar, F. Menoloascina, E. Mangano, et al. “Face Coverings, Aerosol Dispersion and Mitigation of Virus Transmission Risk,” May 19, 2020. https://arxiv.org/abs/2005.10720v1.
  133. Wang, Xiaowen, Enrico G. Ferro, Guohai Zhou, Dean Hashimoto, and Deepak L. Bhatt. “Association Between Universal Masking in a Health Care System and SARS-CoV-2 Positivity Among Health Care Workers.” JAMA 324, no. 7 (August 18, 2020): 703–4. https://doi.org/10.1001/jama.2020.12897.
  134. Wang, Yu, Huaiyu Tian, Li Zhang, Man Zhang, Dandan Guo, Wenting Wu, Xingxing Zhang, et al. “Reduction of Secondary Transmission of SARS-CoV-2 in Households by Face Mask Use, Disinfection and Social Distancing: A Cohort Study in Beijing, China.” BMJ Global Health 5, no. 5 (May 28, 2020). https://doi.org/10.1136/bmjgh-2020-002794.
  135. Ward, Alex. “How Masks Helped Hong Kong Control the Coronavirus.” Vox, May 18, 2020. https://www.vox.com/2020/5/18/21262273/coronavirus-hong-kong-masks-deaths-new-york.
  136. Ward, John. “Stop Forcing People to Wear Useless Masks.” The Slog (blog), July 24, 2020. https://therealslog.com/2020/07/24/stop-forcing-people-to-wear-useless-masks/.
  137. Fortune. “‘We See No Point in Wearing a Face Mask,’ Sweden’s Top Virus Expert Says as He Touts the Country’s Improving COVID Numbers.” Accessed September 9, 2020. https://fortune.com/2020/07/29/no-point-in-wearing-mask-sweden-covid/.
  138. “Wearing a Mask: Myths and Facts.” Accessed September 10, 2020. https://www.umms.org:443/coronavirus/what-to-know/masks/wearing-mask.
  139. “Welcome to CDC Stacks |.” Accessed September 10, 2020. https://stacks.cdc.gov/view/cdc/90553.
  140. ResearchGate. “What’s Your Point towards Facemask Usage under COVID-19?” Accessed September 9, 2020. https://www.researchgate.net/post/Whats_your_point_towards_facemask_usage_under_COVID-19.
  141. Wong, Sunny H., Jeremy Y. C. Teoh, Chi-Ho Leung, William K. K. Wu, Benjamin H. K. Yip, Martin C. S. Wong, and David S. C. Hui. “COVID-19 and Public Interest in Face Mask Use.” American Journal of Respiratory and Critical Care Medicine 202, no. 3 (June 15, 2020): 453–55. https://doi.org/10.1164/rccm.202004-1188LE.
  142. Wu, Katherine J. “Masks May Reduce Viral Dose, Some Experts Say.” The New York Times, July 27, 2020, sec. Health. https://www.nytimes.com/2020/07/27/health/coronavirus-mask-protection.html.
  143. Xiao, Jingyi, Eunice Y. C. Shiu, Huizhi Gao, Jessica Y. Wong, Min W. Fong, Sukhyun Ryu, and Benjamin J. Cowling. “Nonpharmaceutical Measures for Pandemic Influenza in Nonhealthcare Settings—Personal Protective and Environmental Measures – Volume 26, Number 5—May 2020 – Emerging Infectious Diseases Journal – CDC.” Accessed September 8, 2020. https://doi.org/10.3201/eid2605.190994.
  144. Psychology Today. “Yes, Masks Work: Debunking the Pseudoscience.” Accessed September 10, 2020. https://www.psychologytoday.com/blog/logical-take/202007/yes-masks-work-debunking-the-pseudoscience.
  145. Yong, Ed. “Everyone Thinks They’re Right About Masks.” The Atlantic, April 1, 2020. https://www.theatlantic.com/health/archive/2020/04/coronavirus-pandemic-airborne-go-outside-masks/609235/.
  146. UC Davis. “Your Mask Cuts Own Risk by 65 Percent,” July 6, 2020. https://www.ucdavis.edu/coronavirus/news/your-mask-cuts-own-risk-65-percent.
  147. z3508948. “Cloth Masks – Dangerous to Your Health?” Text. UNSW Newsroom, April 23, 2015. https://newsroom.unsw.edu.au/news/health/cloth-masks-%E2%80%93-dangerous-your-health.
  148. Zajenkowski, Marcin, Peter K. Jonason, Maria Leniarska, and Zuzanna Kozakiewicz. “Who Complies with the Restrictions to Reduce the Spread of COVID-19?: Personality and Perceptions of the COVID-19 Situation.” Personality and Individual Differences 166 (November 1, 2020): 110199. https://doi.org/10.1016/j.paid.2020.110199.
  149. “Zeynep Tufekci Advocates Mask Wearing in Open Letter to U.S. Governors and Op-Eds in USA Today and The Atlantic | Sils.Unc.Edu.” Accessed September 30, 2020. https://sils.unc.edu/news/2020/tufecki-masks.
  150. Zhou, S. Steve, Salimatu Lukula, Cory Chiossone, Raymond W. Nims, Donna B. Suchmann, and M. Khalid Ijaz. “Assessment of a Respiratory Face Mask for Capturing Air Pollutants and Pathogens Including Human Influenza and Rhinoviruses.” Journal of Thoracic Disease 10, no. 3 (March 2018): 2059–69. https://doi.org/10.21037/jtd.2018.03.103.
  151. Zwart, Mark P., Lia Hemerik, Jenny S. Cory, J. Arjan G.M. de Visser, Felix J.J.A. Bianchi, Monique M. Van Oers, Just M. Vlak, Rolf F. Hoekstra, and Wopke Van der Werf. “An Experimental Test of the Independent Action Hypothesis in Virus–Insect Pathosystems.” Proceedings of the Royal Society B: Biological Sciences 276, no. 1665 (June 22, 2009): 2233–42. https://doi.org/10.1098/rspb.2009.0064.