Preface

This is a write up on false positive, false negative, true positive, true negative thinking in testing for some condition, trying to detect if it is there or not. It is useful for discussing medical tests, but it applies to any sort of testing where decisions must be made.

I said I would write up testing, so I did. It always comes out seeming more complex than it seems at first glance, even though the calculations are simple arithmetic. I have another one in draft, hitting the material from a different angle, but unfortunately not making it simpler.

This fits within the framework of signal detection theory. When I studied signal detection, I had no idea why it was considered important, particularly in Psychology, but over the years, I have often gone back to it, since it comes up routinely in a number of guises. It has to be statistical signal detection, which would be an obvious extension to binary, deterministic signal detection. This also fits within the framework of fuzzy logic, which I have looked at in years past.

Medical testing makes use of the ideas on signal detection and upon reflection, is no different than the consideration of any other sort of evidence – it has to be obtained, assessed for quality, and interpreted to understand the implications.

The take away is that for low prevalence of a condition (e.g. infection), you can get a lot of false positives (false alarms), even with a fairly specific test.  For a high prevalence of a condition, you can get a lot of false negatives (misses). In addition, for a highly selective test, you get fewer false positives (false alarms), and for a highly sensitive test, you get fewer false negatives (misses).

In the example below, I show how false positives can be far greater than true positives, even for a fairly well performing test, when the prevalence is low:

Parameters	Complement of Parameters
Selectivity = .95	1 – selectivity = .05
Sensitivity = .99	1 – sensitivity = .01
Prevalence = .01	1 – prevalence = .99

We have this formula: Ratio of false positives (FP) to true positives (TP) = ( (1-selectivity) x (1-prevalence) ) / ((sensitivity) x (prevalence) ) .

False positives to true positives = ((.05) x (.99))/(.99) x (.01)) = 5

Using the formula, we get a 5 fold increase of false positives over true positives, for a test that has quite good performance. Below I will show how for any test with error rates, both low prevalence and high prevalence give more false results.

It is nice to now have those words, “specificity” (true negative) and “sensitivity” (true positive) for test performance which I lacked before. I give more explanation below.

Anybody who tells you that such and such a test is any given percentage effective is misleading you somewhat, unintentionally or perhaps even deliberately.

There are four quadrants in the “gold standard” detection matrix: true positive, false positive, true negative, and false negative. There are at least two percentages to be considered, and they vary independently according to the bias in your test, the detection threshold decided upon. You can bias so that you find all the cases, with the consequence that you get a lot of false positives (false alarms). You can bias so that you get few false positives, and as a result, get a lot of false negatives (miss the fire). In addition, the baseline must be considered, with respect to a Bayesian statistical analysis. So, the prior information on general infection rates will make the percentages change. Low base rate of infection gives a high number of false positives. High base rate of infection gives a low number of false positives.

Bayesian reasoning: start with prior probabilities (assessed somehow) and see how probabilities change with new evidence.

If you do not have much noise masking the signal, results are easier to interpret. If you have large numbers and effects that are strong with respect to variability, the statistics should bear you out. Potentially confounding factors can be accounted for.

See http://ephektikoi.ca/blog/2020/10/11/bibliography-on-testing-and-uncertainty/ for more information.

Introduction

It is routine for medical tests to be used to determine the health status of people. The simple view of a test is that it returns a true or false result, using some measure, some test instrument, and some testing protocol. Of course anyone reflecting on the issue even a little bit will realize that this is a much over-simplified view of things. For one thing, what threshold, what cut-off point, is being used to make the decision of true or false? What is the measure being used, and what is the measuring instrument? Most things we measure involve some sort of continuous scale. Are the measurements continuous or is it yes and no? What are the typical values that are used to make the judgement? What are the health implications? How are the numbers to be interpreted as a screening device or diagnostic tool? All of these considerations are important for understanding the test.

The Perfect Test

Here is a diagram of a test which does not consider that there might be some error.

	Does the Condition Exist?
Is the Effect Observed?	Observed	Condition exists
	Not observed	Condition is absent
True or false – assuming no errors

The Imperfect, Real-world Test

With a bit of thought, the question of errors for the test will come up. Is the test perfect? This would seem highly unlikely.

In testing, there are two ways for the results to be true, and two ways for the results to be false.

	Does The Condition Exist?
		Condition Exists	Condition Is Absent
Is The Effect Observed?	Observed	Condition Is Correctly Considered To Exist     (HIT) (TRUE POSITIVE)	Condition Is Falsely Considered To Exist     (FALSE ALARM) (FALSE POSITIVE)
	Not Observed	Condition Is Falsely Considered To Be Absent     (MISS) (FALSE NEGATIVE)	Condition Is Correctly Consider To Be Absent       (TRUE REJECTION) (TRUE NEGATIVE)
True Or False – Assuming Errors

 

1. Observing an effect when the effect exists is a Hit
2. Not observing an effect when the effect exists is a Miss
3. Observing an effect when no effect exists is a False Alarm
4. Observing an effect when no effect exist is a Correct Rejection

Synonyms for these terms are:

 

1. Hit – True Positive (TP), Sensitivity
2. False Alarm – False Positive (FP), Type I Error
3. Miss – False Negative (FN), Type II Error
4. Correct Rejection – True Negative (TN), Specificity

I will use the abbreviations TP, FP, FN, TN in most of the discussion, although the meanings are probably not as easily grasped.

Three factors for calculations:

1. Performance or discrimination ability of the test
2. Prevalence of the condition, present or absent to some percentage
3. Number of tests done independently

Performance of a test: selectivity and sensitivity

Sensitivity applies to a population that all truly have the condition. It is the ratio of hits to misses, usually as a percentage of hits to overall tests. Percentage is calculated (hits / (hits + misses) x 100). Sensitivity gives the percent hits. The one’s complement of sensitivity gives the percent misses. With poorer sensitivity tests, the number of misses gets larger.

Selectivity applies to a population were none have the condition. It is the ratio of correct rejections to false alarms, usually as a percentage of rejections to overall tests. Percentage is calculated as (rejections / (rejections + false alarms) x 100). Selectivity gives the percent correct rejections. The one’s complement of selectivity gives the false alarm percent. With poorer selectivity tests, the number of false alarms gets larger.

Prevalence gives the estimated number where the condition is present. The one’s complement of prevalence gives the estimated number where the condition is absent.

Lower prevalence gives more false positives or false alarms and also lower prevalence gives fewer false negatives or misses.

Higher prevalence gives more false negatives or misses and also higher prevalence gives fewer false positives or false alarms.

False negatives are a function of sensitivity and prevalence whereas false positives are a function of selectivity and prevalence.

The number of tests done independently gives the counts.

In calculating expected values for a condition and errors, use performance, prevalence and the number of tests

Hits	False alarms
Misses	Correct rejections

1. Hits = sensitivity x prevalence x number of tests
2. False alarms = (1- selectivity) x (1 – prevalence) x number of tests
3. Misses = (1 – sensitivity) x prevalence x number of tests
4. Correct rejections = selectivity x (1 – prevalence) x number of tests

Hits versus false alarms are a function of test performance and prevalence.

Considerations: It is a mistake to only use just performance and test count without taking prevalence into account – a big mistake

Given that you know the performance of a test, and the prevalence of the condition, you can calculate the ratios of:

1. False positives to true positives
2. False negatives to true negatives

Ratio of FP to TP ( (1-selectivity) x (1-prevalence) ) / ((sensitivity) x (prevalence) )

Ratio of FN to TN ( (1-sensitivity) x (prevalence) ) / ((selectivity) x (1 – prevalence) )

Example 1

Parameters	Complement
Selectivity = .95	1 – selectivity = .05
Sensitivity = .99	1 – sensitivity = .01
Prevalence = .01	1 – prevalence = .99

False positives to true positives = ((.05) x (.99))/(.99) x (.01)) = 5

Using the formula, we get a 5 fold increase of false positives over true positives

Example 2

Parameters	Complement
Selectivity = 1.00	1 – selectivity = 0.0
Sensitivity = 1.00	1 – sensitivity = 0.0
Prevalence = .50	1 – prevalence = .50

False positives to true positives = ((0) x (.5))/(1) x (.5)) = 0

Using the formula, we get a no increase in false positives over true positives; in fact, there are no false positives or false negatives.

Formulae Summarized

 Test Count = number of tests

 selectivity (fraction of correct rejections for those without condition)

complement of selectivity (fraction of false alarms for those without condition) = (1 – selectivity)

sensitivity (fraction of hits for those with condition)

complement of sensitivity (fraction of misses for those with condition) = (1 – sensitvity)

prevalence is actual incidence as percent of population with condition

complement of prevalance is percent of population without condition = (1 – prevalence)

Estimated True Positive (Hits, TP) = sensitivity x prevalence x case count

Estimated False Positive (False Alarms, FP) = (1 – selectivity) x (1 – prevalence) x (case count)

Estimated False Negatives (Misses, FN) = (1 – sensitivity) * (prevalence) x case count

Estimated True Negatives (Correct Rejections, TN) = (selectivity) x (1 – prevalence) x (case count)

Ratio of FP to TP ( (1-selectivity) x (1-prevalence) ) / ((sensitivity) x (prevalence) )

Ratio of FN to TN ( (1-sensitivity) x (prevalence) ) / ((selectivity) x (1 – prevalence) )

Worked Example For Different Test Parameters and Prevalence

The pattern for all is that the end regions of the curve get high inversely in proportion to the performance of the test. That is for low prevalence and high prevalence both, there are more errors. For low prevalence, the errors are false positives. For high prevalence, the errors are false negatives. The error rates for both types of errors are lower in the middle. High selectivity reduces false positive errors. High sensitivity reduces false negative errors. The ultimate high is the non-existent perfectly performancing test with selectivity of 100% and sensitivity of 100%.

Selectivity 100%, Sensitivity 100%
A Balanced Perfect Test						Counts
Count	Selectivity	Sensitivity	Prevalence	Ratio FP to TP	Ratio FN to TN	TP	FP	FN	TN
1000	1.00	1.00	0.00	0.00	0.00	0.00	0.00	0.00	1000.00
1000	1.00	1.00	0.02	0.00	0.00	20.00	0.00	0.00	980.00
1000	1.00	1.00	0.03	0.00	0.00	30.00	0.00	0.00	970.00
1000	1.00	1.00	0.04	0.00	0.00	40.00	0.00	0.00	960.00
1000	1.00	1.00	0.05	0.00	0.00	50.00	0.00	0.00	950.00
1000	1.00	1.00	0.10	0.00	0.00	100.00	0.00	0.00	900.00
1000	1.00	1.00	0.20	0.00	0.00	200.00	0.00	0.00	800.00
1000	1.00	1.00	0.30	0.00	0.00	300.00	0.00	0.00	700.00
1000	1.00	1.00	0.40	0.00	0.00	400.00	0.00	0.00	600.00
1000	1.00	1.00	0.50	0.00	0.00	500.00	0.00	0.00	500.00
1000	1.00	1.00	0.60	0.00	0.00	600.00	0.00	0.00	400.00
1000	1.00	1.00	0.70	0.00	0.00	700.00	0.00	0.00	300.00
1000	1.00	1.00	0.80	0.00	0.00	800.00	0.00	0.00	200.00
1000	1.00	1.00	0.90	0.00	0.00	900.00	0.00	0.00	100.00
1000	1.00	1.00	0.98	0.00	0.00	980.00	0.00	0.00	20.00
1000	1.00	1.00	1.00	0.00	0.00	1000.00	0.00	0.00	0.00

Selectivity 90%, Sensitivity 99%
An Unbalanced Test with Reasonable Performance						Counts
Count	Selectivity	Sensitivity	Prevalence	Ratio FP to TP	Ratio FN to TN	TP	FP	FN	TN
1000	0.90	0.99	0.01	10.00	0.00	9.90	99.00	0.10	891.00
1000	0.90	0.99	0.02	4.95	0.00	19.80	98.00	0.20	882.00
1000	0.90	0.99	0.03	3.27	0.00	29.70	97.00	0.30	873.00
1000	0.90	0.99	0.04	2.42	0.00	39.60	96.00	0.40	864.00
1000	0.90	0.99	0.05	1.92	0.00	49.50	95.00	0.50	855.00
1000	0.90	0.99	0.10	0.91	0.00	99.00	90.00	1.00	810.00
1000	0.90	0.99	0.20	0.40	0.00	198.00	80.00	2.00	720.00
1000	0.90	0.99	0.30	0.24	0.00	297.00	70.00	3.00	630.00
1000	0.90	0.99	0.40	0.15	0.01	396.00	60.00	4.00	540.00
1000	0.90	0.99	0.50	0.10	0.01	495.00	50.00	5.00	450.00
1000	0.90	0.99	0.60	0.07	0.02	594.00	40.00	6.00	360.00
1000	0.90	0.99	0.70	0.04	0.03	693.00	30.00	7.00	270.00
1000	0.90	0.99	0.80	0.03	0.04	792.00	20.00	8.00	180.00
1000	0.90	0.99	0.90	0.01	0.10	891.00	10.00	9.00	90.00
1000	0.90	0.99	0.98	0.00	0.54	970.20	2.00	9.80	18.00
1000	0.90	0.99	0.99	0.00	1.10	980.10	1.00	9.90	9.00

Selectivity 99%, Sensitivity 90%
An Unbalanced Test with Reasonable Performance						Counts
Count	Selectivity	Sensitivity	Prevalence	Ratio FP to TP	Ratio FN to TN	TP	FP	FN	TN
1000	0.99	0.90	0.01	1.10	0.00	9.00	9.90	1.00	980.10
1000	0.99	0.90	0.02	0.54	0.00	18.00	9.80	2.00	970.20
1000	0.99	0.90	0.03	0.36	0.00	27.00	9.70	3.00	960.30
1000	0.99	0.90	0.04	0.27	0.00	36.00	9.60	4.00	950.40
1000	0.99	0.90	0.05	0.21	0.01	45.00	9.50	5.00	940.50
1000	0.99	0.90	0.10	0.10	0.01	90.00	9.00	10.00	891.00
1000	0.99	0.90	0.20	0.04	0.03	180.00	8.00	20.00	792.00
1000	0.99	0.90	0.30	0.03	0.04	270.00	7.00	30.00	693.00
1000	0.99	0.90	0.40	0.02	0.07	360.00	6.00	40.00	594.00
1000	0.99	0.90	0.50	0.01	0.10	450.00	5.00	50.00	495.00
1000	0.99	0.90	0.60	0.01	0.15	540.00	4.00	60.00	396.00
1000	0.99	0.90	0.70	0.00	0.24	630.00	3.00	70.00	297.00
1000	0.99	0.90	0.80	0.00	0.40	720.00	2.00	80.00	198.00
1000	0.99	0.90	0.90	0.00	0.91	810.00	1.00	90.00	99.00
1000	0.99	0.90	0.98	0.00	4.95	882.00	0.20	98.00	19.80
1000	0.99	0.90	0.99	0.00	10.00	891.00	0.10	99.00	9.90

Selectivity 99%, Sensitivity 99%
A Balanced Test, with Excellent Performance						Counts
Count	Selectivity	Sensitivity	Prevalence	Ratio FP to TP	Ratio FN to TN	TP	FP	FN	TN
1000	0.99	0.99	0.01	1.00	0.00	9.90	9.90	0.10	980.10
1000	0.99	0.99	0.02	0.49	0.00	19.80	9.80	0.20	970.20
1000	0.99	0.99	0.03	0.33	0.00	29.70	9.70	0.30	960.30
1000	0.99	0.99	0.04	0.24	0.00	39.60	9.60	0.40	950.40
1000	0.99	0.99	0.05	0.19	0.00	49.50	9.50	0.50	940.50
1000	0.99	0.99	0.10	0.09	0.00	99.00	9.00	1.00	891.00
1000	0.99	0.99	0.20	0.04	0.00	198.00	8.00	2.00	792.00
1000	0.99	0.99	0.30	0.02	0.00	297.00	7.00	3.00	693.00
1000	0.99	0.99	0.40	0.02	0.01	396.00	6.00	4.00	594.00
1000	0.99	0.99	0.50	0.01	0.01	495.00	5.00	5.00	495.00
1000	0.99	0.99	0.60	0.01	0.02	594.00	4.00	6.00	396.00
1000	0.99	0.99	0.70	0.00	0.02	693.00	3.00	7.00	297.00
1000	0.99	0.99	0.80	0.00	0.04	792.00	2.00	8.00	198.00
1000	0.99	0.99	0.90	0.00	0.09	891.00	1.00	9.00	99.00
1000	0.99	0.99	0.98	0.00	0.49	970.20	0.20	9.80	19.80
1000	0.99	0.99	0.99	0.00	1.00	980.10	0.10	9.90	9.90

Selectivity 95%, Sensitivity 95%
A Balanced Test, with Decent Performance						Counts
Count	Selectivity	Sensitivity	Prevalence	Ratio FP to TP	Ratio FN to TN	TP	FP	FN	TN
1000	0.95	0.95	0.01	5.21	0.00	9.50	49.50	0.50	940.50
1000	0.95	0.95	0.02	2.58	0.00	19.00	49.00	1.00	931.00
1000	0.95	0.95	0.03	1.70	0.00	28.50	48.50	1.50	921.50
1000	0.95	0.95	0.04	1.26	0.00	38.00	48.00	2.00	912.00
1000	0.95	0.95	0.05	1.00	0.00	47.50	47.50	2.50	902.50
1000	0.95	0.95	0.10	0.47	0.01	95.00	45.00	5.00	855.00
1000	0.95	0.95	0.20	0.21	0.01	190.00	40.00	10.00	760.00
1000	0.95	0.95	0.30	0.12	0.02	285.00	35.00	15.00	665.00
1000	0.95	0.95	0.40	0.08	0.04	380.00	30.00	20.00	570.00
1000	0.95	0.95	0.50	0.05	0.05	475.00	25.00	25.00	475.00
1000	0.95	0.95	0.60	0.04	0.08	570.00	20.00	30.00	380.00
1000	0.95	0.95	0.70	0.02	0.12	665.00	15.00	35.00	285.00
1000	0.95	0.95	0.80	0.01	0.21	760.00	10.00	40.00	190.00
1000	0.95	0.95	0.90	0.01	0.47	855.00	5.00	45.00	95.00
1000	0.95	0.95	0.98	0.00	2.58	931.00	1.00	49.00	19.00
1000	0.95	0.95	0.99	0.00	5.21	940.50	0.50	49.50	9.50

Selectivity 90%, Sensitivity 90%
A Balanced Test, with OK Performance						Counts
Count	Selectivity	Sensitivity	Prevalence	Ratio FP to TP	Ratio FN to TN	TP	FP	FN	TN
1000	0.90	0.90	0.01	11.00	0.00	9.00	99.00	1.00	891.00
1000	0.90	0.90	0.02	5.44	0.00	18.00	98.00	2.00	882.00
1000	0.90	0.90	0.03	3.59	0.00	27.00	97.00	3.00	873.00
1000	0.90	0.90	0.04	2.67	0.00	36.00	96.00	4.00	864.00
1000	0.90	0.90	0.05	2.11	0.01	45.00	95.00	5.00	855.00
1000	0.90	0.90	0.10	1.00	0.01	90.00	90.00	10.00	810.00
1000	0.90	0.90	0.20	0.44	0.03	180.00	80.00	20.00	720.00
1000	0.90	0.90	0.30	0.26	0.05	270.00	70.00	30.00	630.00
1000	0.90	0.90	0.40	0.17	0.07	360.00	60.00	40.00	540.00
1000	0.90	0.90	0.50	0.11	0.11	450.00	50.00	50.00	450.00
1000	0.90	0.90	0.60	0.07	0.17	540.00	40.00	60.00	360.00
1000	0.90	0.90	0.70	0.05	0.26	630.00	30.00	70.00	270.00
1000	0.90	0.90	0.80	0.03	0.44	720.00	20.00	80.00	180.00
1000	0.90	0.90	0.90	0.01	1.00	810.00	10.00	90.00	90.00
1000	0.90	0.90	0.98	0.00	5.44	882.00	2.00	98.00	18.00
1000	0.90	0.90	0.99	0.00	11.00	891.00	1.00	99.00	9.00

Selectivity 50%, Sensitivity 50%
A Balanced Test, with Poor Performance						Counts
Count	Selectivity	Sensitivity	Prevalence	Ratio FP to TP	Ratio FN to TN	TP	FP	FN	TN
1000	0.50	0.50	0.01	99.00	0.01	5.00	495.00	5.00	495.00
1000	0.50	0.50	0.02	49.00	0.02	10.00	490.00	10.00	490.00
1000	0.50	0.50	0.03	32.33	0.03	15.00	485.00	15.00	485.00
1000	0.50	0.50	0.04	24.00	0.04	20.00	480.00	20.00	480.00
1000	0.50	0.50	0.05	19.00	0.05	25.00	475.00	25.00	475.00
1000	0.50	0.50	0.10	9.00	0.11	50.00	450.00	50.00	450.00
1000	0.50	0.50	0.20	4.00	0.25	100.00	400.00	100.00	400.00
1000	0.50	0.50	0.30	2.33	0.43	150.00	350.00	150.00	350.00
1000	0.50	0.50	0.40	1.50	0.67	200.00	300.00	200.00	300.00
1000	0.50	0.50	0.50	1.00	1.00	250.00	250.00	250.00	250.00
1000	0.50	0.50	0.60	0.67	1.50	300.00	200.00	300.00	200.00
1000	0.50	0.50	0.70	0.43	2.33	350.00	150.00	350.00	150.00
1000	0.50	0.50	0.80	0.25	4.00	400.00	100.00	400.00	100.00
1000	0.50	0.50	0.90	0.11	9.00	450.00	50.00	450.00	50.00
1000	0.50	0.50	0.98	0.02	49.00	490.00	10.00	490.00	10.00
1000	0.50	0.50	0.99	0.01	99.00	495.00	5.00	495.00	5.00