Editorial in Ganga Ram Journal 2013; 3:66-67
Dicey clinical decisions and biostatistics
A. INDRAYAN
Former Professor and Head,
Department of Biostatistics and Medical Informatics
University College of Medical Sciences, Delhi, India
e-mail: [email protected]
Flipping a coin and rolling a die are statistical favourites to generate uncertainty. All dicey events are inherently uncertain. But can some clinical decisions be just as dicey? On the face of it, it seems oxymoronic as the term ‘clinical’ also means ‘efficient and unemotional’.1 If that be so, clinical decisions should not have much tolerance for uncertainties. In practice, clinical decisions and uncertainties go together.
Admittedly, ‘dicey’ is not the word of choice in the medical research literature. There are some examples, though. The path to lyme disease diagnosis is dicey;2 the role of Dicer expressions in RNAi therapy in cancer is dicey;3 nasal valve compromise can be dicey;4 and neonatal withdrawal syndrome is clinically a dicey call.5 That is just about all that I could locate in PubMed. It seems that most clinical researchers realize that clinical decisions are not so dicey after all, though they are afflicted with uncertainties. Outside medicine, George Cockcroft’s work on making life decisions by rolling a die6 is a popular intellectual gossip.
Medical implications of dicey clinical decisions may be obscure, but consider two patients with the same APACHE-II score, say 18—one survives and the other does not. Certainly the survival is not determined by flipping a coin as survival–death probabilities are not equal at APACHE-II=18, neither it is determined by rolling a die with face probabilities equal to P(survival/APACHE-II=18). The chance of death among operated cases of acute necrotizing pancreatitis is 1/6 and is the same as any particular number from 1–6 showing up on rolling a die. Fortunately, death is not as dicey. It surely depends on host factors to begin with and on the care one receives at the end of it all. Ironically, notwithstanding the respect it commands, APACHE-II is used to predict survival of a critical patient on the basis of physiological measurements at admission,7 as though hospital facilities and critical care are irrelevant! Yet, it has great value in comparing the severity of cases in two or more groups or over time. APACHE-II is actually meant to compare the efficacy of two or more therapies for patients with the same degree of severity but it has come to be widely used to prognosticate mortality. Incidentally, this score is a byproduct of a popular biostatistical method—logistic regression. The associated uncertainties just illustrated underscore the need to be extra careful with such a method.
Statistical methods may look innocuous but they are dangerous tools in the hands of an inexpert.8 In fact, experts too falter when the underlying requirements9 are overlooked and residual uncertainties are discounted. Biostatistical methods claim to be equipped to handle chance as this science uses sampling errors and randomness as its backbone. Many do not realize that this constitutes only one component of uncertainty syndrome. While the aleatory part can be reasonably quantified and handled through statistical methods, my worry has been epistemic uncertainties.8 Belonging to some unknown domain, epistemic uncertainty lends us into a real dicey zone so that some decisions remain no better than rolling a die. The only difficulty is that many of us do not realize that this is so.
Biostatisticians are notorious for extracting information from even random numbers. Random numbers also follow a pattern, called uniform distribution—thus amenable to all kinds of statistical manipulations and inferences. Howsoever random, measurement errors, we all know, mostly follow a Gaussian distribution, and are capable of giving us valid conclusions. Random after all are not as random as they seem. On the other hand, numbers can always be assigned, and justified, to any graded phenomenon, and they may not contain any useful information as stand-alone. For example, risk is a perceptionally useful quantity yet may not provide exciting news. If a procedure has a success rate of 28%, is it good or bad? Isolation can be self-defeating. If you know that an alternative procedure has a success rate of 35%, an inferential value is suddenly attached to 28% success in the first method.10
Coming back to popular scoring systems such as APACHE and Glasgow, how much do they help in removing uncertainty? For the type of critical cases covered by these scores, the chance of survival is just about 80%–85%. Based on this information, if I blindly predict as a non-expert and without the help of any scoring system that a critical case has an 80% chance of survival, I would not be far wrong even in the long run where such probabilities are supposed to work. In that sense, the acclaimed APACHE, for that matter any such scoring system, adds just about 5% to the accuracy of prediction. Imagine the cost in terms of obtaining accurate measurements and feeding them into a scoring system for this paltry gain. This is probably true for many medical tests. The clinical picture may already make you 70% confident of the diagnosis. Tests add just about 5%–10%, but they are still preferred as they are objective and free of bias—and also help to shield one from legal hassles. Perhaps the value of most biostatistical methods also lies in this protection. They assign scientific probabilities to dicey clinical decisions. But statistical decisions can also be falsely positive or falsely negative just as are medical tests. A type I error rate of 5% is enough for ducking when a decision turns out wrong. If a statistical decision does not appeal to your conscience, re-examine it and the chances are that you will find your conscience was right.8
References
1. Oxford Dictionaries. Available at http://oxforddictionaries.
com/definition/american_english/clinical (accessed on 27 March
2013).
2. Pfeiffer MB. Exclusive: Dicey path to lyme disease
diagnosis. Poughkeepsie Journal 2012. Available at
http://www.poughkeepsiejournal.com/article/20121022/
NEWS06/112170001/EXCLUSIVE-DICEY-PATH-LYMEDISEASE-
DIAGNOSIS (accessed on 28 March 2013)
3. Merritt WM, Bar-Eli M, Sood AK. The dicey role of Dicer:
Implications for RNAi therapy. Cancer Res 2010: 70:
2571–2574. Available at http://cancerres.aacrjournals.org/
content/70/7/2571.long (accessed on 15 March 2013).
4. Collins TR. A clinical challenge: Nasal valve compromise can
be a dicey problem, panelists say. ENT Today 2011. Available
at http://www.enttoday.org/details/article/1006491/A_
Clinical_Challenge_Nasal_valve_compromise_can_be_a_dicey_
problem_panelists_say.html (accessed on 28 March 2013).
5. Cohen LS. Neonatal withdrawal syndrome and SSRIs. Healthy
Place – Mental illness overview 2009. Available at http://www.
healthyplace.com/other-info/mental-illness-overview/neonatalwithdrawal-
syndrome-and-ssris/ (accesed on 29 March 2013).
6. Coverdale K. Dicey decisions. Available at http://www.vice.com/
read/dicey-decisions-0000191-v19n6 (accessed on 29 March
2013).
7. Knaus WA, Draper EA, Wagner DP, Zimmerman JE. APACHE
II: A severity of disease classification system. Crit Care Med
1985; 13: 818–29. Available at http://xa.yimg.com/kq/
groups/16749867/1433507048/name/Critical+Care+Medicin
e+1985+Knaus.pdf (accessed on 15 march 2013).
8. Indrayan A. Medical biostatistics. 3rd ed. New York: CRC
Press; 2012. Available at http://www.crcpress.com/product/
isbn/9781439884140. (accessed on 29 March 2013)
9. Kumar R, Indrayan A, Chhabra P. Reporting quality of
multivariable logistic regression in selected Indian medical
journals. J Postgrad Med 2012; 58: 123–126. Available at
http://www.jpgmonline.com/article.asp?issn=0022-3859;year
=2012;volume=58;issue=2;spage=123;epage=126;aulast=Kum
ar (accessed on 29 March 2013).
10. Ubel PA. Beyond costs and benefits: Understanding how patients
make health care decisions. The Oncologist 2010; 15 (Suppl
1):5–10. Available at http://theoncologist.alphamedpress.org/
content/15/suppl_1/5.full (accessed on 29 March 2013).
Admittedly, ‘dicey’ is not the word of choice in the medical research literature. There are some examples, though. The path to lyme disease diagnosis is dicey;2 the role of Dicer expressions in RNAi therapy in cancer is dicey;3 nasal valve compromise can be dicey;4 and neonatal withdrawal syndrome is clinically a dicey call.5 That is just about all that I could locate in PubMed. It seems that most clinical researchers realize that clinical decisions are not so dicey after all, though they are afflicted with uncertainties. Outside medicine, George Cockcroft’s work on making life decisions by rolling a die6 is a popular intellectual gossip.
Medical implications of dicey clinical decisions may be obscure, but consider two patients with the same APACHE-II score, say 18—one survives and the other does not. Certainly the survival is not determined by flipping a coin as survival–death probabilities are not equal at APACHE-II=18, neither it is determined by rolling a die with face probabilities equal to P(survival/APACHE-II=18). The chance of death among operated cases of acute necrotizing pancreatitis is 1/6 and is the same as any particular number from 1–6 showing up on rolling a die. Fortunately, death is not as dicey. It surely depends on host factors to begin with and on the care one receives at the end of it all. Ironically, notwithstanding the respect it commands, APACHE-II is used to predict survival of a critical patient on the basis of physiological measurements at admission,7 as though hospital facilities and critical care are irrelevant! Yet, it has great value in comparing the severity of cases in two or more groups or over time. APACHE-II is actually meant to compare the efficacy of two or more therapies for patients with the same degree of severity but it has come to be widely used to prognosticate mortality. Incidentally, this score is a byproduct of a popular biostatistical method—logistic regression. The associated uncertainties just illustrated underscore the need to be extra careful with such a method.
Statistical methods may look innocuous but they are dangerous tools in the hands of an inexpert.8 In fact, experts too falter when the underlying requirements9 are overlooked and residual uncertainties are discounted. Biostatistical methods claim to be equipped to handle chance as this science uses sampling errors and randomness as its backbone. Many do not realize that this constitutes only one component of uncertainty syndrome. While the aleatory part can be reasonably quantified and handled through statistical methods, my worry has been epistemic uncertainties.8 Belonging to some unknown domain, epistemic uncertainty lends us into a real dicey zone so that some decisions remain no better than rolling a die. The only difficulty is that many of us do not realize that this is so.
Biostatisticians are notorious for extracting information from even random numbers. Random numbers also follow a pattern, called uniform distribution—thus amenable to all kinds of statistical manipulations and inferences. Howsoever random, measurement errors, we all know, mostly follow a Gaussian distribution, and are capable of giving us valid conclusions. Random after all are not as random as they seem. On the other hand, numbers can always be assigned, and justified, to any graded phenomenon, and they may not contain any useful information as stand-alone. For example, risk is a perceptionally useful quantity yet may not provide exciting news. If a procedure has a success rate of 28%, is it good or bad? Isolation can be self-defeating. If you know that an alternative procedure has a success rate of 35%, an inferential value is suddenly attached to 28% success in the first method.10
Coming back to popular scoring systems such as APACHE and Glasgow, how much do they help in removing uncertainty? For the type of critical cases covered by these scores, the chance of survival is just about 80%–85%. Based on this information, if I blindly predict as a non-expert and without the help of any scoring system that a critical case has an 80% chance of survival, I would not be far wrong even in the long run where such probabilities are supposed to work. In that sense, the acclaimed APACHE, for that matter any such scoring system, adds just about 5% to the accuracy of prediction. Imagine the cost in terms of obtaining accurate measurements and feeding them into a scoring system for this paltry gain. This is probably true for many medical tests. The clinical picture may already make you 70% confident of the diagnosis. Tests add just about 5%–10%, but they are still preferred as they are objective and free of bias—and also help to shield one from legal hassles. Perhaps the value of most biostatistical methods also lies in this protection. They assign scientific probabilities to dicey clinical decisions. But statistical decisions can also be falsely positive or falsely negative just as are medical tests. A type I error rate of 5% is enough for ducking when a decision turns out wrong. If a statistical decision does not appeal to your conscience, re-examine it and the chances are that you will find your conscience was right.8
References
1. Oxford Dictionaries. Available at http://oxforddictionaries.
com/definition/american_english/clinical (accessed on 27 March
2013).
2. Pfeiffer MB. Exclusive: Dicey path to lyme disease
diagnosis. Poughkeepsie Journal 2012. Available at
http://www.poughkeepsiejournal.com/article/20121022/
NEWS06/112170001/EXCLUSIVE-DICEY-PATH-LYMEDISEASE-
DIAGNOSIS (accessed on 28 March 2013)
3. Merritt WM, Bar-Eli M, Sood AK. The dicey role of Dicer:
Implications for RNAi therapy. Cancer Res 2010: 70:
2571–2574. Available at http://cancerres.aacrjournals.org/
content/70/7/2571.long (accessed on 15 March 2013).
4. Collins TR. A clinical challenge: Nasal valve compromise can
be a dicey problem, panelists say. ENT Today 2011. Available
at http://www.enttoday.org/details/article/1006491/A_
Clinical_Challenge_Nasal_valve_compromise_can_be_a_dicey_
problem_panelists_say.html (accessed on 28 March 2013).
5. Cohen LS. Neonatal withdrawal syndrome and SSRIs. Healthy
Place – Mental illness overview 2009. Available at http://www.
healthyplace.com/other-info/mental-illness-overview/neonatalwithdrawal-
syndrome-and-ssris/ (accesed on 29 March 2013).
6. Coverdale K. Dicey decisions. Available at http://www.vice.com/
read/dicey-decisions-0000191-v19n6 (accessed on 29 March
2013).
7. Knaus WA, Draper EA, Wagner DP, Zimmerman JE. APACHE
II: A severity of disease classification system. Crit Care Med
1985; 13: 818–29. Available at http://xa.yimg.com/kq/
groups/16749867/1433507048/name/Critical+Care+Medicin
e+1985+Knaus.pdf (accessed on 15 march 2013).
8. Indrayan A. Medical biostatistics. 3rd ed. New York: CRC
Press; 2012. Available at http://www.crcpress.com/product/
isbn/9781439884140. (accessed on 29 March 2013)
9. Kumar R, Indrayan A, Chhabra P. Reporting quality of
multivariable logistic regression in selected Indian medical
journals. J Postgrad Med 2012; 58: 123–126. Available at
http://www.jpgmonline.com/article.asp?issn=0022-3859;year
=2012;volume=58;issue=2;spage=123;epage=126;aulast=Kum
ar (accessed on 29 March 2013).
10. Ubel PA. Beyond costs and benefits: Understanding how patients
make health care decisions. The Oncologist 2010; 15 (Suppl
1):5–10. Available at http://theoncologist.alphamedpress.org/
content/15/suppl_1/5.full (accessed on 29 March 2013).