Overconfidence, Time-on-Task, and Medical Errors: Is There a Relationship?

Background

In the past decades, medical errors have become a center of discussion as all health care systems attempted to achieve a high quality of care. This has been supported by a well-established system that has fostered the incident investigations of every adverse event and the formulation of policies to plug the staff or operation failures.¹ The goal for quality improvement is building a safer health care environment in which attention is given to provide proper care to patients and families affected by medical errors.^2,3

How is the health of the patients affected by medical errors? The Institute of Medicine (IOM) answered this question in 1999 and estimated that the annual incidence of deaths from medical errors in the United States was between 44,000 and 98,000 from 2403351 deaths.⁴ There was strong controversy whether these numbers underestimated or exaggerated the problem. However, this report highlighted the importance and consequences of medical errors. As a result, the US government supported the IOM recommendations as this problem was considered a challenge to the US health care systems. Many follow-up studies have been conducted periodically to observe and assess the improvement and progression of patient safety.^5,6 Despite measures taken, the incidence of medical errors has been increasing. For instance, in 2013, the estimated incidence jumped to 210,000–400,000 deaths a year associated with medical errors among hospitalized patients in the US,⁷ making medical errors the third leading cause of death in the US after heart disease (635,000/year) and cancer (598,000/year). Beyond the loss of lives and the catastrophic effects of patients’ morbidity, these medical errors add a great burden on the economy of approximately $735 to $980 billion, according to the IOM.⁸

To comprehend the impact of medical errors in patient care, the whole process of clinical reasoning and problem solving is better understood.⁹ For instance, one important area where errors have occurred frequently is the area of “diagnostic uncertainty”. Physicians’ work is partly based on personal interpretations and estimating disease probabilities.¹⁰ In studies conducted in the US and the UK, the incidence of diagnostic failure was highest in family medicine (FM), emergency medicine (EM), and internal medicine (IM), disciplines in which diagnosis is indeed subject to higher degrees of diagnostic uncertainty.¹¹

A dual-systems theory was developed by Kahneman and Tversky to explain the decision-making process under uncertainty.^12–15 System 1 is a fast and automated approach. It is unconscious and effortless. This is done through “pattern recognition”, in which quick diagnosis is possible because a physician has encountered similar cases in the past. On the other hand, System 2 is a non-programmed, conscious approach. In contrast to System 1, it is slow, analytical, and effortful. This system is activated when pattern recognition fails. Accordingly, an appropriate amount of data is collected, and a systematic case analysis is performed. Kahneman¹² strongly suggests that System 1 thinking is a source of error, particularly through falling prey to cognitive biases. Over forty cognitive biases have been described in clinical medicine.^12.It is estimated that 75% of medical errors in internal medicine practice are attributed to cognitive biases.¹⁶

Kahneman or Croskerry reported that error is often the result of neglect, time pressure, or lack of appropriate knowledge,¹⁷ it is certainly likely that cognitive biases play an important part in the emergence of errors. Among the most clearly recognized biases is overconfidence.¹⁸ It is considered a major factor that leads to diagnostic error.¹⁹ Premature diagnostic closure is defined as “narrowing the choice of diagnostic hypothesis too early”.¹⁸ As a result, the most likely diagnosis is never considered.^20,21 Premature closure occurs when a physician either gets attached to data collected early from miscellaneous sources of information or does not gather enough data to reach the correct diagnosis.¹¹ As a result, the proposed diagnosis is not verified. Overconfidence is a form of excess and unfounded confidence in one’s abilities. It is a risky personal trait in medicine, especially in fields where patient life is in a critical situation, and prognosis depends on the right medical decisions such as emergency medicine. It occurs when physicians think they perform better than they do. Without relying on enough patient data, the physician’s opinion becomes the base for the diagnosis. Generally, overconfident people have a strong belief that their contribution usually has a positive outcome (regardless of whether it does). Rather than relying on reasonable and relevant information, overconfident people trust their hunches.²²

Although studies suggest that physicians’ high confidence can lead to medical errors through premature closure,^23,24 experimental research is lacking to support this claim. Therefore, in the experiment described below we influenced the amount of confidence in their decisions that physicians have and studied its effect on diagnostic accuracy and time on task in a subsequent diagnostic assignment. We expected physicians who displayed a higher level of confidence to spend less time on the task and be less diagnostically accurate.

Methods

Overview

For the purpose of investigating the effects of high confidence on diagnostic accuracy, the current study was designed as a randomized controlled experiment conducted over a four-week period during October–November 2019. The independent variable was high confidence, and the dependent variables were response time and diagnostic accuracy.

Participants

Forty third-year internal medicine residents participated in this study. All internal medicine residents at different main hospitals in the Riyadh region were invited to participate in this study. Residents were informed that their participation is not mandatory and that their refusal to participate will not affect their learning experience in any way. Those who accepted to be part of the study were randomly assigned to either the experimental (high confidence) group or the control (low-confidence) group. The participants included 27 men and 13 women. Their mean (SD) age was 31.8 (4.1) years, and their mean clinical experience was 7.9 (3.9) years. There were 20 participants in each group. After randomization, no significant difference was found in terms of sex, age, or years of training. Participation was voluntary, and informed consent was obtained from residents who agreed to participate. The specific purpose of the study was not announced to the residents because this would probably affect the validity of the results. After the collection of data, participants were informed about the main aim of the study.

The internal medicine residency program in Saudi Arabia is a four-year program that covers general internal medicine and subspecialties. It has two levels: junior level (first and second year) and senior-level (third and fourth year) residents. Choosing internal medicine residents for the current study is justified given their exposure to diverse and complicated chronic cases. These residents often deal with overlapping chronic medical conditions; thus, they are more subject to the demands of achieving a high level of diagnostic accuracy. Moreover, they are not consistently required to function under the immediate pressure of handling emergencies, a factor that can significantly influence one’s confidence and diagnostic accuracy. This study is particularly pertinent as it provides insights into how confidence levels may impact their diagnostic decision-making in less urgent but equally critical situations.

Materials

In total, eight written clinical vignettes were used for this study with the following diagnoses: stomach cancer, vitamin B12 deficiency, pulmonary thromboembolism, celiac disease, acute viral pericarditis, acute myeloid leukemia, acute bacterial endocarditis, and sarcoidosis. The cases were written by experts in internal medicine. Each case was presented in English and included a patient’s medical history, relevant physical examination findings, and laboratory test results. See Box 1 for an example case.

Box 1 Example of a Medical Case

Procedure

Both groups were informed that the main purpose of the study was to identify factors affecting problem solving. A booklet included cases and short-answer questions given to the two groups. The high-confidence group was informed that the cases in the booklet were given to fourth-year medical students to assess the difficulty of the cases, and we found that the students correctly solved 6 to 7 out of 8 cases. This was done to ensure that the residents feel confident solving the cases. The low-confidence group was informed that those cases were alternatively given to two professors of internal medicine with more than 15 years of experience, and the participants got only 3 out of the 8 cases correct. It was done to ensure that the low-confidence group would experience a lower level of confidence. Inducing low confidence in this group does not imply that we expect the group to be less confident or high confident; rather, it is a planned experiment designed to investigate the consequences of low and high confidence on diagnosis accuracy. After the information of the purported difficulty level of the cases was given, the participants were asked to indicate how many of the cases they expected to diagnose correctly.

Subsequently, the participants were given an example case to practice before solving the eight actual cases. Then, the participants typed in their diagnosis for each individual case. Response time was recorded in minutes and seconds for each case. After completing the task, the candidates were asked four short-answer questions: 1. How suitable do you think these cases are for helping medical students to learn to diagnose cases? 2. How many cases out of 8 you think you have diagnosed correctly? 3. How complex were the cases on average in your opinion? 4. How often have you seen in practice on average cases such as the ones presented? The questions were answered using a scale from 0 to 10, with exception of the question referring to the number of cases diagnosed correctly.

Analysis

Cases were scored for diagnostic accuracy using the following rule: 0 = incorrect answer, 1 = partially correct, and 2 = correct. A diagnosis was considered correct if it included the main component of the true diagnosis (eg, “bacterial endocarditis” in the case of acute bacterial endocarditis). A diagnosis was considered partially correct when part of the diagnosis was mentioned, but the main correct diagnosis was missing (eg, leukemia in the case of acute myeloid leukemia). A diagnosis was considered incorrect when it did not match the main diagnosis or even a part of it (eg, “myocardial infarction” in the case of acute viral pericarditis).

For each participant, the mean score of confidence level, diagnostic accuracy, and mean response time was calculated for all the eight cases. The group’s mean (average) confidence level was obtained by adding the sum of participants’ ratings on different scenarios and dividing it by the total number of participants. The diagnostic accuracy was estimated as a % by calculating the correct diagnoses and dividing them by the total number of diagnoses done. This percentage shows the group’s mean diagnostic accuracy. We measured the amount of time each participant took to react to each diagnosis. The total of all individual reply times was then divided by the total number of participants to obtain the group’s average or mean response time. All differences between groups were assessed through t-tests. The p-value <0.05 was considered significant.

Results

There were a total of 40 participants in the study who were divided equally into the two groups of High confidence (n = 20) and Low confidence (n = 20) as the independent variables. The results of the independent samples t-test revealed that the high confidence group had a significantly higher confidence level compared with the control group: 0.75 compared to 0.61 with a p-value of 0.03. This implies that our experimental manipulation had succeeded. The dependent variables for the study included the Response Time and Diagnostic. See Table 1.

Table 1 Mean Differences with Standard Deviations (SD) Between the High-Confidence and Low-Confidence Groups on Confidence Level, Diagnostic Accuracy, and Response Time

Accuracy

Response Time

There was no significant difference in response time, although the high-confidence group needed on average 12% less time to arrive at a diagnosis.

Diagnostic Accuracy

The results showed that the diagnostic accuracy was almost similar in both groups. The mean score in diagnostic accuracy was equal to 0.65 indicating that the residents were able to respond correctly to 5 out of 8 cases. This shows that the accurate diagnosis can be established by initiative from the less confident group, whereas the overconfident group will end the case quickly, resulting in a diagnostic error. Interestingly, the confidence instilled by telling the participants that the cases to be diagnosed would be either rather easy or quite difficult seems to have spilled over to self-assessment of performance and the frequency by which the cases are said to be seen by both groups. Both groups misjudge the number of cases successfully diagnosed, but it is the high-confidence group that significantly overestimates its own performance. Surprisingly, this group also indicates to have encountered the cases in professional practice significantly more frequently, a statement that is unlikely true because the groups were randomly assigned to the conditions of the experiment. A high confidence seems to alter the perception of reality, even in the light of actual performance. See Table 2.

Table 2 Mean Differences with Standard Deviations (SD) Between the High-Confidence and Low-Confidence Groups on the Post Experiment Variables

Discussion

This study represents a first attempt to experimentally study the effect of overconfidence on premature closure, as expressed by shorter processing times of clinical vignettes, and diagnostic accuracy. To that end, we influenced the amount of confidence that third-year residents in internal medicine had in future diagnostic decisions by presenting cases to be diagnosed as either rather easy or particularly difficult. We studied its effect on confidence, diagnostic accuracy, and time on task in a subsequent diagnostic task. We expected physicians who displayed a higher level of confidence to spend less time on the task and be less diagnostically accurate.

We were successful in inducing appropriate levels of confidence in our randomly assigned groups of third-year residents. The high-confidence group even after the diagnostic task rated its own performance significantly higher than its actual performance warranted. Although high-confidence participants spent somewhat less time on the diagnostic task, however, we failed to find effects of this experimentally induced effect on time on task and diagnostic accuracy. This indicates that the overconfident group is more likely to make a quick, inaccurate diagnosis, and prematurely close the contact with patients, whereas the control group may spend more time ruling out rare diagnoses, making them less susceptible to premature closure. This would necessitate a clinical picture that corresponds to both a simpler, incorrect diagnosis and a more complex accurate diagnosis. The results are consistent to the findings of an experiment that failed to display relationships between overconfidence, premature closure, and diagnostic accuracy, as suggested by the extant literature.²³

The question is why this is so. One possible answer is that such relationship does not exist. Indeed, there is yet no convincing empirical evidence supporting the idea of a causal relationship between overconfidence, premature closure (as expressed by lesser processing time), and diagnostic error, but theoretical analyses and common sense make it likely that overconfident physicians do not adequately appreciate the full findings presented by a patient; they are unable to “swap horses midstream”,²⁵ because they think that their initial interpretations must be correct. This inability would result in premature closure followed by incorrect diagnoses.²³ On the other hand, one to the few studies in which the effect of premature closure was investigated directly failed to find effects on diagnostic accuracy.²⁶ Even a comprehensive review of the relationship between overconfidence and diagnostic error was unable to unearth direct evidence of the causal relationship between overconfidence, premature closure, and diagnostic accuracy.²³

A second answer to the question why we failed to find a causal relationship between overconfidence on the one hand and time on task and diagnostic accuracy on the other may be that overconfidence is a personality trait of some physicians rather than an attribute that can be induced. Cassam²⁷ argues that overconfidence can be considered an “epistemic vice”. The author is of the opinion that bettering physician self-knowledge can reduce diagnostic errors brought on by overconfidence is proven to be dubious. Overconfidence is a cognitive bias that is “stealthy” in the sense that it prevents its own detection. If this is true, if overconfidence is an enduring aspect of one’s personality, an experiment in which confidence in one’s own performance is manipulated by the experimenter may not really be appropriate.

Although the study results are favorable, there are certain limitations that limit the generalizability of the findings that must be addressed in future investigations. First, interns who volunteered to participate were randomly allocated to one of two groups. As a result, the participants may have been more motivated than those who did not participate, however this does not ensure a precise representation of the population. Second, there is the possibility of a lack of ecological validity. The external validity including ecological validity of studies such as the one discussed here is a challenge. A case in point is the use of vignettes rather than “real” patients. Although vignettes in general have been demonstrated to have sufficient external validity, they may be too easy for participants in an overconfidence study, because they present all findings necessary to arrive at an appropriate diagnosis. To ensure internal validity, an experimental approach compels researchers to simplify the event under study. The internal validity ascribed to an experimental approach ensures the accuracy of any causal inferences made from the findings. Premature closure may express itself particularly in situations where the physician decides how much information he or she needs. In such cases, experiments are not the way to go. The third is the failure to account for individual differences in cognitive biases, which may lead to the mistaken belief that all experts are equally susceptible to biases. To overcome this issue, we propose improving or developing reliable, specific measurement methods for cognitive biases. In addition, examine retrospectively the amount of diagnostic error produced as a result of premature closure in each of their own practices. For identifying errors, criteria would need to be defined. This might reduce a relative risk between overconfidence and diagnostic error.

Conclusion

This study shows that cognitive bias can be a source of medical errors. Current research has demonstrated that when health care professionals display high levels of confidence combined with insufficient time spent on performing tasks during patient diagnosis, can increase the risk of medical errors in many ways including premature closure and ignoring the complexities of a patient’s condition. Moreover, the results revealed that the high-confidence group spent less time diagnosing the presented cases, signifying diagnostic errors made due to premature closure. Yet, no differences appeared in diagnostic accuracy between them and the low-confidence (control) group. Further studies should consider confidence as a personal trait instead of an adaptable one as well as to teach doctors how to overcome error-related tendencies.