Research paper
Randomized clinical trial of a customized electronic alert requiring an affirmative response compared to a control group receiving a commercial passive CPOE alert: NSAIDewarfarin co-prescribing as a test case Brian L Strom,1 Rita Schinnar,1 Warren Bilker,1 Sean Hennessy,1 Charles E Leonard,1 Eric Pifer2 1
Correspondence to Brian L Strom, University of Pennsylvania School of Medicine, Center for Clinical Epidemiology and Biostatistics, 824 Blockley Hall, 423 Guardian Drive, Philadelphia, Pennsylvania 19104-6021, USA;
[email protected] Received 16 July 2009 Accepted 30 April 2010
ABSTRACT Background Studies that have looked at the effectiveness of computerized decision support systems to prevent drugedrug interactions have reported modest results because of low response by the providers to the automated alerts. Objective To evaluate, within an inpatient computerized physician order entry (CPOE) system, the incremental effectiveness of an alert that required a response from the provider, intended as a stronger intervention to prevent concurrent orders of warfarin and non-steroidal anti-inflammatory drugs (NSAIDs). Design Randomized clinical trial of 1963 clinicians assigned to either an intervention group receiving a customized electronic alert requiring affirmative response or a control group receiving a commercially available passive alert as part of the CPOE. The study duration was 2 August 2006 to 15 December 2007. Measurements Alert adherence was compared between study groups. Results The proportion of desired ordering responses (ie, not reordering the alert-triggering drug after firing) was lower in the intervention group (114/464 (25%) customized alerts issued) than in the control group (154/ 560 (28%) passive alerts firing). The adjusted OR of inappropriate ordering was 1.22 (95% CI 0.69 to 2.16). Conclusion A customized CPOE alert that required a provider response had no effect in reducing concomitant prescribing of NSAIDs and warfarin beyond that of the commercially available passive alert received by the control group. New CPOE alerts cannot be assumed to be effective in improving prescribing, and need evaluation.
INTRODUCTION Medication errors are a leading cause of death in American hospitals.1 Several studies have shown significant effects on medication errors with the use of computer-based reminders in an order entry system.2e4 However, the effect of computer-based advice in these studies is highly variable 5e14 and little is understood about the determinants of this variability. Recent studies have shown that computerbased recommendations for drugedrug interactions are overridden as much as 90% of the time.15e17 There are several characteristics of the alert itself that may affect the likelihood that the alert will be followed. Such factors as the nature of the alerted condition (eg, the perception by ordering clinicians that allergy alerts J Am Med Inform Assoc 2010;17:411e415. doi:10.1136/jamia.2009.000695
are more important than drug interaction alerts),18 19 how the alert integrates into clinical workflow (eg, an excessive volume of alerts interferes with workflow),20 and whether or not the alert is associated with a user’s action (eg, a drug interaction alert may not be relevant if the patient had previously received the alerted medication without complication)19 have all shown some impact on the likelihood that the advice will be followed.21 22 But a great many questions still remain on how to create the most effective tools for giving advice to practicing clinicians. Commercial computerized provider order entry (CPOE) systems are commonly distributed with built-in programs that provide simple alerts to the providers. These simple alerts take the form of message boxes with a warning. The fact that many alerts may fire simultaneously reduces the likelihood that any individual alert will be followed. Since most alerts do not require an affirmative response by the prescriber, and this approach has been shown to be ineffective,6 the intervention in our study was a customized electronic alert that specifically tested the effectiveness of requiring clinicians to acknowledge the alert. Warfarin management is complex because of the need for tight control of thromboplastins that determine the international normalized ratio (INR) and the many potential interactions between it and other drugs. There are so many potential interactions, in fact, that it is nearly impossible for physicians to remember all of them at any given time. Different drug interactions may have different propensities for causing adverse drug events of varying severity, but there is clear evidence that non-steroidal anti-inflammatory drugs (NSAIDs) given in conjunction with warfarin can increase bleeding risk by as much as fivefold compared to each drug alone.23 Most authors recommend generally avoiding NSAIDs when a patient is also receiving warfarin.24
BACKGROUND Randomized clinical trials in the inpatient setting that specifically required the clinicians’ response to an alert found these alerts to be effective. However, studies mostly focused on alert compliance with reminder guidelines for medical care25e28 rather than alerts to reduce drugedrug interactions. Studies in the outpatient setting focus on alerts to reduce inappropriate prescribing, primarily to the elderly.29e32 411
Downloaded from http://jamia.oxfordjournals.org/ by guest on February 5, 2016
Center for Clinical Epidemiology and Biostatistics, Department of Biostatistics and Epidemiology, Center for Education and Research on Therapeutics, University of Pennsylvania School of Medicine, Pennsylvania, USA 2 El Camino Hospital, Mountain View, California; previously University of Pennsylvania, Philadelphia, Pennsylvania
Research paper
RESEARCH QUESTION This study was designed to test the effectiveness of a customized CPOE alert specifically requesting a response by the provider using the NSAIDewarfarin interaction as a model. This is a common, well-accepted, potentially serious drug interaction 412
and, indeed, is one of the few that are. It also was occurring in our hospital, despite the presence of the conventional alert, suggesting the need to try something new.
METHODS Study setting The study was conducted at the Hospital of the University of Pennsylvania (HUP) and the Penn Presbyterian Medical Center (PMC) where all inpatient orders are entered using the Sunrise Clinical Manager (SCM, Eclipsys Corporation, Atlanta, Georgia, USA) CPOE system.
Study design The study was a randomized controlled trial of all 1865 resident physicians (RPs) and 98 nurse practitioners (NPs) involved in inpatient care. Attendings were not included as a unit of randomization since attendings at our hospitals do not normally enter the orders themselves; that is delegated to others. The trial was active from 2 August 2006 to 15 December 2007. The study was approved by the University of Pennsylvania Committee on Studies Involving Human Beings, which waived the requirement for informed consent and HIPAA authorization.
Study groups Intervention The alert was activated whenever a RP or NP placed an order for a NSAID with an already active warfarin order, if warfarin was started for a patient already on NSAIDs, or when ordering both simultaneously. The intervention in this study was a newly formatted alert that prompted the provider to discontinue the NSAID and order acetaminophen as an alternative. The specific text of the customized alert was: ‘Prescription of warfarin and NSAID together is contra-indicated. Please select an alternative drug, such as acetaminophen, by clicking on View Actions button.’
The provider could then either click the ‘acknowledge’ button to continue with the order for a NSAID, or could click on the ‘view action’ button to discontinue the current order of a NSAID and start a new order for acetaminophen. The control group received the standard passive alerts in the form of a message box warning the provider not to prescribe the combination drugs. A response was not required to allow the combination of drugs to be prescribed.
Endpoints The primary outcome was a new concurrent prescription order for NSAIDs and warfarin accepted through the electronic ordering system. In this context, not reordering the alert-triggering drug within 10 minutes after the alert fired represented a desired clinical response.
Data collection The unit of analysis for this study was the alert that fired, or would have fired, whenever a prescription order for concurrent NSAID and warfarin was encountered during the inpatient stay. Alerts were attributed to the study group of the provider who ordered the second prescriptiond that is, the prescription that triggered the alert. A 5-minute rule was applied to counting the total number of alerts, given that some alerts kept firing multiple times immediately one-after-another because providers, despite the customized alert, attempted to re-enter the order for the triggering J Am Med Inform Assoc 2010;17:411e415. doi:10.1136/jamia.2009.000695
Downloaded from http://jamia.oxfordjournals.org/ by guest on February 5, 2016
A common attribute of alert designs that require physician response is that the computer window cannot be closed without forcing physicians to choose among several decision options. One study27 provided intervention physicians with a pop-up window with automated guideline reminders to order tests or treatments and required them to accept, reject, or modify the suggested guidelines. This study found improved adherence to recommended guidelines, although adherence declined over time. Another trial26 compared providers receiving computer reminders to perform screening tests and required to select a response (‘done/order today,’ or ‘not applicable to this patient,’ or ‘patient refused,’ or ‘next visit’) to controls receiving similar reminders but without a required response. Again, more frequent compliance with reminders was seen among the intervention physicians.26 Another study28 activated and deactivated over a period of several weeks an automated decision support system delivering more conservative recommendations for psychotropic medications for hospitalized elderly patients. The intervention screen similarly required an explicit decision choice (‘change order to recommended medication’, ‘proceed with this order’, or ‘cancel this order’). The authors reported that prescriptions for every class of psychotropic drugs agreed with the system recommendations more frequently during the intervention periods. An interesting study by Paterno et al5 used inpatient alert log data to rank drugedrug interactions by severity and used several different displays to present this information to the clinicians. Serious alerts required clinicians either to cancel the order or discontinue a pre-existing drug order (‘hard stop alert’); less serious alerts also required action by the clinician, either to discontinue one of the drugs or to select an override reason; the least serious alerts were presented as information only, requiring no action of any kind from the clinician. Tiering the presentation of drugedrug interaction alerts by severity level was seen to be associated with higher rate of compliance for alerts leading the authors to conclude that how alerts are prioritized and presented to the user may be as important as which alerts to deliver.5 In contrast, other studies did not find alerts to be effectived even those requiring the clinician’s response.9 12 In their study of the characteristics of drug allergy overrides in a hospital CPOE system that required the clinician to enter a free-text reason for the override, Hsieh et al33 found that overrides were common. However, based on subsequent chart reviews, these overrides were mostly justified because of risk-benefit calculations for the particular patients. Similarly, in several Veterans Affairs medical centers, where entering a reason for the override is requested for critical drugedrug interactions, Grizzle et al12 found that more than half of the responses provided no clinical justification for the override. Thus, as noted earlier, alerts are effective in some domains25e28 and ineffective in others. A recent review by Eslami et al34 and other reviews35 36 also show that the impact of CPOE systems is mixed. However, a majority of studies were not randomized, there were only a few studies specifically focused on alerts to prevent drugedrug interactions in the hospital setting, and the follow-up periods were too short to conclude that the effect was lasting.
Research paper medication to ensure that the order was processed. Accordingly, alerts firing within #5 minutes of each other were counted as a single episode, whereas alerts firing within >5 minutes of each other were counted as separate episodes. If the alert-triggering drug was not reordered within an interval of 10 minutes after the alert fired, it was counted as a ‘desired clinical response’ (ie, not having active concurrent orders for warfarin and NSAID after the alert had fired). As a secondary analysis, we also investigated a 24-hour interval after alert firing.
Statistical analysis ‘Desired ordering rates’ in the intervention and control groups were compared using logistic regression,37 accounting also for resident physician and nurse practitioner clustering38 (ie, the non-independence of outcomes in patients within the same provider), using STATA 10.39
RESULTS
Figure 1 Profile of RCT for the Warfarin-tmp/smx Study. *Duplicate IDs resulted from the same providers working in both study hospitals, having different IDs in each hospital but representing the same persons. None received alerts. J Am Med Inform Assoc 2010;17:411e415. doi:10.1136/jamia.2009.000695
DISCUSSION We observed in a randomized clinical trial that a customized CPOE alert that required a provider response had no effect in reducing concomitant prescribing of NSAIDs and warfarin beyond that of the commercially available passive alert received by the control group. To date, most studies that looked at the effectiveness of computerized decision support systems designed in particular to reduce medication errors of drugedrug interactions have reported only modest results.8 9 18 40 The principal reason for this was the low response by the providers to the automated alerts.33 41 Providers tend to override the alerts because they are perceived to be non-specific, lacking the providers’ additional knowledge of the clinical situation for the specific patient context.18 One area for further research is the best way to display alerts to providers to make drug alerts more effective and reduce overriding. If alerts are ineffective because they are easy to ignore, then attention should be given to finding ways to amplify the effect of the alert. We focused on one feature of such displaysdnamely, the requirement that providers acknowledge the alert as compared to not requiring any active response. We found that this feature in itself as part of a computerized order entry system was not sufficient. The customized alert advising the clinician to order acetaminophen instead of NSAID in patients requiring warfarin and to acknowledge the alert was no more effective than the standard of care passive alert in reducing the undesired prescribing. On the other hand, requiring more interaction from clinicians when alerts fire, in order to increase compliance, may be disruptive to the workflow and may have other unintended consequences. A recent trial by Strom et al42 to evaluate the effectiveness of a nearly ‘hard stop’ CPOE prescribing alert intended to reduce concomitant orders of warfarin and trimethoprim/sulfamethoxazole found it to be extremely effective in changing prescribing. However, this intervention precipitated treatment delays among patients in the intervention group needing immediate drug treatment, leading to early termination 413
Downloaded from http://jamia.oxfordjournals.org/ by guest on February 5, 2016
Figure 1 shows the subject randomization flow, including the 1963 providers in the analysis (960 intervention providers receiving the customized alerts and 1003 control providers receiving the standard-of-care alerts). Overall there were 1218 alerts. After applying the 5-minute rule (namely, counting as a single event alerts that fired within #5 minutes of each other), there were 1024 alerts: 464 in the intervention group and 560 in the control group. The proportion of desired ordering responses (ie, not reordering the alert-triggering drug within 10 minutes of firing) was lower in the intervention group (25% (114/464) customized alerts issued) than in the control group (28% (154/560) passive
alerts firing). The odds of inappropriate ordering by the intervention providers was adjusted OR 1.22 (95% CI 0.69 to 2.16; p¼0.48), indicating that the modified alert had failed to prevent the concomitant ordering of warfarin and NSAIDs (see table 1). These results were adjusted for provider type (RP or NP) and for hospital (HUP or PMC) as potential confounders and accounting for clustering by provider (ie, adjusting for the nonindependence of outcomes in patients within the same provider). It is of interest to note the high mean number of alerts per provider (3.5 vs 4.5 in the intervention and control groups, respectively) even after counting alerts triggered by repeated orders within #5 minutes as a single alert event. The 25th, 50th, and 75th percentiles of the number of alerts triggered by offending orders per provider were 1, 2, and 4, respectively, in each group. The results from the secondary analysis based on behavior within 24 hour after an alert had fired were not substantively different and are not presented. There was a trend over the 17-month study period with decreasing proportion of desired ordering responses over time (OR 1.09 per month; p¼0.007). There was also a significant month-by-study group interaction (p¼0.04). The interaction was the result of multiple crossovers in the desired ordering rates between the two groups over time. Overall, there was no group difference.
Research paper Table 1 Frequency of alerts for warfarin and non-steroidal anti-inflammatory drugs (NSAIDs) that were followed by desired response by study group Study outcome Study group
Total alerts fired* (alerts triggered by concurrent ordering of warfarin and NSAID)
Intervention group Control group
464 560
Statistical Results
Desired ordering responses (proportion of the alerts that were followed by an appropriate action (ie, the alert-triggering drug was not reordered within 10 min after the alert fired) 114 (25%) 154 (28%)
Unadjusted OR (95% CI)
Adjusted ORy (95% CI)
1.16 (0.88 to 1.54)
1.22 (0.69 to 2.16)
*Counting as a single event alerts that were fired within #5 minutes of each other. yAdjusted for study group, hospital, provider type, and clustering by provider type.
The customized alert may have been too similar to the alert in the control group, although we do not think so, since our alert provided an alternative strategy and forced a response. The customized alert may have been flawed if it led to misunderstanding the meaning of the ‘acknowledge’ button. On the other hand, there are data suggesting that clinicians make risk-benefit calculations in deciding whether to accept alerts.27 33 Thus, it may be that the clinicians considered drugedrug interactions to be unimportant despite the stronger intervention, or considered the particular interaction that we selected to study unimportant given the ability to monitor INRs closely in hospitals. There is no way for us to determine this. However, regardless of the mechanism, it is clear that the more invasive alert added nothing to the traditional one.
Limitations
CONCLUSION
A limitation of the study, perhaps resulting from the long duration of the intervention (15 months), is the possibility that the difference of effect between the intervention group and control group for concurrent orders of NSAIDs and warfarin may have diminished over time because residents and nurse practitioners usually work in teams and may work alongside other physicians that may be assigned to a different group in the study. It is common for RPs and NPs to discuss the care plan of patients, which include issues of ordering medications. Moreover, over time, the RPs and NPs from both the intervention group and control groups may have learned not to order NSAIDs with an active order of warfarin due to awareness of the alert. However, our analysis of time trend was not consistent with this being a problem. Another limitation is that because the study took place over 15 months, all residents did not participate in the study for the same amount of time. Fluctuations in the effectiveness of the alert over time could perhaps be explained by new house-staff joining the study population and old house-staff leaving the training program. Nevertheless, the comparison between the study groups was a concurrent comparison regardless. Thus, the overall comparison should still be valid. An important limitation is the absence of information on potential confounders that may have influenced the occurrence of alert overrides. For example, information on varying years of experience of the providers was not collected and the providers were not surveyed about the reasons why they overrode them. Information on the clinical conditions and severity triggering the alert was also not collected. This was felt unnecessary since the study was randomized. Further, collecting some of this information (eg, debriefing the subjects) could have had a substantial Hawthorne effect. However, this means this information was not available to shed light on subsets of subjects, assisting in understanding why the alerts were not followed. Ultimately, it is not possible for us to determine without further studies why the customized alert was not more effective.
In conclusion, a customized CPOE alert that required a provider response had no effect in reducing concomitant prescribing of NSAIDs and warfarin beyond that of usual standard of care. CPOE alerts cannot be assumed to be effective in improving prescribing and need to be formally evaluated.
414
Acknowledgments The authors wish to thank the members of the University of Pennsylvania IRB Oversight Committee, Todd Barton, Jeffrey Staab, and Robert Gross for their valuable suggestions and guidance. In addition, the authors thank Jesse Chittams, Asaf Hanish, Qing Liu, and Catherine T Williams-Smith from the CCEB Biostatistics Analysis Center for creating the analytic data set. We thank also Maggie Massary, from the University of Pennsylvania Health System, for her work in developing the electronic alert and for guiding us through the elements of the electronic intervention and Sunrise data collection. We would like to thank Anne Saint John for her excellent assistance in preparing this manuscript. Finally, this paper has greatly benefited from comments from the JAMIA peer reviewers, especially regarding alternative explanations for why drug alerts may not be effective in our and other studies. Funding This study was funded partly by the University of Pennsylvania Health System, and partly by grant number U18HS016946 from the Agency for Healthcare Research and Quality. Competing interests None. Provenance and peer review Not commissioned; externally peer reviewed.
REFERENCES 1. 2. 3. 4. 5. 6. 7.
Kohn LT, Corrigan JM, Donaldson MS, eds. (Committee on Quality of Health Care in America, Institute of Medicine). To Err Is Human: Building a Safer Health System. Washington, DC, USA: National Academies Press, 1999. Bates DW, Leape LL, Cullen DJ, et al. Effect of computerized physician order entry and a team intervention on prevention of serious medication errors. JAMA 1998;280:1311e16. Bates DW, Cohen M, Leape LL, et al. Reducing the frequency of errors in medicine using information technology. J Am Med Inform Assoc 2001;8:299e308. Teich JM, Merchia PR, Schmiz JL, et al. Effects of computerized physician order entry on prescribing practices. Arch Intern Med 2000;160:2741e7. Paterno MD, Maviglia SM, Gorman PN, et al. Tiering drug-drug interaction alerts by severity increases compliance rates. J Am Med Inform Assoc 2009;16:40e6. Chrischilles EA, Fulda TR, Byrns PJ, et al. The role of pharmacy computer systems in preventing medication errors. J Am Pharm Assoc (Wash) 2002;42:439e48. Judge J, Field TS, DeFlorio M, et al. Prescribers’ responses to alerts during medication ordering in the long term care setting. J Am Med Inform Assoc 2006;13:385e90.
J Am Med Inform Assoc 2010;17:411e415. doi:10.1136/jamia.2009.000695
Downloaded from http://jamia.oxfordjournals.org/ by guest on February 5, 2016
of the study. Of course, given our current results do not indicate that this additional burden on physicians leads to additional effectiveness, that too argues not to implement such alerts. The strength of this study was its randomized design. The decision to randomize clinicians rather than patients was motivated by several considerations. First, since the order is written by these individuals, it is appropriate to consider each order by them as an opportunity for error. Second, each medical practitioner has a unique access code to use the electronic ordering system and the order system menu can be varied by individual user. In addition, we wanted to keep each practitioner in the same study group for the duration of the study to reduce contamination between the two groups. An additional strength is the large number of clinicians included in the study (approximately 2000).
Research paper 8. 9. 10. 11. 12. 13. 14. 15. 16. 17. 18.
20. 21. 22. 23. 24. 25.
J Am Med Inform Assoc 2010;17:411e415. doi:10.1136/jamia.2009.000695
26. 27. 28. 29. 30. 31. 32. 33. 34. 35. 36. 37. 38. 39. 40. 41. 42.
Litzelman DK, Dittus RS, Miller ME, et al. Requiring physicians to respond to computerized reminders improves their compliance with preventive care protocols. J Gen Intern Med 1993;8:311e17. Overhage JM, Tierney WM, Zhou XH, et al. A randomized trial of “corollary orders” to prevent errors of omission. J Am Med Inform Assoc 1997;4:364e75. Peterson JF, Kuperman GJ, Shek C, et al. Guided prescription of psychotropic medications for geriatric inpatients. Arch Intern Med 2005;165:802e7. Terrell KM, Perkins AJ, Dexter PR, et al. Computerized decision support to reduce potentially inappropriate prescribing to older emergency department patients: a randomized, controlled trial. J Am Geriatr Soc 2009;57:1388e94. Raebel MA, Carroll NM, Kelleher JA, et al. Randomized trial to improve prescribing safety during pregnancy. J Am Med Inform Assoc 2007;14:440e50. Simon SR, Smith DH, Feldstein AC, et al. Computerized prescribing alerts and group academic detailing to reduce the use of potentially inappropriate medications in older people. J Am Geriatr Soc 2006;54:963e8. Smith DH, Perrin N, Feldstein A, et al. The impact of prescribing safety alerts for elderly persons in an electronic medical record: an interrupted time series evaluation. Arch Intern Med 2006;166:1098e104. Hsieh TC, Kuperman GJ, Jaggi T, et al. Characteristics and consequences of drug allergy alert overrides in a computerized physician order entry system. J Am Med Inform Assoc 2004;11:482e91. Eslami S, de Keizer NF, de Jonge E, et al. A systematic review on quality indicators for tight glycaemic control in critically ill patients: need for an unambiguous indicator reference subset. Crit Care 2008;12:R139. Conroy S, Sweis D, Planner C, et al. Interventions to reduce dosing errors in children: a systematic review of the literature. Drug Saf 2007;30:1111e25. Kaushal R, Shojania KG, Bates DW. Effects of computerized physician order entry and clinical decision support systems on medication safety: a systematic review. Arch Intern Med 2003;163:1409e16. MuCullagh P, Nelder JA. Generalized linear models. 2nd edn. London: Chapman and Hall Ltd, 1983. Thompson SG, Pyke SD, Hardy RJ. The design and analysis of paired cluster randomized trials: an application of meta-analysis techniques. Stat Med 1997;16:2063e79. Stata 10. Stata Corporation. College Station, Texas, 2007. Glassman PA, Belperio P, Simon B, et al. Exposure to automated drug alerts over time: effects on clinicians’ knowledge and perceptions. Med Care 2006;44:250e6. Lapane KL, Waring ME, Schneider KL, et al. A mixed method study of the merits of e-prescribing drug alerts in primary care. J Gen Intern Med 2008;23:442e6. Strom BL, Schinnar R, Aberra F. Unintended effects of a CPOE nearly hard stop alert to prevent a drug interaction: A RCT. Arch Intern Med. In press.
415
Downloaded from http://jamia.oxfordjournals.org/ by guest on February 5, 2016
19.
Feldstein AC, Smith DH, Perrin N, et al. Reducing warfarin medication interactions: an interrupted time series evaluation. Arch Intern Med 2006;166:1009e15. Tamblyn R, Huang A, Taylor L, et al. A randomized trial of the effectiveness of ondemand versus computer-triggered drug decision support in primary care. J Am Med Inform Assoc 2008;15:430e8. Palen TE, Raebel M, Lyons E, et al. Evaluation of laboratory monitoring alerts within a computerized physician order entry system for medication orders. Am J Manag Care 2006;12:389e95. Eslami S, de Keizer NF, Abu-Hanna A. The impact of computerized physician medication order entry in hospitalized patientsda systematic review. Int J Med Inform 2008;77:365e76. Grizzle AJ, Mahmood MH, Ko Y, et al. Reasons provided by prescribers when overriding drug-drug interaction alerts. Am J Manag Care 2007;13:573e8. Taylor LK, Kawasumi Y, Bartlett G, et al. Inappropriate prescribing practices: the challenge and opportunity for patient safety. Healthc Q 2005;8 Spec No:81e5. van der Sijs H, Lammers L, van den Tweel A, et al. Time-dependent drug-drug interaction alerts in care provider order entry: software may inhibit medication error reductions. J Am Med Inform Assoc 2009;16:864e8. Weingart SN, Toth M, Sands DZ, et al. Physicians’ decisions to override computerized drug alerts in primary care. Arch Intern Med 2003;163:2625e31. van der Sijs H, Aarts J, Vulto A, et al. Overriding of drug safety alerts in computerized physician order entry. J Am Med Inform Assoc 2006;13:138e47. Payne TH, Nichol WP, Hoey P, et al. Characteristics and override rates of order checks in a practitioner order entry system. Proc AMIA Symp 2002:602e6. Taylor LK, Tamblyn R. Reasons for physician non-adherence to electronic drug alerts. Stud Health Technol Inform 2004;107:1101e5. Isaac T, Weissman JS, Davis RB, et al. Overrides of medication alerts in ambulatory care. Arch Intern Med 2009;169:305e11. Weingart SN, Simchowitz B, Shiman L, et al. Clinicians’ assessments of electronic medication safety alerts in ambulatory care. Arch Intern Med 2009;169:1627e32. Bates DW, Teich JM, Lee J, et al. The impact of computerized physician order entry on medication error prevention. J Am Med Inform Assoc 1999;6:313e21. Bates DW, Kuperman GJ, Rittenberg E, et al. A randomized trial of a computerbased intervention to reduce utilization of redundant laboratory tests. Am J Med 1999;106:144e50. Delaney JA, Opatrny L, Brophy JM, et al. Drug drug interactions between antithrombotic medications and the risk of gastrointestinal bleeding. CMAJ 2007;177:347e51. Chan TY. Adverse interactions between warfarin and nonsteroidal antiinflammatory drugs: mechanisms, clinical significance, and avoidance. Ann Pharmacother 1995;29:1274e83. Kucher N, Koo S, Quiroz R, et al. Electronic alerts to prevent venous thromboembolism among hospitalized patients. N Engl J Med 2005;352:969e77.
