When Guidelines Don’t Guide

Research Report

When Guidelines Don’t Guide: The Effect of Patient Context on Management Decisions Based on Clinical Practice Guidelines Mathew Mercuri, PhD, Jonathan Sherbino, MD, Robert J. Sedran, MD, Jason R. Frank, MD, Amiram Gafni, PhD, and Geoffrey Norman, PhD

Abstract Purpose This study examines the influence of patient social context on physicians’ adherence to clinical practice guidelines (CPGs). Method Expert emergency medicine (EM) physicians and novice physicians (EM residents) were surveyed using an Internet-based program between January and July of 2013. Participants were presented clinical cases and were asked to indicate if they would order or prescribe a specified test or treatment. Cases were chosen from four domains

It’s more important to know what sort of person has a disease than to know what sort of disease a person has. —Hippocrates

Clinical practice guidelines (CPGs)

were introduced to assist clinicians in determining the appropriate course of action for a given medical condition.1 The application of CPGs is intended to promote the standardization of medical practice along the lines of scientific principles, or the best (available) evidence of effectiveness. Some believe that by reducing uncertainty, CPGs might operate to reduce variation in medical practice, thereby promoting better patient outcomes.2–5 Studies have found that physician adherence to CPGs is incomplete, and investigated the reasons for why this is Please see the end of this article for information about the authors. Correspondence should be addressed to Dr. Mercuri, Department of Medicine, Division of Cardiology, Columbia University, 51 Audubon Ave. (5th Floor), New York, NY 10032; e-mail: matmercuri@hotmail. com. Acad Med. 2014;XX:00–00. First published online doi: 10.1097/ACM.0000000000000542

Academic Medicine, Vol. XX, No. XX / XXXX XXXX

where CPGs exist, and were constructed to include or exclude a “context variable” (CV). Both expert and novice physicians’ CPG adherence rate in the CV condition was compared with that in the no CV condition. The CPG adherence rates in CV and no CV conditions were also compared between expert and novice EM physicians.

interval [CI]: 0.17–0.53, P < .001). Experts were less likely to adhere to CPGs in the CV condition when compared with novice physicians (n = 28) (56% versus 67%; OR = 0.62, 95% CI: 0.39–1.0, P = .039). Expert and novice EM physicians did not differ in their adherence to CPGs in the no CV condition.

Results Expert EM physicians (n = 28) were less likely to adhere to CPGs in the CV condition compared with the no CV condition (56% versus 80%, respectively; odds ratio [OR] = 0.32, 95% confidence

Conclusions Participants were sensitive to both the best clinical evidence of benefit, as recommended by CPGs, and patient context when determining how care should be managed.

the case.6–8 In two systematic reviews of the literature on CPG adherence, a majority of the studies focused on lack of awareness or knowledge of current CPGs, and lack of belief or trust in published CPGs as the primary culprits for nonadherence.6,7 In a 2005 systematic review, Choudhry et al9(p269) observed an inverse relationship between physician age and experience and quality of care (most often defined as CPG adherence in their study), which the authors attributed to a lack of sufficient “factual knowledge” among expert physicians.

Thus, even when clinical uncertainty is minimized (e.g., when evidencebased CPGs are available to the physician), management decisions may be influenced by other contextual factors. However, the examination of physicians’ deliberate nonadherence to CPGs as a result of patient context, in particular those aspects of context that are personal or “nonmedical” (i.e., social) in nature, receives comparatively less attention in the literature than deviations due to lack of awareness or knowledge deficits.

Whereas medical practice is grounded in clinical science, patient management decisions often are influenced by the individual patient’s circumstance (i.e., context) as well. For example, Andersen and Newman10 (1973) outline a number of patient-related factors that can influence the use of health care services, including a patient’s affordability, access, and attitudes (both personal and family) toward health and health care. Likewise, Ro11 (1969) argued that what the physician recommends is mediated in part by his or her response to each patient’s context, defined by both personal or situational factors.

In this study, we aimed to determine whether factors related to an individual patient’s social context might influence physicians’ departures in management decisions from CPG recommendations. Contextual factors include the patient’s occupation, proximity to care, expectations of treatment, and factors related to home life (e.g., lives alone). This study focused on point-of-care management decisions for common emergency medicine (EM) cases where established CPGs exist. The following hypothesis was tested: 1) Given a case where an established CPG exists, the rate at which expert EM physicians’ management decisions will

1

Copyright © by the Association of American Medical Colleges. Unauthorized reproduction of this article is prohibited.

Research Report

adhere to CPG will be lower when a mediating factor related to the context of the patient is presented compared with when it is not. Whereas information regarding the most effective treatment for a given medical condition might be acquired through study of the medical literature, effectively tailoring care to a given patient’s situation is likely learned through experience with patients. Thus, we expected that more expert physicians are more adept at integrating the patient’s context into their management decisions. This study also tests the following hypothesis related to expertise: 2) In cases where a factor related to the patient’s context is presented, expert EM physicians’ management decisions will less often adhere to CPG recommendations compared with those of novice EM physicians. Conversely, to determine whether nonadherence is a consequence of inadequate knowledge, we also tested the hypothesis that: 3) In cases where factors related to the patient’s context are not presented, expert EM physicians’ management decisions will adhere to CPG recommendations as often as novice EM physicians. Method

an expert and certified EM physician (J.S.). Cases were chosen from four domains—two related to diagnostic testing and two related to treatment. Specifically, cases focused on (1) ordering an x-ray for suspected ankle injury; (2) ordering a computerized tomography (CT) scan for suspected head injury; (3) prescribing antibiotics for sore throat (i.e., suspected streptococcal [strep] pharyngitis); and (4) prescribing warfarin for atrial fibrillation, where the patient has been referred to a cardiologist. The basic cases were developed to meet the terms of one of the following CPGs: Ottawa Ankle Rule12; Canadian CT Head Rule13; Centor Score/McIssac Score for Strep Pharyngitis14,15; or Atrial Fibrilation CHADS2 Score.16 Each case described the presenting illness and its history, the patient’s past medical history, current medications and social history, and results of the physical exam.

jeopardized if an ankle fracture was missed. Likewise, in one sore throat case, the patient, a nurse, who lived in an isolated area, could not easily follow up on an abnormal test. Three cases were developed for each domain. Two additional expert and certified EM physicians reviewed the cases to assess content and face validity. Table 1 shows a brief summary of the cases and their associated CV. In November 2012, we developed a survey containing the 12 cases. The survey was balanced so that half the cases contained an associated CV. We developed a second version that mirrored the first, so cases that contained a CV on the first survey did not include the CV on the second survey, and vice versa. This facilitated an experimental design whereby no physician would see both the CV and no CV versions of the same case when participating in the experiment, so as to avoid revealing the experimental condition.

A “context variable” (CV) that could be inserted into the social history was developed for each case. The CVs were designed to provide a reasonable basis for not following the CPG. For example, in one ankle injury case a professional hockey player’s livelihood may be

The cases appeared on each survey in no particular order according to domain or CV condition. The order at which each case was presented was the same for both surveys. The information presented for each case appeared on a single page.

Table 1 Summary of Cases and Their Associated Context Variables Included in a CPG Adherence Survey Sent to Canadian Physicians, 2013

Participants

Casea

Descriptor

CPG indicates

Context variable

Participants were recruited from both expert and novice EM physicians practicing in one of three cities in Ontario, Canada: Hamilton, London, and Ottawa. We defined expert physicians as those who were residency trained in EM and who had at least five years of EM practice after residency. We defined the novice EM physician group as those who were enrolled in the first two years of a residency program in EM. This was done to ensure that participant novice physicians had a working knowledge of EM-related CPGs, but to minimize the influence of clinical experience on shaping management decisions.

Ankle 1 Ankle 2

46 yr M trauma L ankle 21 yr M trauma R ankle

No X-ray No X-ray

Pursued by police Professional hockey player

Ankle 3

36 yr F trauma R ankle

No X-ray

Civil suit

Head 1

62 yr M minor head injury

No CT

Lives alone

Head 2

47 yr F minor head injury

No CT

Chair of the hospital board

Head 3

93 yr M head injury

CT

Advanced Alzheimer’s

Strep 1

21 yr F sore throat

No Abx

Mother is a nurse who works in the ED

Strep 2

5 yr M sore throat

No Abx

Family members recently treated for suspected strep, anxious parents

Strep 3

46 yr F sore throat

No Abx

Nurse practitioner, leaves for isolated community tomorrow

AF 1

54 yr F palpitations

No warfarin

Vascular surgeon

AF 2

64 yr M palpitations

Warfarin

Alcoholic

AF 3

78 yr F palpitations

Warfarin

Unreliable patient

Survey instrument Twelve clinical scenarios outlining typical cases that EM specialists might encounter in practice were developed in October 2012 in consultation with

2

  Abbreviations: CPG indicates clinical practice guideline; Yr, year; M, male; F, female; L, left; R, right; CT, computed tomography; Abx, antibiotics; ED, emergency department. a Cases were created from four domains, with three cases per domain. Two (Ankle, Head) related to diagnostic testing, and two (Strep, AF) related to treatment. “Ankle” indicates ordering an x-ray for a suspected ankle injury; “head” indicates ordering a CT for a suspected head injury; “strep” indicates prescribing antibiotics for suspected streptococcal (strep) pharyngitis; and “AF” indicates prescribing warfarin for atrial fibrillation, where the patient has been referred to a cardiologist.

Academic Medicine, Vol. XX, No. XX / XXXX XXXX

Copyright © by the Association of American Medical Colleges. Unauthorized reproduction of this article is prohibited.

Research Report

Prior to the presented cases, the survey included questions regarding participant demographics and characteristics of their practice environment. The survey concluded with four questions regarding the participants’ familiarity with each of the above-mentioned CPGs. Participants were unable to access their answers regarding management decisions once they were asked the CPG familiarity questions, to avoid influencing their survey responses. Data acquisition We surveyed participants and collected data between January and July 2013. We used a Web-based survey design administered via LimeSurvey version 2.0, a free, open-source application (LimeSurvey, Hamburg, Germany). All potential participants received an e-mail invitation from the local investigator using a local EM physician e-mail distribution list (one in each of Hamilton, London, and Ottawa), or through an EM residency program e-mail distribution list (one in each of Michael G. DeGroote School of Medicine, McMaster University, Schulich School of Medicine & Dentistry, Western University, or Faculty of Medicine, University of Ottawa). Responding participants were then randomized to one of the two survey instruments, stratified according to experience (novice versus expert EM physician). Once randomized, a unique survey link was prepared and forwarded to each participant’s e-mail. Only a single survey attempt was possible for each link, and the survey needed to be completed in a single session. A reminder e-mail was sent out one week after the initial invitation to encourage participation. For each presented case, the participant was asked to indicate if he or she would order the diagnostic test or prescribe the treatment in question (yes/no). The participant could advance to the next case only after indicating a management decision. The survey was self-paced. Later, the participant was asked to indicate if he or she was familiar with each of the above-mentioned CPGs (yes/no). Participants received a small honorarium for their participation in the study. The dataset was stripped of personal identifiers prior to analysis. The study received approval by the McMaster University/Hamilton Health Sciences research ethics board.

Academic Medicine, Vol. XX, No. XX / XXXX XXXX

Analysis Data related to participant demographics and characteristics of the practice environment were used only to describe the population. A chi-square test was used to compare the frequency of adherence to a CPG when the CV was present versus when the CV was not present. This analysis was conducted in each of the novice and expert physician groups. A chi-square test was performed to determine if the frequency of adherence to a CPG in the CV condition differed between expert and novice EM physicians. This analysis was then repeated in the no CV condition. A P value of .05 was considered significant. All analyses were performed using Microsoft Excel 2008 (Microsoft, Redmond, Washington) or SPSS version 20 (IBM Corp., Armonk, New York) for Macintosh. Results

We surveyed and collected data from 28 expert and 28 novice physicians, evenly balanced between the two survey forms. Three participants indicated they were not familiar with the Centor Score/McIssac

Score for Strep Pharyngitis (2 expert, 1 novice physician), and 3 participants indicated the same for the Atrial Fibrillation CHADS2 Score (2 expert, 1 novice physician). Cases related to these CPGs for participants unfamiliar with them were removed from the dataset prior to analysis. All of the participants indicated that they worked primarily at an urban teaching hospital. Participant demographics and characteristics of work environment are described in Table 2. Expert physicians were significantly less likely to follow CPG recommendations in the presence of a CV compared with when the CV was not present (56% versus 80%, respectively; odds ratio [OR] = 0.32, 95% confidence interval [CI] = [0.17, 0.53], P < .001). The rate at which novice physicians followed CPG recommendations also differed between conditions (67% for CV condition versus 79% for no CV; OR = 0.53, 95% CI = [0.31, 0.9], P = .013). In the absence of CV, expert physicians and novice physicians were equally likely to adhere to CPGs (80% for experts versus 79% for novice physicians, OR = 1.05, 95% CI = [0.59, 1.8], P = .85). However,

Table 2 Demographics, Characteristics of Practice Environment, and Adherence to CPGs of Survey Participants, Canada, 2013 Characteristic Demographics  Female, no. (%)  Years in EM (post certification), mean (SD)

Expert EMa (n = 28)

Novice EMa (n = 28)

Total (N = 56)

8 (29)

13 (46)

21 (38)

14.9 (8.3)

—

—

Practice environment  ED census (cases per month), no. (%)     >25,000–50,000

4 (14)

3 (11)

7 (13)

    >50,000–75,000

17 (61)

10 (36)

27 (48)

    >75,000

7 (25)

15 (54)

22 (39)

 Shifts per month, mean (SD)

13.4 (5.8)

14.4 (3.1)

13.9 (15)

 Patients per shift, mean (SD)

26.7 (7.5)

11.4 (2.4)

19.0 (9.5)

 CV condition, mean (SD)

3.25 (1.5)

3.96 (1.3)

3.61 (1.4)

 No CV condition, mean (SD)

4.64 (1.0)

4.68 (1.4)

4.66 (1.2)

20 (71)

16 (57)

36 (64)

 CV/no CV > 1, no. (%)

2 (7)

4 (14)

6 (11)

 CV/no CV = 1, no. (%)

6 (21)

8 (29)

14 (25)

Followed CPGs

 CV/no CVb < 1, no. (%)

  Abbreviations: CPGs indicates clinical practice guidelines; EM, emergency medicine; ED, emergency department; SD, standard deviation; CV, context variable. a Expert EM was defined as a physician who was trained in an EM residency and had at least five years of EM practice experience; novice EM was defined as those who were enrolled in the first two years of a residency program in EM. b “CV/no CV” is the ratio of how many times the participant followed the CPGs in CV condition cases compared with no CV condition cases.

3

Copyright © by the Association of American Medical Colleges. Unauthorized reproduction of this article is prohibited.

Research Report

expert EM physicians were less likely to follow CPG recommendations in the CV condition when compared with novice physicians (56% versus 67%; OR = 0.62, 95% CI = [0.39, 1.0], P = .039). Subsequent analyses of the data indicated that the results were not driven by any particular physician. Rather, management responses from the majority of participants followed the pattern described above, with a higher proportion doing so in the expert group (see Table 2). Figure 1 shows the rate of adherence to CPGs for both the expert and novice physicians in both the CV and no CV conditions. Discussion

Both expert and novice physicians’ management decisions reflected CPGs in the majority of cases where the CV was not presented. However, their management decisions were less likely to reflect those recommended in CPGs when they were presented cases that included the CV. This observation supports our hypothesis that a patient’s context influences how physicians manage care, even when CPGs are available and known. Although the rate at which expert and novice physicians’ management decisions reflected CPGs did not differ when a CV was not presented (supporting our third hypothesis), expert physicians were less likely to follow CPGs in the CV condition when compared with novice physicians. This finding supports our second hypothesis that the extent to which a physician will adhere to CPGs

when presented information related to a patient’s context is to some degree based on experience. Our findings do not support the hypothesis that adherence is related primarily to physicians’ knowledge of CPGs.9 With the exception of a few participants (described above), the participants indicated near-universal familiarity with the case-related CPGs, and management decisions reflected these CPGs in the majority of cases in the control (i.e., “no CV”) condition, irrespective of EM experience. Despite this demonstrated familiarity, management decisions deviated from CPGs more often in the presence of a contextual factor (i.e., “CV”). These observations imply that the “lack of adherence” among physicians may not be a function of their level of CPG knowledge but, rather, due to their sensitivity to each patient’s unique needs. In other words, management decisions, while based on the best available evidence, are tailored according to the physician’s perception of potential benefit given the whole experience of the patient. Furthermore, the data presented suggest that this “tailoring” becomes more pronounced with experience. Advocates for CPGs acknowledge the tension between guideline provisions and the realities of clinical practice. The Canadian Medical Association Handbook on Clinical Practice Guidelines states “that CPGs are not intended to provide guidance in all circumstances and for all patients,”17(p3) as the authors

Figure 1 Rate of adherence to clinical practice guidelines (CPGs) in both the context variable (CV) and no CV conditions, according to a 2013 survey of “expert” (trained in emergency medicine [EM] residency with at least five years of EM practice experience) and “novice” (enrolled in first two years of an EM residency program) physicians in Hamilton, London, and Ottawa, Ontario, Canada.

4

recognize that “their more general nature renders them insensitive to the particular circumstances of individual cases.”17(p3) Likewise, Woolf et al3(p529) (1999) raise concern that “the frequently touted benefit of clinical guidelines—more consistent practice patterns and reduced variation—may come at the expense of reducing individualized care for patients with special needs.” This position is supported by key advocates of the evidence-based medicine paradigm for management of care. For example, when determining if CPG recommendations are applicable to a patient, Wilson and colleagues18(p1632) (1995) suggest: “You should look for information that must be obtained from and provided to patients for patient preferences that should be considered. It is important to consider whether the values assigned (implicitly or explicitly) to outcomes could differ enough from your patients’ preferences to change a decision about whether to adopt a recommendation.” Despite this awareness, best practices for CPG development endorse a process that emphasizes evidence derived from randomized controlled trials,19–21 which are not designed to take into account potentially important aspects of the individual that cannot be easily quantified.22 For this reason it is understandable that some physicians believe that CPGs are impractical or too rigid to apply to individual patients.23 Limitations This study has several limitations that are typical of experimental designs. Most notably is the extent to which results based on hypothetical cases are generalizable to actual practice. Recent evidence suggests, however, that in acquisition of diagnostic skills, written cases can be considered interchangeable with video or live simulations.24 Thus, these findings may be generalizable to management decision making. The surveys used a fixed case order, creating the possibility of an order effect on management decisions. Our study only looked at management decisions within a single specialty. Additional studies examining the effects of patient context on management decisions relative to CPGs among physicians from a variety of specialties, and from various communities, are required to confirm

Academic Medicine, Vol. XX, No. XX / XXXX XXXX

Copyright © by the Association of American Medical Colleges. Unauthorized reproduction of this article is prohibited.

Research Report

the presented findings. Unfortunately, finer analyses according to management domain were not possible given data limitations due to sample size. Thus, although it appears that no particular domain was driving the lack of adherence to CPGs, such could not be confirmed or denied based on the collected data. Although the physicians indicated familiarity with the included CPGs, our study assumed that they had functional knowledge of each CPG by memory. Finally, as our study used hypothetical cases, patient outcomes resulting from management decisions could not be measured. Thus, it is not known from our study if such deviations from CPGs would indeed result in improved patient outcomes. Implications for education If patient context is a justifiable reason to deviate from CPGs, what is correct for each patient may depend on characteristics related to the “context.”25 The lack of complete agreement among physicians in our study when their recommended management decisions in the presence of the CV deviated from CPGs suggests that there is no obvious single pathway to achieving optimal decisions. Perhaps this is not surprising; much of clinical education is directed at identifying the “best” therapy for clinical conditions, and practice guidelines tend to reinforce the rational basis for selection of therapies. There is no equivalent educational strategy for identifying and managing patients taking contextual factors into account. So it is not unexpected that there is little consensus as to the appropriate action in the light of particular contextual factors. The situation is somewhat ironic. On the one hand, we endorse “patient-centered care” and involve the patient in decision making26; on the other, we assume that optimal care is a direct consequence of scientific evidence.22 Precisely why experts are more likely than novices to deviate from guidelines is not clear. One possibility is that, as expertise is acquired, the clinician moves from an analytical to an experiencebased approach to diagnosis.27 This maturational hypothesis suggests that different educational strategies may be required for experts and novices. For experts, it may require little more than legitimizing the notion that thoughtful departures from guidelines in light of individual patient circumstances are not

Academic Medicine, Vol. XX, No. XX / XXXX XXXX

just acceptable but are expected. On the other hand, some authors have advanced specific faculty development strategies to improve patient-centeredness.27 Although it is tempting to presume that the ultimate solution is to intervene with medical students, while they are still young and malleable, this may not be the case. As we indicated before, novice diagnostic reasoning is primarily analytical.27 Interventions directed at integrating patient contextual factors into management decisions may simply increase cognitive load28 and result in less optimal learning. These implications must, however, be viewed as speculative. As this study implies, our understanding of how to integrate contextual and patient factors into optimal decision making is poorly developed to date. Conclusions

CPGs are developed to assist physicians in determining the best course of care for patients. Although CPGs might provide opportunity to maximize outcomes at the population level, they are not designed to account for the unique needs of individual patients. The data presented here suggest that EM physicians are sensitive to both individual patient context and the best clinical evidence of benefit (as indicated by CPGs) when determining how care should be managed. Additional research is needed to determine the extent to which consideration of “context” does result in better patient-important outcomes. If deemed important, the “context” of patients should be considered when examining why physicians might not adhere to CPG recommendations. How this should be incorporated into medical training will require further study. Acknowledgments: The authors would like to thank Sharyn Kreuger for assistance developing the electronic survey and collecting data, and Shelley McLeod for assistance in recruitment. The authors also thank Dr. Michelle Welsford and Dr. John Opie for help in pilot testing the cases. Funding/Support: This study received funding support from the Program for Educational Research and Development at McMaster University. Other disclosures: None reported. Ethical approval: This study received approval from the McMaster University/Hamilton Health Sciences research ethics board.

Dr. Mercuri is a postdoctoral research fellow, Department of Medicine, Division of Cardiology, Columbia University, New York, New York. Dr. Sherbino is emergency medicine physician and associate professor, Department of Medicine, McMaster University, Hamilton, Ontario, Canada. Dr. Sedran is emergency medicine physician and associate professor, Department of Medicine, Division of Emergency Medicine, Western University, London, Ontario, Canada. Dr. Frank is associate director, Office of Education, Royal College of Physicians and Surgeons of Canada, and director of education and associate professor, Department of Emergency Medicine, University of Ottawa, Ottawa, Ontario, Canada. Dr. Gafni is professor, Department of Clinical Epidemiology and Biostatistics, Centre for Health Economics and Policy Analysis, McMaster University, Hamilton, Ontario, Canada. Dr. Norman is professor, Department of Clinical Epidemiology and Biostatistics, Program for Educational Research and Development, McMaster University, Hamilton, Ontario, Canada.

References 1 Field MJ, Lohr MJ. Clinical Practice Guidelines: Directions for a New Program. Washington, DC: National Academy Press; 1990. 2 Davis DA, Taylor-Vaisey A. Translating guidelines into practice. A systematic review of theoretic concepts, practical experience and research evidence in the adoption of clinical practice guidelines. CMAJ. 1997;157:408–416. 3 Woolf SH, Grol R, Hutchinson A, Eccles M, Grimshaw J. Clinical guidelines: Potential benefits, limitations, and harms of clinical guidelines. BMJ. 1999;318:527–530. 4 Audet AM, Greenfield S, Field M. Medical practice guidelines: Current activities and future directions. Ann Intern Med. 1990;113:709–714. 5 Chassin MR. Practice guidelines: Best hope for quality improvement in the 1990s. J Occup Med. 1990;32:1199–1206. 6 Cabana MD, Rand CS, Powe NR, et al. Why don’t physicians follow clinical practice guidelines? A framework for improvement. JAMA. 1999;282:1458–1465. 7 Cochrane LJ, Olson CA, Murray S, Dupuis M, Tooman T, Hayes S. Gaps between knowing and doing: Understanding and assessing the barriers to optimal health care. J Contin Educ Health Prof. 2007;27:94–102. 8 Lugtenberg M, Zegers-van Schaick JM, Westert GP, Burgers JS. Why don’t physicians adhere to guideline recommendations in practice? An analysis of barriers among Dutch general practitioners. Implement Sci. 2009;4:1–9. 9 Choudhry NK, Fletcher RH, Soumerai SB. Systematic review: The relationship between clinical experience and quality of health care. Ann Intern Med. 2005;142:260–273. 10 Andersen R, Newman JF. Societal and individual determinants of medical care utilization in the United States. Milbank Mem Fund Q Health Soc. 1973;51:95–124. 11 Ro KK. Patient characteristics, hospital characteristics and hospital use. Med Care. 1969;7:295–312.

5

Copyright © by the Association of American Medical Colleges. Unauthorized reproduction of this article is prohibited.

Research Report 12 Stiell IG, McKnight RD, Greenberg GH, et al. Implementation of the Ottawa ankle rules. JAMA. 1994;271:827–832. 13 Stiell IG, Wells GA, Vandemheen K, et al. The Canadian CT Head Rule for patients with minor head injury. Lancet. 2001;357:1391–1396. 14 McIsaac WJ, Goel V, To T, Low DE. The validity of a sore throat score in family practice. CMAJ. 2000;163:811–815. 15 Centor RM, Witherspoon JM, Dalton HP, Brody CE, Link K. The diagnosis of strep throat in adults in the emergency room. Med Decis Making. 1981;1:239–246. 16 Gage BF, Waterman AD, Shannon W, Boechler M, Rich MW, Radford MJ. Validation of clinical classification schemes for predicting stroke: Results from the National Registry of Atrial Fibrillation. JAMA. 2001;285:2864–2870. 17 Davis D, Goldman J, Palda VA. Canadian Medical Association Handbook on Clinical Practice Guidelines. Ottawa, Ontario, Canada: Canadian Medical Association; 2007. 18 Wilson MC, Hayward RS, Tunis SR, Bass EB, Guyatt G. Users’ guides to the

6

19

20

21 22

medical literature. VIII. How to use clinical practice guidelines. B. What are the recommendations and will they help you in caring for your patients? The EvidenceBased Medicine Working Group. JAMA. 1995;274:1630–1632. Brouwers MC, Browman GP, Burgers JS, et al. Appraisal of Guidelines for Research & Evaluation II: AGREE II Instrument. The Agree Research Trust: Canadian Institute of Health Research. 2009. http://www. agreetrust.org/resource-centre/agree-ii/. Accessed August 20, 2014. AGREE Collaboration. Development and validation of an international appraisal instrument for assessing the quality of clinical practice guidelines: The AGREE project. Qual Saf Health Care. 2003;12: 18–23. GRADE Working Group. Grading quality of evidence and strength of recommendations. Br Med J. 2004;328:1490–1498. Tonelli MR. The philosophical limits of evidence-based medicine. Acad Med. 1998;7C3:1234–1240.

23 Farquhar CM, Kofa EW, Slutsky JR. Clinicians’ attitudes to clinical practice guidelines: A systematic review. Med J Aust. 2002;177:502–506. 24 Durning SJ, Dong T, Artino AR Jr, et al. Instructional authenticity and clinical reasoning in undergraduate medical education: A 2-year, prospective, randomized trial. Mil Med. 2012;177;38–43. 25 Mercuri M, Birch S, Gafni A. Using smallarea variations to inform health care service planning: What do we “need” to know? J Eval Clin Pract. 2013;19:1054–1059. 26 Frankel RM, Eddins-Folensbee F, Inui TS. Crossing the patient-centered divide: Transforming health care quality through enhanced faculty development. Acad Med. 2011;86:445–452. 27 Eva KW. Stemming the tide: Cognitive aging theories and their implications for continuing education in the health professions. J Contin Educ Health Prof. 2003;23:133–140. 28 Mayer RE. Applying the science of learning to medical education. Med Educ. 2010;44:543–549.

Academic Medicine, Vol. XX, No. XX / XXXX XXXX

Copyright © by the Association of American Medical Colleges. Unauthorized reproduction of this article is prohibited.