Using administrative data to describe casemix: A comparison with the medical record

Pergamon

08954356(94)EOO56-U

J Clin Epidemiol Vol. 47, No. 9, pp. 1027-1032, 1994 Copyright 0 1994 Elsevier ScienceLtd Printed in Great Britain. All rights reserved

0895-4356194 $7.00+ 0.00

USING ADMINISTRATIVE DATA TO DESCRIBE CASEMIX: A COMPARISON WITH THE MEDICAL RECORD DAVID J MALENKA,‘S~* DALE MCLERRAN,’ NORALOU RGGS,~ ELLIOTT S. FISHER’,~.~ and JOHN E. WENNBERG”’ ‘The Center for the Evaluative Clinical Sciences and the Department of Community and Family Medicine, Dartmouth Medical School, Hanover, NH, *Department of Medicine, DartmouthHitchcock Medical Center, Lebanon, NH 03756, U.S.A., 3Department of Community Health Sciences, University of Manitoba, Winnipeg, Manitoba, Canada and 4White River Junction Veterans Administration Medical Center, White River Junction, VT, U.S.A. (Received in revised form 2 February 1994)

Abstract-We compared the coding of comorbid conditions in an administrative a database to that found in medical records for 485 men who had undergone prostatectomy. Only a few specific conditions showed good agreement between charts and claims. Most showed poor agreement and appeared more frequently in the chart. A comorbidity index calculated from each of these sources was used to explore the differences in mortality for patients who had undergone transurethral vs open prostatectomy. The claims-based comorbidity index most often underestimated the index from the chart. Proportional hazards analysis showed that models including either comorbidity index were better than those without an index and models with information from both indices were best. No analysis eliminated the effect of type of prostatectomy on long-term mortality. Claims-based measures of comorbidity tend to underrepresent some conditions but may be an acceptable first step in controlling for differences across patient populations. Administrative

database

Comorbidity

index

INTRODUCTION

Health care research using administrative databases has received increasing attention over the last decade [l]. A cause of concern in this work has been the question of whether observed differences among physicians, hospitals or treatments are confounded by differences in casemix that are not measurable in administrative claims data [2-51. In previous work using claims data, we reported an increased relative risk (RR) of dying

*All correspondonce should be addressed to: David J. Malenka, M.D., Section of Cardiology, DartmouthHitchcock Medical Center, Lebanon, NH 03756, U.S.A.

Prostatectomy

following transurethral resection of the prostate (TURP) vs open prostatectomy [6]. This remained true despite efforts to control for casemix with information from the claims [6] or medical records [7]. Our measure of casemix was based on the comorbidity index of Charlson et al. [8]. This index was developed for use with patient charts and modified for use with claims data. Therefore, for a subset of patients who underwent prostatectomy, casemix data was available from both medical records and claims. This presented the opportunity to examine how the identification of comorbid conditions through claims data compared to what was documented in the chart and to explore their merits in adjusting for casemix.

1027

1028

DAVID J. MALENKA et al. METHODS

The study population

Patients were initially identified through claims data as previously described [7]. All patients undergoing prostatectomy at the Health Sciences Centre, Winnipeg, Manitoba, between 1 January 1974, and 30 December 1980, were eligible for study provided they were over age 55 at the time of the operation and did not have a history of bladder or prostate cancer. Every patient who had undergone open prostatectomy (n = 248) and a random sample of one out of three patients who had undergone TURP (n = 268) were selected for study. Thirty-one patients were subsequently eliminated from the analysis (26 charts could not be found, two patients were woman, one patient had a history of prostate cancer and for one patient there was disagreement between the claims and the chart as to type of prostatectomy) leaving a study population of 485. Chart -based data

We abstracted data on a broad range of clinical variables from a patient’s history, physical exam and laboratory findings as documented during the admission for prostatectomy. These included the data required to calculate the weighted comorbidity index developed by Charlson et al. (Appendix Table Al, [8]) and an estimate of functional health, Karnofsky’s performance status. Both variables have been shown to be predictive of long-term mortality [8,9]. Only information available prior to the operation was used, though there was no formal attempt to blind abstracters to the remainder of the chart. Details of the chart abstraction have been reported [7]. Claims-based

data

Claim records were obtained from the Manitoba Health Services Commission. For each hospital discharge an abstract is submitted to the commission. It contains unique patient, hospital and physician identifiers, as well as information on diagnoses and surgical procedures. The records we used contained up to three diagnoses and three surgical procedures. Codes were selected from the International Classification of Diseases 8th Revision (ICD-8) for all claims submitted through 3 1 March 1979, and the 9th revision, Clinical Modification (ICD-9-CM) for subsequent claims. Claims were linked to construct a complete history of

hospitalization for each patient. Vital status was obtained from a federal registry. Casemix

A weighted comorbidity index was calculated according to the protocol of Charlson. Each comorbid condition present was assigned a weight and these weights were summed to give an index value (Appendix Table Al). This index was adapted for use with claims data by having three clinicians familiar with both health services research and administrative databases aggregate codes into groups representing, at face value, each of the comorbid conditions. (Groupings are available from the authors on request.) Information recorded on the discharge abstract from the hospitalization for prostatectomy which might have represented an outcome was not used in calculating the index. This included diagnoses of acute myocardial infarction (MI) and congestive heart failure (CHF). The chart-based index used only information abstracted from the admission for prostatectomy. All claims for hospitalizations from 1974 up to the admission for prostatectomy were used to generate the claims-based index. Analytic

methods

The intraclass correlation coefficient [lo] was used as the summary measure of agreement between the chart-based and claims-based index of comorbidity. Agreement among components of the comorbidity indices was measured using kappa [lo], where 0.0 indicates all observed agreement could be chance, values between 0.40 and 0.75 represent fair to good agreement beyond chance and 1.0 indicates perfect agreement. Survival analysis was conducted using Cox regression (SAS Proc PHGLM, [I 11). In these analyses the comorbidity index was used as an ordinal variable. Age was stratified into three groups (55-69, 70-74, 275). Based on previous work [7], Karnofsky’s performance status, scored on a loo-point scale in increments of 10, was categorized as ~70 and > 80. Follow-up for mortality was censored at 5 years. Nested models were compared by the likelihood ratio chi-square test statistic [12]. RESULTS

Table 1 presents the comorbidity index scores derived from charts and claims for the 485 patients who underwent prostatectomy. The claims-based comorbidity index underestimates

1029

Using Administrative Data to Describe Casemix

more often than overestimates the comorbidity index computed from the chart. There are 133 (27.4%) patients for whom the chart-based index is larger than the claims-based index. In contrast, only 45 (9.3%) patients have a claimsbased index greater than the chart-based index. The intraclass correlation measuring agreement between chart- and claims-based indices was only moderate (r = 0.551). There was no systematic difference in the relationship between the claims- and chart-based indices by age or type of operation (data not shown). Table 2 presents similar information for specific comorbid conditions. Diabetes and tumor show good agrement between the chart and the claims (K = 0.856, K = 0.780 respectively). Most other conditions show poor agreement and appear more frequently in the chart. Cerebrovascular disease is unique in that it appears with equal frequency in the chart and the claims but for different groups of patients (rc = 0.288). In identifying comorbid conditions from claims, we were careful not to use data that might represent an outcome. This was especially true for MI and CHF, for which the ICD-8 codes could not distinguish a pre-existing condition from an adverse outcome. However, the charts suggested these conditions were common and there was the possibility of introducing too conservative a bias by ignoring conditions appearing for the first time on the operative (index) admission. Table 2 shows that including index admission codes in the definition of MI and CHF led to no improvement in the agreement between charts and claims for the former and only modest improvement for the latter. Even though coding in the claims may not agree with the data in the charts, the claimsbased index may do as well as the chart-based

Table 1. Frequency of claims-based comorbidity index scores compared to chart-based comorbidity index scores Claims-based index score Chart-based index score 0 1 2 3 4 5 6 7 Total

0 230 60 20 3 0 0 102 1 315

1

2

3

4

5

7

Total

27 49 19 6 2 2

3 6 21 7 3 3

1 0

0 0

0

1

1 106

0 45

1 1 5 0 0100 5000 2000 0010 1 0 0000 14 2

2

1

263 120 61 21 7 6 5 2 485

Kappa = 0.364. Variance (kappa) = 8.95 x 10m4. Intraclass correlation coefficient, r = 0.551.

Table 2. Frequency of specific comorbid conditions mentioned on the chart, in the claims, or both

as

Mentioned on Variable Diabetes Tumor Cirrhosis CTD* Leukemia COPD* MI* MII* CHFI* Metastases CHF* PVD* CVD* Dementia Paralysis Ulcers Portal hyper* Diabetic camp* Renal disease Lymphoma

Chart only

Claims only

Both

Total

kappa

8 4 1 2 1 42 24 24 16 3 20 16 14 8 16 32 2 5 0 0

1 3 0 0 2 18 24 24 20 0 10 4 12 0 0 1 0 1 2 0

29 13 1 2 3 51 25 25 15 1 11 6 6 1 2 3 0 0 0 0

38 20 2 4 6 111 73 73 51 4 41 26 32 9 18 36 2 6 2 0

0.856 0.780 0.666 0.665 0.665 0.557 0.455 0.455 0.415 0.398 0.392 0.357 0.288 0.197 0.194 0.141 0.000 0.000 0.000 -

*CTD = connective tissue disease; COPD = chronic obstructive pulmonary disease; MI = myocardial infarction, not considered a pm-existing condition if it appeared in the claims for the first time during the operative admission; MI1 = myocardial infarction, considered a pre-existing condition if it appeared in the claims for the first time during the operative admission; CHFI = congestive heart failure, considered a preexisting condition if it appeared in the claims for the first time during the operative admission; CHF = congestive heart failure, not considered a pre-existing condition if it appeared in the claims for the first time during the operative admission; PVD = peripheral vascular disease; CVD = cerebrovascular disease; Portal hyper = portal hypertension; Diabetic camp = diabetic complications.

index in adjusting for casemix. Table 3 presents coefficients and measures of fit [12] of a Cox proportional hazards survival model using the chart-based index. Other variables in the model include age, Karnofsky score, and type of procedure (TURP vs open prostatectomy). This is compared with models using the claims-based index alone, the claims- and chart-based indices together, and the maximum of the two indices. The log likelihood of the model with the chartbased index is larger than the log likelihood of the model with the claims-based index. However, parameter estimates for the relative risks of the index and for TURP vs open prostatectomy are similar for both models. Based on log likelihoods, models that contain either the chart- or claims-based indices are significantly better than the model which does not include an index. The relative risk of TURP vs open prostatectomy is also lower when either the chart- or claims-based index is included in the model. A model with both the chart-based and

DAVID

1030

J. MALENKAet al.

claims-based indices included as independent variables is better than the model with the chart-based index alone (x2 = 5.65, df = I), though the relative risks associated with these indices may be imprecise because of collinearity. The same appears true of the model that uses the maximum of the chart- and claims-based indices, although the log likelihood for this model cannot be compared to log likelihoods for other models through a chi-square test.

DISCUSSION

Several methods have been developed to control for casemix in health services research [ 13-211. Some require chart abstraction [17-l 91 while others use routinely collected administrative information (such as claims data). Most [ 13-181 were designed to control for differences in resource consumption, not outcomes, and only a few [19-211 have been validated for the latter. Our goal in this study was to examine how a claims-based index of comorbidity compared to a chart-based index. We found that the claimsbased index tended to underestimate comorbidity compared to the chart-based index (Table 1) and most conditions contributing to the index were underrepresented in the claims (Table 2). The use of the chart-based index added slightly more to the prediction of mortality than did the claims-based index (Table 3). However, the claims may contribute information which is missing from the chart and models having access to both chart- and claims-based indices were the best predictors of mortality. Despite the availability of chart-based information, the relative risk estimate for mortality following

TURP vs open prostatectomy was not substantially altered. Roos et al. [21] also used a claims-based version of the Charlson comorbidity index and considered the extent to which risk prediction improved by adding information from one well known measure found on chart review, the American Society of Anesthesiologists’ (ASA) Physical Status score. They found that in most cases prediction was not improved by including the prospectively collected data. At least two factors may explain why their findings differ from ours. First, the ASA score may not contain as much clinically predictive information as the Charlson index abstracted from the medical record. Second, their claims-based measure of casemix contained some conditions that could be considered as outcomes (e.g. acute MI) and are strongly predictive of death. Including them in their model would have the effect of overestimating the predictive value of the claims. It is important to note the limitations of our study. It did not establish the validity of the claims data to assess for casemix because there was no “gold standard”. The chart review was limited to information available from the surgical admission for prostatectomy and did not include review of data from previous admissions or outpatient visits. Thus, the information from our review of the medical record cannot be considered as a definitive summary of a patient’s condition. More exhaustive review of the medical record [22] may demonstrate greater differbetween it and ences in comorbidities administrative claims. What we did find was that the claims-based index often underestimated comorbidity and was a significant

Table 3. Relative risks, confidence intervals, and measures of fit of four Cox proportional hazards models for survival using claims- and/or chart-based comorbidity index scores for risk adjustment Comorbidity Index

Model

TURP vs OPEN*

Relative risk1

Confidence interval

-2 Log likelihood

Relative risk

Confidence interval

Chart-based index

1.39

1.23, 1.57

1231.46

1.65

1.11, 2.45

Claims-based index

1.51

1.28, 1.78

1234.66

1.72

1.15, 2.55

Chart-based index Claims-based index

1.27 1.30

1.09, 1.46 1.06, 1.59

1225.52

1.63

1.10, 2.42

1.42 -

1.26, 1.60

1226.96

I .60

1.08, 2.38

1254.78

1.91

1.28, 2.83

Model?

Maximum of chart- and claims-based index Index not included

-

*Comparison of the risk of dying following transurethral resection of the prostate compared to open prostatectomy. TModels also contained age (5569, 70-74, a75), Karnofsky score (80), and type of operation (TURP vs OPEN). SIncreased risk associated with each unit increase in comorbidity score. Comparison group is patients with a 0 score.

UsingAdministrativeData to DescribeCasemix predictor of mortality whether or not the chartbased index was included in the model. Therefore, the use of a claims-based index was a reasonable first step in our attempts to adjust for casemix. The claims we studied were from Manitoba, Canada, used ICD-8 and ICD-9-CM codes, and were processed between 1974 and 1980. It is difficult to know how our results might generalize to the use of current claims data. The advent of systems of prospective payment has provided incentive for more complete coding of comorbid conditions and ICD-9-CM codes allow for better distinction of pre-existing problems from outcomes. Thus, it seems probable that current claims data should perform at least as well as the data we used in controlling for casemix. However, this hypothesis remains to be tested. Though the use of a claims-based index was a reasonable first step in attempting to control for casemix, it was not as good as the chartbased index. Given the unexpected finding of increased mortality following TURP vs open prostatectomy, it was important to use the best information available in our efforts to adjust for casemix. However, this may not always be the case. The improvement in model fit achieved through the chart-based index was small and did not alter our results. Moreover, the major limitation of observational research, the lack of randomization, affected both efforts at adjusting for casemix. It is important to note this was not the experience of Concato et al. [22] in their retrospective study of a Yale-New Haven prostatectomy cohort. They were able to lower the point estimate of relative risk for long-term mortality of TURP vs open prostatectomy to 1.03 by using information from the medical record. Unlike our study, they obtained and reviewed all previous medical records, both inpatient and outpatient. We reviewed only information from the surgical admission. They used the diagnoses listed on the discharge form (“face sheet”) from the index hospitalization to represent the claims data. We used all available claims data, giving us in some cases up to 8 years of past medical history. While we may not have obtained all the relevant information from the medical record, the same may be true of their use of claims data. It is possible that in their sample of patients, use of a more complete administrative database also would have reduced the relative risk of dying following TURP vs open prostatectomy.

1031

Our findings show that adjustments for comorbidity were not markedly altered when claims-based data were substituted for chartbased data. Because they are part of an administrative process, claims data are available for any cohort of hospitalized patients and for those over 65 years of age, many cohorts of outpatients (Medicare Part B claims). Their availability, together with our results, suggest that claim-based measures of comorbidity are a reasonable first step in controlling for differences in outcomes across patient populations and may in some circumstances, be adequate. Before this is assumed to be true, other studies must be done that use more current data, show no bias in coding across institutions, demonstrate adequate correlation with other measures of comorbidity and ulimately, show that results using claims-based data are consistent with the results from randomized, controlled trials. Acknowledgements-We are indebted to the Manitoba Health Services Commission, the personnel in the Medical Records Department at the Health Sciences Centre, Winnipeg, Manitoba, and to our nurse abstracters for the successful completion of this project. This work was supported in part by Grant 6607-1197-44 Research Program Directorate, Health and Welfare, Canada, and by Career Scientist Award 6604-1001-48. NPR is an associate of the Canadian Institute for Advanced Research and Director of the Manitoba Center for Health Care and Evaluation.

REFERENCES 1.

2. 3.

4.

5.

6.

I.

8.

9.

Roper WL, Winkenwerder W, Hackbarth GM, Krakauer H. Effectiveness in health care: an initiative to evaluate and improve medical practice. N Engl J Med 1988; 319: 1197-1202. Schroeder SA. Outcomes assessment 70 years later: Are we ready? N Engl J Med 1987; 316: 160-162. Dubois RW, Rogers WH, Moxley III JH, Draper D, Brook RH. Hospital inpatient mortality: Is it a predictor of quality? N Engl J Med 1987; 317: 16741680. Jencks SF, Williams DK, Kay TL. Assessing hospitalassociated deaths from discharge data. JAMA 1988; 260: 224&2246. Greenfield S, Aronow HU, Elashoff RM, Watanabe D. Flaws in mortalitv data. The hazards of ignorine comorbid disease. JAMA 1988; 260: 2253-2255. Roos NP, Wennberg JE, Malenka DJ et al. Mortality and reoperation following _ open re_ and transurethral section of the prostate for benign prostatic hypertroohv. N Enal J Med 1989: 320: 1120-1124. ‘Malenka DJ, Roos N, Fi$her ES et al. Further study of the increased mortality following transurethral prostatectomy: a chart-based analysis. J Urol 1990; 144: 224-228. Charlson ME, Pompei P, Alex KL, MacKenzie CR. A new method of classifying prognostic comorbidity in longitudinal studies: development and validation. J Chron Dis 1987; 70: 373-383. Stanley KE. Prognostic factors for survival in patients with inoperable lung cancer. J Nat1 Cancer Inst 1980; 65: 25-33.

1032

et al. DAVIDJ. MALENKA

10. Fleiss JL. Statistical Methods for Rates sod Proportions, 2nd edition. New York: Wiley; 1981: 212-236. 11. SAS Institute Inc. SUGI Supplemental Library User’s Guide, Version 5 Edition. Cary, NC: SAS Institute Inc.; 1986: 662. 12. Lee ET. Statistical Methods for Survival Data Analysis, 1st Edition. Belmont, CA: Lifetime Learning Publications; 1980: 298-337. 13. Fetter RB, Shin Y, Freeman JL, Averill RF, Thompsom JD. Case mix definition by diagnostic related groups. Med Care 1980; 18 (Suppl. 2): l-53. 14. Freeman JL, Fetter RB, Park H et al. DRG Refinement. In: Fetter RB, Ed. DRGs: Des@, Development and Applications. Health Administration Press; 1991: Chapter 3. 15. Gonnella JS, Hornbrook MC, Louis DZ. Staging of disease: a case-mix measurement. JAMA 1984; 251: 637644. 16. Young WW. Incorporating severity of illness and comorbidity in case-mix. Health Care Fin Rev 1984; 6 (Suppl): 23-3 1. 17. Horn SD, Horn RA, Sharkey PD. The severity of illness index as a severity adjustment to diagnosis-related groups. Health Care Fin Rev (Ann Suppl) 1984: 33-45. 18. Brewster AC, Karlin BG, Hyde LA, Jacobs CM. MEDISGRPS: A clinicallv based aouroach to classifving hospital patients at admission~‘Inqniry 1984; 23: 377-387. 19. Knaus WA, Draper EA, Wagner DP. APACHE II: A severity of disease classification system. Crlt Care Med 1985: 13: 818-829. 20. MO&~ JM, Ross Jr LL. Using insurance claims to measure health status; the illness scale. J Cbron Dis 1987; 40: 4lS-50s.

21. Ross LL, Sharp SM, Cohen MM, Wajda A. Risk adjustment in claims-based research: the search for efficient approaches. J Clin Epidemiol 1989; 42: 1193-1206. 22. Concato J, Horwitz RI, Feinstein AR, Elmore JG, Schiff SF. Problems of comorbidity in mortality after prostatectomy. JAMA 1992;- 267: 1077-1082.

APPENDIX Al. Charlson’s weighted-index of comorbidity* [S] Assigned weights for characteristics 1

2

3 6

Characteristic Myocardial infarction Congestive heart failure Vascular disease Cerebrovascular disease Pulmonary disease Connective tissue disease Ulcer disease Diabetes Paralysis Moderate or severe renal disease Diabetes with end organ damage Solid tumor Leukemia Lymphoma Moderate or severe liver disease Metastatic solid tumor

*Assigned weights for each condition that a patient had. The total equals the score. Example: pulmonary disease (1) and diabetes (2) = total score (3).