Research Article Received 8 September 2011,
Accepted 23 April 2012
Published online in Wiley Online Library
(wileyonlinelibrary.com) DOI: 10.1002/sim.5442
Mixed treatment comparisons using aggregate and individual participant level data Pedro Saramago,a * † Alex J. Sutton,b Nicola J. Cooperb and Andrea Mancaa Mixed treatment comparisons (MTC) extend the traditional pair-wise meta-analytic framework to synthesize information on more than two interventions. Although most MTCs use aggregate data (AD), a proportion of the evidence base might be available at the individual level (IPD). We develop a series of novel Bayesian statistical MTC models to allow for the simultaneous synthesis of IPD and AD, potentially incorporating study and individual level covariates. The effectiveness of different interventions to increase the provision of functioning smoke alarms in households with children was used as a motivating dataset. This included 20 studies (11 AD and 9 IPD), including 11 500 participants. Incorporating the IPD into the network allowed the inclusion of information on subject level covariates, which produced markedly more accurate treatment–covariate interaction estimates than an analysis solely on the AD from all studies. Including evidence at the IPD level in the MTC is desirable when exploring participant level covariates; even when IPD is available only for a fraction of the studies. Such modelling may not only reduce inconsistencies within networks of trials but also assist the estimation of intervention subgroup effects to guide more individualised treatment decisions. Copyright © 2012 John Wiley & Sons, Ltd. Keywords:
evidence synthesis; individual patient data; aggregate data; meta-analysis; mixed treatment comparisons; heterogeneity
1. Introduction As a consequence of the move towards evidence-based healthcare, with its underlying principle that evidence synthesis must be seen as the key to more coherent and efficient research [1], it is necessary to systematically identify and consider evidence from all the relevant studies [2]. In many cases the synthesis of evidence is conducted using pair-wise meta-analysis [3, 4]; a quantitative process that uses summary results (or aggregate data, AD) of studies accessible in the published literature to produce a numerical summary of the overall/pooled effect of an intervention. The methods for meta-analysis are extensively described in the literature [4]. Generally, these use either fixed-effect or random-effect approaches, where estimates from the latter approach are conditional to the assumption that, in addition to sampling error, differences between studies exist because of variations in their design and conduct, and this is often termed between-study heterogeneity. Meta-regression techniques can be further used to explore any apparent between-study heterogeneity, with the aim of estimating treatment– covariate interactions [5, 6]. Methods relying on aggregate information on patient-level covariates have been shown to have low statistical power [7] and to be highly susceptible to ecological fallacy biases [5]. Another way of obtaining combined statistics is to perform meta-analyses over individual patient or participant level data (IPD) [8]. Given that IPD meta-analysis overcomes the above mentioned disadvantages associated to the synthesis of AD [6, 7, 9], this approach has been considered the ‘gold standard’ for meta-analysis. Individual participant data is, however, usually available for only a (small) proportion
a Centre
for Health Economics, University of York, York, U.K. of Health Sciences, University of Leicester, Leicester, U.K. *Correspondence to: Pedro Saramago, Centre for Health Economics, The University of York, Alcuin A, York YO10 5DD, U.K. † E-mail:
[email protected] b Department
Copyright © 2012 John Wiley & Sons, Ltd.
Statist. Med. 2012
P. SARAMAGO ET AL.
of all relevant studies. In an attempt to include all available data in the analysis, recent extensions to this modelling framework have been developed, which allow IPD and AD to be used jointly to estimate the effects of a treatment [10, 11]. Standard approaches to meta-analysis usually consider only pair-wise comparisons. However, in most medical conditions multiple interventions are available, and clinicians and policy makers need to decide on the optimal strategy among all relevant alternatives. Recent extensions to standard pair-wise metaanalysis are indirect comparisons and mixed treatment comparisons (MTCs) (sometimes called network meta-analysis) [12, 13]. These methods combine evidence on multiple alternative interventions, and inform treatment comparisons that may not have been trialled head-to-head; without breaking randomization. Several assumptions are imposed by the MTC approach: (i) such methods can only be applied to connected networks of studies; (ii) the treatment effects are thought to be generalisable across patients from trials included in the network; and (iii) in the presence of evidence loops consistency across the evidence base must exist [14–18]. Inconsistency happens when there are discrepancies between direct and indirect evidence and this may be linked to differences in study-level covariates acting as treatment-effect modifiers in a similar manner to those that cause between study heterogeneity in pair-wise meta-analysis. As with any meta-analysis, in MTCs it is desirable to account for heterogeneity/inconsistency, otherwise results may be biased [14]. Also, the identification of factors contributing to heterogeneity/inconsistency may be valuable clinically because the optimal treatment strategy may vary across different patient groups. Thus, if treatment recommendations are made for subgroups of patients, this will lead to efficiency gains when compared with the suboptimal framework of decisions based on overall (mean) effectiveness. Despite the advantages of IPD for exploring heterogeneity/inconsistency, IPD has rarely been used in the context of MTC. This paper considers synthesis models for a binary outcome where AD and IPD are available (although direct simplifications of the model allow the analysis of just AD or just IPD), and where patient level covariates may be of interest (although study level covariates can also be incorporated). Models are fitted to a motivating dataset on uptake of smoke alarms to prevent accidents in preschool children. After describing the motivating dataset in Section 2, Section 3 outlines existing and novel models for the MTC of AD, IPD, and AD and IPD simultaneously. Models that estimate mean intervention effects, ignoring the influence of the covariates, are considered first. These are followed by MTC models that incorporate both AD and IPD data allowing for both individual and aggregate study level covariate data to be included to estimate treatment–covariate interactions. Results of applying the described methods to the motivating dataset are described in Section 4, which is followed by some discussion topics and concluding remarks in Section 5.
2. Motivating example: The effectiveness of home safety education and the provision of safety equipment for the prevention of accidents in preschool children and the impact of socioeconomic characteristics The motivating example for this paper comes from an evaluation of the effectiveness of home safety education and the provision of functioning smoke alarm safety devices for the prevention of accidents in preschool children. Studies were identified as part of a more general and updated Cochrane systematic review of safety equipment [10, 19]. The primary intention of this review was to obtain IPD from all relevant studies and to subsequently synthesise these in a meta-analysis. Unfortunately, the investigators were only successful in obtaining IPD for a proportion of the studies. Additional details on the included studies can be found in related literature [10, 11, 19, 20]. The review included nonrandomised and randomised controlled trials (RCTs), and controlled beforeand-after studies. Although the initial Cochrane review [10] identified much of the relevant literature base, it did not consider comparative studies that did not have usual care as a comparator (e.g. studies comparing ‘smoke alarms education’ vs ‘smoke alarms education plus low-cost/free fitted equipment’, etc.). Therefore, a supplementary systematic review of existing reviews was conducted [21] to identify further relevant ‘head-to-head’ primary studies that could be included in a network analysis. The exploration of (binary) participant-level socioeconomic characteristics was of primary interest for both reviews because there were concerns that the effectiveness of the interventions under review was dependent on socioeconomic status. As a case study for this paper, the outcome measure assessed was the provision of a functioning smoke alarm (binary — Yes/No) given different interventions designed to increase its prevalence in households Copyright © 2012 John Wiley & Sons, Ltd.
Statist. Med. 2012
P. SARAMAGO ET AL.
with children. The following relevant evidence was used: 9 studies available in IPD format [22–30] and 11 available in AD format [31–41], summing up to approximately 11 500 participants. Seven out of nine available studies in IPD format and 7 out of 11 in AD format were RCTs. Also, two of the IPD studies and six of the AD studies were cluster-allocated trials, but in none of these had the cluster design been accounted for in the original analysis. In all subsequent analyses, the effect of the clustering was modelled for the IPD. For the summary study estimates an approximate adjustment was made through the inflation of treatment effect variances prior to all modelling described below (Sutton et al. [11] made cluster adjustments to the AD within the modelling; this has the advantage of allowing for the uncertainty in the estimation of the intraclass correlation coefficient and could be pursued within an MTC framework if desired. See also Kendrick et al. [10] for further details). Information for these studies is provided in Table I. Seven implementation strategies are defined across the available evidence base, namely: (1) (2) (3) (4) (5) (6) (7)
Usual care (UC) Education (E) Education plus low cost/free safety equipment (E C FE) Education plus low cost/free safety equipment plus home inspection (E C FE C HI) Education plus low cost/free safety equipment plus fitting (E C FE C F) Education plus home inspection (E C HI) Education plus low cost/free safety equipment plus fitting plus home inspection (E C FE C F C HI).
The resulting evidence network is presented in Figure 1. An MTC analysis of this data ignoring the IPD has been published elsewhere [20]. Notice that there are as many comparisons with (1) UC strategy as between non-UC interventions. Out of the 10 direct comparisons expressed by the network, 6 are informed by at least 1 study available in IPD format (continuous line). Figure 1 also indicates (dashed line) the network structure of evidence with information on the covariate of interest — household parent status (i.e. two or single parent household). From the set of covariates thought to be associated with the risk of injuries in children and assed in the above mentioned systematic reviews, single parent status was chosen as a covariate in our exercise. However, because of missing data, a smaller network is obtained (i.e. only six interventions are included in the comparison) when including studies that have information on this covariate, because there is no covariate information available for studies assessing intervention (7) E C FE C F C HI (i.e. the intervention of highest intensity). The implications of these data restrictions/limitations are considered in Section 5.
3. Methods for estimating a pooled treatment effect — indirect comparisons and mixed treatment comparisons This section reviews existing MTC methodology, which we extend in future sections. First, two simple hypothetical cases will be used to exemplify an indirect treatment comparison and an MTC. Figure 2(a) shows a network formed by pair-wise comparisons between three interventions: A, B and C. Direct head-to-head evidence is available for comparisons of A vs C and B vs C. There is no direct evidence between A and B (no solid line connects these two treatments). Treatments A and B are indirectly linked through C, which is the common comparator in this diagram. An indirect comparison estimate of A and B can be derived using evidence from A vs C and B vs C trials. On the log-odds ratio scale, an estimate of A vs B comparison, dAB , can be derived from existing evidence [42]: dAB D dCB dCA . Figure 2(b) represents the case when four interventions are of interest: A, B, C and D. Not only do we have direct evidence available for the comparisons C vs A and C vs B, but also for A vs B and A vs D. Given the network of the existing direct evidence, we can derive indirect estimates for C vs A, C vs B, A vs B and B vs D, using the same rationale as before. In fact, except for some cases (i.e. B vs D and C vs D), Figure 2(b) shows that both types of evidence (i.e. direct and indirect) are available for most pair-wise comparisons. When analysing data in this way, as for standard meta-analysis, a decision needs to be made as to whether each trial is assumed to provide an estimate of exactly the same quantity (fixed effect) or if the studies included in the meta-analysis provide estimates that are realisations from a common distribution of possible outcomes, which are exchangeable (random effect). Such an approach generalises to networks of any complexity; see elsewhere for further details [43, 44]. Copyright © 2012 John Wiley & Sons, Ltd.
Statist. Med. 2012
Copyright © 2012 John Wiley & Sons, Ltd.
NRT
RCT
RCT
RCT
CBA
RCT
RCT
RCT
Miller et al., 1982
Barone, 1988
Johnston et al., 2000
Gielen et al., 2002
Mock et al., 2003
Harvey et al., 2004
Clamp et al., 1998
Hendrickson, 2002
RCT
NRT CBA RCT RCT RCT
Matthews, 1988 Schwars et al., 1993 King et al., 2001 Sangvai et al., 2007 Gielen et al., 2001
Sznajder et al., 2003
Study design
Study lead author, year
IA
IA
IA
CA
CA
CA
CA
CA
CA
IA IA IA IA CA
Allocation type
IPD
IPD
IPD
AD
AD
AD
AD
AD
AD
AD AD AD AD AD
Data available
26/40
71/82
10/297 (2.33/69.18)
5/10 54/56 (52.02/53.95) 46/105 (9.34/21.31)
816/1060
6/50
18/308 (3.03/71.74)
34/38 (20.08/22.45)
77/80 (74.18/77.07)
81/83
47/56 (44.2/52.66)
61/108 (12.38/21.92) 39/41 (23.04/24.22)
37/38
997/1545 (781.6/61211.2)
211/211 (20.05/21.15) 47/58 (44.2/54.54)
394/469 16/17
6/12
27/47
136/143 (31.07/31.14)
406/482
6/12
1421/1583 (1114.0/1241.0)
866/902
Strategies [number of participants with functioning smoke alarms / total number of participants (number adjusting for clustering in parenthesis)] (1) UC (2) E (3) E C FE (4) E C FE C (5) E C (6) E C HI (7) E C FE C HI FE C F F C HI
0.103 (0.122; 0.084) 0.244 (0.132; 0.350) 0.135 (0.120; 0.152)
NA
NA
0.573 (0.486; 0.638 NA
NA
NA NA NA NA 00.848 (0.830; 0.860) NA
Singleparent household (by treatment arm) - %
Table I. Available evidence on interventions seeking to increase the ownership of functioning smoke alarm safety equipment to prevent thermal injuries (burns) in children.
P. SARAMAGO ET AL.
Statist. Med. 2012
Copyright © 2012 John Wiley & Sons, Ltd.
RCT
RCT
NRT
RCT NRT
RCT
Watson et al., 2005
Gielen et al., 2007
Bulzacchelli et al., 2009
Phelan et al., 2010 Kendrick et al., 1999
DiGuiseppi et al., 2002
CA
IA CA
IA
IA
IA
Allocation type
IPD
IPD IPD
IPD
IPD
IPD
Data available
112/138 305/339 (233.4/259.4) 5/30 (5/30)
55/71
325/375
619/737
109/139
345/384
341/385 (260.9/294.6) 8/44 (8/44)
692/764
130/140
0.264 (0.255; 0.273) 0.695 (0.720; 0.669) 0.719 (0.761; 0.698) NA 0.108 (0.104; 0.112 NA
Strategies [number of participants with functioning smoke alarms / total number of participants (number adjusting for clustering in parenthesis)] (1) UC (2) E (3) E C FE (4) E C FE C (5) E C (6) E C HI (7) E C FE C SingleHI FE C F F C HI parent household (by treatment arm) - %
Notes: UC, usual care; E, education; E C FE, education plus low cost / free equipment; E C FE C HI, education plus low cost / free equipment plus home inspection; E C FE C F, education plus low cost / free equipment plus fitting; E C HI, education plus home inspection; E C FE C F C HI, education plus low cost / free equipment plus fitting C home inspection; NRT, non-randomised trial; CBA, controlled before and after trial; RCT, randomised controlled trial; IA, participants are individually allocated; CA, participants are cluster allocated; AD, aggregate data; IPD, individually participant data.
Study design
Study lead author, year
Table I. Continued.
P. SARAMAGO ET AL.
Statist. Med. 2012
P. SARAMAGO ET AL.
Section 3.1 below outlines a standard random effects MTC model for a binary outcome [42], which is expanded upon in future sections (i.e. we do not pursue fixed effects approaches here but this could be attained with straightforward simplifications of the model). As conducted by previous authors [42], because of the flexibility of the modelling allowed, all models described below are fitted using Bayesian MCMC methods as implemented in the software WinBUGS (Imperial College and MRC, Cambridge, U.K.) [45]. All unknown parameters require prior distributions within a Bayesian paradigm, and are given prior distributions, which are intended to be vague throughout.
Figure 1. Network diagram for the functioning smoke alarm outcome with information on the number and format of evidence available for each treatment comparison (continuous line). Additionally, information on the number and format of evidence available for single parent status is also displayed (dashed line).
(a)
dCA
C
dCB dAB
A
dAB = dCB – dCA
B C
(b)
A B dCA dCB dAB
C
dAD
A
B
dCA = dCB – dAB dCB = dCA – dAB dAB = dCB – dCA dBD = dAD – dAB dCD = dCA – dAD
D
C
A B D
Figure 2. Diagrams of evidence structures: (a) indirect comparison of intervention A and B given studies on the comparisons of CA and CB, and (b) network of studies reflecting mixed treatment comparisons of CA, CB, AB and AD trials.
Copyright © 2012 John Wiley & Sons, Ltd.
Statist. Med. 2012
P. SARAMAGO ET AL.
3.1. Synthesizing aggregate data only Following the approach of Ades et al. [42] a random-effects MTC model for binary outcome data, using only AD, can be written as
(
logit pj k D ıjbk
rj k Bi n pj k ; nj k jb
b D A; B; C; : : : : if k D b
jb C ıjbk if k alphabetically after b N dbk ; 2 N dAk dAb ; 2
(1)
where rj k , denotes the number of observed events, and nj k the total number of individuals in the k th treatment arm of the j th trial. The underlying probabilities of an event for each arm in each trial are represented by pj k . The quantity jb represents the log-odds of an event for treatment b in study j , and dbk is the log odds ratio for treatment k relative to the study-specific baseline treatment b. Each ıjbk , the log odds ratio for treatment k relative to treatment b in trial j , is assumed to be normally distributed with mean dbk and variance 2 . Prior distributions need to be specified for jb and d N 0; 106 and for . U nif .0; 2//. Note that dAA D 0. This synthesis model requires modifications to incorporate trials with three or more arms because the model must take into account the correlation structure between arms of the same trials. These alterations rely on the use of the multivariate normal distribution for the intervention effects (see elsewhere for details of implementation [43]). 3.2. Synthesizing individual participant-level data only A random-effects model for MTC using IPD only can be written as
logit pij k D ıjbk
(
Yij k Bernoulli pij k jb
b D A; B; C; : : : : if k D b
jb C ıjbk if k alphabetically after b N dbk ; 2 N dAk dAb ; 2
(2)
For each study, the binary response of the i th participant, in the k th treatment arm of the j th study, Yij k (i.e. 1 D event, 0 D no event), is assumed to follow a Bernoulli distribution with probability of the event of interest occurring of pij k . As above, a standard logistic regression is fitted to each participant i of the j th trial, with jb , representing the log-odds for the control group (baseline b) in study j . The ıjbk , derived from the treatment group indicator for each participant i, is the log odds ratio for treatment k relative to the study-specific (j ) baseline treatment b. Prior distributions for jb , d and are specified as in model (1). A similar modification to allow for studies with more than two arms, akin to that in Section 3.1, is straightforward and can also be applied to all subsequent models described. We are unaware of any published applications of the IPD MTC model as in model (2). 3.3. Synthesizing individual-level and aggregate-level data One practical limitation of carrying out an IPD-only network meta-analysis is that it relies on the availability of IPD datasets for all studies. If this is not possible, one may be faced with the difficulty of having to statistically synthesize evidence in two different formats (i.e. IPD and AD‡ ). The model described here follows the Sutton et al. [11] approach to pair-wise meta-analysis, which allows for the additional level of complexity introduced by different study designs because the motivating example here also includes
‡
Alternative options used in the field of meta-analysis are (a) reduce IPD to AD and then conduct a standard AD metaanalysis; (b) simulate IPD from AD and do an IPD meta-analysis. Either of these is not without limitations (please see Riley and colleagues review paper [53]).
Copyright © 2012 John Wiley & Sons, Ltd.
Statist. Med. 2012
P. SARAMAGO ET AL.
different designs (i.e. cluster and individual allocation trials). This model can be viewed as an integration of the previous two models, (1) and (2), with all notation conventions remaining the same. The model is described in four interrelated parts. Thus, a random-effects MTC model for the combination of IPD and AD evidence for binary outcomes, allowing for different allocation procedures, can be written as: (3) Part I: Model for individually allocated IPD studies
logit pij k D
(
Yij k Bernoulli pij k IPD jb IPD jb
C ıjbk
b D A; B; C; : : : if k D b if k alphabetically after b
(3.1)
6 IPD jb N 0; 10 For i D 1; 2; : : : ; number of participants in the j t h individually allocated IPD study; j D 1, 2, . . . , number of individually allocated IPD studies; and k D 1; 2; : : : ; number of treatments for which participants were allocated to. Part II: Model for cluster allocated IPD studies
logit pi mj k D
Yi mj k Bernoulli pi mj k ( c:IPD b D A; B; C if k D b mjb c:IPD C ıjbk mjb
if k alphabetically after b
(3.2)
2 c:IPD mjb N j ; :cj For i D 1; 2; : : : ; number of participants in the mt h cluster of the j t h cluster-allocated IPD study; m D 1; 2; : : : ; number of clusters in the j t h study; j D (number of individually allocated IPD studies C 1), . . . , (number of individually allocated IPD studies C number of cluster allocated IPD studies); and k D 1, 2, . . . , number of treatments for which participants were allocated to. While Equation (3.1) is used for individually allocated IPD studies, (3.2) allows for clustering effects within studies. In Equation (3.2) a separate unconstrained control group odds for each cluster is estimated on the logit scale within each study, c:IPD , assuming that these are exchangeable within each study, mjb 2 with mean j and variance :cj — both these parameters requiring the specification of (vague) prior dis tributions N 0; 106 and U nif .0; 10/; respectively . Cluster effects between studies are assumed to be independent. Part III: Model for both cluster and individually allocated AD studies rj k Bi n pj k ; nj k ( AD b D A; B; C; : : : : if k D b jb logit pj k D AD C ıjbk if k alphabetically after b jb
(3.3)
For j D .number of individually allocated IPD studies C number of cluster allocated IPD studies C 1/ : : : .total number of studies/. Equation (3.3) combines both individually and cluster allocated studies in AD because premodel data adjustments were made to appropriately inflate the treatment effect variances for the effects of clustering in the cluster-allocated trials. This inflation may be made by estimating the intraclass correlation coefficient, as previously demonstrated in related literature [46]. A prior distribution needs to be specified for 6 AD N 0; 10 . jb Part IV: Combination of estimates of the intervention effect ıjbk N dbk ; 2 N dAk dAb ; 2
(3.4)
For k D 1 : : : ; total number of treatments. Copyright © 2012 John Wiley & Sons, Ltd.
Statist. Med. 2012
P. SARAMAGO ET AL.
Equation (3.4) specifies a random-effect to be placed across all treatment effect estimates from the IPD and AD, ıjb ’s, imposing the exchangeability property. In this way synthesis across both types of data is achieved because Equations (3.1), (3.2) and (3.3) ‘share’ parameters. Prior distributions need to be specified for d N.0; 106 / and . U nif .0; 2//. We note that for an analysis such as this, which does not consider participant level covariates, no loss of information is expected when using AD compared with IPD and while such a model may seem over-elaborate in this circumstances, it is no longer the case when covariate information is included, as we outline in the next section. 3.4. Mixed treatment comparisons models: including covariates The inclusion of study-level covariates may explain some of the between-study heterogeneity and reduce inconsistency in the network [14]. Of course, treatment-effect associations may be explored using AD and average study level covariates. However, this approach is potentially problematic because derived associations may occur purely by chance, as a result of the presence of confounding factors, or ecological bias [5]. Meta-regression techniques can be translated into the MTC situation in quite a straightforward way, although the disadvantages persist [47–49]. Cooper et al. [14] considered three different assumptions that can be made about treatment-covariate interactions, namely: (i) they are independent for every treatment in the network; (ii) they are the same for all treatments in the network; and (iii) they are assumed exchangeable for all treatments in the network. Option (i) is the least stringent but requires more data; option (ii) makes the strongest assumption, while option (iii) is a ‘half-way-house’ between the first two, where interactions can be different across treatments but they borrow strength from one another. When data availability is expected to be limited — as it is in the motivating example — option (iii) may be most appealing and is pursued below. Riley and Steyerberg [50] have shown that IPD allows for the modelling of treatment-covariate interactions using both within-study and between-study variability, and such variability can be partitioned to produce a meta-regression and a ‘pure’ IPD estimate of the interaction of interest. These estimates can subsequently be merged into an overall interaction estimate if deemed appropriate. The following MTC model is described in parts and extends model (3) for the combination of IPD and AD evidence, considering (i) individual level covariate values for the IPD, (ii) study-level covariate information for AD, (iii) exchangeable covariate-treatment interactions, and (iv) the partitioning of the variability regarding the interactions (extending the framework used by Riley and Steyerberg [50] to the MTC setting). Alternative assumptions could be accommodated with relatively straightforward modifications to the model specification. (4) Part I: Model for individually allocated IPD studies including covariates Yij k Bernoulli pij k 8 C ˇ0j xij b D A; B; C; : : : if k D b IPD ˆ ˆ jb < IPD jb C ıjbk C ˇ0j xij C if k alphabetically after b logit pij k D ˆ ˆ : B W Cˇbk x j C ˇbk xij x j 6 IPD jb N 0; 10
(4.1)
For i D 1; 2; : : : ; number of participants in the individually allocated j t h IPD study; and j D 1; 2; : : : ; number of individually allocated IPD studies Part II: Model for cluster allocated IPD studies including covariates Yi mj k Bernoulli pi mj k 8 C ˇ0j xi mj b D A; B; C if k D b c:IPD ˆ mjb ˆ ˆ < C ıjbk C ˇ0j xi mj C if k alphabetically after b c:IPD logit pi mj k D mjb ˆ ˆ ˆ : B W x j C ˇbk xi mj x j Cˇbk 2 c:IPD mjb N j ; :cj
Copyright © 2012 John Wiley & Sons, Ltd.
(4.2)
Statist. Med. 2012
P. SARAMAGO ET AL.
For i D 1; 2; : : : ; number of participants in the mt h cluster of the j t h cluster allocated IPD study; m D 1; 2; : : : ; number of clusters in the j t h study; and j D (number of individually allocated IPD studies C1), . . . , (number of individually allocated IPD studies C number of cluster allocated IPD studies). There are three extra terms in (4.1) and (4.2) compared with (3.1) and (3.2), namely: (i) a studyspecific individual-level covariate regression term, ˇ0j xij (or ˇ0j xi mj for cluster-allocated studies), where ˇ0j is the main covariate effect and xij (xi mj ) refers to the value for the binary covariate in th th the i th participant (in the of the m cluster) j study; (ii) an interaction term to account for the withinW W study interaction, ˇbk xij x j (or ˇbk xi mj x j for cluster-allocated studies), where within-study relationship is modelled by centring xij (xi mj ) about the mean covariate value, x j , in each study and W ˇbk is assumed different for each (active) W treatment vs control comparator but exchangeable throughW W N ˇAk ˇAb ; B2 W , needing a prior distribution to be specified for out all IPD studies (i.e. ˇbk B x j , by B W . U nif .0; 2//); and (iii) an interaction term to model the between-study relationship, ˇbk W B interacting with the mean covariate value, where, like ˇbk , ˇbk is also assumed different but exchangeW able. While within-study relationships may only be estimated through IPD and are captured by ˇbk , indicating variations in an individual’s event risk for a change in xij (xi mj ), the between-study relaB tionships may be estimated by both IPD and AD and are captured by ˇbk , denoting the variations in underlying mean event risk for a change in x j (the mean covariate value for the j th study). The difference between these two terms represents an estimate of the ecological bias [50]. In (4.2) both j and :cj require the specification of vague prior distributions N 0; 106 . Part III: Model for both cluster and individually allocated AD studies including covariates rj k Bi n pj k ; nj k 8 AD < jb logit pj k D B : AD C ı jbk C ˇbk Xj jb
b D A; B; C if k D b (4.3) if k alphabetically after b
j D (number of individually allocated IPD studies C number of cluster allocated IPD studies C 1), . . . , (total number of studies) As in model (1), the likelihood contribution, rj k , is described in the usual way. One additional term B was included, ˇbk Xj , which represents a study-level specific covariate regression term for treatment k relative to the study-specific baseline treatment b for each j trial. This term is equivalent to the exchangeable interaction term estimated in the two IPD statistical models above, which modelthe between-study B relationship, the ˇbk N 0; 106 . x j . Once again, a vague prior distribution is specified for AD jb Part IV: Combination of estimates of intervention effect including covariates B ıjbk N dAk C dAb C ˇbk Xj ; 2 B B B N ˇAk ˇAb ; B2 B ˇbk
(4.4)
k D 1 : : : ; total number of treatments In Equation (4.4), the study-specific individual-level covariate regression terms, ˇ0j ’s, are given vague B prior distributions N.0; 106 / . Exchangeability of ˇbk is declared, with prior distributions needing to be specified for B B . U nif .0; 2//. If independent treatment interactions were desired across the W B network, it would simply require defining ˇbk and ˇbk as following normality with mean 0 and large W B variance. Extension to multiple regression coefficients is straightforward. Note that dAA ; ˇAA ; ˇAA D 0.
4. Application Using the case study described in Section 2, we fit the models considered in Section 3 sequentially, ‘building up’ to the full complexity of model (4) to the outcome of interest ‘possession of a functioning smoke alarm’. Initially, we compare the results of the MTC models for IPD, with AD plus IPD, including Copyright © 2012 John Wiley & Sons, Ltd.
Statist. Med. 2012
P. SARAMAGO ET AL.
the IPD where available, ignoring covariates. We then consider the binary covariate ‘single parent household’ as a potential treatment modifier. This covariate is binary (0 D single parent household; 1 D two parent household) for each individual in the IPD and a percentage when referring to the proportion of single parent households in a study for the AD studies. Given the relatively small number of trials available, we primarily specify the model with exchangeable interaction coefficients separating the estimated treatment–covariate interaction effects from the within-study and between-study information. This is followed by a model with overall exchangeable interaction coefficients and comparisons between (overall) exchangeable and independent interaction effects. These later models are slight modifications of model (4) in which within-study and between-study associations were merged into an overall parameter. Unless stated otherwise, for all models presented, the MCMC sampler was run for 10 000 iterations and these were discarded as ‘burn-in’. Models were run for a further 5000 iterations on which inferences are based. Chain convergence was checked on all presented posterior sample summaries including checking stability across distinct sets of initial values. The WinBUGS code used for model (4) of the synthesis of AD plus IPD including the binary covariate is provided in the Appendix with specific prior distributions used for this example here. 4.1. Analysis of example without covariates Table II shows parameter estimates obtained for the novel approaches without covariates. The first column of results relates to analysis that combines the nine IPD studies using model (2); that is, this is akin to doing an IPD synthesis with the policy of excluding studies for which IPD could not be obtained. On the second results column, and through the use of model (3), all 20 studies are synthesised, using IPD where available (nine studies) and AD where not (11 studies).
Table II. Parameter estimates from fitting different MTC synthesis models without including covariates to the functioning smoke alarm outcome data Model (2) 9 studies included
Model (3) 20 studies included
Random effects MTC of IPD
Random effects MTC of AD and IPD
Median of MCMC posterior sample
95% credible interval
Median of MCMC posterior sample
E E C FE E C FE C HI E C FE C F E C HI E C FE C F C HI 2
0:297 2:036 1:130 0:930 — 1:165 1:677
2:045 to 1:425 1:074 to 5:338 0:86 to 3:391 0:843 to 2:651 — 1:732 to 3:886 0:347 to 3:812
0:130 1:125 0:956 0:962 1:169 1:938 0:651
Odds ratios for intervention effects (vs usual care)
E E C FE E C FE C HI E C FE C F E C HI E C FE C F C HI
0:743 7:664 3:096 2:534 — 3:205
Deviance Information criteria
DIC
Parameter (or funcation of) Log odds ratios for intervention effects (vs usual care) Between-study variance
0:129 to 4:158 0:342 to 208:15 0:423 to 29:69 0:431 to 14:171 — 0:177 to 48:698 2937:67
0:878 3:080 2:601 2:618 3:220 6:944
95% credible interval 1:116 to 0:82 0:05 to 2:427 0:033 to 2:177 0:235 to 2:171 0:407 to 3:167 0:827 to 3:158 0:151 to 2:362 0:328 to 2:271 0:952 to 11:328 1:033 to 8:823 0:791 to 8:768 0:666 to 23:743 2:286 to 23:52 3059:49
Notes: education; E C FE, education plus low cost / free equipment; E C FE C HI, education plus low cost / free equipment plus home inspection; E C FE C F, education plus low cost / free equipment plus fitting; E C HI, education plus home inspection; E C FE C F C HI, education plus low cost / free equipment plus fitting plus home inspection. IPD evidence not available for this treatment comparison. Although presented models are not comparable, the DIC statistic is shown for completeness. E,
Copyright © 2012 John Wiley & Sons, Ltd.
Statist. Med. 2012
P. SARAMAGO ET AL.
Note that when considering just the IPD evidence, a smaller evidence network (i.e. with six rather than seven interventions) is evaluated with intervention E C HI not being considered because there were no IPD studies including this treatment strategy. Carrying out the synthesis of only IPD evidence through the use of model (2), results were found to be similar to the synthesis of the same studies by using model (1) when all trial evidence is reduced to AD (results not shown but available on request). The same was true for the results of synthesising all 20 studies at AD level when compared with results when using synthesis model (3) (results not shown but published elsewhere [20]). This is to be expected because there is no extra information in the IPD when overall mean effects are of interest (i.e. patient level covariates are not considered). The only reason why these two pairs of results do not agree exactly relates to how cluster adjustment is dealt with in the AD and IPD and potentially the slight influence of the prior distributions. Larger differences are apparent between the use of all 20 studies (i.e. IPD C AD) and only the nine for which IPD was available. When synthesising the full set of evidence, the most ‘intense’ intervention (E C FE C F C HI) stands out with an odds ratio of 6.9 (CrI 2.29–23.5). When considering only evidence from the nine IPD studies, the intervention E C FE is estimated as being the most effective compared with standard care although this is highly uncertain (odds ratio of 7.66 (0.34–208.2)).
4.2. Analysis of models including a binary covariate Unfortunately, only 2 of the 11 studies for which only AD was available supplied an estimate of the percent of included subject from single parent families. Additionally, no parent status information was available for two of the nine studies for which IPD was available (Table I). The impact on the network diagram of the forced omission of 11 of the studies is indicated by the dashed lines in Figure 1. Compared with the network for all 20 studies (solid lines in Figure 1), this shows that even though six out of the initial seven interventions are still included, they have ‘weaker links’ in terms of the amount of evidence informing each of the comparisons. The results of estimating treatment interaction terms separating the between-study and the withinstudy associations are shown in Table III. These results refer to the direct implementation of model (4) where IPD is included where possible (seven studies) and AD where not (two studies), and interactions are assumed to be exchangeable. To assess whether ecological bias exists, the difference between the association terms was estimated, ˇ diff ’s. The credibility intervals of the difference between withinstudy and between-study associations for household parent status include 0. Therefore, there is little evidence to suggest the presence of systematic differences between the covariate treatment interactions estimated by the within study and between study variation. Therefore, in all subsequent analyses presented, these two sets of coefficients are replaced with a single one combining the between and within study information. The results of four different random effect approaches to our analysis are presented in Figure 3. ‘MTC AD RE — exchangeable’ refers to the analysis where all nine studies are fitted using AD (and all covariates are study-level) and the treatment–covariate interactions are assumed to be exchangeable; ‘MTC AD RE — independent’ refers to a similar analysis, only this time each treatment–covariate interaction is assumed to be independent. ‘MTC AD C IPD RE — exchangeable’ considers seven IPD and two AD studies and finally ‘MTC AD C IPD RE — independent’ is similar to the previous one, however only with independent interaction terms. While the main treatment effects (the d ’s) are reasonably consistent across all four models, the point estimates and the uncertainty in the interaction terms (the ˇ’s) are considerably different. Uncertainty was much reduced when IPD was available, and where it was not, estimates were shrunken towards the estimates where IPD was available in the model where interactions were assumed to be exchangeable. Interestingly, this resulted in all interaction effects close to 0 for the exchangeable interactions model including IPD where available. This contrasts with parameter estimates from the independent interaction model using all evidence in the AD format, which were particularly large in magnitude and very uncertain. On the basis of the Deviance Information Criteria (DIC) statistic [51] — a measure of model goodness of fit — all models including the covariate provide a ‘better’ fit to the data than the models not including it. This indicates that a proportion of the existent heterogeneity/inconsistency in the synthesis is being explained when incorporating a relevant covariate [14]. Within the models considering the covariate, the ones with lowest DIC statistics are the ones that consider exchangeable treatment - covariate Copyright © 2012 John Wiley & Sons, Ltd.
Statist. Med. 2012
P. SARAMAGO ET AL.
Table III. Parameter estimates from fitting a MTC synthesis model of AD and IPD evidence with a binary covariate related to single parent status to the functioning smoke alarm outcome data considering exchangeable treatment interactions and modelling separately within and between associations. Model (4) - nine studies included
Parameter (or function of)
Random effects MTC of AD and IPD with covariate Median of MCMC posterior sample
95% credible interval
Regression coefficients for within study association with single parent (vs two parent families) - ˇ W ’s
E E C FE E C FE C HI E C FE C F E C HI
0:018 0:363 0:366 0:023 0:143
0:779 to 0:704 0:856 to 3:235 0:663 to 2:48 0:664 to 0:604 1:855 to 2:934
Regression coefficients for between study association with single parent status (vs two parent families) - ˇ B ’s
E E C FE E C FE C HI E C FE C F E C HI
2:667 2:603 2:688 2:521 2:761
1:67 to 6:059 2:426 to 6:957 2:374 to 7:114 2:387 to 6:97 2:146 to 7:167
Difference between regression coefficients for within and between study associations with single parents (vs two parent families) - ˇ diff ’s
E E C FE E C FE C HI E C FE C F E C HI
2:671 1:997 2:151 2:553 2:462
1:692 to 6:186 3:514 to 6:683 3:093 to 6:83 2:423 to 7:002 2:954 to 7:268
Deviance Information criteria
DIC
2452:65
Notes: Model
includes information on covariate related to household having one or two parents - treatment interactions are assumed exchangeable and are split in within- and between-study associations. E, education; E C FE, education plus low cost / free equipment; E C FE C HI, education plus low cost / free equipment plus home inspection; E C FE C F, education plus low cost / free equipment plus fitting; E C HI, education plus home inspection; E C FE C F C HI, education plus low cost / free equipment plus fitting plus home inspection.
associations (i.e. ‘MTC AD RE — exchangeable’ and ‘MTC AD C IPD RE — exchangeable’ with DIC D 105.24 and DIC D 2451.77, respectively).
5. Discussion 5.1. Findings summary This paper contributes to the evidence synthesis methodological literature by describing and applying a series of novel MTC models that allow IPD and both AD and IPD to be included while considering a participant level covariate and making different assumptions about the covariate effects [14]. Modelling of cluster allocation effects [11], and distinct covariate effects based on between-study and within-study variability (because the former is susceptible to ecological biases) were also considered [50]. The motivating example — assessing the effectiveness of interventions to increase the uptake of functioning smoke alarms in households — showed that more ‘intense’ interventions are more effective than less ‘intensive’ ones, with the one providing education C low cost/free equipment C fitting C home inspection having the highest level of effectiveness from the set. 5.2. Evidence synthesis in Health Technology and Public Health programmes appraisal Estimating the effectiveness of alternative healthcare interventions is at the heart of not only clinical but also economic evaluations. The National Institute for Health and Clinical Excellence (NICE) for England and Wales uses economic analyses to recommend healthcare technologies for use in the National Health Service. The guide to methods for health technology assessment (HTA) published by NICE [52] acknowledges that the construction of ‘. . . an analytical framework to synthesize the available Copyright © 2012 John Wiley & Sons, Ltd.
Statist. Med. 2012
P. SARAMAGO ET AL.
E, d E+FE, d E+FE+HI, d E+FE+F, d E+HI, d
E,β E+FE,β E+FE+HI,β E+FE+F,β E+HI,β -8
-6
-4
-2
0
2
4
6
8
10
Log scale (relative to UC) MTC AD RE - exchangeable (DIC = 105.24*) MTC AD RE - independent (DIC = 105.28*) MTC AD+IPD RE - exchangeable (DIC = 2451.77**) MTC AD+IPD RE - independent (DIC = 2451.99**) Posterior median, 95% CrI
Figure 3. Parameter estimates from fitting the MTC of AD and the MTC of AD C IPD synthesis models with arm-level information on the binary covariate related to single parent status to the binary functioning smoke alarm outcome data considering exchangeable and independent treatment interactions, not separating betweenand within-study interactions (Notes: * – comparable DIC statistics; ** – comparable DIC statistics).
evidence in order to estimate clinical and cost-effectiveness. . . ’ should be performed and recommends that ‘. . . all relevant evidence must be identified, quality assessed and pooled using explicit criteria and justifiable and reproducible methods’ (page 27). Meta-analyses techniques are often used to summarize evidence on clinical effectiveness in NICE technology appraisals, and subsequently inform related economic analyses. The use of indirect and MTC methodologies in informing decision-making is also becoming more frequent. The notion of what should be considered to be relevant evidence in HTA and Public Health is yet to be unequivocally determined. Consequently, issues surrounding the use of IPD compared with AD also remain unclear. Resistance from authors/researchers to release IPD data, the costs related to time and computational burdens compared with analysis of AD only, and the delay in producing the evidence for decision making all act against seeking to collect and use IPD routinely. Therefore, the benefits of obtaining IPD over and above the existing AD evidence should be taken into account. Clear benefits such as more accurate estimation of subgroup effects, as demonstrated here, make a strong case for using IPD in synthesis models whenever possible. However, IPD may not be always available for all studies, hence the methods developed here. We believe these are an improvement compared with existing alternatives identified by a recent review of the literature [53]. 5.3. Strengths and limitations This paper did not explore issues related to suboptimum data quality such as the use of nonrandomised studies in the synthesis (although bias adjustment proposals have been published, focusing mainly on reweighting schemes usually attributing lower weight to evidence with a high risk of bias [54–56]) or adjusting for arm imbalances in the covariate of interest (although approaches recently developed for pair-wise meta-analysis could be applied to the MTC models developed here [55]). For heterogeneity to be realistically large, restrictions were imposed in the main analysis through the use of ‘not so vague’ priors, which may limit results interpretability. Nonetheless, we have conducted Copyright © 2012 John Wiley & Sons, Ltd.
Statist. Med. 2012
P. SARAMAGO ET AL.
sensitivity analyses over these prior distributions for between-study variances, which showed that results were not sensitive to the choice of distribution parameterization. A worthwhile extension would be to develop a generalized linear modelling framework to extend our approach to other types of outcome measures (e.g. categorical, continuous) building on recent work [18].
6. Conclusion Using IPD from all studies is desirable, nonetheless this will not be possible in the majority of instances and thus we believe that the presented models have a valuable role in the evaluation of interventions, particularly where there is inconsistency in the network and/or treatment subgroup effects are of interest. We think this paper brings into question the often publicised view that IPD syntheses are the ‘gold standard’ if only a fraction of the available studies can be included, that is, we would argue it is better not to exclude any studies from the analysis, irrespective of the format they are available in. Models herein described were applied to a particular Public Health (PH) example; nevertheless, they are potentially of use in other healthcare contexts, including HTA assessments of drugs and devices where IPD may be available for a particular product, but not for competitor products.
Appendix. WinBUGS code used to combine the two data formats, AD and IPD, including a binary patient covariate This code relates to model (4) described above and is designed to be as generic as possible and easy for the user to modify and adapt to specific applications. For example, if no data exists in one or more of the sections, the corresponding section of code can simply be deleted from the model. Six datasets are required to fit the complete model: two containing constants, one that indexes study treatments and specifies study baseline treatments, two for IPD evidence and one for the AD evidence. We should note again that for the clustered allocated AD evidence, adjustments should be conducted prior to defining the WinBUGS data model. All data should be loaded before the model is compiled. Because of size and agreements of use, the original data sets are not included in their entirety, but a couple of lines of data are supplied for each study/data combination for illustration purposes. Note: simplified code without the inclusion of a covariate is available from the corresponding author on request.
Copyright © 2012 John Wiley & Sons, Ltd.
Statist. Med. 2012
P. SARAMAGO ET AL.
Copyright © 2012 John Wiley & Sons, Ltd.
Statist. Med. 2012
P. SARAMAGO ET AL.
# Vague priors for random-effects m.betaw dnorm(0, 1.0E-6) tau.betaw dunif(0,2) tau.sq.betaw
