Computers in Human Behavior Computers in Human Behavior 23 (2007) 1421–1434 www.elsevier.com/locate/comphumbeh
Internet vs. paper and pencil administration of questionnaires commonly used in panic/agoraphobia research Per Carlbring a,*, Sara Brunt b, Susanne Bohman b, ¨ st d, David Austin c, Jeffrey Richards c, Lars-Go¨ran O Gerhard Andersson a a
Department of Behavioural Sciences, Linko¨ping University, 581 83 Linko¨ping, Sweden b Department of Psychology, Uppsala University, Uppsala, Sweden c School of Primary Health Care, Monash University, Melbourne, Australia d Department of Psychology, Stockholm University, Stockholm, Sweden Available online 11 July 2005
Abstract The aim of this study was to investigate the psychometric properties of Internet administered questionnaires used in panic research. Included were 494 people who had registered for an Internet-based treatment program for panic disorder (PD). Participants were randomly assigned to fill in the questionnaires either on the Internet or the paper-and-pencil versions, and then to fill in the same questionnaires again the next day using the other format. The questionnaires were the body sensations questionnaire [BSQ; Chambless, D. L., Caputo, G. C., Bright, P., & Gallagher, R. (1984). Assessment of fear of fear in agoraphobics: the body sensations questionnaire and the agoraphobic cognitions questionnaire. Journal of Consulting and Clinical Psychology, 52, 1090–1097], agoraphobic cognitions questionnaire [ACQ; Chambless, D. L., Caputo, G. C., Bright, P., & Gallagher, R. (1984). Assessment of fear of fear in agoraphobics: the body sensations questionnaire and the agoraphobic cognitions questionnaire. Journal of Consulting and Clinical Psychology, 52, 1090–1097], mobility inventory [MI; Chambless, D. L., Caputo, G., Jasin, S., Gracely, E. J., & Williams, C. (1985). The mobility inventory for agoraphobia. Behaviour Research and Therapy, 23, 35–44], beck anxiety inventory *
Corresponding author. Fax: +46 1314 9403. E-mail address:
[email protected] (P. Carlbring).
0747-5632/$ - see front matter 2005 Elsevier Ltd. All rights reserved. doi:10.1016/j.chb.2005.05.002
1422
P. Carlbring et al. / Computers in Human Behavior 23 (2007) 1421–1434
[BAI; Beck, A. T., Epstein, N., Brown, G., & Steer, R. A. (1988). An inventory for measuring clinical anxiety: psychometric properties. Journal of Consulting and Clinical Psychology, 56, 893–897], beck depression inventory II [Beck, A. T., & Steer, R. A. (1996). Beck Depression Inventory. Manual, Svensk version (Swedish version). Fagernes, Norway: Psykologifo¨rlaget, AB], quality of life inventory [QOLI; Frisch, M. B., Cornell, J., Villanueva, M., & Retzlaff, P. J. (1992). Clinical validation of the quality of life inventory. A measure of life satisfaction for use in treatment planning and outcome assessment. Psychological Assessment, 4, 92–101], ˚ sberg depression rating scale [MADRS; Svanborg, P., & A ˚ sberg, M. and montgomery A (1994). A new self-rating scale for depression and anxiety states based on the comprehensive psychopathological rating scale. ACTA Psychiatrica Scandinavica, 89, 21–28]. Results showed largely equivalent psychometric properties for the two administration formats (CronbachÕs a between 0.79 and 0.95). The results also showed high and significant correlations between the Internet and the paper-and-pencil versions. Analyses of order effects showed an interaction effect for the BSQ and the MI (subscale Accompanied), a main effect was identified for ACQ, MI-Alone, BAI and BDI II. However, in contrast to previous research, the Internet version did not consistently generate higher scores and effect sizes for the differences were generally low. Given the presence of an interaction effect, we recommend that the administration format should be stable in research across measurement points. Finally, the findings suggest that Internet versions of questionnaires used in PD research can be used with confidence. 2005 Elsevier Ltd. All rights reserved. Keywords: Internet administration; Panic disorder; Questionnaire; Psychometrics
1. Introduction With the advent of modern information technology new opportunities have emerged regarding treatment and assessment of panic disorder (Richards, Klein, & Carlbring, 2003). Although self-help approaches have been developed and evaluated previously (e.g., Gould & Clum, 1993), it was only recently that researchers begun to explore the feasibility of administering self-help treatment with minimal therapist intervention using the Internet (Richards et al., 2003). This novel approach has conveyed several advantages such as overcoming distances and facilitating access to therapist feedback on a rapid basis. In addition, once the costs for transferring the treatment to a website have been deducted, it is apparent that minimal therapist contact self-help via the Internet reduces costs (Carlbring et al., in press), when compared with standard face to face therapy. Although concerns have been raised regarding the use of self-help for panic disorder (Febbraro, Clum, Roodman, & Wright, 1999; Taylor, 2000), it is possible that Internet delivered self-help can offset the disadvantages, since therapist time can be freed up so that the clinician can consult colleagues for supervision and expertise (when responding to e-mails). In parallel with the development of Internet-based treatment, administration of questionnaires are beginning to be transferred onto the Internet. There are obvious advantages with Internet administration of questionnaires. For example they can be filled out in the patientsÕ own homes; by making registration of all items obligatory before submission, missing values can be handled; data will appear directly for
P. Carlbring et al. / Computers in Human Behavior 23 (2007) 1421–1434
1423
transfer to a statistics program; and finally scoring can be facilitated using scripts. Other advantages associated with Internet-administration include reduced costs, as well as the opportunity for researchers to access a larger and potentially more diverse population. However, psychometric properties of Internet administered measures cannot be taken for granted. This was highlighted in a review by Buchanan (2003) who also conducted a series of studies investigating psychometric properties of questionnaires. Although, some studies suggest that Internet-based questionnaires can generate equivalent information as paper-and-pencil tests in terms of psychometric properties and test characteristics (Andersson, Kaldo-Sandstro¨m, Stro¨m, & Stro¨mgren, 2003), it is less certain that norms can be transferred, and indeed adjustments in scoring might be needed (Buchanan, 2003). Therefore, the equivalence of Internetbased to pencil-and-paper versions of questionnaires cannot be assumed overall. Consequently, it has been recommended that each Internet-based measure be independently evaluated (Buchanan, 2003). Research on Internet delivered treatment for panic disorder is currently being conducted by independent research groups (Alcaniz et al., 2003; Carlbring, Ekselius, & Andersson, 2003; Carlbring et al., in press; Carlbring, Westling, Ljungstrand, Ekselius, & Andersson, 2001; Klein & Richards, 2001; Richards & Alvarenga, 2002). In addition, some preliminary work has been published regarding Internetbased psychiatric assessment of panic patients (Carlbring et al., 2002). In the latter case, it turned out that the Internet was less than perfect for diagnostic purposes. However, so far Internet administration of self-report instruments used in panic research has not yet been evaluated. The aim of this study was to test the equivalence of paper-and-pencil and Internet-administered versions of several commonly used measures of panic-related variables. All questionnaires were regarded to be wellsuited to a computerized medium as they are easily understood, brief, and require simple responses (via selection of responses from a Likert-type scale). The present study employed a randomized design in which half of the participants completed an Internet version of the questionnaires first and the paper-and-pencil version next. For the other half the order was reversed. In addition to establishing the psychometric properties of the measures used, we were also in a position to test order effects in a repeated measures design. This was intended to answer the question as to whether response formats (Internet vs. paper-and-pencil) can be regarded as interchangeable.
2. Method 2.1. Participants Participants were 494 people who had registered for an Internet-based treatment program for panic disorder (PD). These people were recruited by means of newspaper articles in Swedish national and regional papers and on the Web pages of the Swedish Anxiety Association. Web pages for the study had been created (Carlbring et al., 2001), including general information about panic disorder, an outline of the study, information regarding consent from the ethics committee, and an application
1424
P. Carlbring et al. / Computers in Human Behavior 23 (2007) 1421–1434
form. All subjects had access to a computer and an Internet connection. The sample consisted of 51% men and 49% women, with a mean age of 37.6 (SD = 10.9). The educational level, marital status, and computer experience are presented in Table 1. 2.2. Measures 2.2.1. Body sensations questionnaire (BSQ; Chambless, Caputo, Bright, & Gallagher, 1984) The BSQ is a 17 item measure of fears associated with physical symptoms of arousal that are commonly experienced during anxiety. Each item is rated on a five-point scale ranging from 1 (not at all) to 5 (extremely). Administration typically takes 5–10 min (Antony, 2001). Chambless et al. (1984) reported a mean score of 3.05 (SD = 0.86) for outpatients with agoraphobia, and 1.80 (SD = 0.59) for a community sample. The BSQ has good internal consistency (CronbachÕs a = 0.89) and high test– retest reliability (r = 0.79; Arrindell, 1993). Finally, the BSQ is sensitive to changes following treatment, making it ideal for measuring treatment outcome (Chambless et al., 1984). 2.2.2. Agoraphobic cognitions questionnaire (ACQ; Chambless et al., 1984) The ACQ is a 14 item self-report questionnaire in which individuals rate how often specific cognitions occur when they are feeling anxious or frightened. Each item is rated on a five-point scale ranging from 1 (thought never occurs) to 5 (thought always occurs when I am nervous). Administration typically takes 5– 10 min (Antony, 2001). Chambless et al. (1984) reported a mean score of 2.32 (SD = 0.66) for outpatients with agoraphobia, and 1.60 (SD = 0.46) for a community sample. The ACQ has shown acceptable internal consistency (CronbachÕs a = 0.82) Table 1 Demographic description of the participantsa Highest educational level Elementary education unfinished finished High school unfinished finished Community college unfinished finished College/University unfinished finished Ph.D. a
Percent
1.6 5.2 8.9 30.6 0.4
19.0 31.0 0.8
Marital status
Subjective computer experience
Percent
Married/living together with children 42.7 without children 19.0
Far above average
13.7
Living apart with children without children
Above average
29.4
2.8 5.2
Single with children without children
Average
32.3
7.7 21.8
Widow/widower with children
Percent
Below average
3.6
0.8 Far below average 2
No significant differences existed between the experimental conditions according to v tests.
0.0
P. Carlbring et al. / Computers in Human Behavior 23 (2007) 1421–1434
1425
and test–retest reliability (r = 0.79; Arrindell, 1993). Finally, the ACQ is sensitive to changes following treatment, making it ideal for measuring outcome (Chambless et al., 1984). 2.2.3. Mobility inventory (MI; Chambless, Caputo, Jasin, Gracely, & Williams, 1985) The MI is a self-report questionnaire, of which 26 items measure avoidance behavior typically associated with agoraphobia ‘‘when accompanied’’ (MI-accomp), and 27 items that measure agoraphobic avoidance ‘‘when alone’’ (MI-alone). Each item is rated on a five-point scale ranging from 1 (never avoid) to 5 (always avoid). Administration typically takes 5–10 min (Antony, 2001). Chambless et al. (1985) reported mean scores on the MI for two samples of agoraphobia patients. The means on the avoidance alone subscale were 3.35 and 3.30, respectively (SD = 1.06 and 0.99). The means on the avoidance accompanied subscale were 2.64 and 2.41, respectively (SD = 0.90 and 0.70). For a normal control sample, the means were 1.25 (SD = 0.24) and 1.07 (SD = 0.08) for the two scales, respectively. The MI has excellent internal consistency (CronbachÕs a = 0.91–0.97). The test–retest reliability is high with rs ranging from 0.89 to 0.90 for the avoidance alone subscale, and from 0.75 to 0.86 for the accompanied subscale (Chambless et al., 1985). 2.2.4. Beck anxiety inventory (BAI; Beck, Epstein, Brown, & Steer, 1988) The BAI is a 21-item self-report inventory for measuring the severity of anxiety in psychiatric populations. Respondents rate how much they have been bothered by a list of symptoms during the past week on a four-point scale ranging from 0 (not at all) to 3 (terrified). The manual reports that the mean total score for people with panic disorder with agoraphobia is M = 27.27 (SD = 13.11). In a normative sample a score of 3 fell at the 50th percentile, and a score of 10 fell at the 80th percentile (Gillis, Haaga, & Ford, 1995). de Beurs, Wilson, Chambless, Goldstein, and Feske (1997) showed that the BAI has high internal consistency in panic patients (CronbachÕs a = 0.92) and good test–retest reliability (0.83). 2.2.5. Beck depression inventory The BDI was originally constructed by Beck, Ward, Mendelson, Mock, and Erbaugh (1961) in order to assess degree of depression in adolescents and adults. The revised version was used in the present study (Beck & Steer, 1996). The BDI consists of 21 variables with operationalized descriptions of four possible scale steps, 0–3. It is possible for patients to choose more than one alternative on each variable, however, only the highest rating is counted when the total score is computed. Although the BDI was initially designed to be administered by trained interviewers, it is most often self-administered. When self-administered, the instrument generally takes 5–10 min to complete and is scored by summing the ratings given to each of the 21 items. The BDI has been extensively validated in various studies. A review of the BDIÕs internal consistency estimates yielded a mean coefficient alpha of 0.86 for psychiatric patients and 0.81 for nonpsychiatric subjects (Beck, Steer, & Garbin, 1988). For psychiatric patients the test–retest reliability has been reported to range
1426
P. Carlbring et al. / Computers in Human Behavior 23 (2007) 1421–1434
from 0.48 to 0.86, whereas the coefficients for nonpsychiatric subjects ranged from 0.60 to 0.83 (Beck, Steer, et al., 1988). 2.2.6. Quality of life inventory (QOLI; Frisch, Cornell, Villanueva, & Retzlaff, 1992) The QOLI consists of 16 areas rated by the subject concerning importance (0 to 2), and satisfaction ( 3 to +3). These two ratings are then multiplied yielding a score between 6 and +6 for each area. To obtain the total quality of life score, the average scores across the areas that the participant has judged as important or very important for his/her life satisfaction are calculated. The instrumentÕs internal consistency is high, between a = 0.77 and 0.89, and the one month test–retest reliability ¨ st, Breitholtz, and Thulin (1997) lies between r = 0.80 and 0.91 (Frisch et al., 1992). O reported a total mean score of 0.84 (SD = 3.05) for Swedish outpatients with anxiety, and 2.76 (SD = 2.29) for normal controls. 2.2.7. Montgomery A˚sberg depression rating scale (MADRS; Svanborg & A˚sberg, 1994) ˚ sberg (1979) and is designed to be MADRS was created by Montgomery and A ˚ sberg (1994) developed a particularly sensitive to treatment effects. Svanborg and A self-assessment version of MADRS named MADRS-S which is utilized in this study. This instrument measures the following symptoms: mood changes, anxiety, changes in sleeping patterns, appetite, ability to concentrate, initiative-taking, emotional ˚ sberg engagement, pessimism and attitude to life. According to Montgomery and A (1979), the instrument is reliable and shows high correlations (from r = 0.80 to ˚ sberg, 1994). r = 0.94) between expert ratings and self-reports (Svanborg & A
3. Procedure All participants had registered for an Internet-based treatment program for PD. Before commencing treatment, participants were instructed to complete 7 self-rated questionnaires twice, but in different administration formats. Half the subjects were randomized to answer the paper-and-pencil version first, and the next day the Internet version (Post-first). The other half did the assessments in a counter-balanced order (IT-first). For the administration of the Internet versions, participants were instructed to go to a dedicated web-page and to complete the questionnaires which were presented in the order BSQ, ACQ, MI, BAI, BDI, QOLI, and MADRS. Password access was required and unique codes served to identify the respondents. Participants were free to backtrack and review all item responses (and amend if desired) before submitting the questionnaire. Paper versions of the measures (presented in the same order as on the web-page) were sent by post to participantsÕ homes and they were instructed to complete and return the measures (in a reply-paid envelope supplied) preferably the same day. There was no experimenter present during paper or Internet administration, and all communication with participants was done remotely (by telephone and/or email).
P. Carlbring et al. / Computers in Human Behavior 23 (2007) 1421–1434
1427
Participants typically completed all questionnaires (both paper and Internet administered) in their homes. The external dropout rate was 23%, leaving 344–350 participants eligible for analysis (depending on the questionnaires that were not completed). 3.1. Statistical analyses Statistical analysis was conducted using SPSS version 11.5. The analyses were conducted on each questionnaire separately, and the MI was, in accordance with the manual, divided into two subscales (alone vs. accompanied). The internal consistency was estimated with CronbachÕs a, and the correlations between Internet and paper-and-pencil questionnaires were calculated with Pearson correlations (two tailed). Z-test was used to test differences in correlations between the two groups (Clark-Carter, 1997). Significance testing of differences in questionnaire administration format (paperand-pencil & Internet) and order (Post-first or IT-first) was done with 2 · 2 mixed ANOVAs. Any interaction effect was subsequently tested with a t-test where the p-level underwent a Bonferroni adjustment to p < 0.0125. g2 served as effect size where according to Clark-Carter (1997), g2 = 0.01 is small; g2 = 0.059 is moderate and g2 > 0.138 is considered to be a large effect.
4. Results 4.1. Internal consistency The questionnairesÕ internal consistencies (CronbachÕs a) across questionnaires and administration formats are presented in Table 2. CronbachÕs a ranged between 0.79 and 0.95. The first two columns show the first administration occasion for each group. The differences in internal consistency between the two administration formats were negligible. The difference was largest for MADRS-S (a = 0.82 vs. 0.87) and smallest for MI-Alone (a = 0.94 vs. 0.94). The Internet version had values between 0.81 and 0.95, and the paper-and-pencil had values between 0.79 and 0.94. The second administration showed the same pattern. Both groups had a slightly higher value the second time. 4.2. Associations Correlations between scores from Internet and paper-and-pencil administrations are reported in Table 3. The results showed high and significant correlations between the Internet and the paper-and-pencil versions of the questionnaires, respectively; from a low of r = 0.81 (BSQ) to a high of r = 0.96 (MI-Alone). The differences in correlations between the two groups were not significant (all pÕs > 0.14). Hence, the first administration modality did not affect the correlations between the two types of administration.
1428
P. Carlbring et al. / Computers in Human Behavior 23 (2007) 1421–1434
Table 2 Internal consistency (CronbachÕs a) for the two administration formats (paper-and-pencil and Internet) and order of administration (Internet first or paper-and-pencil first)
BSQ ACQ MI-accomp MI-alone BAI BDI QOLI MADRS-S
IT-first on Internet (n = 174–175)
Post-first on paper (n = 172–177)
IT-first on paper (n = 172–174)
Post-first on Internet (n = 172–176)
0.84 0.81 0.95 0.94 0.88 0.88 0.81 0.82
0.88 0.84 0.92 0.94 0.90 0.90 0.79 0.87
0.87 0.83 0.94 0.94 0.91 0.89 0.82 0.83
0.88 0.85 0.95 0.94 0.92 0.92 0.82 0.90
4.3. Order and main effects of administration format Mean values for the questionnaires are presented for each type of administration (i.e., Internet, paper-based) and for each group (order of administration) in Table 4. The differences were analyzed with a repeated measures ANOVA with questionnaire type as the within factor, and administration order as a between group factor. In two of the eight ANOVAs an interaction effect was identified (BSQ & MIAccomp), and in four ANOVAs a main effect was identified (ACQ, MI-Alone, BAI & BDI). In two questionnaires (QOLI & MADRS-S), no main- or interaction effect was found and these are therefore not commented on further. 4.3.1. Body sensations questionnaire For BSQ there was a significant main effect indicating a higher value for the paper-and-pencil version (M = 3.02 vs. 2.90). However, the result should be viewed in the light of a significant interaction effect (F(1, 348) = 24.8, p < 0.001). A subsequent analysis, with Bonferroni adjusted p-values, showed that the group that started with BSQ on paper scored significantly lower on the Internet compared to paper-andTable 3 Correlations between Internet and paper-and-pencil versions of the questionnaires (Pearson two-tailed)
BSQ ACQ MI-accomp MI-alone BAI BDI QOLI MADRS-S
IT-first (n = 172–174) r
Post-first (n = 172–177) r
Both groups (IT-first & Post-first) (n = 345–351) r
0.81 0.89 0.96 0.94 0.86 0.93 0.93 0.89
0.83 0.90 0.96 0.95 0.83 0.94 0.91 0.92
0.81 0.89 0.96 0.95 0.84 0.94 0.92 0.91
All correlations are significant (p < 0.001).
P. Carlbring et al. / Computers in Human Behavior 23 (2007) 1421–1434
1429
Table 4 Mean (SD), main effects and interactions for the questionnaires administered both in paper-and-pencil and on the Internet, n (IT-first) = 172–174, n (Post-first) = 172–177 Questionnaire
BSQ n = 350
Group
Paper
Internet
Main effect
M (SD)
M (SD)
Version F
Order F
Interaction F
24.8***
IT-first Post-first
2.99 (0.67) 3.05 (0.69)
2.97 (0.59) 2.82 (0.68)
36.5***
0.50
ACQ IT-first Post-first
2.52 (0.68) 2.54 (0.71)
2.42 (0.61) 2.41 (0.71)
45.4***
0.003
1.3
MI-accomp IT-first Post-first
2.06 (0.81) 2.07 (0.77)
2.10 (0.77) 2.04 (0.77)
0.10
0.06
6.4*
MI-alone IT-first Post-first
2.70 (0.90) 2.65 (0.86)
2.72 (0.88) 2.70 (0.87)
5.9*
0.90
1.5
n = 349
BAI IT-first Post-first
22.04 (10.7) 23.20 (11.9)
19.26 (9.1) 19.99 (11.0)
82.2***
0.74
0.43
n = 350
BDI IT-first Post-first
17.16 (9.5) 17.87 (10.9)
17.89 (9.6) 18.12 (11.2)
6.3*
0.20
1.5
QOLI IT-first Post-first
0.70 (1.8) 0.72 (1.8)
0.72 (1.7) 0.74 (1.8)
0.34
0.01
0.008
MADRS-S IT-first 16.69 (7.4) Post-first 17.11 (9.4)
16.42 (7.1) 16.79 (8.3)
2.3
0.20
0.02
n = 348
n = 347
n = 345
n = 344
n = 345 *p
< 0.05;
**p
< 0.01;
***p
< 0.001.
pencil (t(176) = 7.9, p < 0.001), while the group that started with Internet did not get a significantly different score compared to the paper-and-pencil version (t(172) = 0.75, p = 0.46). Furthermore, there were no differences in scores between the groups on the paper-and-pencil version (t(382) = 0.86, p = 0.39) and no difference in scores between the groups on the Internet version (t(349) = 2.2, p = 0.03). The effect size for the interaction was moderate (g2 = 0.066). 4.3.2. Agoraphobic cognitions questionnaire A significant main effect of administration form was found for ACQ (F(1, 346) = 45.4, p < 0.001). The composite mean for the Internet version was lower than the paper-and-pencil version (M = 2.42 vs. 2.53), and the effect size was moderate (g2 = 0.116).
1430
P. Carlbring et al. / Computers in Human Behavior 23 (2007) 1421–1434
4.3.3. Mobility inventory – when accompanied A significant interaction effect was found for MI-Accomp (F(1, 345) = 6.4, p < 0.05). However, the effect size was small (g2 = 0.018) and the post hoc tests could not identify any significant differences. 4.3.4. Mobility inventory – when alone A significant main effect of administration form was found for MI-Alone (F(1, 343) = 5.9, p < 0.05). The composite mean for the Internet version was higher than the paper-and-pencil version (M = 2.71 vs. 2.68), but the effect size was small (g2 = 0.017). 4.3.5. Beck anxiety inventory A significant main effect of administration form was found for BAI (F(1, 348) = 82.2, p < 0.001). The composite mean for the Internet version was lower than the paper-and-pencil version (M = 19.63 vs. 22.62), and the effect size was large (g2 = 0.191). 4.3.6. Beck depression inventory-II A significant main effect of administration form was found for BDI-II (F(1, 347) = 6.3, p < 0.05). There was a significant higher score in the Internet version than the paper-and-pencil version (M = 18.01 vs. 17.52), but the effect size was small (g2 = 0.018).
5. Discussion This study was undertaken to validate the use of Internet administered questionnaires used in research on panic disorder. In no case was the internal consistency of the questionnaires affected by the administration format. Both the Internet and the paper-and-pencil versions of all questionnaires had alpha values well above 0.70, which is considered good (Clark-Carter, 1997). Hence, the data support earlier studies suggesting that the alpha values of paper-and-pencil questionnaires can be replicated when administered via the Internet (Andersson et al., 2003; Buchanan, 2003; Pasveer & Ellard, 1998). A critical aspect of equivalence is the strength of the relationship between administration formats. The correlations between the two administration formats were high, and in line with test–retest values on the paper-and-pencil version. This could indicate that the importance of administration format is marginal. However, in clinical research differences in mean scores are used to evaluate treatment outcome, and hence differences in mean scores between administration formats could imply that it is not possible to switch between assessment formats at different assessment points. In the present study, with a fairly large sample, we found some differences in mean scores. For two questionnaires (MI-Alone & BDI), higher values were obtained for the Internet compared to the paper-and-pencil administration format. This is consistent with some previous research (e.g., Buchanan, 2003; Buchanan & Smith, 1999)
P. Carlbring et al. / Computers in Human Behavior 23 (2007) 1421–1434
1431
suggesting that Internet administration leads to higher scores overall. In fact, Buchanan (2002) has even suggested that separate norms should be established for Internet-based questionnaires. However, the effect sizes for the differences were small. Therefore, the clinical relevance of these discrepancies is relatively unimportant. In contrast to what has previously been found in research on Internet administration, significantly higher scores were found for the BSQ, BAI and ACQ in their paper-and-pencil versions. Here the effect sizes were moderate to large. The differences in scores were 0.12 (BSQ), 2.99 (BAI), and 0.11 (ACQ), which again from a practical point of view are rather small with the possible exception of the BAI. Therefore, for the BAI it might be important to use ‘‘Internet norms’’ when evaluating test scores obtained online. Finally, three questionnaires seemed unaffected by the administration format (MI-accompanied, QOLI, & MADRS-S). Perhaps even more important for the above question were the significant interaction effects, found for two questionnaires. Although perhaps of not much practical importance, the interactions found for the BSQ and MI-accomp suggest that the order of administering the questionnaire affects the test–retest score difference if different administration formats are used. This implies that it is best to use the same administration format across assessment points. There are some caveats on the generalizability of these results to panic and panic disorder research. First, as participants were self-recruited there is no way to guarantee how representative they were of either applicants for panic disorder treatment in medical settings or even research for that matter. As they were recruited over the Internet we were also not aware of their diagnoses, or even that panic was their primary problem. This could be crucial, in particular as panic symptoms commonly occur across all Axis-I diagnoses of the DSM (e.g., Reed & Wittchen, 1998). However, this recruitment method is also a strength of the study, as it has been argued that the Internet holds promise as a way to recruit more, rather than less, representative samples for research (Reips, 2000; Schmidt, 1997). In Sweden, where this study was conducted, a large majority of the adult population have access to the Internet (Statistics Sweden, 2004), and it is mainly older people who do not access the Internet. As it is known that panic disorder is less often diagnosed among older persons (Taylor, 2000), it is less likely that we have falsely included people without panic disorder than we would have if we had targeted another diagnostic category more common among older persons (e.g., generalized anxiety disorder). Secondly, although we used an experimental design to test order effects, we did not obtain test–retest reliability coefficients for the two presentation formats separately. Previous research has, however, not given any indication that test–retest should differ markedly between administration formats, but of course this remains an open question. However, related research on another diagnostic group has not found any differences in test–retest reliabilities of questionnaire measures administered over the Internet compared to paper and pencil (Andersson et al., 2003). A related issue concerns responsiveness to change, which was not investigated in this study. There is however some indirect evidence for responsiveness, as treatment studies using Internet administered outcome measures have been able to detect treatment
1432
P. Carlbring et al. / Computers in Human Behavior 23 (2007) 1421–1434
effects (e.g., Andersson, Stro¨mgren, Stro¨m, & Lyttkens, 2002; Carlbring et al., 2001; Stro¨m, Pettersson, & Andersson, 2004). A third concern deals with statistical power, and the fact that we had a comparably large sample which might render clinically insignificant effects, statistically significant (Abelson, 1995). Indeed, the significant differences in mean scores between administration formats were often very small. As a way to complement the statistical analyses we also presented effect sizes, and a similar finding was obtained with commonly small differences. A final issue concerns the ethical and clinical implications of using the Internet to collect data. In clinical practice, and given the lack of an established way to ascertain identity over the Internet, we recommend that the Internet should be used second to an in vivo consultation with a psychologist or psychiatrists. Perhaps, this will change in the future when online verification becomes more common. In sum, web-based psychological testing can be reliable and valid, but each questionnaire should be validated for online use. It seems that all the questionnaires examined in this study can be used for both clinical and research purposes, but one should not mix different administration formats within the same study.
References Abelson, R. P. (1995). Statistics as principled argument. Hillsdale, NJ: Lawrence Erlbaum Associates. Alcaniz, M., Botella, C., Banos, R., Perpina, C., Rey, B., Lozano, J. A., et al. (2003). Internet-based telehealth system for the treatment of agoraphobia. Cyberpsychology & Behavior, 355–358. Andersson, G., Kaldo-Sandstro¨m, V., Stro¨m, L., & Stro¨mgren, T. (2003). Internet administration of the Hospital Anxiety and Depression Scale in a sample of tinnitus patients. Journal of Psychosomatic Research, 55, 259–262. Andersson, G., Stro¨mgren, T., Stro¨m, L., & Lyttkens, L. (2002). Randomised controlled trial of Internet based cognitive behavior therapy for distress associated with tinnitus. Psychosomatic Medicine, 64, 810–816. Antony, M. M. (2001). Measures of panic disorder and agoraphobia. In M. M. Antony, S. M. Orsillo, & L. Roemer (Eds.), PractitionerÕs guide to empirically based meassures of anxiety (pp. 95–125). New York: Kluwer Academic/Plenum Press. Arrindell, W. A. (1993). The fear of fear concept: stability, retest artefact and predictive power. Behaviour Research and Therapy, 31, 139–148. Beck, A. T., Epstein, N., Brown, G., & Steer, R. A. (1988). An inventory for measuring clinical anxiety: psychometric properties. Journal of Consulting and Clinical Psychology, 56, 893–897. Beck, A. T., & Steer, R. A. (1996). Beck depression inventory. Manual, Svensk version (Swedish version). Fagernes, Norway: Psykologifo¨rlaget AB. Beck, A. T., Steer, R. A., & Garbin, M. G. (1988). Psychometric properties of the Beck Depression Inventory: twenty five years of evaluation. Clinical Psychology Review, 8, 77–100. Beck, A. T., Ward, C. H., Mendelson, M., Mock, J., & Erbaugh, J. (1961). An inventory for measuring depression. Archives of General Psychiatry, 4, 561–571. Buchanan, T. (2002). Online assessment: desirable or dangerous? Professional Psychology: Research and Practice, 33, 148–154. Buchanan, T. (2003). Internet-based questionnaire assessment: appropriate use in clinical contexts. Cognitive Behaviour Therapy, 32, 100–109. Buchanan, T., & Smith, J. L. (1999). Research on the Internet: validation of a World-Wide Web mediated personality scale. Behavior Research Methods, Instruments and Computers, 31, 565–571.
P. Carlbring et al. / Computers in Human Behavior 23 (2007) 1421–1434
1433
Carlbring, P., Ekselius, L., & Andersson, G. (2003). Treatment of panic disorder via the Internet: a randomized trial of CBT vs. applied relaxation. Journal of Behavior Therapy and Experimental Psychiatry, 34, 129–140. Carlbring, P., Forslin, P., Ljungstrand, P., Willebrand, M., Strandlund, C., Ekselius, L., et al. (2002). Is the Internet-administered CIDI-SF equivalent to a human SCID-interview? Cognitive Behaviour Therapy, 31, 183–189. Carlbring, P., Nilsson-Ihrfelt, E., Waara, J., Kollenstam, C., Buhrman, M., & Kaldo, V., et al. (in press). Treatment of panic disorder: live therapy vs. self-help via Internet. Behaviour Research and Therapy. Carlbring, P., Westling, B. E., Ljungstrand, P., Ekselius, L., & Andersson, G. (2001). Treatment of panic disorder via the Internet – a randomized trial of a self-help program. Behavior Therapy, 32, 751–764. Chambless, D. L., Caputo, G., Jasin, S., Gracely, E. J., & Williams, C. (1985). The Mobility Inventory for agoraphobia. Behaviour Research and Therapy, 23, 35–44. Chambless, D. L., Caputo, G. C., Bright, P., & Gallagher, R. (1984). Assessment of fear of fear in agoraphobics: the Body Sensations Questionnaire and the Agoraphobic Cognitions Questionnaire. Journal of Consulting and Clinical Psychology, 52, 1090–1097. Clark-Carter, D. (1997). Doing quantitative psychological research. Hove: Psychology Press. de Beurs, E., Wilson, K. A., Chambless, D. L., Goldstein, A. J., & Feske, U. (1997). Convergent and divergent validity of the Beck Anxiety Inventory for patients with panic disorder and agoraphobia. Depression and Anxiety, 6, 140–146. Febbraro, G. A. R., Clum, G. A., Roodman, A. A., & Wright, J. H. (1999). The limits of bibliotherapy: a study of differential effectiveness of self-administrated interventions in individuals with panic attacks. Behavior Therapy, 30, 209–222. Frisch, M. B., Cornell, J., Villanueva, M., & Retzlaff, P. J. (1992). Clinical validation of the Quality of Life Inventory. A measure of life satisfaction for use in treatment planning and outcome assessment. Psychological Assessment, 4, 92–101. Gillis, M. M., Haaga, D. A. F., & Ford, G. T. (1995). Normative values for the Beck Anxiety Inventory, Fear Questionnaire, Penn State Worry Questionnaire, and Social Phobia and Anxiety Inventory. Psychological Assessment, 7, 450–455. Gould, R. A., & Clum, G. A. (1993). A meta-analysis of self-help treatment approaches. Clinical Psychology Review, 13, 169–186. Klein, B., & Richards, J. C. (2001). A brief Internet-based treatment for panic disorder. Behavioural and Cognitive Psychotherapy, 29, 113–117. ˚ sberg, M. (1979). A new depression scale designed to be sensitive to change. Montgomery, S. A., & A British Journal of Psychiatry, 134, 382–389. ¨ st, L.-G., Breitholtz, E., & Thulin, U. (1997). Quality of life in anxiety disorder patients. Unpublished O manuscript, Stockholm. Pasveer, K. A., & Ellard, J. H. (1998). The making of a personality inventory: help from the WWW. Behavior Research Methods, Instruments and Computers, 30, 309–313. Reed, V., & Wittchen, H.-U. (1998). DSM-IV panic attacks and panic disorder in a community sample of adolescents and young adults: how specific are panic attacks? Journal of Psychiatric Research, 32, 335–345. Reips, U.-D. (2000). The Web experiment method: advantages, disadvantages, and solutions. In M. H. Birnbaum (Ed.), Psychological experiments on the Internet (pp. 89–117). San Diego: Academic Press. Richards, J. C., & Alvarenga, M. E. (2002). Extention and replication of an Internet-based treatment program for panic disorder. Cognitive Behaviour Therapy, 31, 41–47. Richards, J., Klein, B., & Carlbring, P. (2003). Internet-based treatment for panic disorder. Cognitive Behaviour Therapy, 32, 125–135. Schmidt, W. C. (1997). World-Wide Web survey research: benefits, pottential problems, and solutions. Behavior Research Methods, Instruments, & Computers, 29, 274–279. Statistics Sweden. (2004). Privatpersoners anva¨ndning av datorer och Internet [Use of computers and the Internet by private persons in 2003]. Retrieved 1 March, 2004. Available from http://www.scb.se/ Statistik/IT/IT0102/2003M00/IT0102_2003M00_BR_TKFT0401.pdf.
1434
P. Carlbring et al. / Computers in Human Behavior 23 (2007) 1421–1434
Stro¨m, L., Pettersson, R., & Andersson, G. (2004). Internet-based treatment for insomnia: a controlled evaluation. Journal of Consulting and Clinical Psychology, 72, 113–120. ˚ sberg, M. (1994). A new self-rating scale for depression and anxiety states based on the Svanborg, P., & A comprehensive psychopathological rating scale. ACTA Psychiatrica Scandinavica, 89, 21–28. Taylor, S. (2000). Understanding and treating panic disorder. Cognitive-behavioral approaches. Chichester: Wiley.
