Internet vs. paper and pencil administration of questionnaires commonly used in panic/agoraphobia research

Abstract

The aim of this study was to investigate the psychometric properties of Internet administered questionnaires used in panic research. Included were 494 people who had registered for an Internet-based treatment program for panic disorder (PD). Participants were randomly assigned to fill in the questionnaires either on the Internet or the paper-and-pencil versions, and then to fill in the same questionnaires again the next day using the other format. The questionnaires were the body sensations questionnaire [BSQ; Chambless, D. L., Caputo, G. C., Bright, P., & Gallagher, R. (1984). Assessment of fear of fear in agoraphobics: the body sensations questionnaire and the agoraphobic cognitions questionnaire. Journal of Consulting and Clinical Psychology, 52, 1090–1097], agoraphobic cognitions questionnaire [ACQ; Chambless, D. L., Caputo, G. C., Bright, P., & Gallagher, R. (1984). Assessment of fear of fear in agoraphobics: the body sensations questionnaire and the agoraphobic cognitions questionnaire. Journal of Consulting and Clinical Psychology, 52, 1090–1097], mobility inventory [MI; Chambless, D. L., Caputo, G., Jasin, S., Gracely, E. J., & Williams, C. (1985). The mobility inventory for agoraphobia. Behaviour Research and Therapy, 23, 35–44], beck anxiety inventory [BAI; Beck, A. T., Epstein, N., Brown, G., & Steer, R. A. (1988). An inventory for measuring clinical anxiety: psychometric properties. Journal of Consulting and Clinical Psychology, 56, 893–897], beck depression inventory II [Beck, A. T., & Steer, R. A. (1996). Beck Depression Inventory. Manual, Svensk version (Swedish version). Fagernes, Norway: Psykologiförlaget, AB], quality of life inventory [QOLI; Frisch, M. B., Cornell, J., Villanueva, M., & Retzlaff, P. J. (1992). Clinical validation of the quality of life inventory. A measure of life satisfaction for use in treatment planning and outcome assessment. Psychological Assessment, 4, 92–101], and montgomery Åsberg depression rating scale [MADRS; Svanborg, P., & Åsberg, M. (1994). A new self-rating scale for depression and anxiety states based on the comprehensive psychopathological rating scale. ACTA Psychiatrica Scandinavica, 89, 21–28]. Results showed largely equivalent psychometric properties for the two administration formats (Cronbach’s α between 0.79 and 0.95). The results also showed high and significant correlations between the Internet and the paper-and-pencil versions. Analyses of order effects showed an interaction effect for the BSQ and the MI (subscale Accompanied), a main effect was identified for ACQ, MI-Alone, BAI and BDI II. However, in contrast to previous research, the Internet version did not consistently generate higher scores and effect sizes for the differences were generally low. Given the presence of an interaction effect, we recommend that the administration format should be stable in research across measurement points. Finally, the findings suggest that Internet versions of questionnaires used in PD research can be used with confidence.

Introduction

With the advent of modern information technology new opportunities have emerged regarding treatment and assessment of panic disorder (Richards, Klein, & Carlbring, 2003). Although self-help approaches have been developed and evaluated previously (e.g., Gould & Clum, 1993), it was only recently that researchers begun to explore the feasibility of administering self-help treatment with minimal therapist intervention using the Internet (Richards et al., 2003). This novel approach has conveyed several advantages such as overcoming distances and facilitating access to therapist feedback on a rapid basis. In addition, once the costs for transferring the treatment to a website have been deducted, it is apparent that minimal therapist contact self-help via the Internet reduces costs (Carlbring et al., in press), when compared with standard face to face therapy. Although concerns have been raised regarding the use of self-help for panic disorder (Febbraro et al., 1999, Taylor, 2000), it is possible that Internet delivered self-help can offset the disadvantages, since therapist time can be freed up so that the clinician can consult colleagues for supervision and expertise (when responding to e-mails).

In parallel with the development of Internet-based treatment, administration of questionnaires are beginning to be transferred onto the Internet. There are obvious advantages with Internet administration of questionnaires. For example they can be filled out in the patients’ own homes; by making registration of all items obligatory before submission, missing values can be handled; data will appear directly for transfer to a statistics program; and finally scoring can be facilitated using scripts. Other advantages associated with Internet-administration include reduced costs, as well as the opportunity for researchers to access a larger and potentially more diverse population. However, psychometric properties of Internet administered measures cannot be taken for granted. This was highlighted in a review by Buchanan (2003) who also conducted a series of studies investigating psychometric properties of questionnaires. Although, some studies suggest that Internet-based questionnaires can generate equivalent information as paper-and-pencil tests in terms of psychometric properties and test characteristics (Andersson, Kaldo-Sandström, Ström, & Strömgren, 2003), it is less certain that norms can be transferred, and indeed adjustments in scoring might be needed (Buchanan, 2003). Therefore, the equivalence of Internet-based to pencil-and-paper versions of questionnaires cannot be assumed overall. Consequently, it has been recommended that each Internet-based measure be independently evaluated (Buchanan, 2003).

Research on Internet delivered treatment for panic disorder is currently being conducted by independent research groups (Alcaniz et al., 2003, Carlbring et al., 2003, Carlbring et al., in press, Carlbring et al., 2001, Klein and Richards, 2001, Richards and Alvarenga, 2002). In addition, some preliminary work has been published regarding Internet-based psychiatric assessment of panic patients (Carlbring et al., 2002). In the latter case, it turned out that the Internet was less than perfect for diagnostic purposes. However, so far Internet administration of self-report instruments used in panic research has not yet been evaluated. The aim of this study was to test the equivalence of paper-and-pencil and Internet-administered versions of several commonly used measures of panic-related variables. All questionnaires were regarded to be well-suited to a computerized medium as they are easily understood, brief, and require simple responses (via selection of responses from a Likert-type scale). The present study employed a randomized design in which half of the participants completed an Internet version of the questionnaires first and the paper-and-pencil version next. For the other half the order was reversed. In addition to establishing the psychometric properties of the measures used, we were also in a position to test order effects in a repeated measures design. This was intended to answer the question as to whether response formats (Internet vs. paper-and-pencil) can be regarded as interchangeable.