Comparison of Open and Closed Questionnaire Formats in Obtaining Demographic Information From Canadian General Internists

Abstract

The objective of this study was to compare the impact of closed- versus open-ended question formats on the completeness and accuracy of demographic data collected in a mailed survey questionnaire. We surveyed general internists in five Canadian provinces to determine their career satisfaction. We randomized respondents to receive versions of the questionnaire in which 16 demographic questions were presented in a closed-ended or open-ended format. Two questions required respondents to make a relatively simple computation (ensuring that three or four categories of response added to 100%). The response rate was 1007/1192 physicians (80.0%). The proportion of respondents with no missing data for all 16 questions was 44.7% for open-ended and 67.0% for closed-ended formats (P < 0.001). The odds of having missing items remained higher for open-ended response options after adjusting for a number of respondent characteristics (2.67, 95% confidence interval 2.01 to 3.55). For the two questions requiring computations focused on professional activity and income, there were more missing data (P = 0.02, 0.02, respectively) but fewer inaccurate responses (P = 0.009, 0.20, respectively) for the open-ended compared to the closed-ended format. Investigators can achieve higher response rates for demographic items using closed format response options, but at the risk of increasing inaccuracy in response to questions requiring computation.

Introduction

Comprehensive data collection through survey questionnaires requires both maximizing questionnaire response rates and ensuring questionnaire completeness and accuracy among those responding. Advantages and disadvantages of different methods of questionnaire administration (i.e., face-to-face interview, telephone survey, and mailed self-administered completion) have been well described 1, 2. In two randomized trials of health survey administration comparing these three methods, Hochstein [3] and Siemiatycki [4] both detected very small differences in response rates, but found higher costs generated by face-to-face interviews. Siemiatycki also found that sensitive questions about income and Medicare number were more readily answered by mailed self-administered questionnaire than by telephone and face-to-face interviews. In addition, when adult asthmatics were randomized to receive self-administered versus interviewer-administered questionnaires we found that the proportion of endorsed quality of life symptoms was higher using the former than the latter [5].

Investigators have also examined the effect of item formatting on the completeness and accuracy of survey data. Schumann and Presser [6] reported a public opinion survey asking about the most important problem facing the United States. They found that different formats yielded different answers; using the open-ended format, respondents were more likely to complain about the political leadership and less likely to comment on violence than when using the closed-ended format. In another study, prompting workers with closed-ended questions resulted in identification of occupational exposure to an average of 11.0 potentially hazardous materials per respondent, whereas open-ended questions resulted in reporting of 3.1 materials [7].

Some have suggested that, to reduce the time and effort spent on questionnaire development and to facilitate direct comparisons across questionnaires, investigators should use standardized demographic questions in creating new survey instruments [8]. Demographic information describes survey respondents, allows comparisons with nonrespondents, and facilitates assessment of the generalizability of questionnaire results. In addition, demographic data permit comparisons among important respondent subgroups and adjustment for differences among them. We conducted an investigation designed to examine the relative completeness and accuracy of demographic data from open-ended versus closed-ended formats in a survey of career satisfaction among general internists in five Canadian provinces.