Literature Search

An electronic search strategy (See Appendix S3) was developed by a librarian (TR) and refined using the peer review of electronic search strategies checklist [7]. The search strategy was implemented in the following electronic databases: MEDLINE (1950 to week 12 2011), Healthstar, Cochrane CENTRAL, AMED (1985 to 2011), PsycINFO, The Cochrane Collaboration’s Methodology Register, EMBASE (1980 to 2011) and the Health and Psychosocial Instruments database (1985 to the last week of March 2011). Conference websites, scientific journals and experts in the field of CAM were also consulted in order to identify relevant publications of CAM questionnaires.

Eligibility Criteria

Questionnaires were included if they sought to assess the prevalence of a wide range of CAM use (more than one type or one sub-category of CAM), such as products and services provided by CAM practitioners, in pediatric patients (age 18 years and younger), and if they were reported in English or French language articles. All types of studies (e.g. psychometric studies, prevalence studies and clinical trials) were included if they reported using such a questionnaire. Grey literature, such as abstracts published in scientific journals or proceedings of conferences were excluded due to lack of information concerning questionnaires used. Results from our search were exported into a Reference Manager database.

Identification of Studies

Two members of the team (KTA and AB or BD) independently screened identified titles, abstracts and key words for relevance, and assessed potentially relevant studies for inclusion according to our eligibility criteria. Full text articles of records meeting the screening criteria were obtained. Each reviewer then decided independently which articles met the eligibility criteria and should be included in the systematic review. Disagreements were resolved by discussion with a third member of the team (JS or DM). If some data from the studies were insufficient or missing, we attempted to obtain this information from the authors by personal communication (email) up to three times.

Data Extraction Procedures

Data from included studies were independently extracted by three members of the team using the previously piloted data extraction form (KTA and MW or AB) and then entered into an Excel spread sheet. The extracted data included: characteristics of the studies (study design, objectives, geographical location, target population) and the questionnaires (purpose, child and/or proxy report, content, time to complete). The time to administer a tool was documented and rated as 1) short if it took less than 10 minutes, or 2) long if it took more than 10 minutes to complete [8].

The methodological quality of studies reporting each CAM questionnaire as well as the measurement properties of the CAM questionnaires themselves were assessed. When data were missing from a study, corresponding authors were e-mailed (two attempts) in an effort to obtain questionnaires and additional information. Authors were also e-mailed at the end of our systematic review in order to confirm results.

Synthesis

We completed a narrative summary of the included studies. Content analysis was used to study the content of the included articles. Descriptive statistics were used to describe the characteristics of the studies and questionnaires.

Search Results

The electronic search provided a total of 476 records after eliminating duplicates. Additional consultation of conference websites, scientific journals and experts in the field of CAM yielded another 107 records. From these 583 unique potentially relevant records, 104 met the screening criteria. We included 104 articles in the systematic review which summarized 100 studies from which 96 CAM questionnaires were identified (see Figure S1 for a flow diagram of included studies and Appendix S6 for a list of references for the included studies). Among authors of included studies, 69 (69/96 = 71.9%) responded to our query by providing additional information on their questionnaires’ measurement properties. Some of these authors no longer had access to questionnaires and a total of 50 authors ultimately provided copies for this review. Although we did not receive questionnaires from all authors, six CAM questionnaires were displayed in their respective publications and one was found online. After completing our review, 51 (51/96 = 53.1%) authors confirmed our appraisal of their studies and questionnaires, nine of them gave new information that did not modify our assessment and one author gave us new information that modified our assessment of their questionnaire [15].

Characteristics of the Studies

Included studies sought to assess children’s use of CAM (n = 100), perceptions regarding children’s use of CAM (n = 86) and attitudes towards CAM in general (n = 33). Most studies were cross-sectional surveys of CAM use (n = 98) and only two assessed CAM use longitudinally [16]; [17]. Only 58 studies provided a definition of what they considered to be CAM and definitions varied tremendously. The most common definition was the NCCAM definition (n = 18 studies) [1].

Studies were conducted in 19 countries, the most common being the United States (n = 35), United Kingdom (n = 15), Canada (n = 11), Australia (n = 8), Turkey (n = 6), Israel (n = 3), Ireland (n = 2), Germany (n = 2), Switzerland (n = 2) and New Zealand (n = 2). Three studies did not disclose precisely where they were conducted and three studies were conducted in multiple countries. Studies were conducted in general pediatrics (n = 24), in specific pediatric conditions (n = 63), such as cancer (n = 19), neurological (n = 16), respiratory (n = 11), gastrointestinal [5], rheumatologic (n = 4), dermatologic (n = 3), haematological (n = 2), metabolic (n = 2) and otolaryngological conditions (n = 1), as well as in both general pediatrics and specific diseases (n = 5). Other studies also assessed CAM use in children undergoing surgical procedures (n = 2), children with a range of diverse chronic diseases (n = 3) or acute illnesses (n = 2) as well as homeless youth (n = 1).

Characteristics of the CAM Questionnaires

All included CAM questionnaires sought to discriminate between users and non-users of CAM (i.e. discriminative purpose) and none were reported to be designed to assess changes in CAM use over time (evaluative purpose). Thirty nine percent (n = 37) of the questionnaires assessed disease-specific CAM use while 42% (n = 40) assessed the use of CAM in a general pediatric population without referring to a specific disease, and one questionnaire assessed both current disease-related use as well as use for other reasons. Other questionnaires were not described in enough detail to determine whether they were generic or disease-specific (n = 18, 19%). Most questionnaires were targeted at parents, while 23% of questionnaires gathered children and adolescent self-reports and only two studies mentioned using a questionnaire specifically designed to gather children and adolescents self-report. Children completed questionnaires on their own beginning at 10 years of age while children as young as seven years helped their parents to answer the questions.

The content of the questionnaires varied considerably. The number of types of CAM listed in questionnaires ranges from 3 to 161, some questionnaires listing only broad categories of CAM (n = 7) and others presenting an exhaustive list of herbal products, vitamins and minerals. CAM types and categories were often not adequately reported by studies and their diversity made it difficult to synthesize the data. Questionnaires also included a wide range of domains and items linked to CAM use (see Figure S2). The most common domains and items were: reasons for CAM use (n = 61, 63.5%), helpfulness (n = 58, 60.4%), communication with health providers concerning CAM (n = 51, 53.1%), source of information about CAM (n = 48, 50%), CAM use by parents/family (n = 40, 41.7%) and costs of CAM (n = 38, 39.6%).

Concerning ease of use of the questionnaires, only ten were reported to have a completion time under 10 minutes and the others were reported to take up to 60 minutes to complete. A summary of the characteristics of questionnaires tested for measurement properties is shown in Table S1.

Methodological Quality of the Studies

Using the COSMIN checklist assessment, none of the studies showed adequate methodological quality. Of the 11 questionnaires which were mentioned to have been tested for content validity [18]–[28], no studies were descriptive enough to reach the COSMIN criteria. Most of the publications (82%) describing those questionnaires did not mention whether items refer to and comprehensively represent relevant aspects of the construct to be measured. They also did not state whether items are relevant for the population surveyed or for the purpose of the questionnaire. Two CAM questionnaires [20]; [21] had their content validity assessed by both patients and experts, which is an advantage [13]; [29], while the other nine had it assessed by either patients (n = 2) or experts (n = 7).

Three CAM questionnaires were assessed for construct validity [15]; [30]; [31] but none were adequately reported. Most authors did not provide information on missing items, a priori hypotheses regarding the direction and magnitude of correlations, comparator instrument, sample size and statistical analyses. Authors of one study reported a small sample size (n = 25 patients) and the use of Spearman correlation coefficients by means of personal communication [15].

Of the four questionnaires which were reported to have been assessed for test-retest reliability [15]; [30]; [32]; [33], inter-rater reliability [30] or internal consistency [15], none was described in enough detail to meet the COSMIN criteria. Most of the publications describing those questionnaires did not report how missing items were handled and how repeated measurements were conducted (mode of administration, sample size, statistical analyses and time interval for test-retest). Some questionnaires were also tested with long delays between first and second administration (4 to 6 weeks), different modes of administration between test and retest, small sample size and inadequate statistical analyses. One questionnaire [15] showed variable levels of test-retest agreement using intraclass correlation coefficients as well as internal consistency using Cronbach’s alpha (data provided by authors by means of a personal communication).

Measurement Properties of CAM Questionnaires

When evaluating the quality of measurement properties for each of the 96 included CAM questionnaires using the Terwee criteria, none was rated as thoroughly valid. No questionnaire reported an assessment of floor and ceiling effects. Only 16 CAM questionnaires were assessed for one or more of the following properties: content validity, and test-retest and inter-rater reliability (See summary characteristics of the most promising questionnaires in Table S1).

Ratings for content validity were indeterminate since questionnaires were not reported to have used appropriate methods (e.g. purpose of the questionnaire, concepts to measure, involvement of the target population, methods of item selection and reduction). None of the included questionnaires described their precise purpose (whether they are discriminative, evaluative or predictive) of the questionnaires. Measured concepts were often listed but not in a thorough manner. Only two questionnaires [22]; [25] were reported to be comprehensive (i.e. including all relevant items) and have involved the target population of patients. One showed a high level of agreement between experts ratings for included items [22].

Construct validity was rated as indeterminate for the three questionnaires [15]; [30]; [31]. The domain scores of one questionnaire were compared with measures of parental discipline and spiritual practice/beliefs, which yielded Spearman’s correlation coefficients ranging from −0.52 (p<0.01) to 0.34 (p = 0.11) [15]. The results of the second questionnaire were contrasted with medical and nursing notes documenting CAM use for inpatients [30], while findings from the third questionnaire were evaluated against CAM products found in patients’ homes [31]. The indeterminate rating was granted for each of these because correlation coefficients were not presented, not strong enough or not provided for all domains of the questionnaire.

All ratings for the four questionnaires [15]; [30]; [32]; [33] having assessed test-retest reliability were indeterminate because the majority did not present agreement coefficients such as Intraclass correlation coefficient or Kappa coefficients or confidence intervals. One of these questionnaires [15] presented variable scores of agreement depending on the item (Intraclass correlation coefficients between 0.52 to 0.78), which ranges from a negative to a positive rating, which amounted to an indeterminate rating. The same questionnaire also showed poor to questionable levels of internal consistency (α = 0.28 to 0.64). Inter-rater reliability was also indeterminate for the only questionnaire having assessed it (because the percentage of CAM use according to the different raters was reported but not the agreement between raters) [30].

According to the Cohen score, no questionnaire was considered to be a well-established one, two approached the level of a well-established assessment [33]; [34] and seven were promising assessments [18]; [19]; [21]; [22]; [24]; [28]; [35]. The remainder (n = 87) were either not described enough in publications (and the questionnaires not provided in the publication nor available upon request to the author) or their measurement properties were not reported at all. (See summary characteristics of the most promising questionnaires in Table S1).

Finally, among all included questionnaires, 19 were reported to have good readability/comprehension but only one of them using proper testing. This questionnaire was assessed using the Flesch-Kincaid grade reading level (score = 5.7), a test designed to indicate comprehension difficulty and based on two core measures (word and sentence length).