The National Institutes of Health (NIH) has an extensive history of supporting child health (CH) research through a variety of infrastructure grants, including funding for clinical research centers (CRCs), institutional clinical research curricula and training, and career development programs for CH researchers, pediatric clinical research networks, and education and meeting activities (1,2). In 2006, the NIH launched the Clinical and Translational Science Award (CTSA) program with the vision that these enhanced institutional infrastructure grants would better catalyze the translation of new scientific discoveries into improved health of human populations (3,4). Existing pediatric CRCs and some institutional training and career development programs were subsumed into the CTSAs as linked awards; there was no requirement for a separate CH component. The NIH Reform Act of 2006 changed this framework in order to preserve independent funding and infrastructure for pediatric CRCs (5). Subsequent CTSA Requests for Applications (RFAs) have allowed for the appointment of a pediatric coprincipal investigator under a single CTSA who would have direct authority over a separate budget or would otherwise secure institutional independence of pediatric CRCs with respect to finances, infrastructure, resources, and research agenda. As of 2011, 58 of the 60 CTSAs include CH components and 9 have principal investigators who are themselves pediatricians (6). However, none of the CTSAs have chosen to create a separate budget for CH, which makes it difficult to directly identify the resources devoted to support of CH research in individual institutions and to measure the effectiveness of the CTSA program as a whole in supporting CH research over time.

Concurrent with the funding of the first group of 12 CTSAs in 2006, the CTSA Consortium-Child Health Oversight Committee (CC-CHOC) was established as a part of the broader CTSA Consortium governance structure and was charged with the responsibility of facilitating CH research throughout the Consortium (7,8). A CC-CHOC workgroup was formed to develop and test a measurement strategy to (i) enable CTSA institutions to uniformly assess their current levels of support, research activity, training of new investigators, and multicenter collaborations related to CH research; (ii) track changes in these parameters over time within institutions; and (iii) eventually estimate the impact of the CTSA Consortium, as a whole, on CH research. This report describes the initial development and evaluation of the measurement strategy and presents data on specific metrics from institutions receiving CTSAs during 2006–2009.

Results

The CC-CHOC workgroup developed metrics for CH research that included measurement of support and infrastructure and research activity, career development of new investigators, collaborative research, and leadership ( Table 1 ). All 46 institutions that received CTSAs from 2006 to 2009 received an invitation to submit data for these metrics for the most recent award or calender year; 37 (80%) institutions provided responses. Comparing responding and nonresponding institutions, there was no evidence of a difference in geographic distribution by region (P = 0.96), year of award (P = 0.75), or the amount of the CTSA in dollars by quartiles of percentile rank (P = 0.33). A children’s hospital was listed as a partner institution and a pediatrician was the CTSA principal investigator for 34 (92%) and 5 (14%) responding institutions, respectively, vs. for 6 (67%, P = 0.08) and 0 (P = 0.57) of nonresponding institutions, respectively.

Table 1 Measurement domains and description of metrics
Full size table

The numeric data needed to calculate each metric were provided by most of the responding institutions ( Table 2 ). Except for the collaborative research metric, data were provided similarly for CH research, investigators, or pediatric subjects (the numerators) and for the CTSA or institution as a whole (the denominators). Data regarding the number of multicenter studies involving more than one CTSA were the most difficult for institutions to provide, both for studies involving pediatric subjects and for studies involving subjects of any age.

Table 2 Institutions providing data and descriptive data regarding metrics
Full size table

The numeric data values reported by individual institutions ranged widely ( Table 2 ). Likewise, there was wide variation in the calculated metrics ( Figure 1 ), most notably in the metrics related to active CRC studies, mentored research training awards, and CTSA leadership positions involving CH researchers. There was diversity among the institutions that represented the upper bounds of the ranges, with six different institutions represented (the upper bound was represented by one institution for the metrics related to overall funding, active CRC studies, and mentor research training awards, one institution for both metrics related to research activity, and four different institutions for the other metrics). Nonetheless, despite the overall variation, the medians of the metrics were generally consistent ( Figure 1 , range of medians, 0.11–0.24) and the central portions of their distributions, as represented by the interquartile ranges, were relatively narrow.

Figure 1
figure 1

Distribution of metric proportions across CTSA institutions. Box plots show the distribution of metric proportions reported by the 37 responding institutions (minimum, 25%, 50%, 75%, maximum). Table 1 provides the full description of each metric. Metric III, collaborative research (multicenter studies with ≥1 CTSA institution participating), was not included because few institutions were able to provide data for the institution as a whole (see Table 2 ). The numbers of institutions included in each distribution are overall funding, 36; pilot grants, 33; active CRC studies, 35; IRB-approved studies, 31; IRB-approved and funded studies, 27; mentored clinical/translational research training awards, 27; and leadership positions, 36. CRC, clinical research center; CTSA, Clinical and Translational Science Award; IRB, institutional review board.

Full size image

Challenges in collecting the numeric data were due primarily to issues related to the interpretation of the definitions and the data sources of the metrics ( Table 3 ). Differences between NIH and institutional review board (IRB) definitions of CH research and whether to include other sources of support (e.g., supplemental awards, institutional matching funds) were identified commonly. Less common, but more fundamental, were issues of whether studies that relate to CH but do not involve children as research subjects directly should be viewed as CH research, such as health-care services research and maternal health studies. Inadequate sources of data were a major challenge due to the lack of variables that clearly identify CH research studies and due to difficulties in synchronizing the records of separate institutional databases, such as IRB and grant funding databases, to perform queries using the combined data fields from each database.

Table 3 Description of issues identified while collecting data regarding the metrics
Full size table

Institutions provided additional qualitative statements of how their CTSA prompted the development of new institutional strategies and initiatives to enhance CH research, including establishing multidisciplinary CH research teams or committees to better integrate CH research into the CTSA, enhance linkages with basic science researchers and developing specific foci of expertise in CH research; using evidence of CTSA support to obtain additional matching funds to support CH research and new CH investigators; using indirect funds to increase investment in CH research; recruiting CH researchers to compete for pilot grants; establishing a community-based CH research network; and inaugurating an annual pediatric research day event.

Discussion

The metrics presented in this cross-sectional survey illustrate that institutions that received CTSAs from 2006 to 2009 provide substantial support for CH research through overall funding and pilot study grants, CRC utilization, and mentored research training awards for CH investigators. In these institutions, roughly a quarter of all IRB-approved studies include pediatric subjects, and nearly a fifth of CTSA leadership positions are filled by pediatricians. These findings may be positively skewed by the trend toward a higher proportion of institutions with a children’s hospital listed as a partner institution among responding vs. nonresponding institutions.

Although these findings are generally positive, additional investment by CTSAs in CH research will be required to achieve the major breakthroughs necessary to improve the health of children from infancy to adulthood. As reported in this study, the proportions of overall funding and pilot study grants devoted to CH research in CTSA institutions are consistent with the proportion of NIH funding targeted toward CH research reported recently (9). However, the proportion of overall NIH funding for CH research has declined by nearly a third over the past two decades (9). Moreover, leaders in the pediatric research community have identified major unmet needs for CH research that must be addressed to improve the future health and well-being of children in the United States (9,10).

These results should also be viewed in the context of the limitations of this first attempt to assess the impact of the CTSA program on CH research. The lack of pre-award data and the cross-sectional design of this study, which occurred 1–4 years after the receipt of the CTSA in the cohorts of institutions receiving awards during 2006–2009, do not allow us to assess trends in the support for CH research in relation to the CTSAs in individual institutions or across the Consortium as a whole. Although we provided instructions to standardize data collection, issues related to metric definitions and data sources ( Table 3 ) may have resulted in underestimates or greater variability with respect to the metric data reported by individual institutions. We did not perform formal assessments of the reliability of data collection at individual CTSA institutions or the validity of the data in relation to externally verified measures, although we plan to do so in the future. The difficulties we encountered with definitions and data sources for the metrics illustrate general challenges inherent in quantifying the impact of the CTSAs on clinical and translational research in general, and on CH research in particular. Some of these barriers can be addressed easily. For instance, we have already refined the definitions of the numerators and denominators to be more detailed and specific. Other barriers, such as those involving data sources, are likely to require new or revised databases with additional data fields and improved linkages. The difficulty we encountered in capturing information about research collaborations across CTSA institutions from existing grant management databases deserves particular attention because this metric was one of the most difficult to quantitate and yet it uniquely reflects the power of the Consortium.

Nonetheless, the metrics presented in this study provide an important starting point for evaluating the impact of CTSAs on CH research. First, the metrics were developed intentionally to align with the strategic goals of the CTSA Consortium. Therefore, over time, they should reflect enhancements to the clinical and translational research infrastructure that occur in individual institutions and throughout the Consortium. Second, they provide an opportunity for CTSA institutions to assess their current support for CH research in relation to the Consortium as a whole. Third, the metrics provide a standardized mechanism for individual institutions to track the effect of their efforts to enhance CH research over time through changes in either the numeric data values (i.e., the numerators for the metrics), the calculated proportions, or both. Institution-specific tracking of these metrics is important because factors that limit the validity of comparisons across institutions—particularly those related to sources of data—are likely to be less variable within institutions over time. Moreover, by creating an institutional process for measuring CH research efforts, CTSA institutions can examine whether they are achieving the CH research goals they set. Fourth, the high response rate indicates there is support for this effort across the Consortium.

To better understand the impact of CTSAs on CH research, we also need better approaches for describing CH research portfolios, including the spectrum of translational research (i.e., early- through late-stage translation) conducted in CTSA institutions (11), research that crosses populations (e.g., women’s health research with outcomes relevant to CH, and studies of adult diseases that have their antecedents in conditions affecting children), and research that has an important impact on CH but does not involve pediatric subjects directly.

As the CTSA Consortium matures, its strategic goals evolve, and the definition of CH research is refined, the metrics used to describe the impact of CTSAs on CH research may need to be adapted. We hope these data will also inform a discussion of what may be considered adequate (or optimal) performance with respect to these metrics. Finally, one of the aspirations of this effort is to develop metrics to evaluate the impact of CTSAs on CH, not just CH research. Indeed, the process of developing such a measurement strategy, in and of itself, demonstrates one of the potential strengths of the CTSA Consortium in facilitating CH research (8).

Methods

Definitions

The workgroup used the NIH definitions for clinical and translational research (12) and for inclusion of children as participants in research involving human subjects (13) to define CH research as “all clinical and translational research involving subjects less than 21 years of age that impacts children’s health or addresses childhood diseases, including maternal-fetal research, as well as studies involving infants, children and adolescents.” Research that primarily focused on adults but in which a few older adolescents were enrolled was excluded.

Development of the Metrics

From 2008 to 2009, workgroup members representing 12 CTSAs participated in monthly teleconferences during which measurement domains and specific metrics were proposed, discussed, and selected. Decisions were made by consensus during all phases of this process. The workgroup reviewed the CTSA Consortium Strategic Goals (14) and chose the following measurement domains, which aligned with these Strategic Goals: (i) funding, infrastructure support, and research activity; (ii) career development of new investigators; (iii) collaborative research; and (iv) leadership. Workgroup members were then asked to propose specific metrics in each of these domains and then to rank prospective metrics using the following criteria: (i) capacity to utilize current systems to capture existing data; (ii) generalizability across most institutions; and (iii) ability to identify “new value” attributable the CTSA infrastructure. Metrics that were ranked highly across these criteria were selected for further development. No metrics were added or deleted thereafter. Several members of the workgroup (S.L.B., C.D.S., and W.C.H.) collaborated to draft a written description of the metrics with specific definitions of the numerator and denominator for each metric and instructions for data collection. Other workgroup members provided critique of the draft, and comments were incorporated into a revised version that was presented to the workgroup in the spring of 2009. During the summer of 2009, members of the workgroup pilot-tested procedures for collecting the required data in six of their own institutions and the metric definitions and data collection procedures were revised based on their recommendations. The proposal to implement the data collection strategy for the metrics was reviewed and approved by the CTSA Consortium Steering Committee, a body that consists of all participating CTSA principal investigators and NIH CTSA representatives.

Collection of Metric Data

In November 2009, the final version of the metrics ( Table 1 ) and a worksheet for data collection with instructions were distributed by e-mail to the CC-CHOC member or other relevant contact at each CTSA institution. For each metric, the institution recorded the numeric data and provided qualitative comments, as needed, about how the metric was interpreted and data were collected, as well as any barriers or difficulties encountered in this process. An open-ended field allowed for additional qualitative descriptions of institutional enhancements to promote the conduct of CH research. Given the need to assess administrative financial and IRB databases, data were collected primarily by administrative personnel under the supervision of the CC-CHOC member and either entered directly by each institution using a confidential access code into a Research Electronic Data Capture (REDCap, Vanderbilt Institute for Clinical and Translational Research, Nashville, Tennessee) database (15) or transmitted confidentially to the Institute and subsequently entered into the database. Four follow-up e-mail reminders were sent to CC-CHOC members at all institutions, but nonresponding institutions were not contacted individually either by e-mail or by telephone.

Data and Statistical Analyses

REDCap and SPSS, version 18 (IBM SPSS Statistics, formerly Predictive Analytics SoftWare Statistics, Armonk, NY) were used to generate descriptive statistics. Qualitative data were abstracted and synthesized by members of the workgroup (W.C.H., C.D.S., J.W., S.L.B.) and reviewed by the other members. Fisher’s exact tests were used to compare institutional characteristics between responding and nonresponding CTSA institutions.

Statement of Financial Support

This publication was supported by National Institutes of Health (NIH)/National Center for Research Resources Clinical and Translational Science Award grant nos. UL1 RR024150 (Mayo Center for Translational Science Activities, Mayo Clinic); UL1 RR024975 (Vanderbilt Institute for Clinical and Translational Research, Vanderbilt University); UL1 RR029884 (Arkansas Center for Clinical and Translational Research, University of Arkansas for Medical Sciences); UL1 RR025744 (The Stanford Center for Clinical and Translational Education and Research, Stanford University); UL1 RR025741 (Northwestern University Clinical and Translational Sciences Institute, Northwestern University); UL1 RR024153 (University of Pittsburgh Clinical and Translational Science Institute, University of Pittsburgh); UL1 RR025758 (Harvard Catalyst: The Harvard Clinical and Translational Science Center, Harvard University); UL1 RR024146 (UC Davis Clinical and Translational Science Center, University of California, Davis); UL1 RR024986 (Michigan Institute for Clinical and Health Research, University of Michigan at Ann Arbor); and UL1 RR024160 (University of Rochester Clinical and Translational Sciences Institute, University of Rochester School of Medicine and Dentistry). Its contents are solely the responsibility of the authors and do not necessarily represent the official views of the NIH.

Disclosure

The authors have no relevant financial disclosures or other potential conflicts of interest to report.