Elsevier

The Journal of Pediatrics

Volume 161, Issue 6, December 2012, Pages 1041-1046.e2
The Journal of Pediatrics

Original Article
Improving the Neonatal Research Network Annual Certification for Neurologic Examination of the 18-22 Month Child

https://doi.org/10.1016/j.jpeds.2012.05.048Get rights and content

Objective

To describe the Neonatal Research Network's efforts to improve the certification process for the Follow-Up Study neurologic exam and to evaluate inter-rater agreement before and after two annual training workshops.

Study design

The Neonatal Research Network Follow-Up Study is a multi-center observational study that has examined more than 11 500 infants from 1998-2010 and born ≤26 weeks gestational age at 18-22 months corrected age for neurodevelopmental outcome. The percentages of examiners who agreed with the Gold Standard examiner on 4 neurodevelopmental outcomes on the initial training video and a test video were calculated. Consistency among examiners was assessed with the first-order agreement coefficient statistic.

Results

Improvements in agreement among examiners occurred between 2009 and 2010 and between initial training and test. Examiner agreement with the Gold Standard during the initial training was 83%-91% in 2009 and 89%-99% in 2010. Examiner agreement on the workshop test video increased from 2009-2010 with agreement reaching 100% for all four neurodevelopmental outcomes examined in 2010. First-order agreement coefficient values for the four neurodevelopmental outcomes on the training videos ranged from 0.64-0.82 in 2009 and 0.77-0.97 in 2010.

Conclusions

We demonstrate the importance of annual certification and the benefits of evaluation and revision of certification protocols to achieve high levels of confidence in neurodevelopmental study outcomes for multi-center networks.

Section snippets

Methods

The NICHD NRN maintains a study manual with standard definitions (Table I). A diagnosis of neurodevelopmental impairment (NDI) is implemented for both randomized trials and observational follow-up studies. The definition of NDI is the presence of any of the following: moderate to severe CP, bilateral blindness, bilateral hearing impairment requiring amplification, (GMFCS)6, 7 ≥level 2 reflecting moderate to severe motor impairment, or a Bayley Scales of Infant and Toddler Development 3rd

Results

The 4 neurologic outcomes on the training videos for 2009 and 2010 are shown in Table III to illustrate the diagnoses chosen for the training process. In 2009, the centers' pre-workshop DVD scoring agreed with the Gold Standard rater 83% for normal neurologic exam to 91% for the GMFCS ≥ level 2 across the 6 videos (Table IV). During the workshop for real-time certification training in 2009, agreement on the test video increased to 100% for the diagnoses of abnormal neurologic exam and presence

Discussion

Challenges remain in the attempt to achieve inter-rater reliability for the neurologic exam and for the diagnosis of CP in young children. Although use of the GMFCS in conjunction with the classic neurologic exam has improved examiner accuracy, diagnosing severity of impairment is not a perfect science.11 A number of investigators have made substantial progress in initiatives to simplify and categorize the methodology for diagnosing CP.10, 17, 18 The NRN has begun to use the classification

References (19)

  • Y. Blanchard et al.

    Inter-rater reliability of early intervention providers scoring the Alberta Infant Motor Scale

    Pediatr Phys Ther

    (2004)
  • A. Cusick et al.

    Effect of rater training on reliability of Melbourne Assessment of Unilateral Upper Limb Function scores

    Dev Med Child Neurol

    (2005)
  • D.A. Cook et al.

    Effect of rater training on reliability and accuracy of mini-CEX scores: a randomized, controlled trial

    J Gen Intern Med

    (2009)
  • P. Lyden et al.

    National Institutes of Health Stroke Scale certification is reliable across multiple venues

    Stroke

    (2009)
  • C. Amiel-Tison et al.

    Neurological development from birth to six years; guide for examination and evaluation

    (2001)
  • R.J. Palisano et al.

    Validation of a model of gross motor function for children with cerebral palsy

    Phys Ther

    (2000)
  • R.J. Palisano et al.

    Development and reliability of a system to classify gross motor function in children with cerebral palsy

    Dev Med Child Neurol

    (1997)
  • The Psychological Corporation

    Bayley Scales of Infant and Toddler Development

    (2006)
  • M.J. Briggs-Gowan et al.

    The Brief Infant-Toddler Social and Emotional Assessment: screening for social-emotional problems and delays in competence

    J Pediatr Psychol

    (2004)
There are more references available in the full text version of this article.

Cited by (0)

The National Institutes of Health and the Eunice Kennedy Shriver National Institute of Child Health and Human Development (NICHD) provided grant support for the Neonatal Research Network's (NRN) Follow-Up Study. Data collected at participating sites of the NICHD NRN were transmitted to RTI International, the data coordinating center for the network, which stored, managed, and analyzed the data for this study. J.N.'s time was supported by the NICHD Neonatal Research Network, as the Data Coordinating Center coordinator for the Follow-Up Study. The authors declare no conflicts of interest.

A list of members of the Follow-Up Study Group of the Eunice Kennedy Shriver National Institute of Child Health and Human Development Neonatal Research Network is available at www.jpeds.com (Appendix).

View full text