A comprehensive testing protocol for MRI neuroanatomical segmentation techniques: Evaluation of a novel lateral ventricle segmentation method

Abstract

Although a wide range of approaches have been developed to automatically assess the volume of brain regions from MRI, the reproducibility of these algorithms across different scanners and pulse sequences, their accuracy in different clinical populations and sensitivity to real changes in brain volume have not always been comprehensively examined. Firstly we present a comprehensive testing protocol which comprises 312 freely available MR images to assess the accuracy, reproducibility and sensitivity of automated brain segmentation techniques. Accuracy is assessed in infants, young adults and patients with Alzheimer's disease in comparison to gold standard measures by expert observers using a manual technique based on Cavalieri's principle. The protocol determines the reliability of segmentation between scanning sessions, different MRI pulse sequences and 1.5 T and 3 T field strengths and examines their sensitivity to small changes in volume using a large longitudinal dataset. Secondly we apply this testing protocol to a novel algorithm for segmenting the lateral ventricles and compare its performance to the widely used FSL FIRST and FreeSurfer methods. The testing protocol produced quantitative measures of accuracy, reliability and sensitivity of lateral ventricle volume estimates for each segmentation method. The novel algorithm showed high accuracy in all populations (intraclass correlation coefficient, ICC > 0.95), good reproducibility between MRI pulse sequences (ICC > 0.99) and was sensitive to age related changes in longitudinal data. FreeSurfer demonstrated high accuracy (ICC > 0.95), good reproducibility (ICC > 0.99) and sensitivity whilst FSL FIRST showed good accuracy in young adults and infants (ICC > 0.90) and good reproducibility (ICC = 0.98), but was unable to segment ventricular volume in patients with Alzheimer's disease or healthy subjects with large ventricles. Using the same computer system, the novel algorithm and FSL FIRST processed a single MRI image in less than 10 min while FreeSurfer took approximately 7 h. The testing protocol presented enables the accuracy, reproducibility and sensitivity of different algorithms to be compared. We also demonstrate that the novel segmentation algorithm and FreeSurfer are both effective in determining lateral ventricular volume and are well suited for multicentre and longitudinal MRI studies.

Introduction

A range of automated segmentation algorithms are available for determining the volume of various local brain regions, including widely applied techniques such as FreeSurfer (Fischl et al., 2002), FSL FIRST (Patenaude et al., 2007), ANIMAL (Collins et al., 1999) and the LONI pipeline (Macdonald et al., 1994). Since their development these algorithms have been applied to neurological and psychiatric disorders such as Alzheimer's disease (Cherubini et al., 2010), multiple sclerosis (Benedict et al., 2009) and schizophrenia (Kuperberg et al., 2003) and are also being used to investigate the developing brain in childhood and adolescence (Lenroot et al., 2007). However, early validation studies were limited to healthy young adults and did not report between session, pulse sequence or scanner reproducibility; measures of sensitivity to changes in regional brain volume were rarely presented. These issues are critically important for multi-centre and longitudinal studies, where segmentation algorithms should be sensitive to small changes in brain volume but insensitive to the use of different magnetic resonance imaging (MRI) scanners (reflecting differences in scanner hardware and software and performance differences between otherwise identical scanners). Another consideration is that some algorithms are not able to segment particular types of images, or require varying degrees of user intervention and therefore may become impractical for studies with large cohorts. These problems may explain why manual segmentation of brain regions is still commonplace in the literature (Doty et al., 2008, Dutt et al., 2009, Ettinger et al., 2007, Jack et al., 2008b). Recently, more rigorous studies have been published comparing segmentation algorithms in terms of accuracy (Babalola et al., 2009, Morey et al., 2009), test–retest reproducibility (Morey et al., 2010), sensitivity to changes in brain structure (Bergouignan et al., 2009), and the effect of MRI acquisition parameters on segmentation reproducibility in terms of global (de Boer et al., 2010, Shuter et al., 2008), subcortical and cortical volumes (Jovicich et al., 2009, Wonderlick et al., 2009). However to our knowledge, no publically available dataset exists that may be used to measure segmentation performance in terms of all the above parameters.

The aim of this paper is two-fold, a) to directly address this point by developing a comprehensive testing protocol to determine the accuracy, reproducibility and sensitivity of MRI neuroanatomical segmentation techniques using publicly available data which can be used by other investigators and b) to apply the testing protocol to assess lateral ventricle segmentation using a new fully automated technique and to compare this with two popular freely available packages, FreeSurfer and FSL FIRST.

Specifically with respect to lateral ventricle segmentation:

• The accuracy of the algorithms will be tested in healthy adults, patients with Alzheimer's disease and infants, reflecting a wide range of brain morphology.

• The reproducibility of the algorithms using the same participants will be tested between sessions, across pulse sequences and on data from a 1.5 T and 3 T MRI scanner; reflecting inter-session scanner variability, acquisition protocol variability and hardware variability.

• The sensitivity of the algorithms to changes in ventricular volume will be tested on a longitudinal dataset where age related changes in brain morphology are expected to occur.

Our focus on the lateral ventricles is of clinical relevance and research interest because increased volume of this region has been implicated in a number of psychiatric and neurological disorders. Dilation of the lateral ventricles is one of the most consistent findings in both schizophrenia (Kempton et al., 2010, Wright et al., 2000) and bipolar disorder (Kempton et al., 2008). Although hippocampal volume reduction is the most prominent finding, ventricular volume increase is also a key sign of progression in Alzheimer's disease (Zakzanis et al., 2003) and mild cognitive impairment (Carmichael et al., 2007).