Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Article
  • Published:

CellCognition: time-resolved phenotype annotation in high-throughput live cell imaging

Nature Methods volume 7, pages 747–754 (2010)Cite this article

Subjects

Abstract

Fluorescence time-lapse imaging has become a powerful tool to investigate complex dynamic processes such as cell division or intracellular trafficking. Automated microscopes generate time-resolved imaging data at high throughput, yet tools for quantification of large-scale movie data are largely missing. Here we present CellCognition, a computational framework to annotate complex cellular dynamics. We developed a machine-learning method that combines state-of-the-art classification with hidden Markov modeling for annotation of the progression through morphologically distinct biological states. Incorporation of time information into the annotation scheme was essential to suppress classification noise at state transitions and confusion between different functional states with similar morphology. We demonstrate generic applicability in different assays and perturbation conditions, including a candidate-based RNA interference screen for regulators of mitotic exit in human cells. CellCognition is published as open source software, enabling live-cell imaging–based screening with assays that directly score cellular dynamics.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Figure 1: Supervised machine learning and classification of morphologies.
Figure 2: Hidden Markov modeling of progression through morphology stages.
Figure 3: Automated annotation of mitotic spindle and Golgi dynamics, and replication factory patterns during S-phase progression.
Figure 4: Timing phenotypes and kinetic measurements.
Figure 5: RNAi screen for mitotic exit regulators.

Similar content being viewed by others

References

  1. Conrad, C. & Gerlich, D.W. Automated microscopy for high-content RNAi screening. J. Cell Biol. 188, 453–461 (2010).

    Article  CAS  Google Scholar 

  2. Doil, C. et al. RNF168 binds and amplifies ubiquitin conjugates on damaged chromosomes to allow accumulation of repair proteins. Cell 136, 435–446 (2009).

    Article  CAS  Google Scholar 

  3. Collinet, C. et al. Systems survey of endocytosis by multiparametric image analysis. Nature 464, 243–249 (2010).

    Article  CAS  Google Scholar 

  4. Sonnichsen, B. et al. Full-genome RNAi profiling of early embryogenesis in Caenorhabditis elegans. Nature 434, 462–469 (2005).

    Article  CAS  Google Scholar 

  5. Goshima, G. et al. Genes required for mitotic spindle assembly in Drosophila S2 cells. Science 316, 417–421 (2007).

    Article  CAS  Google Scholar 

  6. Neumann, B. et al. Phenotypic profiling of the human genome by time-lapse microscopy reveals cell division genes. Nature 464, 721–727 (2010).

    Article  CAS  Google Scholar 

  7. Neumann, B. et al. High-throughput RNAi screening by time-lapse imaging of live human cells. Nat. Methods 3, 385–390 (2006).

    Article  CAS  Google Scholar 

  8. Loo, L.H., Wu, L.F. & Altschuler, S.J. Image-based multivariate profiling of drug responses from single cells. Nat. Methods 4, 445–453 (2007).

    CAS  Google Scholar 

  9. Conrad, C. et al. Automatic identification of subcellular phenotypes on human cell arrays. Genome Res. 14, 1130–1136 (2004).

    Article  CAS  Google Scholar 

  10. Glory, E. & Murphy, R.F. Automated subcellular location determination and high-throughput microscopy. Dev. Cell 12, 7–16 (2007).

    Article  CAS  Google Scholar 

  11. Harder, N. et al. Automatic analysis of dividing cells in live cell movies to detect mitotic delays and correlate phenotypes in time. Genome Res. 19, 2113–2124 (2009).

    Article  CAS  Google Scholar 

  12. Zhou, X., Li, F., Yan, J. & Wong, S.T. A novel cell segmentation method and cell phase identification using Markov model. IEEE Trans. Inf. Technol. Biomed. 13, 152–157 (2009).

    Article  Google Scholar 

  13. Gerlich, D. & Ellenberg, J. 4D imaging to assay complex dynamics in live specimens. Nat. Cell Biol. 5, S14–S19 (2003).

    Google Scholar 

  14. Schmitz, M.H. & Gerlich, D.W. Automated live microscopy to study mitotic gene function in fluorescent reporter cell lines. Methods Mol. Biol. 545, 113–134 (2009).

    Article  CAS  Google Scholar 

  15. Boland, M.V. & Murphy, R.F. A neural network classifier capable of recognizing the patterns of all major subcellular structures in fluorescence microscope images of HeLa cells. Bioinformatics 17, 1213–1223 (2001).

    Article  CAS  Google Scholar 

  16. Wahlby, C., Sintorn, I.M., Erlandsson, F. & Borgefors, G. Combining intensity, edge and shape information for 2D and 3D segmentation of cell nuclei in tissue sections. J. Microscopy 215, 67–76 (2004).

    Article  CAS  Google Scholar 

  17. Walker, R. & Jackway, P. Statistical geometric features-extensions for cytological textureanalysis. Proc. 13th Int. Conf. Pattern Recognition 2, 790–794 (1996).

    Article  Google Scholar 

  18. Haralick, R., Dinstein & Shanmugam Textural features for image classification. IEEE Transactions on Systems, Man and Cybernetics 3, 610–621 (1973).

    Article  Google Scholar 

  19. Boser, B.E., Guyon, I. & Vapnik, V. A training algorithm for optimal margin classifiers. COLT ′92: Proceedings of the Fifth Annual Workshop on Computational Learning Theory (1992).

  20. Wang, M. et al. Novel cell segmentation and online SVM for cell cycle phase identification in automated microscopy. Bioinformatics 24, 94–101 (2008).

    Article  CAS  Google Scholar 

  21. Chen, X., Zhou, X. & Wong, S.T. Automated segmentation, classification, and tracking of cancer cell nuclei in time-lapse microscopy. IEEE Trans. Biomed. Eng. 53, 762–766 (2006).

    Article  Google Scholar 

  22. Durbin, R.R., Eddy, S., Krogh, A. & Mitchison, G. Biological sequence analysis: probabilistic models of proteins and nucleic acids. (Cambridge University Press, 1998).

  23. Viterbi, A. Error bounds for convolutional codes and an asymptotically optimum decoding algorithm. IEEE Trans. Inf. Theory 13, 260–269 (1967).

    Article  Google Scholar 

  24. Baum, L.E., Petrie, T., Soules, G. & Weiss, N. A maximization technique occurring in the statistical analysis of probabilistic functions of Markov chains. Ann. Math. Stat. 41, 164–171 (1970).

    Article  Google Scholar 

  25. Dempster, A.P., Laird, N.M. & Rubin, D.B. Maximum likelihood from incomplete data via the EM algorithm. J. R. Stat. Soc. B 39, 1–3 (1977).

    Google Scholar 

  26. Meraldi, P., Draviam, V.M. & Sorger, P.K. Timing and checkpoints in the regulation of mitotic progression. Dev. Cell 7, 45–60 (2004).

    Article  CAS  Google Scholar 

  27. Hagting, A. et al. Human securin proteolysis is controlled by the spindle checkpoint and reveals when the APC/C switches from activation by Cdc20 to Cdh1. J. Cell Biol. 157, 1125–1137 (2002).

    Article  CAS  Google Scholar 

  28. Bollen, M., Gerlich, D.W. & Lesage, B. Mitotic phosphatases: from entry guards to exit guides. Trends Cell Biol. 19, 606–616 (2009).

    Article  Google Scholar 

  29. Wolthuis, R. et al. Cdc20 and Cks direct the spindle checkpoint-independent destruction of cyclin A. Mol. Cell 30, 290–302 (2008).

    Article  CAS  Google Scholar 

  30. Wang, M., Zhou, X., King, R.W. & Wong, S.T. Context based mixture model for cell phase identification in automated fluorescence microscopy. BMC Bioinformatics 8, 32 (2007).

    Article  CAS  Google Scholar 

  31. Erfle, H. et al. Reverse transfection on cell arrays for high content screening microscopy. Nat. Protoc. 2, 392–399 (2007).

    Article  CAS  Google Scholar 

  32. Snapp, E.L. et al. Formation of stacked ER cisternae by low affinity protein interactions. J. Cell Biol. 163, 257–269 (2003).

    Article  CAS  Google Scholar 

  33. Dultz, E. et al. Systematic kinetic analysis of mitotic dis- and reassembly of the nuclear pore in living cells. J. Cell Biol. 180, 857–865 (2008).

    Article  CAS  Google Scholar 

  34. Schaub, B.E., Berger, B., Berger, E.G. & Rohrer, J. Transition of galactosyltransferase 1 from trans-Golgi cisterna to the trans-Golgi network is signal mediated. Mol. Biol. Cell 17, 5153–5162 (2006).

    Article  CAS  Google Scholar 

  35. Leonhardt, H. et al. Dynamics of DNA replication factories in living cells. J. Cell Biol. 149, 271–280 (2000).

    Article  CAS  Google Scholar 

  36. Steigemann, P. et al. Aurora B-mediated abscission checkpoint protects against tetraploidization. Cell 136, 473–484 (2009).

    Article  Google Scholar 

  37. Wu, T.F., Lin, C.J. & Weng, R.C. Probability estimates for multi-class classification by pairwise coupling. J. Machine Learning Res. 5, 975–1005 (2004).

    Google Scholar 

Download references

Acknowledgements

We thank C. Conrad and W.H. Gerlich for critical comments on the manuscript, F.O. Gathmann for helpful discussions about software engineering, N. Graf for outstanding information technology support, G. Csucs, members of the Swiss Federal Institute of Technology (ETHZ) Light Microscopy Center and members of the ETHZ RNAi Screening Center for technical support, J. Rohrer (University of Zurich) for providing GalT-EGFP plasmid, J. Pines (Gurdon Institute, Cambridge, UK) for providing Securin-EYFP and cyclin B1–EGFP plasmids, K. Beck and U. Kutay for providing images of cells expressing fluorescent α-tubulin and histone H2B, and Q. Zhong for generating the plot for Supplementary Figure 1. Work in the Gerlich laboratory is supported by Swiss National Science Foundation (SNF) research grant 3100A0-114120, SNF ProDoc grant PDFMP3_124904, a European Young Investigator award of the European Science Foundation, an EMBO fellowship, Young Investigator Programme and Marine Biological Laboratory Summer Research Fellowship to D.W.G., a grant by the Swiss Federal Institute of Technology (ETH-TH), a grant by the UBS foundation, a Roche Ph.D. fellowship to M.H.A.S. and a Mueller fellowship of the Molecular Life Sciences Ph.D. program Zurich to M.H. B.F. was supported by European Commission's seventh framework program project Cancer Pathways. Work in the Ellenberg laboratory is supported by a European Commission grant within the Mitocheck consortium (LSHG-CT-2004-503464). Work in the Peter laboratory is supported by the ETHZ, Oncosuisse, SystemsX.ch (LiverX) and the SNF.

Author information

Authors and Affiliations

  1. Institute of Biochemistry, Swiss Federal Institute of Technology Zurich, Zurich, Switzerland

    Michael Held, Michael H A Schmitz, Michael H Olma, Matthias Peter & Daniel W Gerlich

  2. Marine Biological Laboratory, Woods Hole, Massachusetts, USA

    Michael Held, Michael H A Schmitz & Daniel W Gerlich

  3. Genome Biology Unit, European Molecular Biology Laboratory, Heidelberg, Germany

    Bernd Fischer

  4. Cell Biology and Biophysics Unit, European Molecular Biology Laboratory, Heidelberg, Germany

    Thomas Walter & Jan Ellenberg

  5. Advanced Light Microscopy Facility, European Molecular Biology Laboratory, Heidelberg, Germany

    Beate Neumann

Authors
  1. Michael Held
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Michael H A Schmitz
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Bernd Fischer
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Thomas Walter
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Beate Neumann
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Michael H Olma
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Matthias Peter
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Jan Ellenberg
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Daniel W Gerlich
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

M.H. designed the image analysis workflow, implemented the software, performed imaging experiments and prepared the manuscript. M.H.A.S. established stable cell lines, performed most imaging and all RNAi experiments. B.F. designed and implemented the hidden Markov model. T.W. designed parts of the feature extraction and of the image analysis workflow. B.N. and J.E. generated the siRNA cell transfection array. M.H.O. and M.P. established live imaging of EGFP-PCNA. D.W.G. designed assays and the general strategy for image processing and wrote the paper.

Corresponding author

Correspondence to Daniel W Gerlich.

Ethics declarations

Competing interests

The authors declare no competing financial interests.

Supplementary information

Supplementary Text and Figures

Supplementary Figures 1–8 and Supplementary Tables 1–6 (PDF 4136 kb)

Supplementary Movie 1

Time-lapse imaging of HeLa cells stably expressing the fluorescent chromatin marker H2B-mCherry (imaged with widefield epifluorescence 20Ă— dry objective). The movie shows a region of interest of 512 Ă— 512 Ă— 30 (x Ă— y Ă— t (pixel Ă— pixel Ă— frames); overall movie dimensions: 1,392 Ă— 1,040 Ă— 206 (x Ă— y Ă— t (pixel Ă— pixel Ă— frames)); time lapse, 4.6 min. (MOV 3869 kb)

Supplementary Movie 2

Object detection and supervised classification of morphologies. The contours were derived by the automated segmentation, and the color code for different morphology classes is as indicated in Figure 1b. Original data are shown in Supplementary Movie 1. (MOV 5220 kb)

Supplementary Movie 3

Automated extraction of mitotic events. The movie displays 100 randomly selected examples for cells progressing through mitosis (same as in Fig. 2a). The cells were in silico synchronized to the prophase-prometaphase transition and sorted based on total prometaphase and metaphase duration. The morphology classes annotated as in Figure 2a are indicated by color-coding as in Figure 1b. (MOV 7115 kb)

Supplementary Movie 4

Classification error correction based on free hidden Markov model. The same cells as shown in Figure 2a and Supplementary Movie 3 were classified based on morphological features as well as the temporal context. (MOV 7102 kb)

Supplementary Movie 5

Time-lapse imaging of HeLa cells stably expressing the fluorescent chromatin marker H2B-mCherry (red) and mEGFP–α-tubulin (green) with widefield epifluorescence 20× dry objective. The movie shows a region of interest of 512 × 512 × 30 (x × y × t (pixel × pixel × frames)). The overall movie dimensions were 1,392 × 1,040 × 206 (x × y × t (pixel × pixel × frames)); time lapse, 4.6 min. (MOV 5368 kb)

Supplementary Movie 6

Annotation of spindle dynamics in movies of cells expressing H2B-mCherry and mEGFP–α-tubulin. The movie displays 100 randomly selected examples for automatically annotated cells progressing through mitosis (same as in Fig. 3d). The cells were in silico synchronized to the prophase-prometaphase transition in the H2B-mCherry channel and sorted by total prometaphase and metaphase duration. The morphology classes are indicated by color-coding as indicated in Figure 3a. (MOV 5250 kb)

Supplementary Movie 7

Time-lapse imaging of HeLa cells stably expressing the fluorescent chromatin marker H2B-mCherry (red) and GalT-EGFP (green) with widefield epifluorescence 10Ă— dry objective. The movie shows a region of interest of 512 Ă— 512 Ă— 30 (x Ă— y Ă— t (pixel Ă— pixel Ă— frames)). The overall movie dimensions were 1,392 Ă— 1,040 Ă— 482 (x Ă— y Ă— t (pixel Ă— pixel Ă— frames)); time lapse, 2.8 min. (MOV 6705 kb)

Supplementary Movie 8

Annotation of Golgi dynamics in movies of cells expressing H2B-mCherry and GalT-EGFP. The movie displays 100 randomly selected examples for automatically annotated cells progressing through mitosis (same as in Fig. 3e). The cells were in silico synchronized to the prophase-prometaphase transition in the H2B-mCherry channel and sorted by total prometaphase and metaphase duration. The morphology classes are indicated by color-coding as indicated in Figure 3b. (MOV 4404 kb)

Supplementary Movie 9

Time-lapse imaging of HeLa cells stably expressing the fluorescent chromatin marker H2B-mCherry (red) and DNA replication factory marker EGFP-PCNA (green) with widefield epifluorescence 10Ă— dry objective. The movie shows a region of interest of 350 Ă— 350 Ă— 54 (x Ă— y Ă— t (pixel Ă— pixel Ă— frames); every second time point shown). The overall movie dimensions were 1,392 Ă— 1,040 Ă— 482 (x Ă— y Ă— t (pixel Ă— pixel Ă— frames)); time lapse, 5.9 min. (MOV 4455 kb)

Supplementary Movie 10

Annotation of S-phase progression in movies of cells expressing H2B-mCherry and EGFP-PCNA. The movie displays 100 randomly selected examples for automatically annotated cells progressing through the cell cycle (same as in Fig. 3f). The cells were in silico synchronized to the G1–early S phase transition in the EGFP-PCNA channel and sorted by total S-phase duration. Every second time point of original data is shown. The morphology classes are indicated by color-coding as indicated in Figure 3c. (MOV 6914 kb)

Supplementary Movie 11

Time-lapse imaging of HeLa cells stably expressing H2B-mCherry and Securin-mEGFP with widefield epifluorescence 20× dry objective, treated with 50 ng ml–1 nocodazole immediately before starting the imaging. The movie shows a region of interest of 400 × 400 × 100 (x × y × t (pixel × pixel × frames)). The overall movie dimensions were 1,392 × 1,040 × 500 (x × y × t (pixel × pixel × frames)); time lapse, 2.7 min. (MOV 7547 kb)

Supplementary Movie 12

Time-lapse imaging of Mad2 siRNA transfected HeLa cells stably expressing H2B-mCherry and Securin-mEGFP with widefield epifluorescence 20Ă— dry objective. The movie shows a region of interest of 400 Ă— 400 Ă— 100 (x Ă— y Ă— t (pixel Ă— pixel Ă— frames)). The overall movie dimensions were 1,392 Ă— 1,040 Ă— 500 (x Ă— y Ă— t (pixel Ă— pixel Ă— frames); time lapse, 2.7 min). (MOV 7584 kb)

Supplementary Movie 13

Time-lapse imaging of untreated control HeLa cells stably expressing H2B-mCherry and Securin-mEGFP with widefield epifluorescence 20Ă— dry objective. The movie shows a region of interest of 400 Ă— 400 Ă— 100 (x Ă— y Ă— t (pixel Ă— pixel Ă— frames)). The overall movie dimensions were 1,392 Ă— 1,040 Ă— 500 (x Ă— y Ă— t (pixel Ă— pixel Ă— frames)); time lapse, 2.7 min. (MOV 7700 kb)

Supplementary Movie 14

Time-lapse imaging of control HeLa cells stably expressing H2B-mCherry and IBB-EGFP transfected with nonsilencing siRNA, using widefield epifluorescence 10Ă— dry objective. The movie shows 80 time frames of the entire imaging field downsampled in x/y by a factor of 2 for display. Original movie dimensions: 1,392 Ă— 1,040 Ă— 744 (x Ă— y Ă— t (pixel Ă— pixel Ă— frames)); time lapse, 3.7 min. We captured 108 movies of different RNAi conditions simultaneously in this experiment by multilocation time-lapse imaging. (MOV 9091 kb)

Supplementary Movie 15

Time-lapse confocal imaging of HeLa cells stably expressing H2B-mCherry and mEGFP-α- tubulin (63× oil immersion objective). Cells were transfected with non-silencing siRNA. Movie dimensions are 512 × 512 × 132 (x × y × t (pixel × pixel × frames)); time lapse, 7.1 min. (MOV 9662 kb)

Supplementary Movie 16

Time-lapse confocal imaging of HeLa cells stably expressing H2B-mCherry and mEGFP–α-tubulin (63× oil immersion objective). Cells were transfected with siRNA targeting Cdc20. Movie dimensions are 512 × 512 × 132 (x × y × t (pixel × pixel × frames)); time lapse, 7.1 min. (MOV 9753 kb)

Supplementary Software

CellCognition software. (ZIP 80982 kb)

Rights and permissions

Reprints and permissions

About this article

Cite this article

Held, M., Schmitz, M., Fischer, B. et al. CellCognition: time-resolved phenotype annotation in high-throughput live cell imaging. Nat Methods 7, 747–754 (2010). https://doi.org/10.1038/nmeth.1486

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/nmeth.1486

This article is cited by

Search

Advanced search

Quick links

Nature Briefing AI and Robotics

Sign up for the Nature Briefing: AI and Robotics newsletter — what matters in AI and robotics research, free to your inbox weekly.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing: AI and Robotics