Skip to main content
Log in

A new equivalence based metric for predictive check to qualify mixed-effects models

  • Published:
The AAPS Journal Aims and scope Submit manuscript

Abstract

The main objective of any modeling exercise is to provide a rationale for effective decision making during drug development. The aim of the current simulation experiment was to evaluate the properties of predictive check as a covariate model qualification technique and, more importantly, to introduce and evaluate alternative criteria to qualify models.

Original concentration-time profiles (yod) were simulated using a 1-compartment model for an intravenous drug administered to 25 men and 25 women. The typical clearance for male subjects (TVCLm) was assumed to be 5-fold higher than that for female subjects (TVCLf). Fifty such trials under the same design were generated randomly. Predictive check was used as the model qualification tool to study predictive performance of true (males ≠ females) and false (males=females) models in the context of maximum likelihood estimation. For each yod, 200 replications were generated to study the properties of a discrepancy variable, a statistic that depends on the model, and a test statistic, a statistic that does not depend on the model. Several qualification criteria were evaluated in assessing predictive performance, such as, predictive p-value (Pp), probability of equivalence (peqv), and probability of rejecting the null hypothesis (data=model) using the Kolmogorov-Smirnov test (pks).

The Pp value was calculated using sum of squared errors as a discrepancy variable. For both of the models, the Pp values uniformly ranged between 0 and 1. The pattern of Pp values suggests that qualification of the false model is unlikely. For both of the models, the range of peqv is about 0.95 to 1.0 for concentration at 0.5 hours. However, this is not the case for the concentration at 4 hours, which is primarily dependent on the clearance. The false model (0.35 to 0.50) has poor predictive performance compared with the true model (0.65 to 0.80) using peqv. The pks suggests no difference in the distributions of replicated and original concentrations at all of the time points for both of the models.

Discrepancy variables cannot aid in rejecting false models, whereas the use of a test statistic can aid in rejecting false models. However, selection of an informative test statistic is challenging. As far as the qualification criteria are considered, the equivalence-based comparison of a test statistic is more informative than a significance-based comparison. No convincing evidence exists in the literature demonstrating the added advantages of predictive check as a routine model qualification tool over the existing tools, such as diagnostic plots or mechanistic reasoning. However, when a model is to be used for designing a trial, it should at least be able to regenerate the data used to build the model. In such cases, predictive check might offer insights into potential inconsistencies.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Sheiner LB, Steimer JL. Pharmacokinetic/pharmacodynamic modeling in drug development.Annu Rev Pharmacol Toxicol. 2000;40:67–95.

    Article  PubMed  CAS  Google Scholar 

  2. Gobburu JV, Marroum PJ. Utilisation of pharmacokinetic-pharmacodynamic modelling and simulation in regulatory decision-making.Clin Pharmacokinet. 2001;40:883–892.

    Article  PubMed  CAS  Google Scholar 

  3. Holford NHG, Hale M, Ko HS, Steimer JL, Sheiner LB. Simulation in Drug development: good practices.Center for Drug Development Science. Available at: http://cdds.georgetown.edu. Accessed January 1, 2004.

  4. Wahlby U, Jonsson EN, Karlsson MO. Assessment of actual significance levels for covariate effects in NONMEM.J Pharmacokinet Pharmacodyn. 2001;28:231–252.

    Article  PubMed  CAS  Google Scholar 

  5. FDA. Guidance: Population Pharmacokinetics.FDA guidance page. Available at:http://www.fda.gov/cder/guidance. Accessed January 13, 2004.

  6. Rubin D. Bayesianly justifiable and relevant frequency calculations for the applied statisitcian.Ann Stat. 1984;12:1151–1172.

    Article  Google Scholar 

  7. Gelman A, Carlin JB, Stern HS, Rubin DB. Model checking and sensitivity analysis. In: Gelman A, ed.Bayesian Data Analysis. London: Chapman & Hall; 1995;161–189.

    Google Scholar 

  8. Gelman A, Meng XL. Model checking and model improvement. In: Gilks WR, Richardson S, Spiegelhalter DJ, eds.Markov Chain and Monte Carlo in Practice. London: Chapman & Hall; 1996:189–202.

    Google Scholar 

  9. Yano Y, Beal SL, Sheiner LB. Evaluating pharmacokinetic/pharmacodynamic models using the posterior predictive check.J Pharmacokinet Pharmacodyn. 2001;28:171–192.

    Article  PubMed  CAS  Google Scholar 

  10. Gelman A, Meng XL, Stern HS. Posterior predictive assessment of model fitness via realized discrepancies.Statistica Sinica. 1996;6:733–807.

    Google Scholar 

  11. Guttman I. The use of a concept of a future observation in the goodness-of-fit problems.J R Statist Soc B. 1967;1:83–100.

    Google Scholar 

  12. Tiao GC, Xu D. Robustness of maximum likelihood estimates for multi-step predictions: the exponential smoothing case.Biometrika. 1993;80:623–641.

    Article  Google Scholar 

  13. Gelman A, Goegebeur Y, Tuerlinckx F, Van Mechelen I. Diagnostic checks for discrete data regression models using posterior predictive simulations.J Royal Stat Soc: C. 2000;49:247–268.

    Article  Google Scholar 

  14. Duffull SB, Aarons L. Development of a sequential linked pharmacokinetic and pharmacodynamic simulation model for ivabradine in healthy volunteers.Eur J Pharm Sci. 2000;10:275–284.

    Article  PubMed  CAS  Google Scholar 

  15. Duffull SB, Chabaud S, Nony P, Laveille C, Girard P, Aarons L. A pharmacokinetic simulation model for ivabradine in healthy volunteers.Eur J Pharm Sci.. 2000;10:285–294.

    Article  PubMed  CAS  Google Scholar 

  16. Friberg LE, Freijs A, Sandstrom M, Karlsson MO. Semiphysiological model for the time course of leukocytes after varying schedules of 5-fluorouracil in rats.J Pharmacol Exp Ther. 2000;295:734–740.

    PubMed  CAS  Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Jogarao V. S. Gobburu.

Additional information

Published: October 7, 2005

The views expressed in this article are those of the authors and do not necessarily reflect the official views of FDA.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Jadhav, P.R., Gobburu, J.V.S. A new equivalence based metric for predictive check to qualify mixed-effects models. AAPS J 7, 53 (2005). https://doi.org/10.1208/aapsj070353

Download citation

  • Received:

  • Accepted:

  • DOI: https://doi.org/10.1208/aapsj070353

Keywords

Navigation