Systematic reviews of prediction models

Many clinical prediction models are developed each year and external validation is considered a crucial aspect in the pathway to a model’s implementation in practice. This is because the prediction model is intended for use in new patients, and therefore needs to perform well outside of the development sample. Prediction model performance is recommended to, at least, be measured in terms of model discrimination and model calibration. Calibration, which measures the agreement between a model’s predictions and patients observed outcomes, has been particularly poorly reported in the past but is critically important for a model to be deemed useful in practice, with miscalibration potentially leading to wrong clinical decisions and potential harm for patients. This review will aim to summarise how often models are being externally validated and the quality of reporting in those that are. There will be a particular focus on model calibration, including how often it is reported, what methods are being used to examine it, and how the results are interpreted. Whether a sample size calculation was used to inform the calibration assessment is also of interest. Once the review is published, we hope it will raise awareness of the importance of calibration assessment being undertaken and following the correct guidelines, particularly in regards to its correct assessment, reporting and interpretation.

British Journal of Dermatology

Background There is no evidence base to support the use of 6-monthly monitoring blood tests for the early detection of liver, blood and renal toxicity during established anti-tumour necrosis factor alpha (TNFα) treatment. Objectives To evaluate the incidence and risk factors of anti-TNFα treatment cessation owing to liver, blood and renal side-effects, and to estimate the cost-effectiveness of alternate intervals between monitoring blood tests. Methods A secondary care-based retrospective cohort study was performed. Data from the British Association of Dermatologists Biologic and Immunomodulators Register (BADBIR) were used. Patients with at least moderate psoriasis prescribed their first anti-TNFα treatment were included. Treatment discontinuation due to a monitoring blood test abnormality was the primary outcome. Patients were followed-up from start …

BackgroundImmune-suppressing drugs can cause liver, kidney or blood toxicity. Prognostic factors for these adverse-events are poorly understood.PurposeTo ascertain prognostic factors associated with liver, blood or kidney adverse-events in people receiving immune-suppressing drugs.Data sourcesMEDLINE, Web of Science, EMBASE and the Cochrane library (01 January 1995 to 05 January 2023), and supplementary sources.Data extraction and synthesisData were extracted by one reviewer using a modified CHARMS-PF checklist and validated by another. Two independent reviewers assessed risk of bias using Quality in Prognostic factor Studies tool and assessed the quality of evidence using a Grading of Recommendations Assessment, Development and Evaluation-informed framework.ResultsFifty-six studies from 58 papers were included. High-quality evidence of the following associations was …

BackgroundDespite many systematic reviews and meta-analyses examining the associations of pregnancy complications with risk of type 2 diabetes mellitus (T2DM) and hypertension, previous umbrella reviews have only examined a single pregnancy complication. Here we have synthesised evidence from systematic reviews and meta-analyses on the associations of a wide range of pregnancy-related complications with risk of developing T2DM and hypertension.MethodsMedline, Embase and Cochrane Database of Systematic Reviews were searched from inception until 26 September 2022 for systematic reviews and meta-analysis examining the association between pregnancy complications and risk of T2DM and hypertension. Screening of articles, data extraction and quality appraisal (AMSTAR2) were conducted independently by two reviewers using Covidence software. Data were extracted for studies that …

Community-based complex interventions to sustain independence in older people, stratified by frailty: a systematic review and network meta-analysis - White Rose Research Online White Rose logo White Rose Research Online Home Search Browse Contact Community-based complex interventions to sustain independence in older people, stratified by frailty: a systematic review and network meta-analysis Crocker, T orcid.org/0000-0001-7450-3143, Lam, N, Ensor, J et al. (18 more authors) (Accepted: 2023) Community-based complex interventions to sustain independence in older people, stratified by frailty: a systematic review and network meta-analysis. Health Technology Assessment. ISSN 1366-5278 (In Press) Metadata Authors/Creators: Crocker, T ORCID logo https://orcid.org/0000-0001-7450-3143 Lam, N Ensor, J Jordão, M Bajpai, R Bond, M Forster, A Riley, R Andre, D Brundle, C Ellwood, A Green, J Hale, …

Biometrical Journal

We propose to combine the benefits of flexible parametric survival modeling and regularization to improve risk prediction modeling in the context of time‐to‐event data. Thereto, we introduce ridge, lasso, elastic net, and group lasso penalties for both log hazard and log cumulative hazard models. The log (cumulative) hazard in these models is represented by a flexible function of time that may depend on the covariates (i.e., covariate effects may be time‐varying). We show that the optimization problem for the proposed models can be formulated as a convex optimization problem and provide a user‐friendly R implementation for model fitting and penalty parameter selection based on cross‐validation. Simulation study results show the advantage of regularization in terms of increased out‐of‐sample prediction accuracy and improved calibration and discrimination of predicted survival probabilities, especially when …

Journal of the College of Community Physicians of Sri Lanka

ImportanceMendelian randomization (MR) studies use genetic variation associated with modifiable exposures to assess their possible causal relationship with outcomes and aim to reduce potential bias from confounding and reverse causation.ObjectiveTo develop the STROBE-MR Statement as a stand-alone extension to the STROBE (Strengthening the Reporting of Observational Studies in Epidemiology) guideline for the reporting of MR studies.Design, Setting, and ParticipantsThe development of the STROBE-MR Statement followed the Enhancing the Quality and Transparency of Health Research (EQUATOR) framework guidance and used the STROBE Statement as a starting point to draft a checklist tailored to MR studies. The project was initiated in 2018 by reviewing the literature on the reporting of instrumental variable and MR studies. A group of 17 experts, including MR methodologists, MR study design …

BackgroundInfants and young children born prematurely are at high risk of severe acute lower respiratory infection (ALRI) caused by respiratory syncytial virus (RSV). In this study, we aimed to assess the global disease burden of and risk factors for RSV-associated ALRI in infants and young children born before 37 weeks of gestation.MethodsWe conducted a systematic review and meta-analysis of aggregated data from studies published between Jan 1, 1995, and Dec 31, 2021, identified from MEDLINE, Embase, and Global Health, and individual participant data shared by the Respiratory Virus Global Epidemiology Network on respiratory infectious diseases. We estimated RSV-associated ALRI incidence in community, hospital admission, in-hospital mortality, and overall mortality among children younger than 2 years born prematurely. We conducted two-stage random-effects meta-regression analyses accounting …

Age and ageing

Background Falls are common in older adults and can devastate personal independence through injury such as fracture and fear of future falls. Methods to identify people for falls prevention interventions are currently limited, with high risks of bias in published prediction models. We have developed and externally validated the eFalls prediction model using routinely collected primary care electronic health records (EHR) to predict risk of emergency department attendance/hospitalisation with fall or fracture within 1 year. Methods Data comprised two independent, retrospective cohorts of adults aged ≥65 years: the population of Wales, from the Secure Anonymised Information Linkage Databank (model development); the population of Bradford and Airedale, England, from Connected Bradford (external validation). Predictors included electronic frailty index components …

bmj

The TRIPOD (Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis) statement was published in 2015 to provide the minimum reporting recommendations for studies developing or evaluating the performance of a prediction model. Methodological advances in the field of prediction have since included the widespread use of artificial intelligence (AI) powered by machine learning methods to develop prediction models. An update to the TRIPOD statement is thus needed. TRIPOD+AI provides harmonised guidance for reporting prediction model studies, irrespective of whether regression modelling or machine learning methods have been used. The new checklist supersedes the TRIPOD 2015 checklist, which should no longer be used. This article describes the development of TRIPOD+AI and presents the expanded 27 item checklist with more detailed explanation of each …

Journal of Clinical Epidemiology

ObjectiveTo describe the frequency of open science practices in a contemporary sample of studies developing prognostic models using machine learning methods in the field of oncology.Study design and settingWe conducted a systematic review, searching the MEDLINE database between December 1, 2022, and December 31, 2022, for studies developing a multivariable prognostic model using machine learning methods (as defined by the authors) in oncology. Two authors independently screened records and extracted open science practices.ResultsWe identified 46 publications describing the development of a multivariable prognostic model. The adoption of open science principles was poor. Only one study reported availability of a study protocol, and only one study was registered. Funding statements and conflicts of interest statements were common. Thirty-five studies (76%) provided data sharing …

bmj

An external validation study evaluates the performance of a prediction model in new data, but many of these studies are too small to provide reliable answers. In the third article of their series on model evaluation, Riley and colleagues describe how to calculate the sample size required for external validation studies, and propose to avoid rules of thumb by tailoring calculations to the model and setting at hand.

Sport healthcare and performance support practitioners require data to inform clinical and performance decisions, identify risk factors and investigate the efficacy and effectiveness of different interventions. 1 Data sharing is a key component to improve sport healthcare decisions and initiatives, particularly in identifying and tackling systemic health priorities. However, sport healthcare data sharing practices are extremely limited due to little uptake and incentives, which inhibits knowledge growth, evidence synthesis and research progress. 1 Incorporating open science practices into sport healthcare can increase the efficiency and quality of research, 2 transparency, 3 and promote public trust 4 in the research process, results and implementation. There are many facets to adopting an open science framework with data sharing being a key component (table 1). However, barriers to creating open and accessible data …

Journal of Clinical Epidemiology

ObjectivesTo develop a framework to identify and evaluate spin practices and its facilitators in studies on clinical prediction model, regardless of the modelling technique.Study DesignWe followed a three-phase consensus process: (1) pre-meeting literature review to generate items to be included; (2) a series of structured meetings to provide comments, discussed and exchanged viewpoints on items to be included with a panel of experienced researchers; and (3) post-meeting review on final list of items and examples to be included. Through this iterative consensus process, a framework was derived after all panel’s researchers agreed.ResultsThis consensus process involved a panel of eight researchers and resulted in SPIN-PM which consists of two categories of spin (misleading interpretation and misleading transportability), and within these categories, two forms of spin (spin practices and facilitators of spin). We …

bmj

External validation studies are an important but often neglected part of prediction model research. In this article, the second in a series on model evaluation, Riley and colleagues explain what an external validation study entails and describe the key steps involved, from establishing a high quality dataset to evaluating a model’s predictive performance and clinical usefulness.

RMD open

BackgroundSulfasalazine-induced cytopenia, nephrotoxicity and hepatotoxicity is uncommon during long-term treatment. Some guidelines recommend 3 monthly monitoring blood tests indefinitely during long-term treatment while others recommend stopping monitoring after 1 year. To rationalise monitoring, we developed and validated a prognostic model for clinically significant blood, liver or kidney toxicity during established sulfasalazine treatment.DesignRetrospective cohort study.SettingUK primary care. Data from Clinical Practice Research Datalink Gold and Aurum formed independent development and validation cohorts.ParticipantsAge ≥18 years, new diagnosis of an inflammatory condition and sulfasalazine prescription.Study period1 January 2007 to 31 December 2019.OutcomeSulfasalazine discontinuation with abnormal monitoring blood-test result.AnalysisPatients were followed up from 6 months …

BackgroundPrognostic models incorporate multiple prognostic factors to estimate the likelihood of future events for individual patients based on their prognostic factor values. Evaluating these models crucially involves conducting studies to assess their predictive performance, like discrimination. Systematic reviews and meta-analyses of these evaluation studies play an essential role in selecting models for clinical practice.MethodsIn this paper, we outline three thresholds to determine the target for certainty rating in the discrimination of prognostic models, as observed across a body of validation studies.Results and ConclusionWe propose three thresholds when rating the certainty of evidence about a prognostic model's discrimination. The first threshold amounts to rating certainty in the model's ability to classify better than random chance. The other two approaches involve setting thresholds informed by other …

bmj

Evaluating the performance of a clinical prediction model is crucial to establish its predictive accuracy in the populations and settings intended for use. In this article, the first in a three part series, Collins and colleagues describe the importance of a meaningful evaluation using internal, internal-external, and external validation, as well as exploring heterogeneity, fairness, and generalisability in model performance.

Statistics in medicine

When designing a study to develop a new prediction model with binary or time‐to‐event outcomes, researchers should ensure their sample size is adequate in terms of the number of participants (n) and outcome events (E) relative to the number of predictor parameters (p) considered for inclusion. We propose that the minimum values of n and E (and subsequently the minimum number of events per predictor parameter, EPP) should be calculated to meet the following three criteria: (i) small optimism in predictor effect estimates as defined by a global shrinkage factor of 0.9, (ii) small absolute difference of 0.05 in the model's apparent and adjusted Nagelkerke's R2, and (iii) precise estimation of the overall risk in the population. Criteria (i) and (ii) aim to reduce overfitting conditional on a chosen p, and require prespecification of the model's anticipated Cox‐Snell R2, which we show can be obtained from previous …

bmj

Most clinical specialties have a plethora of studies that develop or validate one or more prediction models, for example, to inform diagnosis or prognosis. Having many prediction model studies in a particular clinical field motivates the need for systematic reviews and meta-analyses, to evaluate and summarise the overall evidence available from prediction model studies, in particular about the predictive performance of existing models. Such reviews are fast emerging, and should be reported completely, transparently, and accurately. To help ensure this type of reporting, this article describes a new reporting guideline for systematic reviews and meta-analyses of prediction model research.