Wynants L.

ObjectivesData-driven decision support tools have been increasingly recognized to transform health care. However, such tools are often developed on predefined research datasets without adequate knowledge of the origin of this data and how it was selected. How a dataset is extracted from a clinical database can profoundly impact the validity, interpretability and interoperability of the dataset, and downstream analyses, yet is rarely reported. Therefore, we present a case study illustrating how a definitive patient list was extracted from a clinical source database and how this can be reported.Study Design and SettingA single-center observational study was performed at an academic hospital in the Netherlands to illustrate the impact of selecting a definitive patient list for research from a clinical source database, and the importance of documenting this process. All admissions from the critical care database admitted …

ObjectivesTo conduct a systematic review of studies externally validating the ADNEX (Assessment of Different Neoplasias in the adnexa) model for diagnosis of ovarian cancer and to present a meta-analysis of its performance.Design

Prognostic outcomes related to hospital admissions typically do not suffer from censoring, and can be modeled either categorically or as time-to-event. Competing events are common but often ignored. We compared the performance of random forest (RF) models to predict the risk of central line-associated bloodstream infections (CLABSI) using different outcome operationalizations. We included data from 27478 admissions to the University Hospitals Leuven, covering 30862 catheter episodes (970 CLABSI, 1466 deaths and 28426 discharges) to build static and dynamic RF models for binary (CLABSI vs no CLABSI), multinomial (CLABSI, discharge, death or no event), survival (time to CLABSI) and competing risks (time to CLABSI, discharge or death) outcomes to predict the 7-day CLABSI risk. We evaluated model performance across 100 train/test splits. Performance of binary, multinomial and competing risks models was similar: AUROC was 0.74 for baseline predictions, rose to 0.78 for predictions at day 5 in the catheter episode, and decreased thereafter. Survival models overestimated the risk of CLABSI (E:O ratios between 1.2 and 1.6), and had AUROCs about 0.01 lower than other models. Binary and multinomial models had lowest computation times. Models including multiple outcome events (multinomial and competing risks) display a different internal structure compared to binary and survival models. In the absence of censoring, complex modelling choices do not considerably improve the predictive performance compared to a binary model for CLABSI prediction in our studied settings. Survival models censoring the competing events at …

Personalization of psychological therapies has always been used by clinicians and describes all efforts to select, adjust, or modify a treatment for the individual to improve outcomes. Precision mental health care approaches can be considered under the umbrella term personalization and specify methods that are algorithmic, quantitative, and empirically derived. Despite a growing research literature demonstrating the efficacy of these approaches, they are rarely tested in clinical practice. A statistically optimized, targeted clinical recommendation is not by itself sufficient to influence clinical practice in a beneficial way; barriers related to dissemination and implementation require increased attention. This article describes clinical and practical factors, technical aspects, statistical considerations, and fundamental contextual issues that should be considered to facilitate data-driven treatments in mental health care contexts …

The TRIPOD (Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis) statement was published in 2015 to provide the minimum reporting recommendations for studies developing or evaluating the performance of a prediction model. Methodological advances in the field of prediction have since included the widespread use of artificial intelligence (AI) powered by machine learning methods to develop prediction models. An update to the TRIPOD statement is thus needed. TRIPOD+AI provides harmonised guidance for reporting prediction model studies, irrespective of whether regression modelling or machine learning methods have been used. The new checklist supersedes the TRIPOD 2015 checklist, which should no longer be used. This article describes the development of TRIPOD+AI and presents the expanded 27 item checklist with more detailed explanation of each …

Objectives To assess the ability of the International Endometrial Tumor Analysis (IETA)‐1 polynomial regression model to estimate the risk of endometrial cancer (EC) and other intracavitary uterine pathology in women without abnormal uterine bleeding. Methods This was a retrospective study, in which we validated the IETA‐1 model on the IETA‐3 study cohort (n = 1745). The IETA‐3 study is a prospective observational multicenter study. It includes women without vaginal bleeding who underwent a standardized transvaginal ultrasound examination in one of seven ultrasound centers between January 2011 and December 2018. The ultrasonography was performed either as part of a routine gynecological examination, during follow‐up of non‐endometrial pathology, in the work‐up before fertility treatment or before treatment for uterine prolapse or ovarian pathology. Ultrasonographic findings were described …

Background Acutely ill children are at risk of unwarranted antibiotic prescribing. Data on the appropriateness of antibiotic prescriptions provide insights into potential tailored interventions to promote antibiotic stewardship. Objectives To examine factors associated with the inappropriateness of antibiotic prescriptions for acutely ill children presenting to ambulatory care in high-income countries. Methods On 8 September 2022, we systematically searched articles published since 2002 in MEDLINE, Embase, CENTRAL, Web of Science, and grey literature databases. We included studies with acutely ill children presenting to ambulatory care settings in high-income countries reporting on the appropriateness of antibiotic prescriptions. The quality of the studies was evaluated using the Appraisal tool for Cross-Sectional Studies and the Newcastle–Ottawa Scale …

Background With advancing therapeutics, lung cancer (LC) survivors are rapidly increasing in number. Although mounting evidence suggests LC survivors have high risk of second primary lung cancer (SPLC), there is no validated prediction model available for clinical use to identify high-risk LC survivors for SPLC. Methods Using data from 6325 ever-smokers in the Multiethnic Cohort (MEC) study diagnosed with initial primary lung cancer (IPLC) in 1993-2017, we developed a prediction model for 10-year SPLC risk after IPLC diagnosis using cause-specific Cox regression. We evaluated the model’s clinical utility using decision curve analysis and externally validated it using 2 population-based data—Prostate, Lung, Colorectal, and Ovarian Cancer Screening Trial (PLCO) and National Lung Screening Trial (NLST)—that included 2963 and 2844 IPLC (101 and 93 SPLC cases …

In designing external validation studies of clinical prediction models, contemporary sample size calculation methods are based on the frequentist inferential paradigm. One of the widely reported metrics of model performance is net benefit (NB), and the relevance of conventional inference around NB as a measure of clinical utility is doubtful. Value of Information methodology quantifies the consequences of uncertainty in terms of its impact on clinical utility of decisions. We introduce the expected value of sample information (EVSI) for validation as the expected gain in NB from conducting an external validation study of a given size. We propose algorithms for EVSI computation, and in a case study demonstrate how EVSI changes as a function of the amount of current information and future study's sample size. Value of Information methodology provides a decision-theoretic lens to the process of planning a validation study of a risk prediction model and can complement conventional methods when designing such studies.

Objective Previous work has suggested that the ultrasound‐based benign simple descriptors (BDs) can reliably exclude malignancy in a large proportion of women presenting with an adnexal mass. This study aimed to validate a modified version of the BDs and to validate a two‐step strategy to estimate the risk of malignancy, in which the modified BDs are followed by the Assessment of Different NEoplasias in the adneXa (ADNEX) model if modified BDs do not apply. Methods This was a retrospective analysis using data from the 2‐year interim analysis of the International Ovarian Tumor Analysis (IOTA) Phase‐5 study, in which consecutive patients with at least one adnexal mass were recruited irrespective of subsequent management (conservative or surgery). The main outcome was classification of tumors as benign or malignant, based on histology or on clinical and ultrasound information during 1 year of …

BackgroundA number of recent papers have proposed methods to calculate confidence intervals and p values for net benefit used in decision curve analysis. These papers are sparse on the rationale for doing so. We aim to assess the relation between sampling variability, inference, and decision-analytic concepts.Methods and resultsWe review the underlying theory of decision analysis. When we are forced into a decision, we should choose the option with the highest expected utility, irrespective of p values or uncertainty. This is in some distinction to traditional hypothesis testing, where a decision such as whether to reject a given hypothesis can be postponed. Application of inference for net benefit would generally be harmful. In particular, insisting that differences in net benefit be statistically significant would dramatically change the criteria by which we consider a prediction model to be of value. We argue instead …

Objectives: To conduct a systematic review of studies externally validating the ADNEX model for ovarian cancer diagnosis and perform a meta-analysis of its performance. Design: Systematic review, meta-analysis Data sources: Medline, EMBASE, WOS, Scopus, and EuropePMC up to 15/05/2023. Review methods: We included external validation studies of the performance of ADNEX using any study design and any study population comprising patients with an adnexal mass. Two independent reviewers extracted data. Disagreements were resolved through discussion. Reporting quality of the studies was scored using the TRIPOD reporting guideline and methodological conduct and risk of bias using the PROBAST tool. We performed random effects meta-analysis of the AUC, sensitivity and specificity at the 10% risk of malignancy threshold, and Net Benefit and Relative Utility at the 10% risk of malignancy threshold. Results: We included 47 studies (17,007 tumours) with median study sample size 261 (range 24-4905). On average, 61% of TRIPOD items were reported. Handling of missing data, sample size justification, and model calibration were rarely described. 91% of validations were at high risk of bias, mainly due to the unexplained exclusion of incomplete cases, low sample size, or absent calibration assessment. The summary AUC to distinguish benign from malignant tumours in operated patients was 0.93 (95% CI 0.92-0.94, 95% prediction interval 0.85-0.98) for ADNEX with CA125 as a predictor (9202 tumours, 43 centres, 18 countries, 21 studies) and 0.93 (95% CI 0.91-0.94, 95% prediction interval 0.85-0.98) for ADNEX without CA125 …

BackgroundThe COVID-19 pandemic required science to provide answers rapidly to combat the outbreak. Hence, the reproducibility and quality of conducting research may have been threatened, particularly regarding privacy and data protection, in varying ways around the globe. The objective was to investigate aspects of reporting informed consent and data handling as proxies for study quality conduct.MethodsA systematic scoping review was performed by searching PubMed and Embase. The search was performed on November 8th, 2020. Studies with hospitalised patients diagnosed with COVID-19 over 18 years old were eligible for inclusion. With a focus on informed consent, data were extracted on the study design, prestudy protocol registration, ethical approval, data anonymisation, data sharing and data transfer as proxies for study quality. For reasons of comparison, data regarding country income level …

Risk prediction models need thorough validation to assess their performance. Validation of models for survival outcomes poses challenges due to the censoring of observations and the varying time horizon at which predictions can be made. This article describes measures to evaluate predictions and the potential improvement in decision making from survival models based on Cox proportional hazards regression. As a motivating case study, the authors consider the prediction of the composite outcome of recurrence or death (the “event”) in patients with breast cancer after surgery. They developed a simple Cox regression model with 3 predictors, as in the Nottingham Prognostic Index, in 2982 women (1275 events over 5 years of follow-up) and externally validated this model in 686 women (285 events over 5 years). Improvement in performance was assessed after the …

IntroductionIt is known that only a limited proportion of developed clinical prediction models (CPMs) are implemented and/or used in clinical practice. This may result in a large amount of research waste, even when considering that some CPMs may demonstrate poor performance. Cross-sectional estimates of the numbers of CPMs that have been developed, validated, evaluated for impact or utilized in practice, have been made in specific medical fields, but studies across multiple fields and studies following up the fate of CPMs are lacking.Methods and analysisWe have conducted a systematic search for prediction model studies published between January 1995 and December 2020 using the Pubmed and Embase databases, applying a validated search strategy. Taking random samples for every calendar year, abstracts and articles were screened until a target of 100 CPM development studies were identified …

ObjectivesTo systematically review the risk of bias and applicability of published prediction models for risk of central line-associated bloodstream infection (CLA-BSI) in hospitalized patients.Study Design and SettingSystematic review of literature in PubMed, Embase, Web of Science Core Collection, and Scopus up to July 10, 2023. Two authors independently appraised risk models using CHecklist for critical Appraisal and data extraction for systematic Reviews of prediction Modelling Studies (CHARMS) and assessed their risk of bias and applicability using Prediction model Risk Of Bias ASsessment Tool (PROBAST).ResultsSixteen studies were included, describing 37 models. When studies presented multiple algorithms, we focused on the model that was selected as the best by the study authors. Eventually we appraised 19 models, among which 15 were regression models and four machine learning models. All …

In this prepreint we report the estimated the number of clinical prediction models (CPM) development articles available for all medical fields from 1995 until 2020 on the basis of a systematic search of literature.

BackgroundClinical prediction models should be validated before implementation in clinical practice. But is favorable performance at internal validation or one external validation sufficient to claim that a prediction model works well in the intended clinical context?Main bodyWe argue to the contrary because (1) patient populations vary, (2) measurement procedures vary, and (3) populations and measurements change over time. Hence, we have to expect heterogeneity in model performance between locations and settings, and across time. It follows that prediction models are never truly validated. This does not imply that validation is not important. Rather, the current focus on developing new models should shift to a focus on more extensive, well-conducted, and well-reported validation studies of promising models.ConclusionPrincipled validation strategies are needed to understand and quantify heterogeneity …

ObjectivesTo assess improvement in the completeness of reporting coronavirus (COVID-19) prediction models after the peer review process.Study Design and SettingStudies included in a living systematic review of COVID-19 prediction models, with both preprint and peer-reviewed published versions available, were assessed. The primary outcome was the change in percentage adherence to the transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD) reporting guidelines between pre-print and published manuscripts.ResultsNineteen studies were identified including seven (37%) model development studies, two external validations of existing models (11%), and 10 (53%) papers reporting on both development and external validation of the same model. Median percentage adherence among preprint versions was 33% (min-max: 10 to 68%). The percentage …

BackgroundA previously developed risk prediction model needs to be validated before being used in a new population. The finite size of the validation sample entails that there is uncertainty around model performance. We apply value-of-information (VoI) methodology to quantify the consequence of uncertainty in terms of net benefit (NB).MethodsWe define the expected value of perfect information (EVPI) for model validation as the expected loss in NB due to not confidently knowing which of the alternative decisions confers the highest NB. We propose bootstrap-based and asymptotic methods for EVPI computations and conduct simulation studies to compare their performance. In a case study, we use the non-US subsets of a clinical trial as the development sample for predicting mortality after myocardial infarction and calculate the validation EVPI for the US subsample.ResultsThe computation methods generated …