Diagnosis of neurosyphilis in HIV-negative patients with syphilis: development, validation, and clinical utility of a suite of machine learning models

Abstract

Background: The ability to accurately identify the absolute risk of neurosyphilis diagnosis for patients with syphilis would allow preventative and therapeutic interventions to be delivered to patients at high-risk, sparing patients at low-risk from unnecessary care. We aimed to develop, validate, and evaluate the clinical utility of simplified clinical diagnostic models for neurosyphilis diagnosis in HIV-negative patients with syphilis. Methods: We searched PubMed, China National Knowledge Infrastructure and UpToDate for publications about neurosyphilis diagnostic guidelines in English or Chinese from database inception until March 15, 2023. We developed and validated machine learning models with a uniform set of predictors based on six authoritative diagnostic guidelines across four continents to predict neurosyphilis using routinely collected data from real-world clinical practice in China and the United States (through the Dermatology Hospital of Southern Medical University in Guangzhou [659 recruited between August 2012 and March 2022, treated as Development cohort], the Beijing Youan Hospital of Capital Medical University in Beijng [480 recruited between December 2013 and April 2021, treated as External cohort 1], the Zhongshan Hospital of Xiamen University in Xiamen [493 recruited between November 2005 and November 2021, treated as External cohort 2] from China, and University of Washington School of Medicine in Seattle [16 recruited between September 2002 and April 2014, treated as External cohort 3] from United States). We included all these patients with syphilis into our analysis, and no patients were further excluded. We trained eXtreme gradient boosting (XGBoost) models to predict the diagnostic outcome of neurosyphilis according to each diagnostic guideline in two scenarios, respectively. Model performance was measured through both internal and external validation in terms of discrimination and calibration, and clinical utility was evaluated using decision curve analysis. Findings: The final simplified clinical diagnostic models included neurological symptoms, cerebrospinal fluid (CSF) protein, CSF white blood cell, and CSF venereal disease research laboratory test/rapid plasma reagin. The models showed good calibration with rescaled Brier score of 0.99 (95% CI 0.98–1.00) and excellent discrimination (the minimum value of area under the receiver operating characteristic curve, 0.84; 95% CI 0.81–0.88) when externally validated. Decision curve analysis demonstrated that the models were useful across a range of neurosyphilis probability thresholds between 0.33 and 0.66 compared to the alternatives of managing all patients with syphilis as if they do or do not have neurosyphilis. Interpretation: The simplified clinical diagnostic models comprised of readily available data show good performance, are generalisable across clinical settings, and have clinical utility over a broad range of probability thresholds. The models with a uniform set of predictors can simplify the sophisticated clinical diagnosis of neurosyphilis, and guide decisions on delivery of neurosyphilis health-care, ultimately, support accurate diagnosis and necessary treatment.