Utility of prediction model score: a proposed tool to standardize the performance and generalizability of clinical predictive models based on systematic review

View More View Less
  • 1 Department of Neurosurgery, Johns Hopkins University School of Medicine, Baltimore, Maryland;
  • 2 Departments of Neurosurgery and Orthopaedic Surgery, University of Southern California Keck School of Medicine, Los Angeles, California;
  • 3 Department of Neurosurgery, Thomas Jefferson University Hospitals, Philadelphia, Pennsylvania;
  • 4 Department of Neurosurgery, University of Toronto, St. Michael’s Hospital, Toronto, Ontario, Canada;
  • 5 Department of Neurosurgery, Medical College of Wisconsin, Milwaukee, Wisconsin; and
  • 6 Department of Neurosurgery, Lahey Hospital and Medical Center, Burlington, Massachusetts
Restricted access

Purchase Now

USD  $45.00

Spine - 1 year subscription bundle (Individuals Only)

USD  $369.00

JNS + Pediatrics + Spine - 1 year subscription bundle (Individuals Only)

USD  $600.00
Print or Print + Online

OBJECTIVE

The objective of this study was to evaluate the characteristics and performance of current prediction models in the fields of spine metastasis and degenerative spine disease to create a scoring system that allows direct comparison of the prediction models.

METHODS

A systematic search of PubMed and Embase was performed to identify relevant studies that included either the proposal of a prediction model or an external validation of a previously proposed prediction model with 1-year outcomes. Characteristics of the original study and discriminative performance of external validations were then assigned points based on thresholds from the overall cohort.

RESULTS

Nine prediction models were included in the spine metastasis category, while 6 prediction models were included in the degenerative spine category. After assigning the proposed utility of prediction model score to the spine metastasis prediction models, only 1 reached the grade of excellent, while 2 were graded as good, 3 as fair, and 3 as poor. Of the 6 included degenerative spine models, 1 reached the excellent grade, while 3 studies were graded as good, 1 as fair, and 1 as poor.

CONCLUSIONS

As interest in utilizing predictive analytics in spine surgery increases, there is a concomitant increase in the number of published prediction models that differ in methodology and performance. Prior to applying these models to patient care, these models must be evaluated. To begin addressing this issue, the authors proposed a grading system that compares these models based on various metrics related to their original design as well as internal and external validation. Ultimately, this may hopefully aid clinicians in determining the relative validity and usability of a given model.

ABBREVIATIONS AUC = area under the curve; NRS = numeric rating scale; NRS-BP = NRS back pain; NRS-LP = NRS leg pain; ODI = Oswestry Disability Index; PROBAST = Prediction Model Risk of Bias Assessment Tool; PSI = Patient Satisfaction Index; QOD = Quality Outcomes Database; TRIPOD = Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis; UPM = utility of prediction model.

Spine - 1 year subscription bundle (Individuals Only)

USD  $369.00

JNS + Pediatrics + Spine - 1 year subscription bundle (Individuals Only)

USD  $600.00

Contributor Notes

Correspondence Daniel M. Sciubba: Johns Hopkins University School of Medicine, Baltimore, MD. dsciubb1@jhmi.edu.

INCLUDE WHEN CITING Published online February 26, 2021; DOI: 10.3171/2020.8.SPINE20963.

Disclosures Dr. Buser reports being a consultant for Cerapedics, Xenco Medical (past), AO Spine (past), and Scripps Research; receiving clinical or research support for the study from SeaSpine (past), Next Science, and Motion Metrics; being a committee member for the North American Spine Society and the AOSNA Research Committee; being a co-chair of the research committee for the Lumbar Spine Society; and being an associate member of the AO Spine Knowledge Forum Degenerative. Dr. Wilson reports being a consultant for Stryker Canada and Bioventus. Dr. Sciubba reports being a consultant for Baxter, DePuy Synthes, Globus Medical, K2M, Medtronic, NuVasive, and Stryker, and unrelated grant support from Baxter Medical, the North American Spine Society, and Stryker.

  • 1

    Tomita K , Kawahara N , Kobayashi T , Surgical strategy for spinal metastases . Spine (Phila Pa 1976) . 2001 ;26 (3 ):298 306 .

    • Search Google Scholar
    • Export Citation
  • 2

    Tokuhashi Y , Matsuzaki H , Toriyama S , Scoring system for the preoperative evaluation of metastatic spine tumor prognosis . Spine (Phila Pa 1976) . 1990 ;15 (11 ):1110 1113 .

    • Search Google Scholar
    • Export Citation
  • 3

    Bauer HCF , Wedin R . Survival after surgery for spinal and extremity metastases. Prognostication in 241 patients . Acta Orthop Scand . 1995 ;66 (2 ):143 146 .

    • Search Google Scholar
    • Export Citation
  • 4

    Tokuhashi Y , Matsuzaki H , Oda H , A revised scoring system for preoperative evaluation of metastatic spine tumor prognosis . Spine (Phila Pa 1976) . 2005 ;30 (19 ):2186 2191 .

    • Search Google Scholar
    • Export Citation
  • 5

    Sioutos PJ , Arbit E , Meshulam CF , Galicich JH . Spinal metastases from solid tumors. Analysis of factors affecting survival . Cancer . 1995 ;76 (8 ):1453 1459 .

    • Search Google Scholar
    • Export Citation
  • 6

    Rades D , Dunst J , Schild SE . The first score predicting overall survival in patients with metastatic spinal cord compression . Cancer . 2008 ;112 (1 ):157 161 .

    • Search Google Scholar
    • Export Citation
  • 7

    van der Linden YM , Dijkstra SPDS , Vonk EJA , Prediction of survival in patients with metastases in the spinal column: results based on a randomized trial of radiotherapy . Cancer . 2005 ;103 (2 ):320 328 .

    • Search Google Scholar
    • Export Citation
  • 8

    Leithner A , Radl R , Gruber G , Predictive value of seven preoperative prognostic scoring systems for spinal metastases . Eur Spine J . 2008 ;17 (11 ):1488 1495 .

    • Search Google Scholar
    • Export Citation
  • 9

    Katagiri H , Takahashi M , Wakai K , Prognostic factors and a scoring system for patients with skeletal metastasis . J Bone Joint Surg Br . 2005 ;87 (5 ):698 703 .

    • Search Google Scholar
    • Export Citation
  • 10

    Paulino Pereira NR , Janssen SJ , van Dijk E , Development of a prognostic survival algorithm for patients with metastatic spine disease . J Bone Joint Surg Am . 2016 ;98 (21 ):1767 1776 .

    • Search Google Scholar
    • Export Citation
  • 11

    Paulino Pereira NR , Mclaughlin L , Janssen SJ , The SORG nomogram accurately predicts 3- and 12-months survival for operable spine metastatic disease: external validation . J Surg Oncol . 2017 ;115 (8 ):1019 1027 .

    • Search Google Scholar
    • Export Citation
  • 12

    Berger I , Piazza M , Sharma N , Evaluation of the risk assessment and prediction tool for postoperative disposition needs after cervical spine surgery . Neurosurgery . 2019 ;85 (5 ):E902 E909 .

    • Search Google Scholar
    • Export Citation
  • 13

    Stopa BM , Robertson FC , Karhade AV , Predicting nonroutine discharge after elective spine surgery: external validation of machine learning algorithms . J Neurosurg Spine . 2019 ;31 (5 ):742 747 .

    • Search Google Scholar
    • Export Citation
  • 14

    Goyal A , Ngufor C , Kerezoudis P , Can machine learning algorithms accurately predict discharge to nonhome facility and early unplanned readmissions following spinal fusion? Analysis of a national surgical registry . J Neurosurg Spine . 2019 ;31 (4 ):568 578 .

    • Search Google Scholar
    • Export Citation
  • 15

    Ogink PT , Karhade AV , Thio QCBS , Predicting discharge placement after elective surgery for lumbar spinal stenosis using machine learning methods . Eur Spine J . 2019 ;28 (6 ):1433 1440 .

    • Search Google Scholar
    • Export Citation
  • 16

    Devin CJ , Bydon M , Alvi MA , A predictive model and nomogram for predicting return to work at 3 months after cervical spine surgery: an analysis from the Quality Outcomes Database . Neurosurg Focus . 2018 ;45 (5 ):E9 .

    • Search Google Scholar
    • Export Citation
  • 17

    Asher AL , Devin CJ , Archer KR , An analysis from the Quality Outcomes Database, Part 2. Predictive model for return to work after elective surgery for lumbar degenerative disease . J Neurosurg Spine . 2017 ;27 (4 ):370 381 .

    • Search Google Scholar
    • Export Citation
  • 18

    Song X , Mitnitski A , Cox J , Rockwood K . Comparison of machine learning techniques with classical statistical models in predicting health outcomes . Stud Health Technol Inform . 2004 ;107 (Pt 1 ):736 740 .

    • Search Google Scholar
    • Export Citation
  • 19

    Senders JT , Staples PC , Karhade AV , Machine learning and neurosurgical outcome prediction: a systematic review . World Neurosurg . 2018 ;109 :476 486.e1 .

    • Search Google Scholar
    • Export Citation
  • 20

    Karhade AV , Thio QCBS , Ogink PT , Development of machine learning algorithms for prediction of 30-day mortality after surgery for spinal metastasis . Neurosurgery . 2019 ;85 (1 ):E83 E91 .

    • Search Google Scholar
    • Export Citation
  • 21

    Karhade AV , Thio QCBS , Ogink PT , Predicting 90-day and 1-year mortality in spinal metastatic disease: development and internal validation . Neurosurgery . 2019 ;85 (4 ):E671 E681 .

    • Search Google Scholar
    • Export Citation
  • 22

    Choi D , Pavlou M , Omar R , A novel risk calculator to predict outcome after surgery for symptomatic spinal metastases; use of a large prospective patient database to personalise surgical management . Eur J Cancer . 2019 ;107 :28 36 .

    • Search Google Scholar
    • Export Citation
  • 23

    Goodwin CR , Schoenfeld AJ , Abu-Bonsrah NA , Reliability of a spinal metastasis prognostic score to model 1-year survival . Spine J . 2016 ;16 (9 ):1102 1108 .

    • Search Google Scholar
    • Export Citation
  • 24

    Wolff RF , Moons KGM , Riley RD , PROBAST: a tool to assess the risk of bias and applicability of prediction model studies . Ann Intern Med . 2019 ;170 (1 ):51 58 .

    • Search Google Scholar
    • Export Citation
  • 25

    Morgen SS , Fruergaard S , Gehrchen M , A revision of the Tokuhashi revised score improves the prognostic ability in patients with metastatic spinal cord compression . J Cancer Res Clin Oncol . 2018 ;144 (1 ):33 38 .

    • Search Google Scholar
    • Export Citation
  • 26

    Katagiri H , Okada R , Takagi T , New prognostic factors and scoring system for patients with skeletal metastasis . Cancer Med . 2014 ;3 (5 ):1359 1367 .

    • Search Google Scholar
    • Export Citation
  • 27

    Ghori AK , Leonard DA , Schoenfeld AJ , Modeling 1-year survival after surgery on the metastatic spine . Spine J . 2015 ;15 (11 ):2345 2350 .

    • Search Google Scholar
    • Export Citation
  • 28

    Karhade AV , Ahmed AK , Pennington Z , External validation of the SORG 90-day and 1-year machine learning algorithms for survival in spinal metastatic disease . Spine J . 2020 ;20 (1 ):14 21 .

    • Search Google Scholar
    • Export Citation
  • 29

    Ahmed AK , Goodwin CR , Heravi A , Predicting survival for metastatic spine disease: a comparison of nine scoring systems . Spine J . 2018 ;18 (10 ):1804 1814 .

    • Search Google Scholar
    • Export Citation
  • 30

    De Silva T , Vedula SS , Perdomo-Pantoja A , SpineCloud: image analytics for predictive modeling of spine surgery outcomes . J Med Imaging (Bellingham) . 2020 ;7 (3 ):031502 .

    • Search Google Scholar
    • Export Citation
  • 31

    Siccoli A , de Wispelaere MP , Schröder ML , Staartjes VE . Machine learning-based preoperative predictive analytics for lumbar spinal stenosis . Neurosurg Focus . 2019 ;46 (5 ):E5 .

    • Search Google Scholar
    • Export Citation
  • 32

    McGirt MJ , Bydon M , Archer KR , An analysis from the Quality Outcomes Database, Part 1. Disability, quality of life, and pain outcomes following lumbar spine surgery: predicting likely individual patient outcomes for shared decision-making . J Neurosurg Spine . 2017 ;27 (4 ):357 369 .

    • Search Google Scholar
    • Export Citation
  • 33

    Rundell SD , Pennings JS , Nian H , Adding 3-month patient data improves prognostic models of 12-month disability, pain, and satisfaction after specific lumbar spine surgical procedures: development and validation of a prediction model . Spine J . 2020 ;20 (4 ):600 613 .

    • Search Google Scholar
    • Export Citation
  • 34

    Khor S , Lavallee D , Cizik AM , Development and validation of a prediction model for pain and functional outcomes after lumbar spine surgery . JAMA Surg . 2018 ;153 (7 ):634 642 .

    • Search Google Scholar
    • Export Citation
  • 35

    Asher AL , Devin CJ , Kerezoudis P , Predictors of patient satisfaction following 1- or 2-level anterior cervical discectomy and fusion: insights from the Quality Outcomes Database . J Neurosurg Spine . 2019 ;31 (6 ):835 843 .

    • Search Google Scholar
    • Export Citation
  • 36

    Quddusi A , Eversdijk HAJ , Klukowska AM , External validation of a prediction model for pain and functional outcome after elective lumbar spinal fusion . Eur Spine J . 2020 ;29 (2 ):374 383 .

    • Search Google Scholar
    • Export Citation
  • 37

    Massaad E , Fatima N , Hadzipasic M , Predictive analytics in spine oncology research: first steps, limitations, and future directions . Neurospine . 2019 ;16 (4 ):669 677 .

    • Search Google Scholar
    • Export Citation
  • 38

    Westermann L , Olivier AC , Samel C , Analysis of seven prognostic scores in patients with surgically treated epidural metastatic spine disease . Acta Neurochir (Wien) . 2020 ;162 (1 ):109 119 .

    • Search Google Scholar
    • Export Citation
  • 39

    Liu Y , Yang M , Li B , Development of a novel model for predicting survival of patients with spine metastasis from colorectal cancer . Eur Spine J . 2019 ;28 (6 ):1491 1501 .

    • Search Google Scholar
    • Export Citation
  • 40

    Singleton KW , Hsu W , Bui AAT . Comparing predictive models of glioblastoma multiforme built using multi-institutional and local data sources . AMIA Annu Symp Proc . 2012 ;2012 :1385 1392 .

    • Search Google Scholar
    • Export Citation
  • 41

    Steyerberg EW , Harrell FE Jr . Prediction models need appropriate internal, internal-external, and external validation . J Clin Epidemiol . 2016 ;69 :245 247 .

    • Search Google Scholar
    • Export Citation
  • 42

    Collins GS , de Groot JA , Dutton S , External validation of multivariable prediction models: a systematic review of methodological conduct and reporting . BMC Med Res Methodol . 2014 ;14 (1 ):40 .

    • Search Google Scholar
    • Export Citation
  • 43

    Steyerberg EW , Moons KGM , van der Windt DA , Prognosis research strategy (PROGRESS) 3: prognostic model research . PLoS Med . 2013 ;10 (2 ):e1001381 .

    • Search Google Scholar
    • Export Citation
  • 44

    Siontis GCM , Tzoulaki I , Castaldi PJ , Ioannidis JPA . External validation of new risk prediction models is infrequent and reveals worse prognostic discrimination . J Clin Epidemiol . 2015 ;68 (1 ):25 34 .

    • Search Google Scholar
    • Export Citation
  • 45

    Lee YH , Bang H , Kim DJ . How to establish clinical prediction models . Endocrinol Metab (Seoul) . 2016 ;31 (1 ):38 44 .

  • 46

    Steyerberg EW , Vergouwe Y . Towards better clinical prediction models: seven steps for development and an ABCD for validation . Eur Heart J . 2014 ;35 (29 ):1925 1931 .

    • Search Google Scholar
    • Export Citation
  • 47

    Collins GS , Reitsma JB , Altman DG , Moons KGM . Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD Statement . BMJ . 2015 ;350 (4 ):g7594 .

    • Search Google Scholar
    • Export Citation
  • 48

    White HJ , Bradley J , Hadgis N , Predicting patient-centered outcomes from spine surgery using risk assessment tools: a systematic review . Curr Rev Musculoskelet Med . 2020 ;13 (3 ):247 263 .

    • Search Google Scholar
    • Export Citation
  • 49

    Chung AS , Copay AG , Olmscheid N , Minimum clinically important difference: current trends in the spine literature . Spine (Phila Pa 1976) . 2017 ;42 (14 ):1096 1105 .

    • Search Google Scholar
    • Export Citation

Metrics

All Time Past Year Past 30 Days
Abstract Views 39 39 39
Full Text Views 11 11 11
PDF Downloads 23 23 23
EPUB Downloads 0 0 0