Can machine learning algorithms accurately predict discharge to nonhome facility and early unplanned readmissions following spinal fusion? Analysis of a national surgical registry

Presented at the 2019 AANS/CNS Section on Disorders of the Spine and Peripheral Nerves

Anshit Goyal Mayo Clinic Neuro-Informatics Laboratory,
Department of Neurosurgery, and

Search for other papers by Anshit Goyal in
jns
Google Scholar
PubMed
Close
 MBBS
,
Che Ngufor Robert D. and Patricia E. Kern Center for the Science of Health Care Delivery, Mayo Clinic, Rochester, Minnesota

Search for other papers by Che Ngufor in
jns
Google Scholar
PubMed
Close
 PhD
,
Panagiotis Kerezoudis Mayo Clinic Neuro-Informatics Laboratory,
Department of Neurosurgery, and

Search for other papers by Panagiotis Kerezoudis in
jns
Google Scholar
PubMed
Close
 MD, MS
,
Brandon McCutcheon Department of Neurosurgery, and

Search for other papers by Brandon McCutcheon in
jns
Google Scholar
PubMed
Close
 MD
,
Curtis Storlie Robert D. and Patricia E. Kern Center for the Science of Health Care Delivery, Mayo Clinic, Rochester, Minnesota

Search for other papers by Curtis Storlie in
jns
Google Scholar
PubMed
Close
 PhD
, and
Mohamad Bydon Mayo Clinic Neuro-Informatics Laboratory,
Department of Neurosurgery, and

Search for other papers by Mohamad Bydon in
jns
Google Scholar
PubMed
Close
 MD
Full access

OBJECTIVE

Nonhome discharge and unplanned readmissions represent important cost drivers following spinal fusion. The authors sought to utilize different machine learning algorithms to predict discharge to rehabilitation and unplanned readmissions in patients receiving spinal fusion.

METHODS

The authors queried the 2012–2013 American College of Surgeons National Surgical Quality Improvement Program (ACS-NSQIP) for patients undergoing cervical or lumbar spinal fusion. Outcomes assessed included discharge to nonhome facility and unplanned readmissions within 30 days after surgery. A total of 7 machine learning algorithms were evaluated. Predictive hierarchical clustering of procedure codes was used to increase model performance. Model performance was evaluated using overall accuracy and area under the receiver operating characteristic curve (AUC), as well as sensitivity, specificity, and positive and negative predictive values. These performance metrics were computed for both the imputed and unimputed (missing values dropped) datasets.

RESULTS

A total of 59,145 spinal fusion cases were analyzed. The incidence rates of discharge to nonhome facility and 30-day unplanned readmission were 12.6% and 4.5%, respectively. All classification algorithms showed excellent discrimination (AUC > 0.80, range 0.85–0.87) for predicting nonhome discharge. The generalized linear model showed comparable performance to other machine learning algorithms. By comparison, all models showed poorer predictive performance for unplanned readmission, with AUC ranging between 0.63 and 0.66. Better predictive performance was noted with models using imputed data.

CONCLUSIONS

In an analysis of patients undergoing spinal fusion, multiple machine learning algorithms were found to reliably predict nonhome discharge with modest performance noted for unplanned readmissions. These results provide early evidence regarding the feasibility of modern machine learning classifiers in predicting these outcomes and serve as possible clinical decision support tools to facilitate shared decision making.

ABBREVIATIONS

ACC = accuracy; ACS-NSQIP = American College of Surgeons National Surgical Quality Improvement Program; ALP = alkaline phosphatase; ANN = artificial neural network; ASA = American Society of Anesthesiologists; AUC = area under the receiver operating characteristic curve; BUN = blood urea nitrogen; CHF = congestive heart failure; COPD = chronic obstructive pulmonary disease; CPT = Current Procedural Terminology; GBM = gradient boosting machine; GLM = generalized linear model; GLMnet = elastic-net GLM; HTN = hypertension; INR = international normalized ratio; LASSO = least absolute shrinkage and selection operator; NPV = negative predictive value; NSQIP = National Surgical Quality Improvement Program; ODI = Oswestry Disability Index; PHC = predictive hierarchical clustering; pLDA = penalized linear discriminant analysis; PPV = positive predictive value; PTT = partial thromboplastin time; ROC = receiver operating characteristic; RF = random forest; SGOT = serum glutamic oxaloacetic transaminase; WBC = white blood cell count.

OBJECTIVE

Nonhome discharge and unplanned readmissions represent important cost drivers following spinal fusion. The authors sought to utilize different machine learning algorithms to predict discharge to rehabilitation and unplanned readmissions in patients receiving spinal fusion.

METHODS

The authors queried the 2012–2013 American College of Surgeons National Surgical Quality Improvement Program (ACS-NSQIP) for patients undergoing cervical or lumbar spinal fusion. Outcomes assessed included discharge to nonhome facility and unplanned readmissions within 30 days after surgery. A total of 7 machine learning algorithms were evaluated. Predictive hierarchical clustering of procedure codes was used to increase model performance. Model performance was evaluated using overall accuracy and area under the receiver operating characteristic curve (AUC), as well as sensitivity, specificity, and positive and negative predictive values. These performance metrics were computed for both the imputed and unimputed (missing values dropped) datasets.

RESULTS

A total of 59,145 spinal fusion cases were analyzed. The incidence rates of discharge to nonhome facility and 30-day unplanned readmission were 12.6% and 4.5%, respectively. All classification algorithms showed excellent discrimination (AUC > 0.80, range 0.85–0.87) for predicting nonhome discharge. The generalized linear model showed comparable performance to other machine learning algorithms. By comparison, all models showed poorer predictive performance for unplanned readmission, with AUC ranging between 0.63 and 0.66. Better predictive performance was noted with models using imputed data.

CONCLUSIONS

In an analysis of patients undergoing spinal fusion, multiple machine learning algorithms were found to reliably predict nonhome discharge with modest performance noted for unplanned readmissions. These results provide early evidence regarding the feasibility of modern machine learning classifiers in predicting these outcomes and serve as possible clinical decision support tools to facilitate shared decision making.

In Brief

The authors determined the performance of complex machine learning algorithms to predict nonhome discharge and unplanned readmissions following spinal fusion. In today's era, it has become increasingly important for novel clinical decision support tools to facilitate preoperative risk stratification and early discharge planning. While previous studies have explored the feasibility of machine learning algorithms to predict spine surgical outcomes, this study provides an important evaluation of discharge to nonhome facility and unplanned readmissions in the same context.

Spinal fusion is one of the most common surgical procedures in the US, with more than 450,000 surgeries being performed on an annual basis, and the number continues to grow at an astonishing rate.18,20,21,46 While fusion remains an effective surgical strategy to treat degenerative conditions of the cervical and lumbar spine, it can be associated with various adverse outcomes, including postsurgical complications (3%–6%), the need for discharge to skilled rehabilitation facilities (9%–18%), and unplanned readmissions (5%–9%).1,3,14,19,32,33 These adverse outcomes indirectly translate to rising costs associated with this group of procedures, which has attracted the attention of national health policy makers, putting surgical outcomes under greater scrutiny.6

With increasing efforts aimed at bending the healthcare cost curve, emphasis is being placed on registries and databases that provide reliable data for tracking and calculating risk-adjusted estimates for these outcomes. Thus clinicians today must handle enormous amounts of complex data, which requires use of robust analytical strategies.26

Machine learning methods can harness high-dimensional and medical data to generate accurate patient risk stratification models, contribute to the development of smart guidelines, and shape healthcare decisions by customizing care to individual patients. Although there have been recent reports with analyses from large datasets attempting to identify predictors of discharge to rehabilitation and unplanned readmission following spine surgery,32,36 there is a dearth of literature in this area and, more specifically, a paucity of studies investigating predictive modeling using complex machine learning algorithms as a method to predict these outcomes.32,36 In this retrospective cohort analysis, we aimed to evaluate and compare the predictive performance of various machine learning models fitted for discharge disposition other than home and unplanned readmission following spinal fusion using a multi-institutional surgical registry.

Methods

Data Source

We queried the American College of Surgeons National Surgical Quality Improvement Program (ACS-NSQIP) participant user files for the years 2012–2013 to conduct this retrospective cohort analysis. NSQIP is a surgical registry with prospectively collected data that currently holds information for over 1.7 million patients from more than 500 hospitals.31 More than 300 variables are recorded in the database, including demographic characteristics, comorbidities, and intraoperative data as well as perioperative outcomes.25 Postoperative complications recorded within 30 days of surgery are clearly defined. Data collectors at each participating institution undergo extensive training for accurately performing data abstraction. In order to ensure data quality and fidelity, regular interrater reliability audits are performed by certified ACS-NSQIP Surgical Clinical Reviewers, along with additional quality control processes.25,43 The data recorded in NSQIP are deidentified and hence our study was exempt from Institutional Review Board (IRB) approval.

Inclusion and Exclusion Criteria

Current Procedural Terminology (CPT), Fourth Edition codes were used to identify patients that underwent any form of cervical or lumbar spinal fusion surgery. The list of codes is provided in Supplemental Digital Content 1 Table 1.

Outcomes of Interest

The primary outcome of interest was discharge disposition to a nonhome facility, and a secondary outcome was unplanned readmission within 30 days of surgical intervention.

Covariates

Covariates of interest included the following: age, race, sex, BMI, functional status, inpatient/outpatient setting, history of diabetes, hypertension (HTN) requiring medication, history of severe chronic obstructive pulmonary disease (COPD), smoking status, history of chronic corticosteroid use within 30 days of the operation, dyspnea at baseline, history of a bleeding disorder, American Society of Anesthesiologists (ASA) class, and CPT codes classified by use of predictive hierarchical clustering (PHC).30 Although PHC mimics traditional hierarchical clustering, the goal of PHC is to cluster subgroups instead of individual observations found within the dataset, such that the discovered clusters result in improved performance of a classification model.30 Moreover, preoperative laboratory values of sodium, hematocrit, white blood cell count (WBC), platelets, albumin, bilirubin, serum glutamic oxaloacetic transaminase (SGOT), alkaline phosphatase (ALP), creatinine, blood urea nitrogen (BUN), international normalized ratio (INR), and partial thromboplastin time (PTT) were also analyzed.

Handling Missing Values

As with most clinical data, the NSQIP data contained a significant amount of missing values. Specifically, the considered variables in the sample are missing in 0% to 60% of cases. To efficiently handle the missing values in the NSQIP data, we applied the random forest (RF) imputation method missForest44 implemented in the R statistical programming language to impute variables with fewer than 25% missing cases. Compared to complete case analysis, using imputation has been shown to yield better, more accurate, and better-performing models.4 To investigate potential loss in predictive power and bias of complete case analysis, we also performed experiments using the unimputed data, i.e., for the same set of variables; we dropped cases with missing values.

Preprocessing

Unlike traditional statistical modeling methods, some machine learning methods require that the data be formatted in certain ways to conform to mathematical assumptions. For instance certain elements of the objective function of a learning algorithm (such as artificial neural networks [ANNs] or the ridge and least absolute shrinkage and selection operator [LASSO] regularizers of the elastic-net generalized linear model [GLMnet] and penalized linear discriminant analysis [pLDA] models, see below) assume that all predictors are centered around zero and have variance in the same order. If a predictor has a variance that is orders of magnitude larger than others, it might dominate the objective function and make the model unable to learn from other predictors correctly as expected. Therefore, we preprocessed the data such that all numerical variables were normalized to zero mean and unit variance. Categorical variables were converted into binary format using one-hot encoding. That is, a categorical predictor with k possible values was transformed into k − 1 binary predictors, with only one active.

Statistical Analysis

All statistical analyses were performed using R version 3.4.1 (R Foundation for Statistical Computing). We performed descriptive statistics to assess baseline patient demographic and clinical characteristics. Continuous variables were compared using an unpaired, two-tailed Student t-test (parametric) while categorical variables were analyzed using the chi-square test (parametric). All p values were two-sided. Statistical significance was defined as p < 0.05.

Predictive Modeling

A total of 7 classification algorithms (GLM, GLMnet, pLDA, naïve Bayes, ANNs, RF, and gradient boosting machines [GBMs]) were trained to predict discharge to nonhome facility and unplanned readmission following spine surgery based on the patient characteristics described in the Covariates section. A description of each algorithm is presented below, and details regarding optimization formulas and possible tuning parameters for each algorithm are available in Supplemental Digital Content 1 Table 2 and Supplemental Digital Content 2. Other than predicting the response from a set of predictors, another common and important step in data-driven modeling is to identify which predictors are most relevant to the prediction task. These variables can be used for interpretation or dimension reduction. The RF and GBM algorithms are capable of producing a variable ranking score or variable importance based on their association with other variables and with the response variable. It should be noted that the variable importance or influence does not provide any explanation about how the variable actually affects the response. For example, a large variable importance score does not necessarily mean large values of the variable will increase/decrease the risk of an outcome. Rather, the actual relationship may be complex, with nonlinearity and interaction.

Selection of Optimal Classification Threshold

Using the predicted risk scores from each model, sensitivity and specificity were computed across all possible cutoff or threshold values that defined class allocations and used to construct the receiver operating characteristic (ROC) curve. The optimal classification threshold was defined as the cutoff risk value on the curve that maximized Youden’s index: J = (sensitivity + specificity) − 1.

Training and Evaluation

Most machine learning algorithms require choosing one or more hyperparameters for optimal performance of the learning algorithm and avoiding overfitting of the data. For the considered machine learning algorithms requiring hyperparameter tuning (e.g., GLMnet, pLDA, ANN, RF, and GBM), a grid search was set up for each combination of hyperparameters and the best combination selected by a 10-fold cross-validation procedure. After selection of optimal tuning parameters, these were then used to train and evaluate the algorithms through 10-fold cross-validation. Ten percent of the training portion of each cross-validation was set aside for selecting the optimal classification threshold and the rest for the final evaluation. Variable importance was assessed for both GBM and RF algorithms, which can directly output a variable importance score for each predictor as a side effect of training. In addition, relative predictor importance was also assessed using GLMs. PHC was employed for agglomerative hierarchical clustering of CPT codes with the goal to increase model performance.30

The accuracy (ACC), area under the ROC curve (AUC), sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) were used to evaluate the models. Multiple imputation was performed for missing data, and model performance was evaluated using both imputed and unimputed datasets. Validation results from the unimputed datasets are available within Supplemental Digital Content 1 Tables 3 and 4.

Results

Cohort Characteristics

A total of 59,145 patients were analyzed. The mean age of the cohort was 57 ± 14.3 years. Over half of the included patients were male (51.5%), and the majority were Caucasian (82.6%, n = 48,899) and non-Hispanic (89%, n = 52,636). Most patients were admitted from home (97.7%, n = 57,799), were independent per functional status (96.6%, n = 57,141), and belonged to ASA class II (51.2%, n = 30,330). Among comorbidities, COPD, congestive heart failure (CHF), and acute renal failure were uncommon, with estimated percentages of 4%, 0.3%, and 0.07%, respectively. Diabetes mellitus was noted in 16.7% (n = 9359) of patients, while 23.5% (n = 13,493) were active smokers within 1 year of surgery. Corticosteroid use within 30 days of the procedure was noted in 3.9% (n = 2350) of patients. HTN requiring medication was observed in 49% (n = 29052) of patients. Nearly one-third of procedures (31.4%, n = 18,575) were performed by orthopedic surgery while the rest were performed by neurosurgery (68%, n = 40,197).

Discharge to Nonhome Facility

The overall rate of nonhome discharge was 12.6% (n = 7487). Table 1 summarizes baseline demographics and comorbidities between patients with discharge disposition other than home and those discharged to home.

TABLE 1.

Discharge to nonhome facility: summary of patient demographic characteristics, comorbidities, and preoperative laboratory values

Discharge to Nonhome Facility
VariableNo (n = 51,658)Yes (n = 7487)p Value
Patient characteristics
 Age group in yrs<0.001
  18–24731 (1)23 (0)
  25–343174 (6)111 (1)
  35–447605 (15)239 (3)
  45–5412,373 (24)696 (9)
  55–6413,018 (25)1484 (20)
  65–7410,206 (20)2471 (33)
  75–844082 (8)1996 (27)
  85+469 (1)467 (6)
 Male27,218 (53)3270 (44)<0.001
 Race
  Asian929 (2)175 (2)<0.001
  African-American3477 (7)829 (11)
  Native American/Alaska Native224 (0)23 (0)
  Caucasian42,928 (83)5971 (80)
  Hawaiian/Pacific Islander142 (0)32 (0)
 Source of admission<0.001
  From acute care, inpatient228 (0)363 (5)
  Admitted from home51,078 (99)6721 (90)
  Nursing home (chronic/intermediate care)79 (0)151 (2)
  Outside emergency department207 (0)206 (3)
  Transfer from other51 (0)42 (1)
  Unknown15 (0)4 (0)
 Functional status<0.001
  Independent50,500 (98)6641 (89)
  Partially dependent746 (1)716 (10)
  Totally dependent68 (0)85 (1)
  Unknown344 (1)45 (1)
 Outpatient procedures14,041 (27)192 (3)
 Surgical specialty<0.001
  Neurosurgery35,384 (68)4813 (64)
  Orthopedics15,932 (31)2643 (35)
  Other342 (1)31 (1)
 Anesthesia0.017
  General51,412 (100)7464 (100)
  Epidural58 (0)5 (0)
  Spinal85 (0)8 (0)
  Other81 (0)10 (0)
 BMI, median (IQR)29.2 (25.6–33.5)29.5 (25.6–34.5)<0.001
Comorbidities
 Active smoking w/in 1 yr12,729 (25)1214 (16)
 Diabetes
  Insulin2267 (4)770 (10)
  Non-insulin5098 (10)1224 (16)
 HTN requiring medication23,886 (46)5166 (69)<0.001
 Dyspnea<0.001
  At rest144 (0)61 (1)
  Moderate exertion2504 (5)676 (9)
 COPD1870 (4)555 (7)<0.001
 CHF78 (0)90 (1)<0.001
 Disseminated malignancy341 (1)278 (4)<0.001
 Bleeding disorder626 (1)326 (4)<0.001
 On dialysis97 (0)84 (1)<0.001
 Steroid use within 30 days1807 (3)543 (7)<0.001
Preoperative lab values, median (IQR)
 Sodium139 (138–141)139 (137–141)<0.001
 BUN, mg/dl15 (12–18)17 (13–22)<0.001
 Creatinine, mg/dl0.88 (0.74–1)0.89 (0.72–1.06)<0.001
 Albumin, g/dl4.2 (4.1–4.3)4 (3.7–4.2)<0.001
 Bilirubin, mg/dl0.54 (0.46–0.64)0.53 (0.44–0.65)0.004
 SGOT, IU/L24 (21–27)24 (21–28)<0.001
 ALP73 (67–80)78 (70–89)<0.001
 WBC, ×103/μl7.1 (6.0–8.5)7.4 (6.0–9.1)<0.001
 Hematocrit, %42 (39–44)39 (36–42)<0.001
 Platelets, ×103/μl239 (203–278)234 (192–278)<0.001
 PTT29 (28–30)29 (27–31)<0.001
 INR1.00 (0.97–1.01)1.00 (0.99–1.08)<0.001

Values are presented as number of patients (%) unless otherwise indicated. Boldface type indicates statistical significance.

For nonhome discharge, AUC ≥ 0.85 was observed for all classification algorithms following training and evaluation through 10-fold cross-validation using the imputed dataset, thus representing excellent discriminatory performance (Table 2). The ACC, representing the percentage of cases correctly classified, which serves as an overall measure of model accuracy, approached 80% for all models. Similarly, a sensitivity and specificity of > 75% was observed for all algorithms. A high negative predictive value (96%) was also observed for all models. Interestingly, logistic regression (GLM) showed predictive performance similar to those of the remaining algorithms (AUC = 0.87). Overall sensitivity ranged between 77% and 80%, with ANN showing highest sensitivity. With the exception of sensitivity, performance metrics were found to be better using imputed data (ANN AUC, 0.87 vs 0.86; ACC, 0.78 vs 0.77; sensitivity, 0.80 vs 0.81; specificity, 0.78 vs 0.76).

TABLE 2.

Prediction of nonhome discharge: performance of different machine learning approaches

ClassifierAUCACCSensSpecPPVNPV
GLM0.870.790.790.790.350.96
GBM0.870.770.790.770.340.96
GLMnet0.870.780.790.780.340.96
ANN0.870.780.800.780.340.96
RF0.850.770.790.760.320.96
pLDA0.860.780.770.780.340.96
VarBayes0.870.780.790.780.340.96

Sens = sensitivity; Spec = specificity; VarBayes = Variance Bayes.

Interpreting the GBM, RF, and GLM Models for Predicting Nonhome Discharge

For each prediction from the GBM, RF, and GLM models, we extracted the variable importance score and averaged over the 10-fold cross-validation. The variable importance of GLM corresponds to the absolute coefficients of the model. Although the ranking order of relative importance of each variable was different between the 3 algorithms, in general, advanced age, inpatient surgery, days from hospital admission to operation, low preoperative albumin, dependent functional status, higher ASA class (≥ 3), and preoperative lab values such as serum albumin, ALP, hematocrit, and INR were noted to be significant predictors of discharge to a rehabilitation/skilled nursing facility. A list of variables in terms of predictor importance is available as Supplemental Digital Content 1 Table 5.

Unplanned Readmission

The observed rate of unplanned readmission within 30 days of surgery was 4.5% (n = 2681). Table 3 summarizes baseline clinical characteristics of patients readmitted and not readmitted within 30 days.

TABLE 3.

Unplanned 30-day readmission: summary of patient demographic characteristics, comorbidities, and preoperative laboratory values

Unplanned Readmission
VariableNo (n = 57,464)Yes (n = 2681)p Value
Patient characteristics
 Age group in yrs<0.001
  18–24737 (1)17 (1)
  25–343196 (6)89 (3)
  35–447638 (14)206 (8)
  45–5412,578 (22)491 (18)
  55–6413,805 (24)697 (26)
  65–7411,976 (21)701 (26)
  75–845679 (10)399 (15)
  85+855 (2)81 (3)
 Male27,218 (53)3270 (44)<0.001
 Race
  Asian1058 (2)46 (2)<0.001
  African-American4045 (7)261 (10)
  Native American/Alaska Native232 (0)15 (1)
  Caucasian46,703 (83)2196 (82)
  Hawaiian/Pacific Islander166 (0)8 (0)
 Transfer<0.001
  From acute care, inpatient542 (1)49 (2)
  Admitted from home55,247 (98)2552 (95)
  Nursing home, chronic/intermediate care193 (0)37 (1)
  Transfer from other85 (0)8 (0)
  Unknown18 (0)1 (0)
 Functional status<0.001
  Independent54,649 (97)2492 (93)
  Partially dependent1325 (2)137 (5)
  Totally dependent133 (0)1 (20)
  Unknown357 (1)32 (1)
 Outpatient procedures13,859 (25)374 (14)
 Surgical specialty<0.001
  Neurosurgery38,354 (68)1843 (69)
  Orthopedics17,748 (31)827 (31)
  Other362 (1)11 (0)
 Anesthesia<0.017
  General51,412 (100)7464 (100)
  Epidural58 (0)5 (0)
  Spinal85 (0)8 (0)
  Other81 (0)10 (0)
 BMI, median (IQR)29.27 (25.6–33.7)29.5 (25.8–34.5)0.003
Comorbidities
 Active smoking w/in 1 year13,341 (24)602 (22)0.16
 Diabetes<0.001
  Insulin2788 (5)249 (9)
  Non-insulin5995 (11)327 (12)
 HTN requiring medication27,433 (49)1619 (60)<0.001
 Dyspnea<0.001
  At rest189 (0)16 (1)
  Moderate exertion2959 (5)221 (8)
 COPD2212 (4)213 (8)<0.001
 CHF145 (0)23 (1)<0.001
 Disseminated malignancy527 (1)92 (3)<0.001
 Bleeding disorder873 (2)79 (3)<0.001
 On dialysis146 (0)35 (1)<0.001
 Steroid use w/in 30 days2146 (4)204 (8)<0.001
Preoperative lab values, median (IQR)
 Sodium139 (138–141)139 (137–141)<0.001
 BUN, mg/dl15 (12–19)16 (13–21)<0.001
 Creatinine, mg/dl0.88 (0.74–1.00)0.90 (0.75–1.07)<0.001
 Albumin, g/dl4.2 (4.0–4.3)4.1 (3.9–4.3)<0.001
 Bilirubin, mg/dl0.54 (0.46–0.64)0.53 (0.45–0.65)0.15
 SGOT, IU/L24 (21–27)24 (21–28)0.51
 ALP73 (67–80)76 (69–85)<0.001
 WBC, ×103/μl7.2 (6.0–8.6)7.4 (6.0–9.1)<0.001
 Hematocrit, %41 (39–44)40 (37–43)<0.001
 Platelets, ×103/μl238 (202–278)238 (197–282)0.29
 PTT29 (28–30)20 (27–30)0.73
 INR1.00 (0.97–1.01)1.00 (0.98–1.04)<0.001

Values are presented as number of patients (%) unless otherwise indicated. Boldface type indicates statistical significance.

Compared to discharge to rehabilitation/skilled nursing facility, predictive performance for readmission was significantly lower (using the imputed dataset) with AUC ranging from 0.63–0.66 (moderate discriminatory performance) and ACC between 0.59 and 0.71 (Table 4). Generally, sensitivity was observed to be low, ranging from 46% to 61%, while specificity varied between 59% and 72%. In terms of ACC and specificity, ANN had the most superior performance (ACC = 0.71, specificity = 0.72). On the other hand, the GLMnet had higher sensitivity (64%) and discrimination (AUC = 0.66). Comparable performance was also observed for logistic regression (GLM) (AUC = 0.66, sensitivity = 61%). Given the low rate of readmission, PPV was very low (7%) while NPV was observed to be high (97%) for all evaluated models. As observed with discharge to rehabilitation, model performance declined when using the unimputed dataset (ANN AUC, 0.63 vs 0.61; ACC, 0.71 vs 0.62; sensitivity, 0.46 vs 0.53; specificity, 0.72 vs 0.62).

TABLE 4.

Prediction of unplanned 30-day readmission: performance of different machine learning approaches

ClassifierAUCACCSensSpecPPVNPV
GLM0.660.610.610.610.070.97
GBM0.660.590.630.590.070.97
GLMnet0.660.600.640.600.070.97
ANN0.630.710.460.720.070.97
RF0.640.640.550.640.070.97
pLDA0.660.660.560.660.070.97
VarBayes0.660.620.610.620.070.97

Discussion

While a large body of evidence dedicated to postoperative complications following spine surgery is available, analyses for discharge disposition remain rare in the current literature.6,23,36,41 To the best of our knowledge, this is the first study to use machine learning to analyze nonhome discharge and unplanned readmission following spinal fusion. Both cervical and lumbar fusions were included in our analysis. The results of the present study are important as they provide preliminary evidence with regard to the feasibility of modern machine learning techniques in predicting these outcomes following spine surgery.

For discharge to nonhome facility, we found excellent discrimination (AUC > 0.85) using all learning approaches. Logistic regression demonstrated equivalence although ANNs showed slightly higher overall performance compared to all other classification algorithms. Multiple imputation contributed toward increased predictive performance.

Studies for training and evaluating prediction models fitted for unplanned readmission following spine surgery are also scarce in the literature. McGirt et al. used prospectively collected data from over 1800 patients to develop and validate a predictive model for complication, readmission, return to work, and 12-month Oswestry Disability Index (ODI) score improvement following lumbar spine surgery.33 Important variables, such as presence and duration of acute/chronic low-back and leg pain, disability, presence of myelopathy, and radiculopathy, were also taken into account. The authors noted excellent discrimination (AUC = 0.79) for readmission prediction despite a much smaller dataset using a logistic regression model. This result demonstrates the importance of the quality and nature of training data, which determines the performance of any classifier.

For unplanned readmission, we observed that the performance for all 7 algorithms was relatively poor compared to that observed with discharge to nonhome facility. This finding could be attributable to the relatively low occurrence of unplanned readmission (5%) in our cohort. Low occurrence rates lead to class imbalances that are a significant impediment to machine learning.13,27 Therefore, larger datasets are required to increase model performance for relatively rare outcomes. It is important to note, however, that most predictive models for hospital readmission published in the literature perform poorly, with AUCs in the range of 0.56–0.72, as noted in a systematic review published in the Journal of the American Medical Association.24 The authors concluded that results might be improved by including functional and social variables in addition to variables for medical comorbidities. Other studies from NSQIP and retrospective cohort analysis of single-institutional data suggest that postoperative complications such as wound infection, pain, and comorbidities such as COPD and CHF are significant predictors of readmission. It is important to note that socioeconomic variables such as insurance status, income, and education are absent in NSQIP. These factors, especially insurance status, are also significant determinants in influencing risk of unplanned readmission, according to a recent systematic review.11

Predictive Modeling in Neurosurgery

Within neurosurgery, a vast array of studies have been performed evaluating the utility of machine learning to predict surgical outcomes.42 Machine learning has been used to predict the likelihood of seizure-free outcome after epilepsy surgery, survival after glioma resection, and stroke/complications in cerebrovascular surgery.2,7,12,16,34,35,37,47 Within the realm of spine surgery, few studies have evaluated machine learning techniques for predicting outcomes. Kim et al.26 evaluated and compared the predictive performance of ANNs and logistic regression models to predict complications following posterior lumbar fusion using the NSQIP dataset. The authors found ANNs and logistic regression to be largely comparable, with either technique showing marginally greater predictive performance over the other for different complications, although the results were almost similar. Using an administrative claims database, Ratliff et al. attempted predictive modeling for complications following spine surgery using logistic regression with and without LASSO regularization and noted similar predictive performance when they compared the two approaches.40 Machine learning has also been used to predict patient satisfaction and costs after spinal fusion.9,28 Table 5 summarizes the results of studies of the utilization of machine learning algorithms to predict outcomes following spine surgery. Generalized linear modeling/logistic regression, however, remains the most commonly studied predictive modeling technique in the medical literature,26 and it remains important to note that this method can often perform as well as more sophisticated machine learning methods. More advanced algorithms might be useful in cases of important nonlinearities and interactions between the predictors and outcome. Therefore, careful data evaluation and decision making regarding the best predictive modeling approach are becoming increasingly important for those assessing neurosurgical outcomes in patient populations.

TABLE 5.

Published studies on predictive modeling using machine learning for spine surgical outcomes

Authors & YearData SourceNo. of PtsTraining & Validation MethodProcedureOutcome of InterestTechnique Employed
Kuo et al., 2018Single-institutional53210-fold cross validationSpinal fusionCosts during hospital admissionNaïve Bayesian, SVMs, logistic regression, C4.5 decision tree, RF
Passias et al., 2018Multi-institutional101Not reportedCervical deformity correctionDistal junctional kyphosisRF algorithm
Watad et al., 2018Single-institutional3070:30 data splitElective spine op with ASA class I or IIIntracranial pressure during surgeryANNs
Kim et al., 2018ACS-NSQIP22,62970:30Posterior lumbar fusionComplications w/in 30 days postop*ANNs
Oh et al., 2017Multi-institutional23470:30 data splitAdult spinal deformity opQALYsC5.0 algorithm (type of decision tree model)
Durand et al., 2018ACS-NSQIP102980:20 data splitAdult spinal deformity opBlood transfusions w/in 30 days postopSingle classification tree & random forest
Ratliff et al., 2016Truven Market Scan database279,13580:20 data splitAll cervical & lumbar spine op proceduresComplications w/in 30 daysLASSO (GLMnet) & multivariable logistic regression
Hoffman et al., 2015Single-institutional27Not reportedSurgery for CSMFunctional outcome (ODI)SVM & multivariable linear regression
Azimi et al., 2015Single-institutional4022:1:1 (training, testing, validation)DiscectomyRecurrent lumbar disc herniationANN & logistic regression
Azimi et al., 2014Single-institutional1262:1 data splitLaminectomyPatient satisfaction at last follow-upANNs
Azimi et al., 2016Single-institutional2032:1:1 (training, testing, validation)Lumbar disc herniation opOp outcome per Macnab classificationANNs

CSM = cervical spondylotic myelopathy; pts = patients; QALY = quality-adjusted life year; SVM = support vector machine.

Wound complications, cardiac complications, venous thromboembolism, mortality.

Wound complications, pneumonia, renal failure, myocardial infarction, pulmonary complications, neurological complications, CHF, venous thromboembolism, urinary tract infection, cardiac dysrhythmia, delirium.

Significance

This is the first study to demonstrate the feasibility of machine learning techniques for discharge destination prediction following spine surgery. We found that machine learning models demonstrate excellent performance for discharge disposition, although more modest results were observed for early hospital readmission. Given the excellent discrimination and accuracy resulting from the use of these techniques, they could be easily be implemented as a clinical tool for shared decision making and early postdischarge planning, thereby allowing conservative resource allocation and cost savings.

Strengths and Limitations

Our study results represent a significant improvement over previous results reported in the existing literature. We employed “big data” modeling from a national surgical registry with data of higher quality than those found in the traditional claims databases17,29 that have been utilized in some previous studies.40 As opposed to studies mainly employing linear models to analyze discharge to rehabilitation/skilled nursing facility,32,36 we utilized multiple machine learning algorithms and compared their predictive performance to each other as well as a traditional generalized linear model. Both cervical and lumbar spine procedures were included, thereby increasing the generalizability of our findings. PHC, a recently reported novel method to provide agglomerative clustering of CPT codes, was utilized to increase model performance.30 We also used multiple measures to evaluate model performance, such as sensitivity, specificity, and overall accuracy in addition to AUC. However, our PPV for the outcome was consistently low while the NPV was high. PPVs and NPVs are generally dependent on the actual incidence of the outcome in the population. This remains as a persistent challenge to the clinical translation of all predictive modeling strategies.

This study has limitations as well. First, abstraction errors can occur within the database. However, according to a recent report by the ACS, it has been found that reliability between abstractors reaches 98%.5 Second, the dataset is not specific to spine surgery and lacks many important variables which could be possible determinants of outcomes, such as acute/chronic back pain and leg pain, baseline disability, presence/absence of myelopathy, and radiculopathy. Having the right clinically meaningful set of variables and optimal data quality is also important in addition to the size of the training dataset. Third, NSQIP captures outcomes until 30 days after surgery and not discharge, hence length of stay needs to be taken into account while considering postdischarge outcomes. We hope that the maturity of the Quality and Outcomes Database (QOD) registry, formerly known as N2QOD will allow for long-term risk assessment in such patient populations and also account for important confounding variables. Lastly, our machine learning algorithms have only been validated internally and further work needs to be carried out in order to validate the predictive model in a separate dataset of surgical patients. Nevertheless, we hope that the results from our current study will lay the groundwork for further studies to utilize machine learning as a risk stratification tool for spine surgery.

Conclusions

Machine learning algorithms represent a valid technique to predict nonhome discharge and unplanned readmission following spine surgery. While excellent performance was noted for discharge to rehabilitation/skilled nursing facility, a more modest performance was noted for unplanned readmission. Traditional classification algorithms for these outcomes, such as logistic regression, may also demonstrate equivalent predictive performance. Further studies are needed with larger datasets using other clinically relevant variables to compare the performance of traditional predictive models and modern machine learning techniques in order to predict outcomes following spine surgery.

Disclosures

The authors report no conflict of interest concerning the materials or methods used in this study or the findings specified in this paper.

Author Contributions

Conception and design: Goyal, McCutcheon. Acquisition of data: Goyal, Kerezoudis, McCutcheon. Analysis and interpretation of data: Ngufor. Drafting the article: Goyal. Critically revising the article: Goyal, Ngufor, Kerezoudis, McCutcheon, Storlie. Reviewed submitted version of manuscript: all authors. Study supervision: Bydon.

Supplemental Information

Online-Only Content

Supplemental material is available with the online version of the article.

References

  • 1

    Adogwa O, Elsamadicy AA, Han JL, Karikari IO, Cheng J, Bagley CA: 30-day readmission after spine surgery: an analysis of 1400 consecutive spine surgery patients. Spine (Phila Pa 1976) 42:520524, 2017

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 2

    Akbari H, Macyszyn L, Da X, Bilello M, Wolf RL, Martinez-Lage M, et al.: Imaging surrogates of infiltration obtained via multiparametric imaging pattern analysis predict subsequent location of recurrence of glioblastoma. Neurosurgery 78:572580, 2016

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 3

    Aldebeyan S, Aoude A, Fortin M, Nooh A, Jarzem P, Ouellet J, et al.: Predictors of discharge destination after lumbar spine fusion surgery. Spine (Phila Pa 1976) 41:15351541, 2016

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 4

    Ambler G, Omar RZ, Royston P: A comparison of imputation techniques for handling missing predictor values in a risk model with a binary outcome. Stat Methods Med Res 16:277298, 2007

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 5

    American College of Surgeons National Surgical Quality Improvement: User Guide for the 2014 ACS NSQIP Participant Use Data File (PUF). Chicago: American College of Surgeons, 2015 (https://www.facs.org/∼/media/files/quality%20programs/nsqip/nsqip_puf_userguide_2014.ashx) [Accessed April 8, 2019]

    • PubMed
    • Search Google Scholar
    • Export Citation
  • 6

    Appaduray SP, Lo P: Effects of diabetes and smoking on lumbar spinal surgery outcomes. J Clin Neurosci 20:17131717, 2013

  • 7

    Asadi H, Kok HK, Looby S, Brennan P, O’Hare A, Thornton J: Outcomes and complications after endovascular treatment of brain arteriovenous malformations: a prognostication attempt using artificial intelligence. World Neurosurg 96:562569, 569.e1, 2016

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 8

    Azimi P, Benzel EC, Shahzadi S, Azhari S, Mohammadi HR: The prediction of successful surgery outcome in lumbar disc herniation based on artificial neural networks. J Neurosurg Sci 60:173177, 2016

    • PubMed
    • Search Google Scholar
    • Export Citation
  • 9

    Azimi P, Benzel EC, Shahzadi S, Azhari S, Mohammadi HR: Use of artificial neural networks to predict surgical satisfaction in patients with lumbar spinal canal stenosis: clinical article. J Neurosurg Spine 20:300305, 2014

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 10

    Azimi P, Mohammadi HR, Benzel EC, Shahzadi S, Azhari S: Use of artificial neural networks to predict recurrent lumbar disk herniation. J Spinal Disord Tech 28:E161E165, 2015

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 11

    Bernatz JT, Anderson PA: Thirty-day readmission rates in spine surgery: systematic review and meta-analysis. Neurosurg Focus 39(4):E7, 2015

  • 12

    Bernhardt BC, Hong SJ, Bernasconi A, Bernasconi N: Magnetic resonance imaging pattern learning in temporal lobe epilepsy: classification and prognostics. Ann Neurol 77:436446, 2015

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 13

    Bilimoria KY, Liu Y, Paruch JL, Zhou L, Kmiecik TE, Ko CY, et al.: Development and evaluation of the universal ACS NSQIP surgical risk calculator: a decision aid and informed consent tool for patients and surgeons. J Am Coll Surg 217:833842, 842.e1–842.e3, 2013

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 14

    Di Capua J, Somani S, Kim JS, Phan K, Lee NJ, Kothari P, et al.: Analysis of risk factors for major complications following elective posterior lumbar fusion. Spine (Phila Pa 1976) 42:13471354, 2017

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 15

    Durand WM, DePasse JM, Daniels AH: Predictive modeling for blood transfusion after adult spinal deformity surgery: a tree-based machine learning approach. Spine (Phila Pa 1976) 43:10581066, 2018

    • PubMed
    • Search Google Scholar
    • Export Citation
  • 16

    Emblem KE, Pinho MC, Zöllner FG, Due-Tonnessen P, Hald JK, Schad LR, et al.: A generic support vector machine model for preoperative glioma survival associations. Radiology 275:228234, 2015

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 17

    Etzioni DA, Lessow CL, Lucas HD, Merchea A, Madura JA, Mahabir R, et al.: Infectious surgical complications are not dichotomous: characterizing discordance between administrative data and registry data. Ann Surg 267:8187, 2018

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 18

    Fingar KR, Stocks C, Weiss AJ, Steiner CA: Most Frequent Operating Room Procedures Performed in U.S. Hospitals, 2003–2012. Statistical Brief 186. Rockville, MD: Agency for Healthcare Research and Quality, Healthcare Cost and Utilization Project, 2014

    • PubMed
    • Search Google Scholar
    • Export Citation
  • 19

    Garcia RM, Khanna R, Dahdaleh NS, Cybulski G, Lam S, Smith ZA: Thirty-day readmission risk factors following single-level transforaminal lumbar interbody fusion (TLIF) for 4992 patients from the ACS-NSQIP database. Global Spine J 7:220226, 2017

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 20

    Goz V, Rane A, Abtahi AM, Lawrence BD, Brodke DS, Spiker WR: Geographic variations in the cost of spine surgery. Spine (Phila Pa 1976) 40:13801389, 2015

  • 21

    Goz V, Weinreb JH, McCarthy I, Schwab F, Lafage V, Errico TJ: Perioperative complications and mortality after spinal fusions: analysis of trends and risk factors. Spine (Phila Pa 1976) 38:19701976, 2013

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 22

    Hoffman H, Lee SI, Garst JH, Lu DS, Li CH, Nagasawa DT, et al.: Use of multivariate linear regression and support vector regression to predict functional outcome after surgery for cervical spondylotic myelopathy. J Clin Neurosci 22:14441449, 2015

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 23

    Kalakoti P, Missios S, Maiti T, Konar S, Bir S, Bollam P, et al.: Inpatient outcomes and postoperative complications after primary versus revision lumbar spinal fusion surgeries for degenerative lumbar disc disease: a National (Nationwide) Inpatient Sample analysis, 2002–2011. World Neurosurg 85:114124, 2016

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 24

    Kansagara D, Englander H, Salanitro A, Kagen D, Theobald C, Freeman M, et al.: Risk prediction models for hospital readmission: a systematic review. JAMA 306:16881698, 2011

  • 25

    Kerezoudis P, McCutcheon B, Murphy ME, Rajjoub KR, Ubl D, Habermann EB, et al.: Thirty-day postoperative morbidity and mortality after temporal lobectomy for medically refractory epilepsy. J Neurosurg 128:11581164, 2018

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 26

    Kim JS, Merrill RK, Arvind V, Kaji D, Pasik SD, Nwachukwu CC, et al.: Examining the ability of artificial neural networks machine learning models to accurately predict complications following posterior lumbar spine fusion. Spine (Phila Pa 1976) 43:853860, 2018

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 27

    Krell MM, Wilshusen N, Seeland A, Kim SK: Classifier transfer with data selection strategies for online support vector machine classification with class imbalance. J Neural Eng 14:025003, 2017

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 28

    Kuo CY, Yu LC, Chen HC, Chan CL: Comparison of models for the prediction of medical costs of spinal fusion in Taiwan diagnosis-related groups by machine learning algorithms. Healthc Inform Res 24:2937, 2018

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 29

    Lawson EH, Louie R, Zingmond DS, Brook RH, Hall BL, Han L, et al.: A comparison of clinical registry versus administrative claims data for reporting of 30-day surgical complications. Ann Surg 256:973981, 2012

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 30

    Lorenzi EC, Brown SL, Huang ES, Sun Z, Heller K: Prediction via clusters of CPT codes for improving surgical outcomes. arXiv (https://arxiv.org/abs/1604.07031) [Accessed April 8, 2019]

    • PubMed
    • Search Google Scholar
    • Export Citation
  • 31

    McCutcheon BA, Kerezoudis P, Porter AL, Rinaldo L, Murphy M, Maloney P, et al.: Coma and stroke following surgical treatment of unruptured intracranial aneurysm: an American College of Surgeons National Surgical Quality Improvement Program Study. World Neurosurg 91:272278, 2016

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 32

    McGirt MJ, Parker SL, Chotai S, Pfortmiller D, Sorenson JM, Foley K, et al.: Predictors of extended length of stay, discharge to inpatient rehab, and hospital readmission following elective lumbar spine surgery: introduction of the Carolina-Semmes Grading Scale. J Neurosurg Spine 27:382390, 2017

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 33

    McGirt MJ, Sivaganesan A, Asher AL, Devin CJ: Prediction model for outcome after low-back surgery: individualized likelihood of complication, hospital readmission, return to work, and 12-month improvement in functional disability. Neurosurg Focus 39(6):E13, 2015

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 34

    Memarian N, Kim S, Dewar S, Engel J Jr, Staba RJ: Multimodal data and machine learning for surgery outcome prediction in complicated cases of mesial temporal lobe epilepsy. Comput Biol Med 64:6778, 2015

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 35

    Munsell BC, Wee CY, Keller SS, Weber B, Elger C, da Silva LAT, et al.: Evaluation of machine learning algorithms for treatment outcome prediction in patients with epilepsy based on structural connectome data. Neuroimage 118:219230, 2015

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 36

    Murphy ME, Maloney PR, McCutcheon BA, Rinaldo L, Shepherd D, Kerezoudis P, et al.: Predictors of discharge to a nonhome facility in patients undergoing lumbar decompression without fusion for degenerative spine disease. Neurosurgery 81:638649, 2017

    • PubMed
    • Search Google Scholar
    • Export Citation
  • 37

    Oermann EK, Rubinsteyn A, Ding D, Mascitelli J, Starke RM, Bederson JB, et al.: Using a machine learning approach to predict outcomes after radiosurgery for cerebral arteriovenous malformations. Sci Rep 6:21161, 2016

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 38

    Oh T, Scheer JK, Smith JS, Hostin R, Robinson C, Gum JL, et al.: Potential of predictive computer models for preoperative patient selection to enhance overall quality-adjusted life years gained at 2-year follow-up: a simulation in 234 patients with adult spinal deformity. Neurosurg Focus 43(6):E2, 2017

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 39

    Passias PG, Vasquez-Montes D, Poorman GW, Protopsaltis T, Horn SR, Bortz CA, et al.: Predictive model for distal junctional kyphosis after cervical deformity surgery. Spine J 18:21872194, 2018

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 40

    Ratliff JK, Balise R, Veeravagu A, Cole TS, Cheng I, Olshen RA, et al.: Predicting occurrence of spine surgery complications using “big data” modeling of an administrative claims database. J Bone Joint Surg Am 98:824834, 2016

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 41

    Sebastian A, Huddleston P III, Kakar S, Habermann E, Wagie A, Nassr A: Risk factors for surgical site infection after posterior cervical spine surgery: an analysis of 5,441 patients from the ACS NSQIP 2005–2012. Spine J 16:504509, 2016

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 42

    Senders JT, Staples PC, Karhade AV, Zaki MM, Gormley WB, Broekman MLD, et al.: Machine learning and neurosurgical outcome prediction: a systematic review. World Neurosurg 109:476486, 486.e1, 2018

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 43

    Shiloach M, Frencher SK Jr, Steeger JE, Rowell KS, Bartzokis K, Tomeh MG, et al.: Toward robust information: data quality and inter-rater reliability in the American College of Surgeons National Surgical Quality Improvement Program. J Am Coll Surg 210:616, 2010

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 44

    Stekhoven DJ, Bühlmann P: MissForest—non-parametric missing value imputation for mixed-type data. Bioinformatics 28:112118, 2012

  • 45

    Watad A, Bragazzi NL, Bacigaluppi S, Amital H, Watad S, Sharif K, et al.: Artificial neural networks can be effectively used to model changes of intracranial pressure (ICP) during spinal surgery using different non invasive ICP surrogate estimators. J Neurosurg Sci [epub ahead of print], 2018

    • PubMed
    • Search Google Scholar
    • Export Citation
  • 46

    Weiss AJ, Elixhauser A: Trends in Operating Room Procedures in US Hospitals, 2001–2011. HCUP Statistical Brief 171. Rockville, MD: Agency for Healthcare Research and Quality, Healthcare Cost and Utilization Project, 2014 (https://www.hcup-us.ahrq.gov/reports/statbriefs/sb171-Operating-Room-Procedure-Trends.jsp) [Accessed April 8, 2019]

    • PubMed
    • Search Google Scholar
    • Export Citation
  • 47

    Yankam Njiwa J, Gray KR, Costes N, Mauguiere F, Ryvlin P, Hammers A: Advanced [18F]FDG and [11C]flumazenil PET analysis for individual outcome prediction after temporal lobe epilepsy surgery for hippocampal sclerosis. Neuroimage Clin 7:122131, 2014

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • Collapse
  • Expand

Metrics

All Time Past Year Past 30 Days
Abstract Views 3800 337 0
Full Text Views 1216 296 97
PDF Downloads 857 189 36
EPUB Downloads 0 0 0