Registry Methodology

Every consecutive patient undergoing elective lumbar spine surgery over a 4-year period at our center was enrolled into a prospective, Web-based, single-center spine registry, regardless of diagnosis or specific surgical treatment. Registry data were gathered at preoperative, 3-month postoperative, and 12-month postoperative clinic visits. Electronic medical record (EMR) review was used to collect variables such as patient comorbidities, details of the individual surgery, and the perioperative course. Perioperative morbidity, the need for inpatient rehabilitation after surgery, and mortality were assessed via EMR review and confirmed on patient interviews. Questionnaires were administered via one-on-one patient interviews to collect baseline and postoperative patientreported outcomes. They included measures of pain (visual analog scales [VASs]), and disease-specific physical disability (the Oswestry Disability Index [ODI]). In addition, the mental component of quality of life and somatic perception and anxiety were assessed. By virtue of an institutional review board waiver, verbal consent during the interview was sufficient for data collection from patients. This waiver was granted because, although the registry does constitute a form of research, the outcome assessments do not alter standard care or treatments delivered and therefore were not deemed a significant risk to patients. All clinical data were prospectively collected and entered into the REDCap portal and stored in a HIPAA (Health Insurance Portability and Accountability Act)-se-cure fashion.

Model Development

Depending on the outcome measure of interest, the sample sizes used for model development ranged from 750 to 1200. This variability stems from the existence of some missing values for the outcomes of interest, which were handled using list-wise deletion (no imputation was performed). These samples were obtained by randomly selecting 80% of the full data to serve as “training” data sets. All baseline input variables were selected a priori based on recognized clinical importance. Linear regression was used to model 12-month ODI after verifying using LOESS fit that a linear relationship with various explanatory variables was reasonable. Because our goal was to optimize prediction while properly accommodating model uncertainty, the linear regression was fitted using Bayesian model averaging.

To model the other 5 categorical outcome variables (complications, readmission, inpatient rehabilitation, return to work, and a composite measure of unplanned outcome), multiple logistic regression with Bayesian model averaging was used. The goodness-of-fit for the ODI model was assessed using R² (explained variation). For the models of categorical endpoints mentioned above, the c-statistic, or the area under the receiver operating characteristic (ROC) curve (AUC), was used. Multi-collinearity is always a concern when multiple regression is performed, however it is less so here because our primary goal is to make predictions on new data (as opposed to understanding the influence of any particular covariate). Multi-collinearity does not generally affect the efficacy of extrapolating a given model to external data.

Model Validation

For each outcome variable, the 20% of the full data that remained after random selection of the training data set was used as the validation data set. Our use of 80% of the data for model development and 20% for model validation is a standard practice in predictive modeling efforts. Each model was used to predict values for its corresponding outcome, using the validation data set, and those predictions were compared with the true observed values. Discrimination, which is one aspect of predictive performance, was assessed using R² for the ODI model and using the c-statistic for the categorical endpoint models. Calibration, another aspect of performance, was assessed using a plot of predicted versus observed values for the ODI model and using the Hosmer-Lemeshow test for the categorical endpoint models. All statistical analyses were performed using R, with a significance level of 0.05. Of note, the final performance measures were actually averaged over 10 successive rounds of 80/20 partitions of the data. This is known as 10-fold cross validation, and it has the effect of reducing the variability in the performance estimates that are reported here.

Registry Cohort

A total of 1803 patients undergoing lumbar spine surgery were prospectively enrolled at the time of this study. Table 1 summarizes the baseline characteristics for this cohort, many of which serve as explanatory variables in our predictor models, as well as the incidence of major clinical outcomes, such as morbidity, hospital readmission, reoperation, and disability level at 12 months after surgery. The average age of the patients at the time of surgery was 55.92, nearly half were male, nearly 90% were Caucasian, 44% were employed before surgery, 74% could ambulate without assistance, and their average baseline ODI was 50.44%. Within 30 days of surgery, the mortality rate was 0.3%, the reoperation rate was 4.3%, and the readmission rate was 5.9%. Transfer from the hospital to inpatient rehabilitation or a skilled nursing facility was noted in 11.6% of patients.

Overall, the average ODI 1 year after surgery was significantly improved (50.4% vs 29.5%, p < 0.05) and 88.9% of patients returned to work, Fig. 1 displays a Kaplan-Meier curve for the number of days until return to work. Significant (p < 0.05) improvements in all PROs were reported 12 months after surgery for all outcome measures: back pain (12-month VAS: 3.5 ± 3.2), leg pain (12-month VAS: 2.8 ± 3.4), disability (12-month ODI: 29.6 ± 20.1) and EQ-5D (12-month QALY [quality-adjusted life year]: 0.7 ± 0.2). Nevertheless, at the individual patient level, wide variation was seen in PROs at 12 months for all diagnoses captured in the registry, as depicted in Fig. 2. Fourteen percent of patients did not show any improvement in disability (ODI) and 24% did not achieve a minimum clinically important difference (MCID: 15% improvement).¹⁴ Moreover, 449 patients (24.5%) experienced an unplanned outcome after surgery. For purposes of this model development, unplanned outcome was defined as a surgical complication, readmission to the hospital, or lack of any improvement in ODI by 12 months after surgery.

Kaplan-Meier curve for days until return to work. The x-axis on this plot represents the number of days until return to work. The y-axis represents the proportion of patients, among those who were employed before surgery, who had returned to work. Three-quarters of those patients returned to work within a month, and no patients returned to work more than 100 days after surgery.

Variability in patientreported outcomes (PROs). The top 3 plots show mean values of ODI, SF-12, and EQ-5D (PROs) at baseline, 3 months, and 12 months postoperatively. There is clear improvement in the mean values over time. However, the bottom 3 plots show the individual patient-level variability in PRO change from baseline to 12 months. Each point represents 1 patient- points below the diagonal line signify improvements in ODI and worsening of SF-12 and EQ-5D. Green points represent patients who achieved the MCID, and red points represent those who did not.

Model Development

For each outcome variable, Table 2 lists the beta coefficients and goodness-of-fit measures for the model development step. Because Bayesian model averaging was employed, the coefficients were weighted by the significance level of their corresponding variables. However, all variables (not just those that are significant) contribute to the overall fit and predictions—the omission of any one variable from the models would reduce predictive performance. Certain predictor variables were selected a priori for mandatory inclusion in the models based on a belief (based on clinical experience) that they influence the outcomes of interest. Using Table 2, personalized predictions for any outcome can be computed by multiplying a patient’s unique value for each explanatory variable by its corresponding coefficient, summing those terms, and then adding the intercept (if one exists). For each categorical variable (all of which are binary), a value of “1” is chosen if patients fit into the category, and a value of “0” is chosen if they do not. The result of this calculation will yield a 12-month ODI score or a predicted probability (categorical outcomes) of complication, hospital readmission, return to work, or a composite unplanned outcome. Please contact the authors for more information on these calculations. Figure 2 graphically represents predictions for 2 hypothetical patients, based on these calculations.

The R² goodness of fit of the model was 0.51, suggesting that over half of the observed variance could be explained by the model. No nonrandom patterns were seen in the residual plot for this model. The AUC values ranged 0.72 to 0.84 for complication, readmission, inpatient rehabilitation, and return to work. The AUC for an unplanned postoperative outcome was 0.82.

Model Validation

Table 2 also provides performance measures for the model validation step. An R² of 0.47 was achieved for 12-month ODI. The ROC characteristics were even higher in the validation study for complications (0.82), readmission (0.79), need for inpatient rehabilitation (0.84), return to work (0.83), and unplanned outcome (0.78). Logistic regression for each of the 5 categorical outcomes satisfied the Hosmer-Lemeshow goodness-of-fit test (p > 0.05). Figure 3 shows a calibration plot of predicted versus observed 12-month ODI values as a demonstration of our model performance. The correlation between predictions and observations is 0.72.

Calibration plot. The y-axis represents observed values for 12-month ODI in our registry, while the x-axis represents the corresponding predicted values for 12-month ODI based on our regression models. The dotted line has a 45° slope through the origin and is where vertical and horizontal values are equal.

Robust Predictive Models

It has been established that considerable patient-level variability exists in clinical outcomes after elective spine surgery. In light of this, we have introduced models which use a large array of baseline patient characteristics to predict ODI at 12 months, postoperative complications, hospital readmission, need for inpatient rehabilitation, and return to work after surgery. Linear regression and Bayesian model averaging were used to develop the models, based on data from a single center spine registry. The models explain approximately one-half of the variation in 12-month ODI and achieve diagnostic accuracy ranging from 72%- 84% for surgical morbidity and return to work. Using standard measures of discrimination and calibration, we have affirmed the predictive performance of the models. They are a significant advancement relative to models published by Lee and colleagues, which have lower ROC characteristics and only predict 2 outcomes—surgical-site infections and medical complications.^7,8 Our models’ performance measures are similar, however, to that of recently published American College of Surgeons National Surgical Quality Improvement Program (ACS NSQIP) models for complications after spine surgery. The ACS NSQIP models are generated from a much larger national data set, but they do not address any patientreported outcomes.

Our decision to employ Bayesian model averaging stems from the fact that traditional modeling techniques require the selection of one model from a set of models that often have similar levels of goodness-of-fit. There is a layer of statistical uncertainty inherent in that selection that is routinely ignored. Bayesian model averaging addresses this uncertainty by creating weighted-average predictions over a whole set of possible models, which has the potential to improve overall predictive performance.

To illustrate how models such as ours could one day be applied in a clinical setting, Fig. 4 presents the predicted outcomes for 2 hypothetical patients whom a neurosurgeon might see in clinic. The patients have differing baseline characteristics, as see in the panel on the left, and the panel on the right shows that they have divergent predictions for every outcome of interest. A Web- or phone-based application that generates graphical predictions such as this, after querying patients about preoperative factors in the clinic waiting room, would allow neurosurgeons and any other spine care providers to create comprehensive risk profiles that are tailored to specific patients. These equations can easily be applied to a calculator or Web-based tool for application in practice. In reality, decisions should not be made solely on evidence-based historical outcomes in similar or matched patients as this model/tool provides, but rather based on the combination of experience, real-time interpersonal judgment, and decision support tools like this. It can draw the surgeon’s attention to outliers of particular risk they may have gone unappreciated in everyday practice. It is meant to supplement the knowledge of surgeon and patient in the decision-making process at the individualized level.

Divergent predictions for 2 hypothetical cases. Here we present 2 hypothetical patients with differing baseline characteristics, as seen in the panel on the left. The panel on the right shows that the 2 patients have divergent predictions of 12-month ODI as well as the probabilities of complication, readmission, need for inpatient rehabilitation, return to work, and unplanned outcome based on our regression models. Predictions such as these can help surgeons understand benefits and risks of surgery that are specific to an individual patient.

Influence of Preoperative Factors

In addition to generating predictions of clinical outcomes after spine surgery, our modeling efforts also reveal which baseline factors are most influential in relation to those outcomes. A higher preoperative ODI, a longer duration of narcotic use, hypertension, atrial fibrillation, more severe extremity pain, and depression/anxiety are significantly associated with a higher 12-month ODI (less effective care). In contrast, preoperative employment, ability to walk, acute back or leg pain, better general health (SF-12 PCS), a diagnosis of disc herniation, and the absence of prior spine surgery, are associated with a lower 12-month ODI (more effective care). These findings correlate with clinical intuition—patients who have less pain, fewer co-morbidities, and better functional status at baseline tend to have less disability after surgery. Patients with acute-onset pain may do better than those with chronic symptoms, because the cause of the pain is more likely due to a structural pathology that is successfully correctable with surgery. Those without prior surgeries also tend to do better because revision surgeries can be more technically difficult, have pain generators that are not surgically correctable, and involve more complications.

Our predictive models demonstrate that a history of smoking, atrial fibrillation, a higher ASA (American Society of Anesthesiologists) grade, a history of myocardial infarction, diabetes, or osteoporosis are associated with a higher risk of postoperative complications and adverse events. This is understandable, as higher ASA grades represent physical status that is less “fit for surgery,” and the other comorbidities have previously been demonstrated to raise the risk of intra- and postoperative adverse events. Osteoporosis, in particular, can impede bony stability and fusion after surgery, potentially leading to hardware failure and possible reoperations. Furthermore, we found that the ability to ambulate before surgery, private insurance, and acute versus chronic back or leg pain are associated with a reduced likelihood of the need for inpatient rehabilitation after surgery. In contrast, increasing age, increasing number of operated levels, a history of atrial fibrillation or arthritis, and a diagnosis of spinal deformity are associated with an increased likelihood of being discharged to inpatient rehabilitation. Deformity surgeries can be some of the most complex and extensive procedures performed in hospitals, involving considerable blood loss and longer lengths of stay, which explains the higher probability of inpatient rehabilitation. Finally, for working patients without workers’ compensation claims who are planning on returning to work after surgery, a preoperative diagnosis of depression, arthritis, and prolonged preoperative opioid use significantly reduced the likelihood of returning to work. Patients with a diagnosis of spondylolisthesis or disc herniation are more likely to return to work. These findings are understandable, given that patients with mood difficulties or long-term opioid consumption may have reduced functional status and/or motivation that prevents reemployment.

Limitations of Study

There are limitations to consider when interpreting these results. First, an R² of 0.51 was achieved for 12-month ODI during model development. This means that our model explains only half the variability in 12-month ODI, leaving room for improvement. This is not surprising, given the challenge of modeling a continuous variable like ODI that has 100 possible values, but it highlights the fact that our model is not yet complete. Inclusion of additional predictor variables in our registry, as well as consideration of nonlinear regression strategies, will likely be necessary to increase predictive performance. The model in its current form, however, is significant in that it represents the first known attempt to predict disability 1 year after elective lumbar surgery, and in doing so provides a set of preoperative factors that are influential in the outcome.

Another limitation is that while our models perform well at predicting observed categorical outcomes when applied to an internal validation data set, their performance remains untested in an external data set from a different medical center. The models presented here are built solely on data from a single-institution registry—one that reflects a particular patient population and a particular set of surgeons and techniques. Therefore, it is unclear how applicable our models may be outside of the Vanderbilt University Medical Center, and a multi-institutional prospective study is needed to investigate this. It is quite possible a patient at another medical center, with a similar risk profile to a patient in the Vanderbilt registry, may have outcomes that differ from the predictions generated by our models. However, we expect that the relative position of the Vanderbilt patient will likely be equivalent to the relative position of the external patient when compared with their respective cohorts. In other words, if a Vanderbilt patient is predicted to be in the top quantile of outcomes in our registry, a similar patient elsewhere will likely also be in that same quantile. In this manner, our models should provide some insight into general risk stratification of patients outside Vanderbilt.

Because large longitudinal data sets that include the breadth and depth of patientreported physical and mental status such as this are rare, it may be difficult to effectively test these models in currently existing registries or research data sets. Additionally, the data inputs that power these models are fairly involved and require the focused attention of the patient for at least 20 minutes with 45 unique baseline variable inputs derived from 39 clinical variables and 38 questionnaire items (ODI, SF-12, MSPQ, VAS-BP, VAS-LP, VAS-NP). Therefore, reproducing the predictive accuracy reported here may be too time intensive and cost prohibitive. The 39 basic clinical variables, such as age, demographics, comorbidities, diagnosis, extent of surgery, and others, can be derived from the electronic medical record. However, the ODI, SF-12, MSPQ, and pain VAS scores are patientreported data inputs that need to be collected via 38 questionnaire items. Future efforts to create “abridged” models that achieve similar predictive performance while requiring less time-intensive data entry will be important. However, it is important to establish the potential that truly granular modeling can have in forecasting long-term outcomes.