A traumatic brain injury prognostic model to support in-hospital triage in a low-income country: a machine learning–based approach

Thiago Augusto Hernandes Rocha Duke Division of Global Neurosurgery and Neurology;

Search for other papers by Thiago Augusto Hernandes Rocha in
Current site
Google Scholar
PubMed
Close
 MSc
,
Cyrus Elahi Duke Division of Global Neurosurgery and Neurology;
Duke University Global Health Institute, Durham, North Carolina;

Search for other papers by Cyrus Elahi in
Current site
Google Scholar
PubMed
Close
 BA
,
Núbia Cristina da Silva Duke University Global Health Institute, Durham, North Carolina;

Search for other papers by Núbia Cristina da Silva in
Current site
Google Scholar
PubMed
Close
 PhD
,
Francis M. Sakita Kilimanjaro Christian Medical Centre, Moshi, Tanzania;

Search for other papers by Francis M. Sakita in
Current site
Google Scholar
PubMed
Close
 MD
,
Anthony Fuller Duke Division of Global Neurosurgery and Neurology;
Duke University Global Health Institute, Durham, North Carolina;
Department of Neurosurgery, Duke University Medical Center;

Search for other papers by Anthony Fuller in
Current site
Google Scholar
PubMed
Close
 MD
,
Blandina T. Mmbaga Duke University Global Health Institute, Durham, North Carolina;
Kilimanjaro Christian Medical Centre, Moshi, Tanzania;
Kilimanjaro Clinical Research Institute, Moshi, Tanzania

Search for other papers by Blandina T. Mmbaga in
Current site
Google Scholar
PubMed
Close
 MD
,
Eric P. Green Duke University Global Health Institute, Durham, North Carolina;

Search for other papers by Eric P. Green in
Current site
Google Scholar
PubMed
Close
 PhD
,
Michael M. Haglund Duke Division of Global Neurosurgery and Neurology;
Duke University Global Health Institute, Durham, North Carolina;
Department of Neurosurgery, Duke University Medical Center;

Search for other papers by Michael M. Haglund in
Current site
Google Scholar
PubMed
Close
 MD, PhD, MACM
,
Catherine A. Staton Duke Division of Global Neurosurgery and Neurology;
Duke University Global Health Institute, Durham, North Carolina;
Division of Emergency Medicine, Duke University Medical Center, Durham, North Carolina; and

Search for other papers by Catherine A. Staton in
Current site
Google Scholar
PubMed
Close
 MD
, and
Joao Ricardo Nickenig Vissoci Duke Division of Global Neurosurgery and Neurology;
Duke University Global Health Institute, Durham, North Carolina;

Search for other papers by Joao Ricardo Nickenig Vissoci in
Current site
Google Scholar
PubMed
Close
 PhD
Full access

OBJECTIVE

Traumatic brain injury (TBI) is a leading cause of death and disability worldwide, with a disproportionate burden of this injury on low- and middle-income countries (LMICs). Limited access to diagnostic technologies and highly skilled providers combined with high patient volumes contributes to poor outcomes in LMICs. Prognostic modeling as a clinical decision support tool, in theory, could optimize the use of existing resources and support timely treatment decisions in LMICs. The objective of this study was to develop a machine learning–based prognostic model using data from Kilimanjaro Christian Medical Centre in Moshi, Tanzania.

METHODS

This study is a secondary analysis of a TBI data registry including 3138 patients. The authors tested nine different machine learning techniques to identify the prognostic model with the greatest area under the receiver operating characteristic curve (AUC). Input data included demographics, vital signs, injury type, and treatment received. The outcome variable was the discharge score on the Glasgow Outcome Scale–Extended.

RESULTS

The AUC for the prognostic models varied from 66.2% (k-nearest neighbors) to 86.5% (Bayesian generalized linear model). An increasing Glasgow Coma Scale score, increasing pulse oximetry values, and undergoing TBI surgery were predictive of a good recovery, while injuries suffered from a motor vehicle crash and increasing age were predictive of a poor recovery.

CONCLUSIONS

The authors developed a TBI prognostic model with a substantial level of accuracy in a low-resource setting. Further research is needed to externally validate the model and test the algorithm as a clinical decision support tool.

ABBREVIATIONS

AUC = area under the receiver operating characteristic curve; ED = emergency department; GCS = Glasgow Coma Scale; GOSE = Glasgow Outcome Scale–Extended; HIC = high-income country; ICU = intensive care unit; KCMC = Kilimanjaro Christian Medical Centre; LMICs = low- and middle-income countries; ML = machine learning; MOI = mechanism of injury; PERRLA = pupils equal, round, and reactive to light and accommodation; TBI = traumatic brain injury.

OBJECTIVE

Traumatic brain injury (TBI) is a leading cause of death and disability worldwide, with a disproportionate burden of this injury on low- and middle-income countries (LMICs). Limited access to diagnostic technologies and highly skilled providers combined with high patient volumes contributes to poor outcomes in LMICs. Prognostic modeling as a clinical decision support tool, in theory, could optimize the use of existing resources and support timely treatment decisions in LMICs. The objective of this study was to develop a machine learning–based prognostic model using data from Kilimanjaro Christian Medical Centre in Moshi, Tanzania.

METHODS

This study is a secondary analysis of a TBI data registry including 3138 patients. The authors tested nine different machine learning techniques to identify the prognostic model with the greatest area under the receiver operating characteristic curve (AUC). Input data included demographics, vital signs, injury type, and treatment received. The outcome variable was the discharge score on the Glasgow Outcome Scale–Extended.

RESULTS

The AUC for the prognostic models varied from 66.2% (k-nearest neighbors) to 86.5% (Bayesian generalized linear model). An increasing Glasgow Coma Scale score, increasing pulse oximetry values, and undergoing TBI surgery were predictive of a good recovery, while injuries suffered from a motor vehicle crash and increasing age were predictive of a poor recovery.

CONCLUSIONS

The authors developed a TBI prognostic model with a substantial level of accuracy in a low-resource setting. Further research is needed to externally validate the model and test the algorithm as a clinical decision support tool.

In Brief

The authors used machine learning to develop an accurate prognostic model for traumatic brain injury. They developed this model using data from a low-income country where there is an immense need for data-driven triage decisions.

Each year, an estimated 69 million people experience a traumatic brain injury (TBI) worldwide, and about 10 million cases result in hospitalization or death.5,20 While timely surgical intervention can significantly reduce morbidity and mortality,9,32 some health systems in low- and middle-income countries (LMICs) do not have the resources to offer comprehensive or timely care. Compared to those in high-income countries (HICs), healthcare facilities in LMICs have almost three times the total TBI volume, lack a sufficient number of trained providers, and have less access to diagnostic technologies.5,28 These challenges contribute to diagnostic and treatment delays, which are known contributors to poor TBI outcomes.16,38 Structural interventions to increase both available diagnostic technologies and hospital capacity can help but represent long-term, resource-intensive solutions. Prognostic modeling as a clinical decision support tool, in theory, could optimize the use of existing resources and support timely treatment decisions in LMICs.

Prognostic models are a type of statistical model that combines two or more input variables of patient information to predict a clinical outcome.24 A prognostic model for TBI could provide an objective estimate of prognosis with readily available information upon presentation to the hospital.36,37,41 Initial management of TBI patients at the hospital is dependent on accurate categorization of injury severity.8 According to a survey of mostly LMIC doctors who routinely treat head-injury patients, only 37% agreed that they currently assess patient prognosis accurately.25 Limited access to diagnostic technologies and trained physicians in LMICs may contribute to a reduced capacity for complex decision-making.

Despite the potential for prognostic models to support TBI clinical decision-making, there remains limited effort to develop these algorithms using data from LMICs.24 The last major systematic review on prognostic models for TBI identified 102 such models.24 Only five of these models included populations from middle-income countries and two included populations from a low-income country.6,12,17,19,26,27 There is a need to develop prognostic models using populations from LMICs because models for HICs may perform badly when extrapolated to poorer settings.36 Another limitation of previous prognostic models is the use of clinical trial data for model development, which could limit external validity.23 An increased prevalence of prospectively collected registry data and greater electronic health record use in LMICs provide an opportunity to develop culturally and contextually relevant prognostic models.

Machine learning (ML), a branch of artificial intelligence, is a powerful approach to model development able to automatically extract meaningful patterns from clinical data that are hard for humans to see.22,33 This technique can reliably predict outcomes from vast numbers of predictors, provide real-time output, and reduce the chance for experts, using more rigid modeling approaches, to introduce bias.29 While previous researchers have applied regression techniques for prognostic model development, to our knowledge, no prognostic models for TBI in low-resource settings have used an ML approach.

The objective of this study was to develop a prognostic model using ML techniques on a TBI registry from a tertiary care hospital in Moshi, Tanzania. It is our hope that the findings from this study will support work to improve prognostication of TBI in LMICs.

Methods

Study Design

We conducted a retrospective analysis of clinical outcome data to determine key predictors of patient prognosis. In designing the analysis, we followed the Guidelines for Developing and Reporting Machine Learning Predictive Models in Biomedical Research as well as the Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD) guideline.3,18

Clinical Setting

The data for this study come from Kilimanjaro Christian Medical Centre (KCMC), a 630-bed, tertiary care hospital serving a population of over 15 million people in Northwest Tanzania (https://www.kcmc.ac.tz). This hospital is the primary point of treatment for TBI in Tanzania. TBI patients arriving at KCMC are evaluated and triaged in the emergency department (ED),35 which is staffed with one residency-trained emergency medicine (EM) physician and general surgery interns. The general surgery interns, residents, and attendings are trained to conduct basic neurosurgical procedures including burr hole procedures and, sparingly, craniotomies and craniectomies. Surgery does not include placement of an intracranial pressure (ICP) monitor. No patients at KCMC undergo ICP monitoring. There are no formally trained neurosurgeons at KCMC.

Initial assessment of injury severity and the risk of a poor outcome is largely based on the healthcare provider’s skill and intuition. The use of resource-intensive diagnostics (e.g., head CT) is limited by cost, which must be paid by a family prior to delivery of a service (e.g., performing the imaging), as well as the availability or functioning of the diagnostic equipment. The decision to transfer a patient from the ED to the intensive care unit (ICU), operating room, or surgical ward is determined by the availability of that resource or by a general surgeon’s or an EM physician’s evaluation. The surgeon’s evaluation, however, often occurs after the patient is admitted.

While critically ill patients can have their definitive treatment expedited, moderately and some severely injured patients are a second priority to critically ill patients. This results in diagnostic and treatment delays for patients with moderate or mild TBIs, which research has shown to increase the chances of poor outcomes.16

Source of Data and Unit of Analysis

This study used a de-identified, prospectively collected registry composed of 3209 patients admitted for TBI at KCMC in Moshi, Tanzania, in the period from 2013 to 2017.35 Patients admitted with TBI had their information collected from a survey, which included information on demographics, vital signs, characterization of injury type, treatment received, and characterization of the outcome observed. The registry does not include posthospitalization information because of limited patient follow-up after discharge.

Prognostic Variables

We initially selected the following prognostic variables: age, sex, mechanism of injury (MOI), intention of injury, presence of alcohol, initial vital signs collected in the ED (temperature, respiratory rate, pulse, systolic blood pressure, diastolic blood pressure, pulse oxygenation), glucose level, pupils’ size and reactiveness (pupils equal, round, and reactive to light and accommodation [PERRLA]), initial Glasgow Coma Scale (GCS) score, surgery for the TBI, and surgery unrelated to TBI. Predictors were selected based on emergency medicine and neurosurgery expert consensus. Additionally, we only considered predictor variables practical for the low-resource setting. “Practical” for this project meant variable collection did not involve resource-intensive equipment, variables were available shortly after patient presentation to the hospital, and low- and high-skilled personnel could collect the information. We did not include CT findings in the model because providers in this setting have inconsistent access to CT scanners.

Outcome Selection

The endpoint for the prognostic model was the Glasgow Outcome Scale–Extended (GOSE) score at patient discharge.13 The discharging doctor or a research nurse calculated the GOSE score for all patients at hospital discharge unless a patient died during the hospitalization. The GOSE is an ordered outcome from 1 (worst outcome, death) to 8 (best outcome, upper good recovery). We dichotomized the GOSE scores into good and poor recovery by classifying scores of 1–6 as poor recovery and scores of 7–8 as good recovery. We chose a dichotomous outcome rather than an ordinal one because, in our clinical setting, there are few patients with moderate outcomes. We selected scores less than 7, rather than less than 5, as a poor recovery to increase the number of patients with poor recovery. This step improved the balance between good and poor recovery in our data set.

Preprocessing of Data for Model Building

All data were processed using the statistical language R (R Foundation for Statistical Computing). The first step was handling missing data. Any input variable for which more than 80% of cases were missing the relevant data were removed from the model. The only variable removed during this first step was the glucose level. Next, we performed multiple imputation using chained equations (MICE) to impute missing values from variables with less than 20% of cases missing.40 We separated each variable considering the measurement level (e.g., numeric, categorical) for different imputation approaches. We considered all the variables for the imputation process. We obtained the resulting data set after 10 iterations.39

A high correlation analysis was performed following the missing imputation steps to identify high correlation variables (i.e., greater than 0.9). No variable was dropped during this step. An analysis to exclude outliers among the numeric variables was performed, and the following cases were dropped: age > 75 years, respiratory rate > 75, and systolic blood pressure > 220 mm Hg.

Several variables considered in the analysis were ordinal or categorical. As a result, the initial group of 21 variables expanded to 56 after dummy variable conversion. To maintain data integrity after this step, we investigated the data for the presence of near zero variance, high correlation, and linear combos. Eighteen variables were removed after the near zero variance test, seven were removed after the analysis of high correlation, and the linear combo analysis did not exclude any variables. Ultimately, we used data from 3138 patients in the database and 31 variables after all preprocessing steps.

Internal Validation Strategy

We used a patient-based cross-validation with 10 folds and five repetitions for data splitting. Since the outcome of interest was imbalanced (14.4% had a poor recovery), we used a regularization for imbalanced procedure called Synthetic Minority Over-sampling Technique (SMOTE).2

Modeling Techniques and Performance Metrics

We tested nine different models: k-nearest neighbors, ridge regression, neural network, bagged tree, Bayesian generalized linear model, Bayesian additive regression trees, gradient boosting machine, single C5.0 ruleset, and random forest.10 All methods were validated using an internal approach. The kappa statistic was the metric used to assess the prognostic models built. We did not have an external data set, so we could not test the precision of the model with unseen data. The metrics for comparison among the models were based on confusion matrix statistics: area under the receiver operating characteristic curve (AUC),7 sensitivity, specificity, positive predictive value, and negative predictive value.1,4,14 All AUC comparative analyses were performed using the R pROC package.30

Institutional Review Board

Our study received ethical clearance from the institutional review boards of KCMC, Moshi, Tanzania, and Duke University.

Results

Of the 3138 patients admitted to the KCMC in the study period, 2685 (85.6%) had a good recovery and 453 (14.4%) had a poor recovery (Table 1). There were 321 (71%) mortality cases in the poor recovery group. Both the good and poor recovery groups were predominantly male (82% and 83%, respectively) with unintentional injuries (81% and 87%). Mean age was also similar, 30.7 and 33.9 years for the good and poor recovery groups, respectively. Seven hundred two (26.1%) patients in the good recovery group and 112 (24.7%) in the poor recovery group reported the use of alcohol at the time of injury.

TABLE 1.

TBI patient profile at KCMC

VariableAll Patients (GOSE scores 1–8)Good Recovery (GOSE scores 7–8)Poor Recovery (GOSE scores 1–6)p Value
No. of patients (%)31382685 (86)453 (14)
Mean age (SD)30.7 (15.2)33.9 (15.7)
Male sex, no. (%)2574 (82)2198 (82)376 (83)0.604
Intention of injury, no. (%)<0.001
 Unintentional2584 (82)2188 (81)396 (87)
 Self-inflicted1 (0)0 (0)1 (0)
 Inflicted by other533 (17)487 (18)46 (10)
 Unknown20 (1)10 (0)10 (2)
Alcohol, no. (%)0.111
 Yes814 (26)702 (26)112 (25)
 No1518 (48)1345 (50)173 (38)
 Unknown806 (26)638 (24)168 (37)
Vital signs, mean (SD)
 Temperature in °C36.5 (0.8)36.4 (0.7)36.5 (1)
 Respiratory rate in breaths per min21.7 (3.8)21.6 (3.6)22.5 (4.7)
 Heart rate in beats per min87.8 (18)87.3 (16.8)91.2 (23.6)
 Systolic BP in mm Hg121.3 (20.4)121.5 (19.4)119.7 (25.5)
 Pulse oximetry in %95.5 (7.3)96.4 (5)89.9 (13.7)
Total GCS score, mean (SD)13.2 (3.2)14 (2.2)8.7 (4.4)
Pupils, no. (%)<0.001
 Bilat reactive2959 (94)2641 (98)318 (70)
 Unilat reactive72 (2)19 (1)53 (12)
 Nonreactive85 (3)12 (0)73 (16)
 Unknown22 (1)13 (0)9 (2)
PERRLA, no. (%)<0.001
 Yes901 (29)787 (29)114 (25)
 No22 (1)8 (0)14 (3)
 Unknown2215 (71)1890 (70)325 (72)
Disposition from ED, no. (%)<0.001
 ICU92 (3)28 (1)64 (14)
 Surgery2684 (86)2322 (86)362 (80)
 Operating theater16 (1)13 (0)3 (1)
 Home323 (10)321 (12)2 (0)
 Death23 (1)1 (0)22 (5)
TBI surgery, no. (%)<0.001
 Yes700 (22)551 (21)149 (33)
 No2107 (67)1813 (68)294 (65)
 Unknown331 (11)321 (12)10 (2)
Other surgery, no. (%)377 (12)312 (12)65 (14)0.439
Surgical ward to ICU, no. (%)796 (25)554 (21)242 (53)<0.001

BP = blood pressure.

Vital signs between the good and poor recovery groups were not significantly different. However, this was not the case for the clinical examination variables. The average GCS score for the poor recovery group was 8.7, whereas the good recovery group had a mean score of 14. In the good recovery group, 2641 (98.4%) patients had bilaterally reactive pupils compared to 318 (70.2%) in the poor recovery group. In the good recovery group, 554 (21%) patients were transferred from the surgical ward to the ICU compared to 242 (53%) in the poor recovery group. One hundred forty-nine (33%) patients in the poor recovery group underwent TBI surgery compared to 551 (21%) in the good recovery group.

We tested nine different ML models on the data set (Table 2). The AUC for the models varied from 66.2% (k-nearest neighbors) to 86.5% (Bayesian generalized linear model; Fig. 1). Despite this range, several models obtained similar AUC values. A confidence interval analysis of AUCs identified the Bayesian generalized linear model as the best approach to categorize the outcomes. For TBI patients receiving care at KCMC, this model produces a 95% CI 85.6–87.4 and an AUC of 86.5%.

TABLE 2.

Machine learning models and the AUC

MethodAUC (%)95% CI, Lower Limit95% CI, Upper Limit
Bayesian generalized linear model86.585.687.4
Random forest84.984.685.3
Ridge regression84.884.585.3
Gradient boosting machine85.184.985.3
Bayesian additive regression trees84.584.384.8
Bagged tree83.682.784.6
Single C5.0 ruleset79.878.880.9
Neural network78.877.880.0
K-nearest neighbors66.266.165.5
FIG. 1.
FIG. 1.

Comparison of ML models applied to predict TBI outcome. Each line represents the receiver operating characteristic (ROC) curve for a different ML technique. The Bayesian generalized linear model (Bay. Gen. Lin. Mod) was the top performing technique. Grad. boost. machine = gradient boosting machine; Bay. Add. Reg. Tr. = Bayesian additive regression trees. Figure is available in color online only.

The Bayesian generalized linear model had a sensitivity of 0.89 and a specificity of 0.67. The precision was 0.94. The accuracy of the model was 0.86 and the kappa statistic was 0.49. The best model achieved a moderate to high capability of predicting a good recovery and an intermediate ability of predicting a bad recovery.

We also extracted the predictive weight of each variable in the Bayesian generalized linear model (Fig. 2). The top four predictors of a good outcome were an increasing GCS score, an increasing blood oxygen level, an increasing systolic blood pressure, and an unintentional injury. The top four predictors of a poor outcome were no TBI surgery, a domestic MOI, a gun MOI, and a car MOI.

FIG. 2.
FIG. 2.

Bar chart for the extracted beta coefficients (Coeff) from the top performing prognostic model, the Bayesian generalized linear model. Orange lines represent variables associated with a poor recovery, whereas blue lines represent variables associated with a good recovery. BP = blood pressure; intercept = value in regression model, included for the sake of transparency. Figure is available in color online only.

Discussion

To our knowledge, this is the first TBI prognostic model using ML in Sub-Saharan Africa. We tested nine different ML techniques. The Bayesian generalized linear modeling approach provided the best performance with an AUC of 86.5% (95% CI 85.6–87.4). By using a generalized linear model, we were able to extract the predictive potential of the model’s variables. Increasing GCS score, increasing pulse oximetry values, and undergoing TBI surgery were predictive of a good recovery while injuries suffered from a motor vehicle crash and increasing age were predictive of a poor recovery. This study applied ML in a low-resource setting, produced a transparent and readable model, and represents a major step toward developing clinical decision support technology to augment inpatient TBI care in LMICs.

Model Performance

This study used ML to produce a TBI prognostic model that can identify upon hospital admission a patient’s risk for poor inpatient recovery. The importance of successfully applying ML to TBI prognostic modeling is two-fold. First, data constraints have limited prognostic models utilizing ML to predominantly high-income and upper middle-income countries. Our access to the TBI patient registry collected by Staton et al. provided a sufficiently robust data set to apply ML techniques.35 Second, previous studies have found ML to outperform traditional statistical methodologies and, at times, even clinical experts.34 The best performing model in this study had an AUC of 86.5% (95% CI 85.6–87.4). In comparison, the only ML-based TBI prognostic model identified in Senders and colleagues’33 review had an AUC of 85.5%.34 Their model used retrospectively collected data to predict inpatient mortality as compared to our model, which used prospectively collected data to predict poor recovery.

Black Box Models

Physician buy-in is paramount for technology adoption. In a survey about prognostic models, most physicians (67%) reported that a more accurate prognostic model would change their current patient management.25 We can encourage physician acceptance of prognostic modeling by maintaining transparency when possible. Some ML approaches, however, generate “black box models,” meaning that the modeling techniques used to build relationships among the examined variables are not easily interpretable. A black box model is likely to be met with skepticism from clinicians, slowing clinical adoption.11 Models based on neural networks, deep learning, and random forest, for example, create intermediary steps hard for the end user to understand.

The Bayesian generalized linear model, the model with the best performance in this study, is not considered a black box model since the weight of each model predictor can be extracted (Fig. 2). The GCS total score, oximetry, and knife, fist, and foot MOIs, as well as the availability of surgery, were the main predictors of a good recovery for TBI patients admitted to KCMC. Our findings are aligned with the literature in terms of the importance of the GCS score, vital signs, and characterization of injury.15,31

Clinical Application and Adoption

Machine learning–based prognostic models have the potential to increase the efficiency and precision of decision-making regarding neurosurgical conditions33 including TBI. In a survey of HIC and LMIC physicians who treat patients with head injury, accurate prognostic information was considered very important for the following: deciding to undertake a decompressive craniotomy, deciding who should receive intensive care, communicating with patient families, and deciding which patient treatment should be withdrawn.25 A previous study exploring how a prediction system influences the intensity of management based on injury severity supports the potential for a more rational use of hospital resources.21 The researchers found more intensive management for patients with a moderate or good prognosis. Conversely, for those with a worse prognosis, the frequency of using resources for intensive management decreased by 39%. In resource-poor settings, the prediction of a poor outcome may result in a more prudent allocation of limited resources. Conversely, the prediction of a good outcome may increase the resources dedicated to a patient, which can be problematic in a setting with few resources.

An example application for a prognostic model in the hospital in this study is the decision to send a patient to the ICU (intensive management) or surgical ward (less intensive management). We found that 53% of the patients who were ultimately classified as having a poor outcome at discharge had been transferred from the surgical ward to the ICU prior to discharge as compared to only 21% of the patients classified as having a good outcome at discharge. This finding means that patients were assessed in the ED, deemed sufficiently stable for care on the surgical ward, and later required transfer to the ICU for more aggressive care. The significant difference between the good and poor recovery groups in requiring this transfer represents an opportunity to improve initial triaging decisions in the ED. This example reveals a potential application for prognostic models to support complex clinical decision-making in hospital settings with a limited number of highly skilled healthcare providers.

Next Steps

Our objectives moving forward include a feasibility study to assess implementation of the model in a low-resource setting and external validation of our model. For the former, we developed a user-friendly website (Fig. 3A) and mobile-based application (Fig. 3B) to access the individual risk scores of patients and their probability of a poor outcome. The end user enters the patient information on the left pane, and the tool provides the patient risk for a poor outcome with or without a certain intervention. We will assess the usability and acceptability of this decision support tool in Sub-Saharan Africa. We expect this feasibility work to address a neglected but necessary step to translate prognostic models from research to practice.

FIG. 3.
FIG. 3.

The web-based application (A) and the mobile-based application (B). Figure is available in color online only

We will also externally validate and augment model performance by incorporating data from other LMICs. The aggregation of data from different LMICs will enrich the training data set for our model and increase the model’s capacity to handle different clinical scenarios. Additionally, we are interested in testing the performance of our model against the CRASH (corticosteroid randomisation after significant head injury) and IMPACT (International Mission for Prognosis and Analysis of Clinical Trials in TBI) prognostic models.23,36

Study Strengths and Weaknesses

A major strength of this study was the quality of data used to develop the prognostic model. The registry data were prospectively collected, had few missing values, and included over 3000 patients. To our knowledge, this is the largest data set used for a TBI prognostic model that did not include clinical trial data. When values were missing, they were statistically estimated using MICE. Perel and colleagues’ systematic review on TBI prognostic models found that three-quarters of the models included fewer than 500 patients and that less than 15% of the models handled missing data statistically.24

This study did have some limitations. First, there was a low volume of patients with a poor recovery in the data set (14.4%). Our use of the SMOTE technique to handle the imbalance during the training phase of the models was intended to overcome this limitation. This imbalance was also the reason for classifying GOSE scores 1–6 as poor recovery. Previous work has used GOSE scores 1–4 for poor recovery.23,36 The incorporation of more data and more patients with poor recovery should help to increase the specificity levels achieved by our models. Next, the generalizability of our results is also limited because of our inability to externally validate our model. The presented model has the potential to be suitable for other contexts, but more data from different sources are necessary to strengthen the potential revealed here. Finally, we included the receipt of TBI surgery as a prognostic variable. The decision to operate is not a neutral decision and thus may introduce bias into our model. Also, we do not know if the patients undergoing TBI surgery and having a good outcome would have had a good outcome without surgical intervention. Despite these limitations, we included this variable in order to estimate patient prognosis with or without surgical intervention.

Conclusions

Machine learning is increasingly used in neurosurgical research, producing excellent results in predicting clinical outcomes.33 This study contributes an application of ML in a novel setting, i.e., inpatient TBI care in a low-resource setting. In LMICs, TBI is a leading cause of death and disability. The use of support systems based on ML techniques can help professionals make more rational clinical decisions. The requisite next step is to test the acceptability, feasibility, and impact of these technologies on patient care. The model developed in this study is an early effort to unlock the potential of decision support technologies to improve the quality of emergency and neurosurgical care.

Acknowledgments

Thank you to the entire Duke Division of Global Neurosurgery and Neurology group, Duke University Global Health Institute, and KCMC for their support and guidance in shaping this manuscript.

Disclosures

Dr. Staton acknowledges salary support funding from the Fogarty International Center (K01, TW010000-01A1), Mr. Rocha acknowledges scholarship support funding from the CAPES foundation (88881.131508/2016-01), and Dr. Haglund acknowledges support from NuVasive and LifeNet for non–study-related clinical or research effort.

Author Contributions

Conception and design: all authors. Acquisition of data: Mmbaga, Staton, Nickenig Vissoci. Analysis and interpretation of data: Elahi, Rocha, Sakita, Mmbaga, Green, Nickenig Vissoci. Drafting the article: Elahi, Rocha, Silva. Critically revising the article: all authors. Reviewed submitted version of manuscript: all authors. Statistical analysis: Elahi, Nickenig Vissoci. Administrative/technical/material support: Silva, Fuller, Mmbaga, Haglund, Staton, Nickenig Vissoci. Study supervision: Elahi, Fuller, Green, Haglund, Staton, Nickenig Vissoci.

Supplemental Information

Previous Presentations

Portions of this work were presented in abstract form at the Annual Meeting of the Society of Academic Emergency Medicine held in Indianapolis, Indiana, on May 15–18, 2018.

References

  • 1

    Carpenter J, Bithell J: Bootstrap confidence intervals: when, which, what? A practical guide for medical statisticians. Stat Med 19:11411164, 2000

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 2

    Chawla NV, Bowyer KW, Hall LO, Kegelmeyer WP: SMOTE: synthetic minority over-sampling technique. J Artif Intell Res 16:321357, 2002

  • 3

    Collins GS, Reitsma JB, Altman DG, Moons KG: Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD Statement. BMC Med 13:1, 2015

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 4

    DeLong ER, DeLong DM, Clarke-Pearson DL: Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics 44:837845, 1988

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 5

    Dewan MC, Rattani A, Gupta S, Baticulon RE, Hung YC, Punchak M, et al.: Estimating the global incidence of traumatic brain injury. J Neurosurg [epub ahead of print April 1, 2018. DOI: 10.3171/2017.10.JNS17352]

    • PubMed
    • Search Google Scholar
    • Export Citation
  • 6

    Eftekhar B, Mohammad K, Ardebili HE, Ghodsi M, Ketabchi E: Comparison of artificial neural network and logistic regression models for prediction of mortality in head trauma based on initial clinical data. BMC Med Inform Decis Mak 5:3, 2005

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 7

    Fawcett T: An introduction to ROC analysis. Pattern Recognit Lett 27:861874, 2006

  • 8

    Frattalone AR, Ling GSF: Moderate and severe traumatic brain injury: pathophysiology and management. Neurosurg Clin N Am 24:309319, 2013

  • 9

    Haselsberger K, Pucher R, Auer LM: Prognosis after acute subdural or epidural haemorrhage. Acta Neurochir (Wien) 90:111116, 1988

  • 10

    Hastie T, Tibshirani R, Friedman J: The Elements of Statistical Learning, ed 2. Berlin: Springer, 2001

  • 11

    Holmes DE, Jain LC, Beaulieu-Jones B: Machine Learning for Structured Clinical Data. Cham, Switzerland: Springer International Publishing, 2018

  • 12

    Hsu MH, Li YC, Chiu WT, Yen JC: Outcome prediction after moderate and severe head injury using an artificial neural network. Stud Health Technol Inform 116:241245, 2005

    • PubMed
    • Search Google Scholar
    • Export Citation
  • 13

    Jennett B, Snoek J, Bond MR, Brooks N: Disability after severe head injury: observations on the use of the Glasgow Outcome Scale. J Neurol Neurosurg Psychiatry 44:285293, 1981

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 14

    Kairalla JA, Coffey CS, Muller KE: GLUMIP 2.0: SAS/IML software for planning internal pilots. J Stat Softw 28:1, 2008

  • 15

    Kalpakis K, Yang S, Hu PF, Mackenzie CF, Stansbury LG, Stein DM, et al.: Permutation entropy analysis of vital signs data for outcome prediction of patients with severe traumatic brain injury. Comput Biol Med 56:167174, 2015

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 16

    Kuo BJ, Vaca SD, Vissoci JRN, Staton CA, Xu L, Muhumuza M, et al.: A prospective neurosurgical registry evaluating the clinical care of traumatic brain injury patients presenting to Mulago National Referral Hospital in Uganda. PLoS One 12:e0182285, 2017

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 17

    Lai YC, Chen FG, Goh MH, Koh KF: Predictors of long-term outcome in severe head injury. Ann Acad Med Singapore 27:326331, 1998

  • 18

    Luo W, Phung D, Tran T, Gupta S, Rana S, Karmakar C, et al.: Guidelines for developing and reporting machine learning predictive models in biomedical research: a multidisciplinary view. J Med Internet Res 18:e323, 2016

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 19

    Mukherjee KK, Sharma BS, Ramanathan SM, Khandelwal N, Kak VK: A mathematical outcome prediction model in severe head injury: a pilot study. Neurol India 48:4348, 2000

    • PubMed
    • Search Google Scholar
    • Export Citation
  • 20

    Murray CJL: The Global Burden of Disease: A Comprehensive Assessment of Mortality and Disability from Diseases, Injuries, and Risk Factors in 1990 and Projected to 2020: Summary. Cambridge, MA: Harvard School of Public Health, 1996

    • PubMed
    • Search Google Scholar
    • Export Citation
  • 21

    Murray LS, Teasdale GM, Murray GD, Jennett B, Miller JD, Pickard JD, et al.: Does prediction of outcome alter patient management? Lancet 341:14871491, 1993

  • 22

    Obermeyer Z, Emanuel EJ: Predicting the future—big data, machine learning, and clinical medicine. N Engl J Med 375:12161219, 2016

  • 23

    Perel P, Arango M, Clayton T, Edwards P, Komolafe E, Poccock S, et al.: Predicting outcome after traumatic brain injury: practical prognostic models based on large cohort of international patients. BMJ 336:425429, 2008

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 24

    Perel P, Edwards P, Wentz R, Roberts I: Systematic review of prognostic models in traumatic brain injury. BMC Med Inform Decis Mak 6:38, 2006

  • 25

    Perel P, Wasserberg J, Ravi RR, Shakur H, Edwards P, Roberts I: Prognosis following head injury: a survey of doctors from developing and developed countries. J Eval Clin Pract 13:464465, 2007

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 26

    Pillai SV, Kolluri VR, Praharaj SS: Outcome prediction model for severe diffuse brain injuries: development and evaluation. Neurol India 51:345349, 2003

    • PubMed
    • Search Google Scholar
    • Export Citation
  • 27

    Ratanalert S, Chompikul J, Hirunpat S, Pheunpathom N: Prognosis of severe head injury: an experience in Thailand. Br J Neurosurg 16:487493, 2002

  • 28

    Raykar NP, Yorlets RR, Liu C, Greenberg SLM, Kotagal M, Goldman R, et al.: A qualitative study exploring contextual challenges to surgical care provision in 21 LMICs. Lancet 385 (Suppl 2):S15, 2015

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 29

    Reddy CK, Aggarwal CC: Healthcare Data Analytics. Boca Raton, FL: CRC Press, 2015

  • 30

    Robin X, Turck N, Hainard A, Tiberti N, Lisacek F, Sanchez JC, et al.: pROC: an open-source package for R and S+ to analyze and compare ROC curves. BMC Bioinformatics 12:77, 2011

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 31

    Sadaka F, Patel D, Lakshmanan R: The FOUR score predicts outcome in patients after traumatic brain injury. Neurocrit Care 16:95101, 2012

  • 32

    Seelig JM, Becker DP, Miller JD, Greenberg RP, Ward JD, Choi SC: Traumatic acute subdural hematoma: major mortality reduction in comatose patients treated within four hours. N Engl J Med 304:15111518, 1981

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 33

    Senders JT, Staples PC, Karhade AV, Zaki MM, Gormley WB, Broekman MLD, et al.: Machine learning and neurosurgical outcome prediction: a systematic review. World Neurosurg 109:476486, 486.e1, 2018

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 34

    Shi HY, Hwang SL, Lee KT, Lin CL: In-hospital mortality after traumatic brain injury surgery: a nationwide population-based comparison of mortality predictors used in artificial neural network and logistic regression models. J Neurosurg 118:746752, 2013

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 35

    Staton CA, Msilanga D, Kiwango G, Vissoci JR, de Andrade L, Lester R, et al.: A prospective registry evaluating the epidemiology and clinical care of traumatic brain injury patients presenting to a regional referral hospital in Moshi, Tanzania: challenges and the way forward. Int J Inj Contr Saf Promot 24:6977, 2017

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 36

    Steyerberg EW, Mushkudiani N, Perel P, Butcher I, Lu J, McHugh GS, et al.: Predicting outcome after traumatic brain injury: development and international validation of prognostic scores based on admission characteristics. PLoS Med 5:e165, 2008

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 37

    Subaiya S, Roberts I, Komolafe E, Perel P: Predicting intracranial hemorrhage after traumatic brain injury in low and middle-income countries: a prognostic model based on a large, multi-center, international cohort. BMC Emerg Med 12:17, 2012

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 38

    Vaca SD, Kuo BJ, Nickenig Vissoci JR, Staton CA, Xu LW, Muhumuza M, et al.: Temporal delays along the neurosurgical care continuum for traumatic brain injury patients at a tertiary care hospital in Kampala, Uganda. Neurosurgery 84:95103, 2019

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 39

    van Buuren S, Groothuis-Oudshoorn K: Multivariate imputation by chained equations in R. J Stat Softw 45:167, 2011

  • 40

    White IR, Royston P, Wood AM: Multiple imputation using chained equations: issues and guidance for practice. Stat Med 30:377399, 2011

  • 41

    Zador Z, Sperrin M, King AT: Predictors of outcome in traumatic brain injury: new insight using receiver operating curve indices and Bayesian network analysis. PLoS One 11:e0158762, 2016

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • Collapse
  • Expand

Cortical visual prostheses offer the potential to translate video into patterned visual cortex stimulation to produce predictable and consistent visual percepts. Artist and copyright Kenneth Probst. Published with permission. See the article by Niketeghad et al. (pp 2000–2007).

  • FIG. 1.

    Comparison of ML models applied to predict TBI outcome. Each line represents the receiver operating characteristic (ROC) curve for a different ML technique. The Bayesian generalized linear model (Bay. Gen. Lin. Mod) was the top performing technique. Grad. boost. machine = gradient boosting machine; Bay. Add. Reg. Tr. = Bayesian additive regression trees. Figure is available in color online only.

  • FIG. 2.

    Bar chart for the extracted beta coefficients (Coeff) from the top performing prognostic model, the Bayesian generalized linear model. Orange lines represent variables associated with a poor recovery, whereas blue lines represent variables associated with a good recovery. BP = blood pressure; intercept = value in regression model, included for the sake of transparency. Figure is available in color online only.

  • FIG. 3.

    The web-based application (A) and the mobile-based application (B). Figure is available in color online only

  • 1

    Carpenter J, Bithell J: Bootstrap confidence intervals: when, which, what? A practical guide for medical statisticians. Stat Med 19:11411164, 2000

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 2

    Chawla NV, Bowyer KW, Hall LO, Kegelmeyer WP: SMOTE: synthetic minority over-sampling technique. J Artif Intell Res 16:321357, 2002

  • 3

    Collins GS, Reitsma JB, Altman DG, Moons KG: Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD Statement. BMC Med 13:1, 2015

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 4

    DeLong ER, DeLong DM, Clarke-Pearson DL: Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics 44:837845, 1988

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 5

    Dewan MC, Rattani A, Gupta S, Baticulon RE, Hung YC, Punchak M, et al.: Estimating the global incidence of traumatic brain injury. J Neurosurg [epub ahead of print April 1, 2018. DOI: 10.3171/2017.10.JNS17352]

    • PubMed
    • Search Google Scholar
    • Export Citation
  • 6

    Eftekhar B, Mohammad K, Ardebili HE, Ghodsi M, Ketabchi E: Comparison of artificial neural network and logistic regression models for prediction of mortality in head trauma based on initial clinical data. BMC Med Inform Decis Mak 5:3, 2005

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 7

    Fawcett T: An introduction to ROC analysis. Pattern Recognit Lett 27:861874, 2006

  • 8

    Frattalone AR, Ling GSF: Moderate and severe traumatic brain injury: pathophysiology and management. Neurosurg Clin N Am 24:309319, 2013

  • 9

    Haselsberger K, Pucher R, Auer LM: Prognosis after acute subdural or epidural haemorrhage. Acta Neurochir (Wien) 90:111116, 1988

  • 10

    Hastie T, Tibshirani R, Friedman J: The Elements of Statistical Learning, ed 2. Berlin: Springer, 2001

  • 11

    Holmes DE, Jain LC, Beaulieu-Jones B: Machine Learning for Structured Clinical Data. Cham, Switzerland: Springer International Publishing, 2018

  • 12

    Hsu MH, Li YC, Chiu WT, Yen JC: Outcome prediction after moderate and severe head injury using an artificial neural network. Stud Health Technol Inform 116:241245, 2005

    • PubMed
    • Search Google Scholar
    • Export Citation
  • 13

    Jennett B, Snoek J, Bond MR, Brooks N: Disability after severe head injury: observations on the use of the Glasgow Outcome Scale. J Neurol Neurosurg Psychiatry 44:285293, 1981

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 14

    Kairalla JA, Coffey CS, Muller KE: GLUMIP 2.0: SAS/IML software for planning internal pilots. J Stat Softw 28:1, 2008

  • 15

    Kalpakis K, Yang S, Hu PF, Mackenzie CF, Stansbury LG, Stein DM, et al.: Permutation entropy analysis of vital signs data for outcome prediction of patients with severe traumatic brain injury. Comput Biol Med 56:167174, 2015

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 16

    Kuo BJ, Vaca SD, Vissoci JRN, Staton CA, Xu L, Muhumuza M, et al.: A prospective neurosurgical registry evaluating the clinical care of traumatic brain injury patients presenting to Mulago National Referral Hospital in Uganda. PLoS One 12:e0182285, 2017

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 17

    Lai YC, Chen FG, Goh MH, Koh KF: Predictors of long-term outcome in severe head injury. Ann Acad Med Singapore 27:326331, 1998

  • 18

    Luo W, Phung D, Tran T, Gupta S, Rana S, Karmakar C, et al.: Guidelines for developing and reporting machine learning predictive models in biomedical research: a multidisciplinary view. J Med Internet Res 18:e323, 2016

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 19

    Mukherjee KK, Sharma BS, Ramanathan SM, Khandelwal N, Kak VK: A mathematical outcome prediction model in severe head injury: a pilot study. Neurol India 48:4348, 2000

    • PubMed
    • Search Google Scholar
    • Export Citation
  • 20

    Murray CJL: The Global Burden of Disease: A Comprehensive Assessment of Mortality and Disability from Diseases, Injuries, and Risk Factors in 1990 and Projected to 2020: Summary. Cambridge, MA: Harvard School of Public Health, 1996

    • PubMed
    • Search Google Scholar
    • Export Citation
  • 21

    Murray LS, Teasdale GM, Murray GD, Jennett B, Miller JD, Pickard JD, et al.: Does prediction of outcome alter patient management? Lancet 341:14871491, 1993

  • 22

    Obermeyer Z, Emanuel EJ: Predicting the future—big data, machine learning, and clinical medicine. N Engl J Med 375:12161219, 2016

  • 23

    Perel P, Arango M, Clayton T, Edwards P, Komolafe E, Poccock S, et al.: Predicting outcome after traumatic brain injury: practical prognostic models based on large cohort of international patients. BMJ 336:425429, 2008

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 24

    Perel P, Edwards P, Wentz R, Roberts I: Systematic review of prognostic models in traumatic brain injury. BMC Med Inform Decis Mak 6:38, 2006

  • 25

    Perel P, Wasserberg J, Ravi RR, Shakur H, Edwards P, Roberts I: Prognosis following head injury: a survey of doctors from developing and developed countries. J Eval Clin Pract 13:464465, 2007

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 26

    Pillai SV, Kolluri VR, Praharaj SS: Outcome prediction model for severe diffuse brain injuries: development and evaluation. Neurol India 51:345349, 2003

    • PubMed
    • Search Google Scholar
    • Export Citation
  • 27

    Ratanalert S, Chompikul J, Hirunpat S, Pheunpathom N: Prognosis of severe head injury: an experience in Thailand. Br J Neurosurg 16:487493, 2002

  • 28

    Raykar NP, Yorlets RR, Liu C, Greenberg SLM, Kotagal M, Goldman R, et al.: A qualitative study exploring contextual challenges to surgical care provision in 21 LMICs. Lancet 385 (Suppl 2):S15, 2015

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 29

    Reddy CK, Aggarwal CC: Healthcare Data Analytics. Boca Raton, FL: CRC Press, 2015

  • 30

    Robin X, Turck N, Hainard A, Tiberti N, Lisacek F, Sanchez JC, et al.: pROC: an open-source package for R and S+ to analyze and compare ROC curves. BMC Bioinformatics 12:77, 2011

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 31

    Sadaka F, Patel D, Lakshmanan R: The FOUR score predicts outcome in patients after traumatic brain injury. Neurocrit Care 16:95101, 2012

  • 32

    Seelig JM, Becker DP, Miller JD, Greenberg RP, Ward JD, Choi SC: Traumatic acute subdural hematoma: major mortality reduction in comatose patients treated within four hours. N Engl J Med 304:15111518, 1981

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 33

    Senders JT, Staples PC, Karhade AV, Zaki MM, Gormley WB, Broekman MLD, et al.: Machine learning and neurosurgical outcome prediction: a systematic review. World Neurosurg 109:476486, 486.e1, 2018

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 34

    Shi HY, Hwang SL, Lee KT, Lin CL: In-hospital mortality after traumatic brain injury surgery: a nationwide population-based comparison of mortality predictors used in artificial neural network and logistic regression models. J Neurosurg 118:746752, 2013

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 35

    Staton CA, Msilanga D, Kiwango G, Vissoci JR, de Andrade L, Lester R, et al.: A prospective registry evaluating the epidemiology and clinical care of traumatic brain injury patients presenting to a regional referral hospital in Moshi, Tanzania: challenges and the way forward. Int J Inj Contr Saf Promot 24:6977, 2017

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 36

    Steyerberg EW, Mushkudiani N, Perel P, Butcher I, Lu J, McHugh GS, et al.: Predicting outcome after traumatic brain injury: development and international validation of prognostic scores based on admission characteristics. PLoS Med 5:e165, 2008

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 37

    Subaiya S, Roberts I, Komolafe E, Perel P: Predicting intracranial hemorrhage after traumatic brain injury in low and middle-income countries: a prognostic model based on a large, multi-center, international cohort. BMC Emerg Med 12:17, 2012

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 38

    Vaca SD, Kuo BJ, Nickenig Vissoci JR, Staton CA, Xu LW, Muhumuza M, et al.: Temporal delays along the neurosurgical care continuum for traumatic brain injury patients at a tertiary care hospital in Kampala, Uganda. Neurosurgery 84:95103, 2019

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 39

    van Buuren S, Groothuis-Oudshoorn K: Multivariate imputation by chained equations in R. J Stat Softw 45:167, 2011

  • 40

    White IR, Royston P, Wood AM: Multiple imputation using chained equations: issues and guidance for practice. Stat Med 30:377399, 2011

  • 41

    Zador Z, Sperrin M, King AT: Predictors of outcome in traumatic brain injury: new insight using receiver operating curve indices and Bayesian network analysis. PLoS One 11:e0158762, 2016

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation

Metrics

All Time Past Year Past 30 Days
Abstract Views 3245 433 0
Full Text Views 1214 928 431
PDF Downloads 686 180 24
EPUB Downloads 0 0 0