Using artificial neural networks to identify patients with concussion and postconcussion syndrome based on antisaccades

View More View Less
  • 1 Injury Prevention Research Office, St. Michael’s Hospital;
  • 2 Faculty of Medicine, University of Toronto;
  • 3 Department of Computer Science, Ryerson University;
  • 4 Division of Neurosurgery, Department of Surgery, St. Michael’s Hospital; and
  • 5 Dalla Lana School of Public Health, University of Toronto, Toronto, Ontario, Canada
Full access

OBJECTIVE

Artificial neural networks (ANNs) have shown considerable promise as decision support tools in medicine, including neurosurgery. However, their use in concussion and postconcussion syndrome (PCS) has been limited. The authors explore the value of using an ANN to identify patients with concussion/PCS based on their antisaccade performance.

METHODS

Study participants were prospectively recruited from the emergency department and head injury clinic of a large teaching hospital in Toronto. Acquaintances of study participants were used as controls. Saccades were measured using an automated, portable, head-mounted device preprogrammed with an antisaccade task. Each participant underwent 100 trials of the task and 11 saccade parameters were recorded for each trial. ANN analysis was performed using the MATLAB Neural Network Toolbox, and individual saccade parameters were further explored with receiver operating characteristic (ROC) curves and a logistic regression analysis.

RESULTS

Control (n = 15), concussion (n = 32), and PCS (n = 25) groups were matched by age and level of education. The authors examined 11 saccade parameters and found that the prosaccade error rate (p = 0.04) and median antisaccade latency (p = 0.02) were significantly different between control and concussion/PCS groups. When used to distinguish concussion and PCS participants from controls, the neural networks achieved accuracies of 67% and 72%, respectively. This method was unable to distinguish study patients with concussion from those with PCS, suggesting persistence of eye movement abnormalities in patients with PCS. The authors’ observations also suggest the potential for improved results with a larger training sample.

CONCLUSIONS

This study explored the utility of ANNs in the diagnosis of concussion/PCS based on antisaccades. With the use of an ANN, modest accuracy was achieved in a small cohort. In addition, the authors explored the pearls and pitfalls of this novel approach and identified important future directions for this research.

ABBREVIATIONS ANN = artificial neural network; AS = antisaccade; AUC = area under the curve; MLP = multilayer perceptron; mTBI = mild TBI; NPV = negative predictive value; PCS = postconcussion syndrome; PER = prosaccade error rate; PPV = positive predictive value; PS = prosaccade; ROC = receiver operating characteristic; SMH = St. Michael’s Hospital; TBI = traumatic brain injury.

OBJECTIVE

Artificial neural networks (ANNs) have shown considerable promise as decision support tools in medicine, including neurosurgery. However, their use in concussion and postconcussion syndrome (PCS) has been limited. The authors explore the value of using an ANN to identify patients with concussion/PCS based on their antisaccade performance.

METHODS

Study participants were prospectively recruited from the emergency department and head injury clinic of a large teaching hospital in Toronto. Acquaintances of study participants were used as controls. Saccades were measured using an automated, portable, head-mounted device preprogrammed with an antisaccade task. Each participant underwent 100 trials of the task and 11 saccade parameters were recorded for each trial. ANN analysis was performed using the MATLAB Neural Network Toolbox, and individual saccade parameters were further explored with receiver operating characteristic (ROC) curves and a logistic regression analysis.

RESULTS

Control (n = 15), concussion (n = 32), and PCS (n = 25) groups were matched by age and level of education. The authors examined 11 saccade parameters and found that the prosaccade error rate (p = 0.04) and median antisaccade latency (p = 0.02) were significantly different between control and concussion/PCS groups. When used to distinguish concussion and PCS participants from controls, the neural networks achieved accuracies of 67% and 72%, respectively. This method was unable to distinguish study patients with concussion from those with PCS, suggesting persistence of eye movement abnormalities in patients with PCS. The authors’ observations also suggest the potential for improved results with a larger training sample.

CONCLUSIONS

This study explored the utility of ANNs in the diagnosis of concussion/PCS based on antisaccades. With the use of an ANN, modest accuracy was achieved in a small cohort. In addition, the authors explored the pearls and pitfalls of this novel approach and identified important future directions for this research.

ABBREVIATIONS ANN = artificial neural network; AS = antisaccade; AUC = area under the curve; MLP = multilayer perceptron; mTBI = mild TBI; NPV = negative predictive value; PCS = postconcussion syndrome; PER = prosaccade error rate; PPV = positive predictive value; PS = prosaccade; ROC = receiver operating characteristic; SMH = St. Michael’s Hospital; TBI = traumatic brain injury.

In Brief

In this study the authors applied machine learning techniques to eye movement data in order to identify patients with concussion and postconcussion syndromes. This approach may lead to the development of a method that can be used in the hospital or the community to diagnose concussions, for which no such objective tests are currently available.

Traumatic brain injury (TBI) is a significant public health concern due to its prevalence and potential for harm. A 2015 report to the US Congress by the Centers for Disease Control and Prevention12 estimated that 2.5 million people in the US sought medical attention for TBI in 2010. It has been further reported that at least 75% of TBIs are classified as mild (mTBI; which includes concussion), although this is likely a gross underestimate as it does not account for injuries in patients who do not seek medical care.13 Though most patients with concussion recover within a few weeks, up to 20% experience persistent physical, cognitive, and emotional symptoms beyond 3 months, termed postconcussion syndrome (PCS).25 Repeated concussions have also been linked to chronic traumatic encephalopathy (CTE), a dementia-like neurodegenerative disease,22 and second-impact syndrome, a very rare phenomenon wherein cerebral dysautoregulation leads to malignant cerebral edema and often death.32 Reliable identification of patients with concussion is crucial for timely initiation of management in order to reduce symptom burden and prevent further harm. Unfortunately, no objective biomarkers are available for use in clinical practice, and diagnosis currently relies on history and physical examination. However, eye movements (specifically antisaccades [ASs]) have been shown to be abnormal in concussion and PCS patients and may be useful in the development of objective diagnostic and prognostic tools.3,15–20,30

Saccades are rapid, conjugate eye movements that serve to transition between 2 areas of focus. We considered 2 subtypes: prosaccades (PSs) and ASs. In the former, a reflexive saccade is generated toward a visual stimulus. In the latter, the eye moves instead toward the mirror opposite spot, relative to a point of central fixation. The AS is neurologically more demanding, requiring inhibition of PS-generating neurons prior to the initiation of a voluntary AS, which is capable of overriding the opposing reflexive PS.8,24 This relies on coordination between several areas of the brain, including the dorsolateral prefrontal cortex (DLPFC), which is chiefly responsible for this preemptive inhibition; the frontal, parietal, and supplemental eye fields; the basal ganglia; the thalamus; the cerebellum; and the brainstem.8 Concussion results in widespread white-matter pathology,1,2,10,29–31,33 and therefore it is likely that trauma-related disruption of saccade-dependent circuits (including DLPFC-mediated PS inhibition in the case of the AS task) underlies the abnormalities observed in this population.

Artificial neural networks (ANNs) have shown promise as clinical decision support tools for decades, owing to their powerful pattern recognition and discriminant analysis abilities.9,11 Constructed to learn in a fashion similar to that of a living brain (albeit simplified), ANNs consist of layered data processing units called neurons, coupled with variable “weights.” In the simplest form of ANN, known as a multilayer perceptron (MLP), data are propagated unidirectionally from an input layer through one or more “hidden” layers before reaching an output layer (Fig. 1). In the untrained state, weights take arbitrary values; they are adjusted during training as the system learns from data it is presented. For this learning to occur, an input vector is fed into the network and the resultant output is compared to a known, correct output. The difference is proportional to the network’s error, which is projected back through the system of weights. Over several iterations, the weights are adjusted to minimize this error using an established technique known as backpropagation.9,14 This experience-based learning allows neural networks to determine relevant interactions between several clinical parameters and their effects on output, without requiring a priori knowledge of the data’s complexity or advanced statistical methods.4,9 Consequently, ANNs have proven to be useful in the field of neurosurgery to predict clinical and CT scan outcomes in patients with TBI.7,26–28 However, these studies are limited to more severely injured patients, and there is a paucity of literature regarding the use of ANNs in the field of concussion. Specifically, the purpose of this study was to explore the utility of an ANN in the diagnosis of concussion and PCS by analyzing ASs.

FIG. 1.
FIG. 1.

General configuration of a 3-layer MLP (single hidden layer) with binary output, which is a common configuration for clinical use. The input nodes (x) are on the left of the diagram, hidden nodes (h) in the middle, and output nodes (y) on the right. Each line is associated with a weight, which is adjusted during training. The arrow represents the unidirectional information flow from input to output. Figure is available in color online only.

Methods

This work is part of a larger study, which received research ethics board (REB) approval (no. 11-147) from St. Michael’s Hospital (SMH). AS data were originally collected for use in a previously reported study,30 upon which the first 4 subsections of this section are based.

Study Participants

Patients from the emergency department and Outpatient Head Injury Clinic at SMH were screened for eligibility criteria, and controls were recruited from acquaintances informally contacted by study participants. Controls were matched to concussion/PCS participants by age (± 5 years) and years of education (± 5 years). All study participants provided informed written consent before data were gathered.

Inclusion/Exclusion Criteria

This study used the World Health Organization’s definition of mTBI6 as the basis for participant selection, with the following qualifications: participants were required to be at least 16 years of age, be able to provide informed consent, and have adequate command of the English language. For the PCS group, we required that the provoking head injury was sustained at least 3 months before our study and that symptoms were persistent for at least 3 months after the injury.

Participants in this study were excluded if they had any of the following comorbid conditions: horizontal diplopia, severe loss of visual acuity, multiple sclerosis, early or alcohol-related dementia, a neurodegenerative disease (including Alzheimer’s disease), uncontrolled diabetes, or orbital trauma from the present injury. Those with a history of incisional brain surgery, brain irradiation, hydrocephalus, stroke, or eye disease (amblyopia or glaucoma) were also excluded, as were individuals who were medically unstable or intoxicated (with alcohol or drugs) at the time of recruitment. Finally, anyone taking medications known to interfere with saccades (specifically benzodiazepines and antipsychotics) was excluded from the study. All participants were given a structured interview to rule out these exclusion criteria.

Saccade Measurement

All participants were asked to attend SMH to complete 100 trials of an AS task. The participants who were concussion patients were asked to undergo testing as soon as possible to minimize latency between injury and testing. Data were gathered using a portable research device, the Saccadometer Advanced (Ober Consulting). This device automatically excludes blinks, adjusts for minor head movements, and has an AS task preprogrammed by the manufacturer. Three low-powered red laser lights, serving as left, right, and central targets, were projected onto a matte white wall 1.5–3.5 m from the participant’s eyes under darkened conditions, following previous work from our group.23,30

Participants initially underwent calibration and tutorial sessions to ensure understanding of the task before data were recorded. We asked participants with refractive errors to wear contacts or examined them to ensure sufficient visual acuity and lack of horizontal diplopia. Participants were instructed to fixate on a centrally projected laser light until it was extinguished, before immediately reappearing 10° horizontally (with equal probability of being to the left or right of center). There was a fixed foreperiod of 1000 msec followed by a random foreperiod of ≤ 1000 msec. Once the light jumped, participants were instructed to look to the mirror opposite spot, relative to the original central target, as quickly and accurately as possible. For each saccade, the waiting time, defined as the maximum amount of time the saccadometer will wait for an eye movement response after the stimulus is presented, was 2000 msec. The trial break time, defined as the length of time between AS trials, was 1000 msec.

Data Processing

Raw data from the saccadometer (including saccade direction, latency, duration, amplitude, and peak velocity) were transferred to a personal computer with LatencyMeter Software version 5.2. Trials with latencies < 50 msec (anticipatory saccades) or > 600 msec (inattention/possible malingering) were excluded from analysis.23,30 Prosaccade error rate (PER) was taken as the number of incorrect PSs divided by the number of accepted trials for each participant. The trial result data from each participant were then divided into 2 groups—incorrect PSs and correct ASs—and the medians of each saccade parameter were taken separately from both groups. In addition, amplitudes were converted into variables independent of the stimulus amplitude: gain (the quotient of saccade amplitude and stimulus amplitude, where a value < 1 indicates a hypometric saccade and a value > 1 indicates a hypermetric saccade) and absolute position error (a measure of the absolute distance between the final saccade position and the stimulus).

For each participant, 14 potential ANN input variables were considered: age, sex, years of education, PER, and 5 median saccade parameters from both the correct AS and incorrect PS sets (latency, duration, gain, absolute position error, and peak velocity). Comparisons were made in turn between concussion and control, PCS and control, and concussion and PCS. Parallel analysis using data from all AS trials was also completed, using the 5 AS-specific parameters as potential inputs. Interparticipant mean values were used to explore differences in saccades between control, concussion, and PCS groups.

Artificial Neural Network

We used an MLP, which had a single hidden layer and was trained with scaled conjugate backpropagation in MATLAB’s Neural Network Toolbox. To avoid bias during training, the size of both input groups being compared was equalized by randomly omitting data points from the larger group. Data at each input neuron were normalized to a mean of zero and a covariance of 1. Input data were then divided randomly into training, validation, and testing sets (70%, 10%, and 15% of data, respectively). The ANN was trained until 6 successively worsening results were detected in the validation set (per MATLAB default settings). This procedure functions as a safeguard against overlearning, wherein the network learns the noise of the training data and becomes less effective at analyzing novel data.11 A further safeguard was implemented to ensure that results were not generated from the initial, random weights of the system by rejecting results from networks with fewer than 10 training iterations. The testing set was used to probe the trained network’s accuracy, sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV). Mean performance results were generated from 100 neural networks, each with a different random set of input data omitted from the larger group (such that no data are omitted from ANN analysis).

Selection of input variables relied on a sensitivity analysis using forward selection. In this approach, the networks were built by iteratively adding individual input parameters based on their effects on the overall network’s accuracy. Letting n be the number of potential inputs for a given network, we began by training n different single-input networks. The input yielding the highest accuracy was added permanently to the network, and n − 1 two-input ANNs were trained, each using the previously selected input and one of the remaining n − 1 candidate parameters. If the most accurate bivariate network was more accurate, on average, than the univariate network, the second variable of this network was added permanently to the full ANN. This process was continued until no additional input variables improved accuracy, at which point the network configuration was final.21 Since there are no established rules for determining the optimal number of hidden nodes in a neural network,14 the final configuration was determined empirically for each network configuration by varying the number of hidden nodes from 2 to 5 times the number of input nodes and selecting the configuration that yielded the highest accuracy.

Once the final ANN configurations were determined, performance results (the mean from 100 ANNs) were generated 10 times. The median of these 10 accuracies and its associated sensitivity, specificity, PPV, and NPV were reported. These results were then compared with results from (otherwise equivalent) single-input ANNs, and receiver operating characteristic (ROC) curves, of each input variable selected during sensitivity analysis.

Statistical Analysis

Cross-sectional AS data were analyzed using ANOVA with Fisher’s least significant difference (LSD) in post hoc comparisons. Skewness and kurtosis were calculated to show the distribution of AS parameters for concussion, PCS, and control groups. Pearson correlation coefficients were calculated to show the association between AS parameters and age/level of education/test latency. The point-biserial correlation coefficient was calculated to show the association between AS parameters and sex. Logistic regressions were constructed to model the predictors for diagnosis and to compare the results produced by ANN. All statistical analysis was conducted using Stata 15 (StataCorp). A p value less than 0.05 was considered statistically significant.

Results

Participant Demographics

Refer to Table 1 for an overview of participant demographics. The median duration between injury and testing for the concussion group was 6 days (range 0–11 days), and 22% of these participants (n = 7) had their first visit > 7 days after injury. Correspondingly, the PCS group had their first visit a median of 16.7 months after injury (range 3–61.6 months). Two participants in the concussion group had a Glasgow Coma Scale (GCS) score < 15 in the emergency department (GCS score 13 and 14), but both regained a GCS of 15 by the time they left the emergency department, and therefore all had a GCS score of 15 at the time of testing. Two control participants were omitted from analysis as outliers, as there may have been exogenous factors (such as lack of motivation) contributing to poor performance. There were also 2 potential concussion outliers identified, though these were kept due to potential contributions from endogenous factors such as injury severity. Omission of these concussion participants, importantly, does not affect the statistical significance of group results.

TABLE 1.

Participant demographic information

Concussion (n = 32)PCS (n = 25)Control (n = 15)
Age in yrs, mean (SD)40 (15.3)44.2 (14.0)38.5 (19.7)
Sex, M/F17:158:177:8
Education (full yrs from grade 1), mean (SD)15.0 (3.4)16.2 (3.4)15.4 (1.6)
History of prior mTBI53.3%48.0%0%

Artificial Neural Network

Sensitivity analysis yielded the following ANN configurations. The neural network comparing concussion and control participants had 4 input neurons (PER, median AS latency, median AS absolute positional error, and age) and 3 hidden neurons. The PCS versus control participant network had 2 inputs (PER and median AS latency) and a single hidden neuron. Both ANNs comparing concussion/PCS to control trials used AS latency and duration as inputs and had a single hidden neuron. During sensitivity analysis of the concussion/PCS comparison, it was observed that the optimal network configuration in the participant analysis could not be deduced due to inconsistency in results. In the trial analysis, AS peak velocity is the only input, with 3 hidden nodes.

Results generated by these ANNs are summarized in Table 2 and compared to the ROC curves in Fig. 2. Our ANNs achieved a classification accuracy of 67% between concussion and control participants and 72% between PCS and control participants. They were unable to distinguish concussion from PCS participants.

TABLE 2.

ANN outputs

Accuracy, %Sensitivity, %Specificity, %PPV, %NPV, %
Concussion/control
 Participants67 (19)63 (29)74 (29)73 (30)66 (29)
 Trials60 (3)55 (10)66 (10)62 (5)60 (5)
PCS/control
 Participants72 (20)71 (32)75 (28)74 (30)74 (29)
 Trials59 (3)54 (9)65 (8)61 (5)58 (4)
Concussion/PCS
 Trials56 (3)70 (15)41 (18)55 (4)62 (10)

Values are presented as mean (SD).

FIG. 2.
FIG. 2.

ROC curves for each saccade parameter used as a neural network input, with ANN performance results superimposed for (A) participant analysis and (B) trial analysis. Error bars represent standard deviation. Closed circles represent the multi-input neural network results from Table 2. Open circles represent results from neural networks trained with only the input variable being plotted, with all other parameters unchanged. Figure is available in color online only.

Group Results

Refer to Table 3 for group mean comparisons. Two saccade parameters, PER and median AS latency, were significantly different between control and concussion/PCS participants (ANOVA, p = 0.04 and p = 0.03, respectively). The frequency distributions for saccade parameters used in the ANN analysis are presented in Fig. 3. There is a weak (r < 0.4) positive correlation between age and each saccade parameter involved in ANN analysis (PER and medians of AS latency, AS duration, AS absolute position error, and AS peak velocity). These variables also demonstrate a weak negative correlation (except AS duration, r = 0.006) with years of education and testing latency in the concussion group.

TABLE 3.

Comparison of interparticipant mean values using ANOVA

Concussion (n = 32)PCS (n = 25)Control (n = 15)Result
PER0.40 (0.23)0.39 (0.21)0.23 (0.20)F(2,69) = 3.33, p = 0.04
Median AS latency, msec292.34 (50.18)282.30 (34.33)254.90 (41.38)F(2,69) = 3.81, p = 0.02
Median AS duration, msec64.67 (25.52)64.06 (20.89)58.83 (16.84)F(2,69) = 0.37, p = 0.69
Median AS gain1.20 (0.57)1.26 (0.54)1.08 (0.39)F(2,69) = 0.52, p = 0.60
Median AS position error, %54.23 (36.21)52.57 (36.06)38.30 (20.52)F(2,69) = 1.24, p = 0.30
Median AS peak velocity, °/sec410.08 (100.30)435.55 (216.67)409.19 (122.90)F(2,69) = 0.23, p = 0.80
Median PS latency, msec193.36 (48.28)184.56 (28.64)192.23 (39.58)F(2,69) = 0.36, p = 0.70
Median PS duration, msec47.46 (9.35)46.04 (4.36)44.20 (7.34)F(2,69) = 0.98, p = 0.38
Median PS gain0.93 (0.32)0.94 (0.22)0.83 (0.17)F(2,69) = 0.93, p = 0.40
Median PS position error, %29.23 (22.36)21.90 (15.14)26.08 (11.15)F(2,69) = 1.14, p = 0.32
Median PS peak velocity, °/sec430.42 (99.55)447.93 (198.87)398.80 (106.94)F(2,69) = 0.55, p = 0.58

Values are presented as mean (SD) unless otherwise indicated.

FIG. 3.
FIG. 3.

Frequency distribution of (A) participant and (B) trial saccade parameters used as ANN inputs. K = kurtosis; ms = milliseconds; S = skewness. Figure is available in color online only.

Regression Analysis

Results of the logistic regression analysis are presented in Table 4. Note that in the concussion versus control analysis, AS latency is a significant predictor of concussion when controlling for PER, median AS absolute position error, and age (OR 1.02, p ≤ 0.05).

TABLE 4.

Results of logistic regression analysis

Saccade ParameterConcussion/Control, OR (95% CI)PCS/Control, OR (95% CI)
PER23.18 (0.36–1488.46)43.02 (0.64–2892.76)
Median AS latency1.02 (1.00–1.04)*1.02 (1.00–1.04)
Median absolute AS position error1.02 (0.99–1.06)NA
Participant age in yrs0.96 (0.91–1.02)NA

NA = not applicable.

p ≤ 0.05.

Discussion

Overview of ANN Results

ANN analysis was able to distinguish concussion and PCS participants from controls with accuracies of 67% and 72%, sensitivities of 63% and 71%, and specificities of 74% and 75%, respectively. Attempts to distinguish concussion from PCS participants proved unsuccessful, corroborating previous work describing persistent AS abnormalities in PCS.16,17,30 Parallel analysis using all AS trials (which reduces the number of available parameters) yielded lower accuracy but higher precision.

We noted a particularly high false-negative rate in these results, which could be associated with several factors. First, there might be a threshold of impact to the brain, below which any (saccade-dependent) neurobiological changes are minimal. This finding could provide further insight into the pathophysiology of concussion and help better define it objectively. Host features, such as the preinjury connectome, may also play a role by imparting differential resilience to injury, and baseline saccades could have differed between cases and controls. Similarly, co-occurring psychiatric conditions could mimic the symptoms of concussion/PCS and confound results (many of these conditions have also been associated with saccade abnormalities5). Finally, a delay of several days between injury and testing for the concussion group could have allowed for some recovery, and our small sample size may have inhibited the ANN’s ability to learn more subtle changes to its input.

Key Saccade Parameters

We determined that PER and median AS latency were significantly different between concussion/PCS and control groups (Table 3). This result is consistent with results of previous studies that found one or both of these parameters to be affected in concussion or PCS,3,15,17,18,20,30 though there is considerable heterogeneity in the literature. Not surprisingly, both PER and median AS latency were selected as ANN inputs during sensitivity analysis. We note that PER is associated with the largest odds ratio in the regression analysis, though a very large confidence interval limits our ability to draw conclusions. The PER ROC curves produced the largest area under the curve (AUC) values, suggesting that it is the best discriminating ANN input (Fig. 2). Comparatively, median AS latency yielded lower AUC values even though it was the only parameter to yield a statistically significant effect in the regression analysis (concussion vs control). Median AS absolute position error, an input variable in the concussion versus control participant analysis, was not significantly different between the concussion/PCS and control groups and had lower AUC values than PER and median AS latency. Interestingly, age was also selected as an input variable in this network, despite our groups being age matched. This is likely a result of imperfect age matching or may indicate a synergistic relationship between age and other inputs, though the effect is small.

In the trial-based analysis, AS latency is the best discriminating input variable based on ROC curves (Fig. 2; note that PER is not a potential input). Interestingly, AS duration was selected as an input for both concussion versus control and PCS versus control comparisons, despite AUC values of only 0.53. AS peak velocity, as a single input, was the best configuration to distinguish concussion and PCS trials, though with very low accuracy. Further, the 3-hidden-neuron optimal state suggests a degree of overlearning by the network. We conclude that neural network analysis was unable to meaningfully distinguish concussion from PCS in both the participant and trial-based analyses.

Pearls and Pitfalls of ANN in Concussion

The most important benefit of an ANN is its ability to amalgamate multiple input parameters of varying discriminant abilities to optimize results, while requiring little input from the user. As shown in Fig. 2, we observed that in all concussion/PCS versus control comparisons, multi-input ANNs yielded stronger results than any single-input network. Importantly, even very weakly discriminating parameters exerted a positive effect on results in the trial analysis. The experience-based learning model of an ANN makes it simple to implement, as it does not require nonlinear statistical methods or significant assumptions about the data or its complexity. Similarly, the ability of ANNs to learn higher-order patterns within data is not limited by the complexity level of a statistical model. These characteristics of flexibility and practical simplicity are considerable benefits of ANNs compared to more conventional techniques.

Several criticisms of ANNs are also noted. In our study, neural networks struggled to overcome the considerable variability within the data (Fig. 3) and the small participant sample size, yielding results comparable to those obtained using ROC curves of the best individual AS parameters (PER in the participant analysis and AS latency in the trial analysis; see Fig. 2). Compared to conventional methods, an ANN model lacks statistical rigor9 and does not readily impart its knowledge of the data onto the user. Consequently, there is a risk of both undertraining, wherein the hidden layer is insufficiently large to accommodate the degree of nonlinearity within the data, and overtraining, wherein the network inappropriately learns the noise within the data. The flexibility of a neural network means that its optimal state must often be deduced by painstakingly varying several free parameters. Due to the sheer size of this parameter space, it is impractical to test every feasible configuration, and results may not reflect those of a fully optimized network. Similarly, ANN analysis does not allow computation of the sample size required to generate a certain confidence level,9 making the planning of future studies challenging.

We suggest the potential for improved ANN accuracy given a larger participant sample, based on observations from Fig. 2. Single-input ANNs yield results falling below their respective ROC curves in the participant analysis (small sample size), whereas parallel results in the trial analysis (large sample size) fall on their respective curves. We suspect, therefore, that a larger participant cohort would improve the discriminatory effect of each input variable, leading to improved overall accuracy. We also note that variables with AUC values as low as 0.53 added benefit to the trial-based ANN, while only stronger variables (AUC ≥ 0.65) were selected in the participant analysis. It may follow that a larger participant cohort (with correspondingly smaller uncertainties) would permit the additional inclusion of less discriminating inputs, which could further improve performance. These observations are limited by considerable uncertainty in our results. An ANN’s ability to continue learning as it is presented new data is a significant potential strength, though very large data sets are needed for this strength to be fully exploited.

Study Limitations and Future Directions

The most important limitation of this study is the small sample size, which yielded large uncertainties and may have limited the discriminatory power of our neural networks. Similarly, we may not have fully captured the diversity of patients with concussion/PCS. For example, our ANN’s inability to distinguish concussion from PCS may indicate that our PCS cohort was enriched with patients who suffered particularly significant injuries or were associated with some other intrinsic factor that led to prolonged reporting of symptoms. Milder forms of PCS may have earlier resolution of saccade abnormalities, which could be better assessed with larger studies. Regarding our ANN construction, reliance on a simple MLP may have been less effective than using a more complex model. In addition, forward selection is not an exhaustive input variable selection method,21 and better input configurations may therefore exist. Regarding saccades, some of our results in the acutely injured group may have been attenuated given the delay between injury and testing, though correlation between testing latency and neural network input parameters was weak. Additionally, the results of our testing are highly dependent on participant motivation. While we are not aware of validated tools to measure saccade effort reliably, the lack of such tools was partially mitigated by excluding trials with latencies > 600 msec. Including more passive oculomotor assessments, such as PSs and smooth pursuit, may provide more insight into the role of motivation.

Follow-up studies with a larger sample size would be useful to further explore the potential utility of ANN-based analysis in this population. Inclusion of multimodal input data, such as neuropsychological testing and neuroimaging, would further test the diagnostic power and flexibility of ANN in concussion and PCS. Additionally, exploring the potential prognostic abilities of ANNs by correlating initial AS findings with outcome may provide greater insight into the mechanisms underlying PCS and how best to predict and manage it. The goals of TBI management are expanding beyond saving lives to improving long-term quality of life. As an objective representation of higher cortical function, ASs may serve to supplement current outcome prediction models focusing on survival and significant disability. By expanding the feature space of TBI diagnostics, this could drive the discovery of new disease patterns relevant to treatment planning and outcome prediction.

We think that this study is an important exploration of the pros and cons of using an ANN to examine AS changes in concussion and PCS, and that it lays the groundwork for future studies.

Conclusions

This study, to our knowledge, is the first to use an ANN to identify concussion and PCS patients based on saccades. Our results demonstrate that ANNs show initial promise as a diagnostic support tool, but our findings are limited by low sample sizes and the significant variability in this type of data. Further study with larger trials will be required.

Acknowledgments

The authors thank Ling Chen for her help with statistical analysis.

This work was supported by the Canadian Institutes of Health Research (MOP 123371) and the Canadian Institute for Military and Veteran Health Research (Advanced Analytic Initiative).

Disclosures

The authors report no conflict of interest concerning the materials or methods used in this study or the findings specified in this paper.

Author Contributions

Conception and design: Landry, Cusimano. Acquisition of data: Ting. Analysis and interpretation of data: Landry. Drafting the article: Landry. Critically revising the article: Landry, Ting, Sadeghian, Cusimano. Reviewed submitted version of manuscript: all authors. Approved the final version of the manuscript on behalf of all authors: Landry. Study supervision: Sadeghian, Cusimano.

References

  • 1

    Aoki Y, Inokuchi R: A voxel-based meta-analysis of diffusion tensor imaging in mild traumatic brain injury. Neurosci Biobehav Rev 66:119126, 2016

  • 2

    Asken BM, DeKosky ST, Clugston JR, Jaffee MS, Bauer RM: Diffusion tensor imaging (DTI) findings in adult civilian, military, and sport-related mild traumatic brain injury (mTBI): a systematic critical review. Brain Imaging Behav 12:585612, 2018

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 3

    Balaban C, Hoffer ME, Szczupak M, Snapp H, Crawford J, Murphy S, : Oculomotor, vestibular, and reaction time tests in mild traumatic brain injury. PLoS One 11:e0162168, 2016

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 4

    Baxt WG: Application of artificial neural networks to clinical medicine. Lancet 346:11351138, 1995

  • 5

    Bittencourt J, Velasques B, Teixeira S, Basile LF, Salles JI, Nardi AE, : Saccadic eye movement applications for psychiatric disorders. Neuropsychiatr Dis Treat 9:13931409, 2013

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 6

    Carroll LJ, Cassidy JD, Holm L, Kraus J, Coronado VG: Methodological issues and research recommendations for mild traumatic brain injury: the WHO Collaborating Centre Task Force on Mild Traumatic Brain Injury. J Rehabil Med 36 (43 Suppl):113125, 2004

    • Crossref
    • Search Google Scholar
    • Export Citation
  • 7

    Chong SL, Liu N, Barbier S, Ong MEH: Predictive modeling in pediatric traumatic brain injury using machine learning. BMC Med Res Methodol 15:22, 2015

  • 8

    Coe BC, Munoz DP: Mechanisms of saccade suppression revealed in the anti-saccade task. Philos Trans R Soc Lond B Biol Sci 372:20160192, 2017

  • 9

    Cross SS, Harrison RF, Kennedy RL: Introduction to neural networks. Lancet 346:10751079, 1995

  • 10

    Delouche A, Attyé A, Heck O, Grand S, Kastler A, Lamalle L, : Diffusion MRI: pitfalls, literature review and future directions of research in mild traumatic brain injury. Eur J Radiol 85:2530, 2016

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 11

    Forsström JJ, Dalton KJ: Artificial neural networks for decision support in clinical medicine. Ann Med 27:509517, 1995

  • 12

    Frieden TR, Houry D, Baldwin G: Traumatic Brain Injury in the United States: Epidemiology and Rehabilitation. Atlanta: CDC, 2015 (https://www.cdc.gov/traumaticbraininjury/pdf/tbi_report_to_congress_epi_and_rehab-a.pdf) [Accessed August 20, 2018]

    • Export Citation
  • 13

    Gerberding JL, Binder S: Report to Congress on Mild Traumatic Brain Injury in the United States: Steps to Prevent a Serious Public Health Problem. Atlanta: CDC, 2003 (https://www.cdc.gov/traumaticbraininjury/pdf/mtbireport-a.pdf) [Accessed August 20, 2018]

    • Export Citation
  • 14

    Gupta KC: Neural network structures, in Gupta KC, Zhang QJ (eds): Neural Networks for RF and Microwave Design. Boston: Artech House, 2000, pp 61103

    • Search Google Scholar
    • Export Citation
  • 15

    Heitger MH, Anderson TJ, Jones RD, Dalrymple-Alford JC, Frampton CM, Ardagh MW: Eye movement and visuomotor arm movement deficits following mild closed head injury. Brain 127:575590, 2004

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 16

    Heitger MH, Jones RD, Anderson TJ: A new approach to predicting postconcussion syndrome after mild traumatic brain injury based upon eye movement function, in Proceedings of the 30th Annual International Conference of the IEEE Engineering in Medicine and Biology Society. Piscataway, NJ: IEEE Engineering in Medicine and Biology Society, 2008, pp 35703573

    • Search Google Scholar
    • Export Citation
  • 17

    Heitger MH, Jones RD, Macleod AD, Snell DL, Frampton CM, Anderson TJ: Impaired eye movements in post-concussion syndrome indicate suboptimal brain function beyond the influence of depression, malingering or intellectual ability. Brain 132:28502870, 2009

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 18

    Hoffer ME, Balaban C, Szczupak M, Buskirk J, Snapp H, Crawford J, : The use of oculomotor, vestibular, and reaction time tests to assess mild traumatic brain injury (mTBI) over time. Laryngoscope Investig Otolaryngol 2:157165, 2017

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 19

    Hunt AW, Mah K, Reed N, Engel L, Keightley M: Oculomotor-based vision assessment in mild traumatic brain injury: a systematic review. J Head Trauma Rehabil 31:252261, 2016

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 20

    Johnson B, Zhang K, Hallett M: Functional neuroimaging of acute oculomotor deficits in concussed athletes. Brain Imaging Behav 9:564573, 2015

  • 21

    May R, Dandy G, Maier H: Review of input variable selection methods for artificial neural networks, in Suzuki K (ed): Artificial Neural Networks: Methodological Advances and Biomedical Applications. London: IntechOpen, 2011, pp 1944

    • Crossref
    • Search Google Scholar
    • Export Citation
  • 22

    Mez J, Daneshvar DH, Kiernan PT, Abdolmohammadi B, Alvarez VE, Huber BR, : Clinicopathological evaluation of chronic traumatic encephalopathy in players of American football. JAMA 318:360370, 2017

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 23

    Mullen SJ, Yücel YH, Cusimano M, Schweizer TA, Oentoro A, Gupta N: Saccadic eye movements in mild traumatic brain injury: a pilot study. Can J Neurol Sci 41:5865, 2014

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 24

    Nyffeler T, Müri RM, Bucher-Ottiger Y, Pierrot-Deseilligny C, Gaymard B, Rivaud-Pechoux S: Inhibitory control of the human dorsolateral prefrontal cortex during the anti-saccade paradigm—a transcranial magnetic stimulation study. Eur J Neurosci 26:13811385, 2007

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 25

    Ruff R: Two decades of advances in understanding of mild traumatic brain injury. J Head Trauma Rehabil 20:518, 2005

  • 26

    Rughani AI, Dumont TM, Lu Z, Bongard J, Horgan MA, Penar PL, : Use of an artificial neural network to predict head injury outcome. J Neurosurg 113:585590, 2010

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 27

    Shi HY, Hwang SL, Lee KT, Lin CL: In-hospital mortality after traumatic brain injury surgery: a nationwide population-based comparison of mortality predictors used in artificial neural network and logistic regression models. J Neurosurg 118:746752, 2013

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 28

    Sinha M, Kennedy CS, Ramundo ML: Artificial neural network predicts CT scan abnormalities in pediatric patients with closed head injury. J Trauma 50:308312, 2001

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 29

    Studerus-Germann AM, Thiran JP, Daducci A, Gautschi OP: Diagnostic approaches to predict persistent post-traumatic symptoms after mild traumatic brain injury—a literature review. Int J Neurosci 126:289298, 2016

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 30

    Ting WKC, Schweizer TA, Topolovec-Vranic J, Cusimano MD: Antisaccadic eye movements are correlated with corpus callosum white matter mean diffusivity, Stroop performance, and symptom burden in mild traumatic brain injury and concussion. Front Neurol 6:271, 2016

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 31

    Veeramuthu V, Narayanan V, Kuo TL, Delano-Wood L, Chinna K, Bondi MW, : Diffusion tensor imaging parameters in mild traumatic brain injury and its correlation with early neuropsychological impairment: a longitudinal study. J Neurotrauma 32:14971509, 2015

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 32

    Wetjen NM, Pichelmann MA, Atkinson JLD: Second impact syndrome: concussion and second injury brain complications. J Am Coll Surg 211:553557, 2010

  • 33

    Yuh EL, Cooper SR, Mukherjee P, Yue JK, Lingsma HF, Gordon WA, : Diffusion tensor imaging for outcome prediction in mild traumatic brain injury: a TRACK-TBI study. J Neurotrauma 31:14571477, 2014

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation

If the inline PDF is not rendering correctly, you can download the PDF file here.

Contributor Notes

Correspondence Alexander P. Landry: St. Michael’s Hospital, Toronto, ON, Canada. alex.landry@mail.utoronto.ca.

INCLUDE WHEN CITING Published online November 23, 2018; DOI: 10.3171/2018.6.JNS18607.

Disclosures The authors report no conflict of interest concerning the materials or methods used in this study or the findings specified in this paper.

  • View in gallery

    General configuration of a 3-layer MLP (single hidden layer) with binary output, which is a common configuration for clinical use. The input nodes (x) are on the left of the diagram, hidden nodes (h) in the middle, and output nodes (y) on the right. Each line is associated with a weight, which is adjusted during training. The arrow represents the unidirectional information flow from input to output. Figure is available in color online only.

  • View in gallery

    ROC curves for each saccade parameter used as a neural network input, with ANN performance results superimposed for (A) participant analysis and (B) trial analysis. Error bars represent standard deviation. Closed circles represent the multi-input neural network results from Table 2. Open circles represent results from neural networks trained with only the input variable being plotted, with all other parameters unchanged. Figure is available in color online only.

  • View in gallery

    Frequency distribution of (A) participant and (B) trial saccade parameters used as ANN inputs. K = kurtosis; ms = milliseconds; S = skewness. Figure is available in color online only.

  • 1

    Aoki Y, Inokuchi R: A voxel-based meta-analysis of diffusion tensor imaging in mild traumatic brain injury. Neurosci Biobehav Rev 66:119126, 2016

  • 2

    Asken BM, DeKosky ST, Clugston JR, Jaffee MS, Bauer RM: Diffusion tensor imaging (DTI) findings in adult civilian, military, and sport-related mild traumatic brain injury (mTBI): a systematic critical review. Brain Imaging Behav 12:585612, 2018

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 3

    Balaban C, Hoffer ME, Szczupak M, Snapp H, Crawford J, Murphy S, : Oculomotor, vestibular, and reaction time tests in mild traumatic brain injury. PLoS One 11:e0162168, 2016

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 4

    Baxt WG: Application of artificial neural networks to clinical medicine. Lancet 346:11351138, 1995

  • 5

    Bittencourt J, Velasques B, Teixeira S, Basile LF, Salles JI, Nardi AE, : Saccadic eye movement applications for psychiatric disorders. Neuropsychiatr Dis Treat 9:13931409, 2013

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 6

    Carroll LJ, Cassidy JD, Holm L, Kraus J, Coronado VG: Methodological issues and research recommendations for mild traumatic brain injury: the WHO Collaborating Centre Task Force on Mild Traumatic Brain Injury. J Rehabil Med 36 (43 Suppl):113125, 2004

    • Crossref
    • Search Google Scholar
    • Export Citation
  • 7

    Chong SL, Liu N, Barbier S, Ong MEH: Predictive modeling in pediatric traumatic brain injury using machine learning. BMC Med Res Methodol 15:22, 2015

  • 8

    Coe BC, Munoz DP: Mechanisms of saccade suppression revealed in the anti-saccade task. Philos Trans R Soc Lond B Biol Sci 372:20160192, 2017

  • 9

    Cross SS, Harrison RF, Kennedy RL: Introduction to neural networks. Lancet 346:10751079, 1995

  • 10

    Delouche A, Attyé A, Heck O, Grand S, Kastler A, Lamalle L, : Diffusion MRI: pitfalls, literature review and future directions of research in mild traumatic brain injury. Eur J Radiol 85:2530, 2016

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 11

    Forsström JJ, Dalton KJ: Artificial neural networks for decision support in clinical medicine. Ann Med 27:509517, 1995

  • 12

    Frieden TR, Houry D, Baldwin G: Traumatic Brain Injury in the United States: Epidemiology and Rehabilitation. Atlanta: CDC, 2015 (https://www.cdc.gov/traumaticbraininjury/pdf/tbi_report_to_congress_epi_and_rehab-a.pdf) [Accessed August 20, 2018]

    • Export Citation
  • 13

    Gerberding JL, Binder S: Report to Congress on Mild Traumatic Brain Injury in the United States: Steps to Prevent a Serious Public Health Problem. Atlanta: CDC, 2003 (https://www.cdc.gov/traumaticbraininjury/pdf/mtbireport-a.pdf) [Accessed August 20, 2018]

    • Export Citation
  • 14

    Gupta KC: Neural network structures, in Gupta KC, Zhang QJ (eds): Neural Networks for RF and Microwave Design. Boston: Artech House, 2000, pp 61103

    • Search Google Scholar
    • Export Citation
  • 15

    Heitger MH, Anderson TJ, Jones RD, Dalrymple-Alford JC, Frampton CM, Ardagh MW: Eye movement and visuomotor arm movement deficits following mild closed head injury. Brain 127:575590, 2004

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 16

    Heitger MH, Jones RD, Anderson TJ: A new approach to predicting postconcussion syndrome after mild traumatic brain injury based upon eye movement function, in Proceedings of the 30th Annual International Conference of the IEEE Engineering in Medicine and Biology Society. Piscataway, NJ: IEEE Engineering in Medicine and Biology Society, 2008, pp 35703573

    • Search Google Scholar
    • Export Citation
  • 17

    Heitger MH, Jones RD, Macleod AD, Snell DL, Frampton CM, Anderson TJ: Impaired eye movements in post-concussion syndrome indicate suboptimal brain function beyond the influence of depression, malingering or intellectual ability. Brain 132:28502870, 2009

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 18

    Hoffer ME, Balaban C, Szczupak M, Buskirk J, Snapp H, Crawford J, : The use of oculomotor, vestibular, and reaction time tests to assess mild traumatic brain injury (mTBI) over time. Laryngoscope Investig Otolaryngol 2:157165, 2017

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 19

    Hunt AW, Mah K, Reed N, Engel L, Keightley M: Oculomotor-based vision assessment in mild traumatic brain injury: a systematic review. J Head Trauma Rehabil 31:252261, 2016

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 20

    Johnson B, Zhang K, Hallett M: Functional neuroimaging of acute oculomotor deficits in concussed athletes. Brain Imaging Behav 9:564573, 2015

  • 21

    May R, Dandy G, Maier H: Review of input variable selection methods for artificial neural networks, in Suzuki K (ed): Artificial Neural Networks: Methodological Advances and Biomedical Applications. London: IntechOpen, 2011, pp 1944

    • Crossref
    • Search Google Scholar
    • Export Citation
  • 22

    Mez J, Daneshvar DH, Kiernan PT, Abdolmohammadi B, Alvarez VE, Huber BR, : Clinicopathological evaluation of chronic traumatic encephalopathy in players of American football. JAMA 318:360370, 2017

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 23

    Mullen SJ, Yücel YH, Cusimano M, Schweizer TA, Oentoro A, Gupta N: Saccadic eye movements in mild traumatic brain injury: a pilot study. Can J Neurol Sci 41:5865, 2014

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 24

    Nyffeler T, Müri RM, Bucher-Ottiger Y, Pierrot-Deseilligny C, Gaymard B, Rivaud-Pechoux S: Inhibitory control of the human dorsolateral prefrontal cortex during the anti-saccade paradigm—a transcranial magnetic stimulation study. Eur J Neurosci 26:13811385, 2007

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 25

    Ruff R: Two decades of advances in understanding of mild traumatic brain injury. J Head Trauma Rehabil 20:518, 2005

  • 26

    Rughani AI, Dumont TM, Lu Z, Bongard J, Horgan MA, Penar PL, : Use of an artificial neural network to predict head injury outcome. J Neurosurg 113:585590, 2010

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 27

    Shi HY, Hwang SL, Lee KT, Lin CL: In-hospital mortality after traumatic brain injury surgery: a nationwide population-based comparison of mortality predictors used in artificial neural network and logistic regression models. J Neurosurg 118:746752, 2013

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 28

    Sinha M, Kennedy CS, Ramundo ML: Artificial neural network predicts CT scan abnormalities in pediatric patients with closed head injury. J Trauma 50:308312, 2001

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 29

    Studerus-Germann AM, Thiran JP, Daducci A, Gautschi OP: Diagnostic approaches to predict persistent post-traumatic symptoms after mild traumatic brain injury—a literature review. Int J Neurosci 126:289298, 2016

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 30

    Ting WKC, Schweizer TA, Topolovec-Vranic J, Cusimano MD: Antisaccadic eye movements are correlated with corpus callosum white matter mean diffusivity, Stroop performance, and symptom burden in mild traumatic brain injury and concussion. Front Neurol 6:271, 2016

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 31

    Veeramuthu V, Narayanan V, Kuo TL, Delano-Wood L, Chinna K, Bondi MW, : Diffusion tensor imaging parameters in mild traumatic brain injury and its correlation with early neuropsychological impairment: a longitudinal study. J Neurotrauma 32:14971509, 2015

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 32

    Wetjen NM, Pichelmann MA, Atkinson JLD: Second impact syndrome: concussion and second injury brain complications. J Am Coll Surg 211:553557, 2010

  • 33

    Yuh EL, Cooper SR, Mukherjee P, Yue JK, Lingsma HF, Gordon WA, : Diffusion tensor imaging for outcome prediction in mild traumatic brain injury: a TRACK-TBI study. J Neurotrauma 31:14571477, 2014

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation

Metrics

All Time Past Year Past 30 Days
Abstract Views 241 103 0
Full Text Views 314 288 13
PDF Downloads 167 128 4
EPUB Downloads 0 0 0