Outcome prediction of intracranial aneurysm treatment by flow diverters using machine learning

Nikhil PaliwalDepartments of Mechanical & Aerospace Engineering,
Canon Stroke and Vascular Research Center, University at Buffalo, the State University of New York, Buffalo, New York

Search for other papers by Nikhil Paliwal in
jns
Google Scholar
PubMed
Close
 BS
,
Prakhar JaiswalDepartments of Mechanical & Aerospace Engineering,

Search for other papers by Prakhar Jaiswal in
jns
Google Scholar
PubMed
Close
 BS
,
Vincent M. TutinoCanon Stroke and Vascular Research Center, University at Buffalo, the State University of New York, Buffalo, New York
Biomedical Engineering, and

Search for other papers by Vincent M. Tutino in
jns
Google Scholar
PubMed
Close
 PhD
,
Hussain ShallwaniNeurosurgery,

Search for other papers by Hussain Shallwani in
jns
Google Scholar
PubMed
Close
 MD
,
Jason M. DaviesNeurosurgery,
Biomedical Informatics and

Search for other papers by Jason M. Davies in
jns
Google Scholar
PubMed
Close
 MD, PhD
,
Adnan H. SiddiquiCanon Stroke and Vascular Research Center, University at Buffalo, the State University of New York, Buffalo, New York
Neurosurgery,

Search for other papers by Adnan H. Siddiqui in
jns
Google Scholar
PubMed
Close
 MD, PhD
,
Rahul RaiDepartments of Mechanical & Aerospace Engineering,

Search for other papers by Rahul Rai in
jns
Google Scholar
PubMed
Close
 PhD
, and
Hui MengDepartments of Mechanical & Aerospace Engineering,
Biomedical Engineering, and

Search for other papers by Hui Meng in
jns
Google Scholar
PubMed
Close
 PhD
Free access

OBJECTIVE

Flow diverters (FDs) are designed to occlude intracranial aneurysms (IAs) while preserving flow to essential arteries. Incomplete occlusion exposes patients to risks of thromboembolic complications and rupture. A priori assessment of FD treatment outcome could enable treatment optimization leading to better outcomes. To that end, the authors applied image-based computational analysis to clinically FD-treated aneurysms to extract information regarding morphology, pre- and post-treatment hemodynamics, and FD-device characteristics and then used these parameters to train machine learning algorithms to predict 6-month clinical outcomes after FD treatment.

METHODS

Data were retrospectively collected for 84 FD-treated sidewall aneurysms in 80 patients. Based on 6-month angiographic outcomes, IAs were classified as occluded (n = 63) or residual (incomplete occlusion, n = 21). For each case, the authors modeled FD deployment using a fast virtual stenting algorithm and hemodynamics using image-based computational fluid dynamics. Sixteen morphological, hemodynamic, and FD-based parameters were calculated for each aneurysm. Aneurysms were randomly assigned to a training or testing cohort in approximately a 3:1 ratio. The Student t-test and Mann-Whitney U-test were performed on data from the training cohort to identify significant parameters distinguishing the occluded from residual groups. Predictive models were trained using 4 types of supervised machine learning algorithms: logistic regression (LR), support vector machine (SVM; linear and Gaussian kernels), K-nearest neighbor, and neural network (NN). In the testing cohort, the authors compared outcome prediction by each model trained using all parameters versus only the significant parameters.

RESULTS

The training cohort (n = 64) consisted of 48 occluded and 16 residual aneurysms and the testing cohort (n = 20) consisted of 15 occluded and 5 residual aneurysms. Significance tests yielded 2 morphological (ostium ratio and neck ratio) and 3 hemodynamic (pre-treatment inflow rate, post-treatment inflow rate, and post-treatment aneurysm averaged velocity) discriminants between the occluded (good-outcome) and the residual (bad-outcome) group. In both training and testing, all the models trained using all 16 parameters performed better than all the models trained using only the 5 significant parameters. Among the all-parameter models, NN (AUC = 0.967) performed the best during training, followed by LR and linear SVM (AUC = 0.941 and 0.914, respectively). During testing, NN and Gaussian-SVM models had the highest accuracy (90%) in predicting occlusion outcome.

CONCLUSIONS

NN and Gaussian-SVM models incorporating all 16 morphological, hemodynamic, and FD-related parameters predicted 6-month occlusion outcome of FD treatment with 90% accuracy. More robust models using the computational workflow and machine learning could be trained on larger patient databases toward clinical use in patient-specific treatment planning and optimization.

ABBREVIATIONS

AR = aspect ratio; AUC = area under the ROC curve; AV = averaged velocity; CFD = computational fluid dynamics; DSA = digital subtraction angiography; FD = flow diverter; IA = intracranial aneurysm; ICA = internal carotid artery; IR = inflow rate; K-NN = K-nearest neighbor; LR = logistic regression; MCR = metal coverage rate; ML = machine learning; ND = neck diameter; NN = neural network; NR = neck ratio; OsR = ostium ratio; PD = pore density; PED = Pipeline embolization device; ROC = receiver operating characteristic; SE = standard error; SHR = shear rate; SR = size ratio; SVM = support vector machine; TT = turnover time.

OBJECTIVE

Flow diverters (FDs) are designed to occlude intracranial aneurysms (IAs) while preserving flow to essential arteries. Incomplete occlusion exposes patients to risks of thromboembolic complications and rupture. A priori assessment of FD treatment outcome could enable treatment optimization leading to better outcomes. To that end, the authors applied image-based computational analysis to clinically FD-treated aneurysms to extract information regarding morphology, pre- and post-treatment hemodynamics, and FD-device characteristics and then used these parameters to train machine learning algorithms to predict 6-month clinical outcomes after FD treatment.

METHODS

Data were retrospectively collected for 84 FD-treated sidewall aneurysms in 80 patients. Based on 6-month angiographic outcomes, IAs were classified as occluded (n = 63) or residual (incomplete occlusion, n = 21). For each case, the authors modeled FD deployment using a fast virtual stenting algorithm and hemodynamics using image-based computational fluid dynamics. Sixteen morphological, hemodynamic, and FD-based parameters were calculated for each aneurysm. Aneurysms were randomly assigned to a training or testing cohort in approximately a 3:1 ratio. The Student t-test and Mann-Whitney U-test were performed on data from the training cohort to identify significant parameters distinguishing the occluded from residual groups. Predictive models were trained using 4 types of supervised machine learning algorithms: logistic regression (LR), support vector machine (SVM; linear and Gaussian kernels), K-nearest neighbor, and neural network (NN). In the testing cohort, the authors compared outcome prediction by each model trained using all parameters versus only the significant parameters.

RESULTS

The training cohort (n = 64) consisted of 48 occluded and 16 residual aneurysms and the testing cohort (n = 20) consisted of 15 occluded and 5 residual aneurysms. Significance tests yielded 2 morphological (ostium ratio and neck ratio) and 3 hemodynamic (pre-treatment inflow rate, post-treatment inflow rate, and post-treatment aneurysm averaged velocity) discriminants between the occluded (good-outcome) and the residual (bad-outcome) group. In both training and testing, all the models trained using all 16 parameters performed better than all the models trained using only the 5 significant parameters. Among the all-parameter models, NN (AUC = 0.967) performed the best during training, followed by LR and linear SVM (AUC = 0.941 and 0.914, respectively). During testing, NN and Gaussian-SVM models had the highest accuracy (90%) in predicting occlusion outcome.

CONCLUSIONS

NN and Gaussian-SVM models incorporating all 16 morphological, hemodynamic, and FD-related parameters predicted 6-month occlusion outcome of FD treatment with 90% accuracy. More robust models using the computational workflow and machine learning could be trained on larger patient databases toward clinical use in patient-specific treatment planning and optimization.

In endovascular treatment of intracranial aneurysms (IAs), flow diverters (FDs) have emerged as an alternative paradigm to coil embolization, particularly in treating wide-neck and challenging aneurysm morphologies.6,28 Deployed across an aneurysm ostium, the densely woven mesh of FD induces flow stasis in the aneurysmal sac, promoting thrombotic conditions and eventual occlusion of the IA. The mesh-like structure of FDs also facilitates endoluminal reconstruction of the parent artery.18,34 Over the past few years, FDs have been one of the mainstays of endovascular intervention. However, despite the effectiveness of FDs in many cases, approximately 25% of FD-treated IAs fail to reach complete occlusion even after 6 months.6,15 In these cases, the FD-treated patients who experience persistent residual filling in the aneurysm sac are at risk for thromboembolic complications and aneurysm rupture.16,32 A priori assessment of FD treatment outcomes could aid clinicians in treatment optimization and lead to better outcomes.

For coil embolization, recanalization has been shown to correlate with aneurysm morphometrics such as size, neck-to-dome ratio, and neck size, which have been used to gauge coil treatment outcome.11 However, for FD-treated IAs, these metrics have not been found to correlate with occlusion outcome.3,6 Instead, other morphological and hemodynamic metrics have been proposed specifically for FDs that correlate with occlusion outcome. Gentric et al.12 showed that a large aneurysm ostium is associated with incomplete occlusion after FD treatment. Mut et al.21 demonstrated that pre- and post-treatment inflow rates, post-treatment aneurysm averaged velocity, and the post-treatment shear rate were significantly different between occluded and nonoccluded IAs after 6 months of FD treatment. However, it remains unclear if these parameters can predict the FD treatment outcome.

In order to develop models for predicting clinical outcome of FD-treated IAs, we surveyed potential candidate algorithms. In IA research, multivariate logistic regression analysis of untreated morphological and hemodynamic parameters has been used to classify aneurysm rupture status.39 In other areas of medical research, novel machine learning (ML) algorithms have emerged as alternatives to traditional statistical methods to predict clinical outcomes, e.g., using medical imaging data to classify brain tumors and heart diseases.2,31,36 Unlike univariate statistical analysis that focuses on identifying the independently significant differences between averages of parameters in different populations, ML algorithms use given features of the available data on a case-by-case basis to predict an outcome.2,19,22,27 Furthermore, studies have also shown that ML algorithms could extract nonintuitive linear and nonlinear combinations of parameters, which themselves may be insignificant in univariate statistical analyses.22

In this study we explored 4 ML algorithms and compared their performances for use in training predictive models for FD treatment outcome. These models included not only the most recent algorithms, such as support vector machine (SVM), K-nearest neighbor (K-NN), and neural network (NN), but also traditional multivariate logistic regression. The purpose was to establish a proof of concept for mining bigger data toward building models to potentially aid clinicians in a priori treatment planning and optimization.

Methods

Patient Selection

Data for patients treated using the commercial FD Pipeline embolization device (PED) (Medtronic) at the Gates Vascular Institute between 2009 and 2017 were retrospectively collected for this study. The inclusion criteria were the presence of a sidewall aneurysm located at the internal carotid artery (ICA) and treated using a single PED and availability of 6-month follow-up angiographic images and pre-treatment 3D rotational digital subtraction angiography (DSA) images Aneurysms with prior treatment and those with insufficient quality of pre-treatment DSA were excluded from this study. Approval for the collection and review of patient data was obtained from the institutional review board at the University at Buffalo. Patient consent was waived by the board due to the retrospective use of de-identified data. Pre-treatment 3D DSA images, 6-month clinical outcome data, and general clinical and demographic data were collected for cases that satisfied the inclusion criteria. Based on their angiographic outcome at 6-month follow-up, aneurysms were dichotomized as occluded (complete occlusion) or residual (contrast filling at the neck/dome). Aneurysms were randomly assigned to either the training or the testing cohort in approximately a 3:1 ratio, keeping the ratio of occluded to residual cases equal in the 2 cohorts.

Virtual FD Deployment and Computational Fluid Dynamics Setup

The 3D DSA images of FD-treated IAs obtained before treatment were segmented using an open-source software package vascular modeling tool kit (vmtk, www.vmtk.org)1 to obtain surface representation of the vascular geometry of each aneurysm. We could not directly obtain the deployed FDs in post-treatment images due to lack of adequate imaging resolution and artifacts. Therefore, to deploy the FD device in the computational model for each aneurysm, we used our previously reported virtual stenting workflow.25,26 Details of the workflow are provided in the Supplementary Material. To accurately represent the patient-specific treatment, the actual specifications of the FD device (diameter and length) were imported into the workflow. To obtain hemodynamic information, we ran image-based computational fluid dynamics (CFD) simulations. For each aneurysm, two simulations were performed: untreated and treated with the modeled FD. Technical details of the CFD simulation setup are provided in the Supplementary Material.

Parameters Calculated: Morphological, Hemodynamic, and FD-Related

From 3D IA models, virtual FD deployment, and untreated and treated CFD simulations, we calculated morphological, FD-related, and hemodynamic parameters for each aneurysm. Previously studied morphology metrics,10 including aneurysm size, neck diameter (ND), size ratio (SR), aspect ratio (AR), and the novel morphometrics neck ratio (NR) and ostium ratio (OsR), were calculated. NR is defined as the ratio of the clinical aneurysm neck diameter to the parent vessel diameter, and OsR is defined as the ratio of the surface area of the aneurysmal ostium surface to the remaining circumferential surface area of the parent artery. Size, ND, SR, and AR were calculated on the 3D IA models using the integrated clinical software AView;37 NR was calculated on the pre-treatment 2D DSA images of patients; and OsR was calculated on the 3D surface IA models using a standalone MATLAB workflow.

FD-related parameters, metal coverage rate (MCR), and pore density (PD) were quantified based on virtual FD deployment results on patient IA models. MCR and PD were calculated across the aneurysm orifice with the deployed FD visible to capture its deployment at the neck.9 MCR quantifies the relative area covered by the FD struts at the neck as opposed to open space (pores), and PD quantifies the number of pores per unit area.17 Virtual FD deployment images of each patient at the aneurysmal neck were obtained in STAR-CCM+, and a standalone MATLAB code was written to quantify the MCR and PD from the deployment images.

From CFD simulation results, time-averaged flow parameters were quantified to measure the effect of FDs on aneurysm hemodynamics. To quantify the pre- and post-FD intra-aneurysmal flow activity, magnitudes of aneurysm averaged velocity (AV) and shear rate (SHR) were volume-averaged inside the aneurysm sac. Flow stasis was quantified by aneurysm inflow rate (IR) and turnover time (TT), defined as the aneurysm sac volume divided by the inflow rate at the neck plane. Increasing aneurysmal flow turnover time can accelerate blood clotting and thrombotic occlusion of the aneurysms.17 These hemodynamic parameters have previously shown association with the occlusion outcome of FD-treated IAs.21,38 The subscripts “pre” and “post” were used to distinguish between the untreated and treated hemodynamic values for each parameter.

A total of 16 parameters were calculated for each aneurysm, including 6 morphology-based parameters—size, ND, SR, AR, NR, and OsR; 2 FD-related parameters—MCR and PD; and 8 hemodynamic-based parameters—AVpre, AVpost, SHRpre, SHRpost, IRpre, IRpost, TTpre, and TTpost.

Statistical Analysis

Statistical analysis was performed on the patient clinical, demographic, morphological, FD-related, and hemodynamic parameters in the training cohort to identify those that differed significantly between the occluded and residual groups. A Shapiro-Wilk test was performed to check for normality of the continuous variables. Differences in parameters between the 2 groups were tested using the Mann-Whitney U-test (for nonnormally distributed data) or Student t-test (for normally distributed data). For categorical variables, a chi-square test was used to test for significant differences between the groups. Statistical significance was defined as p < 0.05. All continuous parameter values were subsequently expressed as mean ± standard error (SE). Before training of the ML models, values of each parameter in the testing and training cohorts were normalized to have a mean value of zero and a standard deviation of 1.

Machine Learning Algorithms

Supervised ML algorithms with binary classification were used to build predictive models. Four ML algorithms were selected for model building since these have shown good performance in clinical healthcare classification studies.20,30,31,35,36 The models included the standard statistical logistic regression (LR), support vector machine (SVM; with linear [L-SVM] and Gaussian [G-SVM] kernels), K-nearest neighbor (K-NN), and neural network (NN). An illustration of the concept of each algorithm is shown in Fig. 1. As shown in Fig. 1, LR uses a linear classification line to separate the two groups, SVM uses either a linear (L-SVM) or nonlinear (Gaussian, G-SVM) kernel to identify the hyperplane that maximizes the distance between the two groups, K-NN performs predictions on a new data point based on its euclidean distance from its “K” neighbors, and NN uses a system of interconnected layers that use back-propagation while training to generate nonintuitive combinations of parameters to optimize the classification model. Descriptions of each algorithm are provided in the Supplementary Material. Each algorithm was used to train two sets of predictive models: 1) using all 16 parameters as input, and 2) using only the statistically significant parameters.

Fig. 1.
Fig. 1.

Illustration of different ML classification algorithms. A: Logistic regression, where a classification line is fitted on the data. B: Support vector machine, where the best hyperplane that separates the data is identified by maximizing the margins on either side. C: K-nearest neighbor, where the predictions of a new point are made based on its distance from the points in the exiting database. D: Neural network with 2 hidden layers, where a system of interconnected neurons uses back-propagation to learn from the training data.

In-house codes were written to normalize the parameter data and train the predictive models for LR, L-SVM, G-SVM, and K-NN in MATLAB (v9.3, R2017, MathWorks) using the Machine Learning Toolbox. For the NN algorithm, Python 3.6 code was developed using the open-source Tensorflow (v1.4.1, Google) and Keras (v2.0.8, https://keras.io/) libraries. A 4-fold cross-validation was used during model training to avoid overfitting on the training cohort.

Training and Testing Accuracy Estimation

Area under the receiver operating characteristic (ROC) curve (AUC) and 95% confidence intervals were quantified to assess the performance of the models on the training cohort for each algorithm.13 The predictive performance of each model was quantified by its accuracy on the independent testing cohort. The flowchart of the model training and testing for all models is shown in Fig. 2.

Fig. 2.
Fig. 2.

Flowchart for building and testing the predictive models for occlusion outcome of FD-treated IAs. After extraction of morphological, FD-related, and hemodynamic parameters, patients are randomly divided into the training and testing cohorts. Two sets of models are then trained using 1) all parameters and 2) significant parameters on the training cohort. The predictive performances of these models are then tested on the testing cohort. Tx = treatment.

Results

Patient Population and Statistical Analysis

Based on the inclusion criteria, 84 IAs in 80 patients were enrolled in this study. At 6-month follow-up, 63 aneurysms had complete occlusion (occluded), whereas 21 aneurysms had residual contrast filling (residual). The clinical and demographic information of patients in each group is listed in Table 1. The patients in the occluded group had a mean age (± SE) of 56.7 ± 1.8 years; this group included 51 females, 20 patients with hypertension, and 23 patients who were current smokers. The residual group had an average age of 58.5 ± 2.8 years and included 19 females, 10 patients with hypertension, and 14 patients who were current smokers. There were no significant differences in age, sex, hypertension, and smoking status between the 2 groups. Upon randomization, the training cohort included 64 aneurysms (48 occluded and 16 residual), with 20 aneurysms reserved to the testing cohort (15 occluded and 5 residual).

TABLE 1.

Summary of characteristics of patients in the occluded and residual aneurysm groups

CharacteristicOccluded (n = 63)Residual (n = 21)p Value
Mean age (yrs)56.7 ± 1.858.5 ± 2.80.53
Female sex51190.31
Hypertension20100.49
Smoking23140.39

Values are numbers of patients unless otherwise indicated. Mean values are presented with SEs.

Table 2 lists the mean and standard error for morphology, FD-device, and hemodynamic-based parameters in the 2 groups in the training cohort, with their respective p values. Mean values of all morphological parameters were higher in the residual group except AR. However, only NR and OsR were statistically different (p = 0.01 and p < 0.001, respectively) between the groups. Both FD-related parameters, MCR and PD, were higher in the occluded group than the residual group, but the differences were not statistically significant. In terms of pre-treatment hemodynamic parameters, aneurysms that occluded within 6-months had lower mean values of AVpre, SHRpre, and IRpre, and higher mean values of TTpre compared to those that did not occlude. However, only IRpre was significantly different between the 2 groups (p = 0.04). Similar to the pre-treatment hemodynamic parameters, the mean values of AVpost, SHRpost, and IRpost were lower and the mean value of TTpost was higher in the occluded group than in the residual group, with AVpost and IRpost showing statistically significant differences between the two groups (p = 0.02 and p = 0.02, respectively). NR, OsR, IRpre, AVpost, and IRpost had statistically significant differences between the 2 groups and were used for training models with significant parameters only; all 16 parameters were used in training all-parameter models.

TABLE 2.

Statistical analysis of morphological, FD, and hemodynamic parameters in the training cohort

ParameterOccluded (n = 48)Residual (n = 16)p Value
Aneurysm morphology
 Size (mm)4.99 ± 0.595.04 ± 0.890.71
 ND (mm)4.42 ± 0.235.45 ± 0.680.10
 SR1.32 ± 0.181.52 ± 0.360.47
 AR0.98 ± 0.090.97 ± 0.150.93
 NR1.17 ± 0.051.51 ± 0.160.01*
 OsR0.36 ± 0.010.55 ± 0.04<0.001*
FD-device characteristics
 MCR (%)27.84 ± 0.6027.54 ± 1.050.80
 PD (1/mm2)29.60 ± 1.1127.03 ± 1.640.20
Pre-treatment hemodynamic parameters
  AVpre (cm/sec)6.86 ± 0.7411.67 ± 2.750.07
  SHRpre (1/sec)177.45 ± 20.68304.56 ± 84.580.22
  IRpre (ml/sec)0.51 ± 0.090.87 ± 0.250.04*
  TTpre (sec)0.29 ± 0.090.16 ± 0.040.64
Post-treatment hemodynamic parameters
  AVpost (cm/sec)1.97 ± 0.305.55 ± 1.730.02*
  SHRpost (1/sec)52.14 ± 9.08150.27 ± 54.670.08
  IRpost (ml/sec)0.20 ± 0.030.51 ± 0.170.02*
  TTpost (sec)0.74 ± 0.210.41 ± 0.100.40

AR = aspect ratio; AV = aneurysm averaged velocity; IR = inflow rate; MCR = metal coverage rate; ND = neck diameter; NR = neck ratio; OsR = ostium ratio; PD = pore density; SHR = shear rate; SR = size ratio; TT = turnover time.

Data are presented as mean ± SE. Subscripted “pre” and “post” indicate pre-treatment and post-treatment hemodynamic values.

Statistically significant (p < 0.05).

Final Converged Models

Through training, we obtained the final converged models for all algorithms. The training loss functions for the LR, L-SVM, G-SVM, K-NN, and NN models trained using all parameters were 0.10, 0.125, 0.156, 0.171, and 0.10, respectively. For the LR, L-SVM, G-SVM, K-NN, and NN models trained with significant parameters, the loss functions were 0.121, 0.156, 0.140, 0.203, and 0.214, respectively.

Model Performance on the Training Cohort

To compare the use of significant parameters versus all parameters on the performance of ML models on the training cohort, ROC analysis was performed for all models. As shown in Fig. 3, all-parameter models had better performance for each algorithm compared to significant-parameter models. Among all-parameter models, NN had the highest training performance (AUC = 0.967) followed by LR (AUC = 0.941), whereas G-SVM model had the lowest AUC (0.841). However, among significant-parameter models, K-NN had the highest performance (AUC = 0.875) followed by the NN model (AUC = 0.854).

Fig. 3.
Fig. 3.

ROC curves for each ML model on the training cohort built using (A) all the parameters and (B) significant parameters only. AUC = area under the ROC curve; G-SVM = Gaussian support vector machine; K-NN = K-nearest neighbor; LR = logistic regression; L-SVM = linear support vector machine; NN = neural network.

Model Accuracy in the Testing Cohort

The predictive accuracy of each model in the testing cohort is shown in Table 3. Overall, models with significant parameters as input had lower accuracy in the testing cohort compared to the models trained using all parameters. The all-parameter models’ predictive accuracy ranged from 85% to 90%, with LR, L-SVM, and K-NN having 85% and G-SVM and NN having 90% prediction accuracies. On the other hand, the significant-parameter models’ accuracy ranged from 55% to 75%, with LR, L-SVM, and G-SVM having the highest accuracy of 75%.

To further analyze the predictive performance of all-parameter models in the testing cohort, individual predictions on 20 IAs in the testing cohort were plotted in a confusion matrix in Fig. 4. The confusion matrix plots the number of correct and incorrect model-predicted outcomes (horizontal axis) against the actual outcomes (vertical axis) for each aneurysm. The cells with correct and incorrect predictions were shaded green and red, respectively. The most accurate NN model had 2 incorrect predictions, where 1 residual IA was predicted as occluded and 1 occluded IA predicted as residual. The predicted versus actual outcomes of other models are also shown in Fig. 4.

TABLE 3.

Accuracy of all-parameter and significant-parameter models in the testing cohort

ML AlgorithmAccuracy
All-Parameter ModelSignificant-Parameter Model 
Logistic regression85%75%
Linear SVM85%75%
Gaussian SVM90%75%
K-nearest neighbor85%70%
Neural network90%55%
Fig. 4.
Fig. 4.

Confusion matrix for predictions of ML models (trained using all parameters) on the testing cohort. The vertical axis represents the actual clinical outcome and the horizontal axis represents the model-predicted outcome. Green indicates correct predictions; red indicates incorrect predictions.

Discussion

In endovascular intervention of IAs by flow diverters, clinicians use pre-treatment and immediate post-treatment DSA images of aneurysms to assess the flow stasis induced by the FD.3,6 However, these images do not provide enough information to assess the long-term outcome of the IA healing. For illustration purposes, Fig. 5 shows two representative FD-treated ICA aneurysms from our cohort, with pre-treatment DSA shown in the left panel and 6-month follow-up DSA in the far right panel. Although the FD placement was successful in both cases, their outcomes at 6 months were quite different: the top aneurysm was completely occluded (red circle), while the bottom aneurysm had significant residual filling. This example highlights the fact that despite the successful FD placement, a treatment might be ultimately unsuccessful due to persistent filling (i.e., IA is not occluded over a long period of time), which exposes the patient to the risk of thromboembolic complications and rupture.16,32 We believe that an ability to predict potential failure of FD treatment prior to the intended intervention will improve treatment planning and thus minimize complications and optimize outcomes.

Fig. 5.
Fig. 5.

Concept of the computational analysis methodology for generating clinically practical predictive models for FD-treated IAs. The left panel shows 2 representative patient-specific ICA aneurysms, which were treated using a single FD. The far right panel shows the 6-month occlusion outcome of both cases: the aneurysm in the upper images was completely occluded (red circle), whereas the aneurysm in the lower images had persistent residual filling. The middle panel shows the computational analysis workflow, which uses pre-treatment 3D DSA images to generate a 3D aneurysm model, and then models the FD deployment and hemodynamics using virtual stenting and CFD, respectively. The morphological, hemodynamic, and candidate FD characteristics are used as features to train machine learning algorithms that can predict the 6-month occlusion outcome of FD-treated IAs. Note that the image of neural network is shown as a representative machine learning algorithm for predictive model building.

To that end, we have developed a computational analysis workflow that extracts information from the pre-treatment 3D DSA to potentially predict the treatment outcome. Conceptually, this computational workflow is an extension of the pre-treatment 3D DSA (middle panel in Fig. 5), which extracts pertinent features that include aneurysm morphology, pre- and post-treatment hemodynamics, and characteristics of the candidate FD device. These features lend themselves to building ML models that could predict long-term outcome when trained on a large number of retrospective FD-treated IA cases.

As a proof of concept of the proposed methodology, we retrospectively collected 84 FD-treated ICA aneurysms from our center. We applied the computational workflow to extract 16 features of aneurysm morphology, FD characteristics, and pre- and post-treatment hemodynamics for each IA. We then trained ML algorithms on 63 FD-treated IAs (training cohort) based on these features and produced models that were 85%–90% accurate in predicting the 6-month occlusion outcome in an independent testing cohort, including correct predictions of the 2 example cases shown in Fig. 5. Application of this methodology to larger databases could generate, as well as validate, more robust predictive models, which could potentially help in assessing the outcome of FD treatment a priori.

It has been argued in the literature that relevant features must be carefully selected for training ML models, especially for larger datasets.5 The reason for this selectivity is that including irrelevant features might incur extra computational cost during training, with little contribution to enhance the performance of the models.5 However, since our feature space is not very large (16 features), we asked if the so-called “irrelevant” features should be discarded. Therefore, we first performed univariate statistical analysis of significant parameters (could be considered as relevant) and insignificant parameters (could be slated as irrelevant). Then, we trained two sets of models, one using all parameters and one with significant parameters only and compared their performance to evaluate whether the inclusion of insignificant parameters helped to increase the model performance. Comparison results showed that all models trained with all 16 parameters had better accuracy (≥ 85%) than all models trained with only 5 significant ones (≤ 75%), and for each ML algorithm, all-parameter models outperformed their significant-parameter counterparts. This indicates that the nonsignificant parameters are not truly irrelevant; they could be relevant through nonintuitive combinations. However, we cannot determine if that was the case due to the small sample size of the training cohort. Studies of a larger patient database are required to elucidate the role of irrelevant parameters in predictive model building.

The healing mechanism of FD-treated aneurysms is currently poorly understood.8,21,23,24,38 Based on our statistical results, we found that pre- and post-treatment inflow rate, post-treatment aneurysm averaged velocity, ostium ratio, and neck ratio were significantly higher in the residual group. These findings could provide some insight into the reason why some FD-treated IAs do not heal. FD intervention aims at diverting the flow away from the IA sac, inducing flow stasis and eventual thrombotic occlusion of the IA. Larger post-treatment inflow rate in the residual group indicates that the FD implantation did not divert enough flow away from the IAs in this group, resulting in ineffective flow diversion. This flow condition could lead to constant replenishment of fresh blood into the IA, which could be the reason for the persistent residual filling into the IA sac even after 6 months in the residual group. Interestingly, we also found the pre-treatment inflow rate to be significantly higher for aneurysms in the residual group compared to aneurysms in the occluded group. For these IAs, FD placement evidently cannot provide sufficient flow diversion away from the IA sac to ensure the long-term thrombotic occlusion of the IAs. Even more interestingly, the residual group also had significantly higher ostium and neck ratios. This means that a larger portion of the parent vessel was the opening into the aneurysms in this group, which allowed more flow into the aneurysmal sac (unpublished data). In these cases, use of alternative strategies like overlapping FDs, compacted FDs, or FDs with adjunctive coils could be more beneficial for successful outcome, namely occlusion.9,38 In the ideal case, our models will identify IAs that may not occlude with a single, uniformly implanted FD, promoting the interventionalist to consider these alternative methods.

To find the best predictive models, we have also compared the performance of 4 different ML algorithms for FD treatment outcome prediction, including LR, SVM, K-NN, and NN. Our results show that NN and G-SVM (90% accuracy) performed slightly better than LR, L-SVM, and K-NN (85% accuracy). The almost similar performances of all of the algorithms suggest that a larger dataset is required to identify the best algorithm among these.

A common concern with training predictive models is overfitting to the training cohort when there are a large number of variables considered. However, we took steps to mitigate this risk, through both 4-fold cross-validation and holding out a test set, to which the models were naïve.29 The models trained using all parameters performed well in training, as well as on the independent testing cohort, confirming that our models are generalizable.

Limitations

This study had the following limitations. First, the CFD had to make simplifying assumptions, like assumed inlet flow waveforms, because patient-specific waveforms were not available. Second, our sample size was small and limited to a single center. Future multicenter collaborations could help generate more robust predictive models. Third, the parameters used in our study were entirely derived based on findings from previous studies, which do not exhaustively represent the relevant morphological, device, and hemodynamic features in FD healing. Fourth, this study is focused on sidewall aneurysms located at the ICA only, and the derived models may not be applicable for FD treatment of IAs at other locations. Last, we did not include patients’ clinical comorbidities and medications in our study, and these could influence the occlusion of FD-treated IAs. Future studies should include clinical comorbidities and medications as input parameters for training the predictive models.

Conclusions

We used ML algorithms to build predictive models for occlusion outcomes of FD-treated sidewall IAs located at the ICA. Our results show that models incorporating all 16 investigated morphological, hemodynamic, and FD-related parameters performed best, with 90% predictive accuracy in an independent testing cohort. The implications are 2-fold: 1) ML algorithms that use all parameters, not just the statistically significant ones, may be utilizing some underlying morphological factors or hemodynamic factors for better outcome classification. 2) Using the computational analysis on larger patient-specific databases, future studies to build robust predictive ML-based models are necessary to allow clinicians to better plan and triage for appropriate treatment of IAs.

Acknowledgments

This work was supported by the National Institutes of Health grant R01-NS-091075 and Canon Medical System Corporation. V.M.T. was supported by the Brain Aneurysm Foundation (Carol W. Harvey Chair of Research Grant, Dawn Brejcha Chair of Research Grant, and Robert Southerland Chair of Research Grant). J.M.D. was supported by the National Center for Advancing Translational Sciences of the National Institutes of Health under award number KL2TR001413 to the University at Buffalo.

Disclosures

V.M.T.—Cofounder: Neurovascular Diagnostics, Inc.

J.M.D.—Research grant: National Center for Advancing Translational Sciences of the National Institutes of Health under award number KL2TR001413 to the University at Buffalo. Speakers’ bureau: Penumbra. Honoraria: Neurotrauma Science.

A.H.S.—Current research grant: Co-investigator: NIH/NINDS 1R01NS091075 Virtual Intervention of Intracranial Aneurysms. Financial interest/investor/stock options/ownership: Amnis Therapeutics, Apama Medical, BlinkTBI, Inc., Buffalo Technology Partners, Inc., Cardinal Health, Cerebrotech Medical Systems, Inc., Claret Medical, Cognition Medical, Endostream Medical, Ltd., Imperative Care, International Medical Distribution Partners, Rebound Therapeutics Corp., Serenity Medical, Inc., Silk Road Medical, StimMed, Synchron, Three Rivers Medical, Inc., Viseon Spine, Inc. Consultant/advisory board: Amnis Therapeutics, Boston Scientific, Canon Medical Systems USA, Inc., Cerebrotech Medical Systems, Inc., Cerenovus, Claret Medical, Corindus, Inc., Endostream Medical, Ltd., Guidepoint Global Consulting, Imperative Care, Integra, Medtronic, MicroVention, Northwest University–DSMB Chair for HEAT Trial, Penumbra, Rapid Medical, Rebound Therapeutics Corp., Serenity Medical, Inc., Silk Road Medical, StimMed, Stryker, Three Rivers Medical, Inc., VasSol, W.L. Gore & Associates. National PI/steering committees: Cerenovus LARGE Trial and ARISE II Trial, Medtronic SWIFT PRIME and SWIFT DIRECT Trials, MicroVention FRED Trial and CONFIDENCE Study, MUSC POSITIVE Trial, Penumbra 3D Separator Trial, COMPASS Trial, INVEST Trial.

H.M.—Principal investigator: NIH grants R01-NS-091075 and R03-NS-090193. Grant support: Canon Medical Systems. Cofounder: Neurovascular Diagnostics, Inc.

Author Contributions

Conception and design: Meng, Paliwal. Acquisition of data: Meng, Paliwal, Shallwani, Davies, Siddiqui. Analysis and interpretation of data: all authors. Drafting the article: all authors. Critically revising the article: all authors. Approved the final version of the manuscript on behalf of all authors: Meng. Statistical analysis: Meng, Paliwal, Jaiswal, Rai. Study supervision: Meng.

Supplemental Information

Online-Only Content

Supplemental material is available online.

References

  • 1

    Antiga L, Piccinelli M, Botti L, Ene-Iordache B, Remuzzi A, Steinman DA: An image-based modeling framework for patient-specific computational hemodynamics. Med Biol Eng Comput 46:10971112, 2008

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 2

    Austin PC, Tu JV, Ho JE, Levy D, Lee DS: Using methods from the data-mining and machine-learning literature for disease classification and prediction: a case study examining classification of heart failure subtypes. J Clin Epidemiol 66:398407, 2013

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 3

    Becske T, Kallmes DF, Saatci I, McDougall CG, Szikora I, Lanzino G, et al.: Pipeline for uncoilable or failed aneurysms: results from a multicenter clinical trial. Radiology 267:858868, 2013

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 4

    Berg P, Iosif C, Ponsonnard S, Yardin C, Janiga G, Mounayer C: Endothelialization of over- and undersized flow-diverter stents at covered vessel side branches: an in vivo and in silico study. J Biomech 49:412, 2016

    • Crossref
    • Search Google Scholar
    • Export Citation
  • 5

    Blum AL, Langley P: Selection of relevant features and examples in machine learning. Artif Intell 97:245271, 1997

  • 6

    Brinjikji W, Murad MH, Lanzino G, Cloft HJ, Kallmes DF: Endovascular treatment of intracranial aneurysms with flow diverters: a meta-analysis. Stroke 44:442447, 2013

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 7

    Cebral JR, Castro MA, Putman CM, Alperin N: Flow-area relationship in internal carotid and vertebral arteries. Physiol Meas 29:585594, 2008

  • 8

    Chung B, Mut F, Kadirvel R, Lingineni R, Kallmes DF, Cebral JR: Hemodynamic analysis of fast and slow aneurysm occlusions by flow diversion in rabbits. J Neurointerv Surg 7:931935, 2015

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 9

    Damiano RJ, Tutino VM, Paliwal N, Ma D, Davies JM, Siddiqui AH, et al.: Compacting a single flow diverter versus overlapping flow diverters for intracranial aneurysms: a computational study. AJNR Am J Neuroradiol 38:603610, 2017

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 10

    Dhar S, Tremmel M, Mocco J, Kim M, Yamamoto J, Siddiqui AH, et al.: Morphology parameters for intracranial aneurysm rupture risk assessment. Neurosurgery 63:185197, 2008

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 11

    Fernandez Zubillaga A, Guglielmi G, Viñuela F, Duckwiler GR: Endovascular occlusion of intracranial aneurysms with electrically detachable coils: correlation of aneurysm neck size and treatment results. AJNR Am J Neuroradiol 15:815820, 1994

    • Search Google Scholar
    • Export Citation
  • 12

    Gentric JC, Darsaut TE, Makoyeva A, Salazkin I, Raymond J: The success of flow diversion in large and giant sidewall aneurysms may depend on the size of the defect in the parent artery. AJNR Am J Neuroradiol 35:21192124, 2014

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 13

    Hanley JA, McNeil BJ: The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology 143:29 36, 1982

  • 14

    Kadirvel R, Ding YH, Dai D, Rezek I, Lewis DA, Kallmes DF: Cellular mechanisms of aneurysm occlusion after treatment with a flow diverter. Radiology 270:394399, 2014

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 15

    Kallmes DF, Brinjikji W, Cekirge S, Fiorella D, Hanel RA, Jabbour P, et al.: Safety and efficacy of the Pipeline embolization device for treatment of intracranial aneurysms: a pooled analysis of 3 large studies. J Neurosurg 127:775780, 2017

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 16

    Kulcsár Z, Houdart E, Bonafé A, Parker G, Millar J, Goddard AJP, et al.: Intra-aneurysmal thrombosis as a possible cause of delayed aneurysm rupture after flow-diversion treatment. AJNR Am J Neuroradiol 32:2025, 2011

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 17

    Lieber BB, Sadasivan C: Endoluminal scaffolds for vascular reconstruction and exclusion of aneurysms from the cerebral circulation. Stroke 41 (10 Suppl):S21S25, 2010

    • Search Google Scholar
    • Export Citation
  • 18

    Lylyk P, Miranda C, Ceratto R, Ferrario A, Scrivano E, Luna HR, et al.: Curative endovascular reconstruction of cerebral aneurysms with the Pipeline embolization device: the Buenos Aires experience. Neurosurgery 64:632643, 2009

    • Crossref
    • Search Google Scholar
    • Export Citation
  • 19

    Lyon A, Ariga R, Mincholé A, Mahmod M, Ormondroyd E, Laguna P, et al.: Distinct ECG phenotypes identified in hypertrophic cardiomyopathy using machine learning associate with arrhythmic risk markers. Front Physiol 9:213, 2018

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 20

    Montoya JC, Li Y, Strother C, Chen GH: 3D deep learning angiography (3D-DLA) from C-arm conebeam CT. AJNR Am J Neuroradiol 39:916922, 2018

  • 21

    Mut F, Raschi M, Scrivano E, Bleise C, Chudyk J, Ceratto R, et al.: Association between hemodynamic conditions and occlusion times after flow diversion in cerebral aneurysms. J Neurointerv Surg 7:286290, 2015

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 22

    Obermeyer Z, Emanuel EJ: Predicting the future—big data, machine learning, and clinical medicine. N Engl J Med 375:12161219, 2016

  • 23

    Ouared R, Larrabide I, Brina O, Bouillot P, Erceg G, Yilmaz H, et al.: Computational fluid dynamics analysis of flow reduction induced by flow-diverting stents in intracranial aneurysms: a patient-unspecific hemodynamics change perspective. J Neurointerv Surg 8:12881293, 2016

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 24

    Paliwal N, Damiano RJ, Davies JM, Siddiqui AH, Meng H: Association between hemodynamic modifications and clinical outcome of intracranial aneurysms treated using flow diverters. Proc SPIE Int Soc Opt Eng 10135:101352F, 2017

    • PubMed
    • Search Google Scholar
    • Export Citation
  • 25

    Paliwal N, Yu H, Damiano R, Xiang J, Yang X, Siddiqui A, et al.: Fast virtual stenting with vessel-specific initialization and collision detection, in Proceedings of the ASME International Design Engineering Technical Conferences and Computers and Information in Engineering Conference, 2014. New York: ASME, 2014, Vol 3 (http://proceedings.asmedigitalcollection.asme.org/proceeding.aspx?articleid=2090696) [Accessed September 7, 2018]

    • Search Google Scholar
    • Export Citation
  • 26

    Paliwal N, Yu H, Xu J, Xiang J, Siddiqui A, Yang X, et al.: Virtual stenting workflow with vessel-specific initialization and adaptive expansion for neurovascular stents and flow diverters. Comput Methods Biomech Biomed Engin 19:14231431, 2016

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 27

    Poudel RPK, Lamata P, Montana G: Recurrent fully convolutional neural networks for multi-slice MRI cardiac segmentation, in Zuluaga MA, Bhatia K, Kainz B, et al. (eds): Reconstruction, Segmentation, and Analysis of Medical Images. Cham, Switzerland: Springer, 2016, pp 8394

    • Search Google Scholar
    • Export Citation
  • 28

    Rajah G, Narayanan S, Rangel-Castilla L: Update on flow diverters for the endovascular management of cerebral aneurysms. Neurosurg Focus 42(6):E2, 2017

  • 29

    Refaeilzadeh P, Tang L, Liu H: Cross-validation, in Liu L, Özsu MT (eds): Encyclopedia of Database Systems. New York: Springer, 2016

  • 30

    Senders JT, Staples PC, Karhade AV, Zaki MM, Gormley WB, Broekman MLD, et al.: Machine learning and neurosurgical outcome prediction: a systematic review. World Neurosurg 109:476486, 486.e1, 2018

    • Search Google Scholar
    • Export Citation
  • 31

    Senders JT, Zaki MM, Karhade AV, Chang B, Gormley WB, Broekman ML, et al.: An introduction and overview of machine learning in neurosurgical care. Acta Neurochir (Wien) 160:2938, 2018

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 32

    Siddiqui AH, Abla AA, Kan P, Dumont TM, Jahshan S, Britz GW, et al.: Panacea or problem: flow diverters in the treatment of symptomatic large or giant fusiform vertebrobasilar aneurysms. J Neurosurg 116:12581266, 2012

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 33

    Spiegel M, Redel T, Zhang YJ, Struffert T, Hornegger J, Grossman RG, et al.: Tetrahedral vs. polyhedral mesh size evaluation on flow velocity and wall shear stress for cerebral hemodynamic simulation. Comput Methods Biomech Biomed Engin 14:922, 2011

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 34

    Szikora I, Berentei Z, Kulcsar Z, Marosfoi M, Vajda ZS, Lee W, et al.: Treatment of intracranial aneurysms by functional reconstruction of the parent artery: the Budapest experience with the Pipeline Embolization Device. AJNR Am J Neuroradiol 31:11391147, 2010

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 35

    Tesche C, De Cecco CN, Baumann S, Renker M, McLaurin TW, Duguay TM, et al.: Coronary CT angiography-derived fractional flow reserve: machine learning algorithm versus computational fluid dynamics modeling. Radiology 288:6472, 2018

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 36

    Wang S, Summers RM: Machine learning and radiology. Med Image Anal 16:933951, 2012

  • 37

    Xiang J, Antiga L, Varble N, Snyder KV, Levy EI, Siddiqui AH, et al.: AView: an image-based clinical computational tool for intracranial aneurysm flow visualization and clinical management. Ann Biomed Eng 44:10851096, 2016

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 38

    Xiang J, Damiano RJ, Lin N, Snyder KV, Siddiqui AH, Levy EI, et al.: High-fidelity virtual stenting: modeling of flow diverter deployment for hemodynamic characterization of complex intracranial aneurysms. J Neurosurg 123:832840, 2015

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 39

    Xiang J, Natarajan SK, Tremmel M, Ma D, Mocco J, Hopkins LN, et al.: Hemodynamic-morphologic discriminants for intracranial aneurysm rupture. Stroke 42:144152, 2011

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation

Supplementary Materials

  • Collapse
  • Expand
  • View in gallery

    Illustration of different ML classification algorithms. A: Logistic regression, where a classification line is fitted on the data. B: Support vector machine, where the best hyperplane that separates the data is identified by maximizing the margins on either side. C: K-nearest neighbor, where the predictions of a new point are made based on its distance from the points in the exiting database. D: Neural network with 2 hidden layers, where a system of interconnected neurons uses back-propagation to learn from the training data.

  • View in gallery

    Flowchart for building and testing the predictive models for occlusion outcome of FD-treated IAs. After extraction of morphological, FD-related, and hemodynamic parameters, patients are randomly divided into the training and testing cohorts. Two sets of models are then trained using 1) all parameters and 2) significant parameters on the training cohort. The predictive performances of these models are then tested on the testing cohort. Tx = treatment.

  • View in gallery

    ROC curves for each ML model on the training cohort built using (A) all the parameters and (B) significant parameters only. AUC = area under the ROC curve; G-SVM = Gaussian support vector machine; K-NN = K-nearest neighbor; LR = logistic regression; L-SVM = linear support vector machine; NN = neural network.

  • View in gallery

    Confusion matrix for predictions of ML models (trained using all parameters) on the testing cohort. The vertical axis represents the actual clinical outcome and the horizontal axis represents the model-predicted outcome. Green indicates correct predictions; red indicates incorrect predictions.

  • View in gallery

    Concept of the computational analysis methodology for generating clinically practical predictive models for FD-treated IAs. The left panel shows 2 representative patient-specific ICA aneurysms, which were treated using a single FD. The far right panel shows the 6-month occlusion outcome of both cases: the aneurysm in the upper images was completely occluded (red circle), whereas the aneurysm in the lower images had persistent residual filling. The middle panel shows the computational analysis workflow, which uses pre-treatment 3D DSA images to generate a 3D aneurysm model, and then models the FD deployment and hemodynamics using virtual stenting and CFD, respectively. The morphological, hemodynamic, and candidate FD characteristics are used as features to train machine learning algorithms that can predict the 6-month occlusion outcome of FD-treated IAs. Note that the image of neural network is shown as a representative machine learning algorithm for predictive model building.

Metrics

All Time Past Year Past 30 Days
Abstract Views 0 0 0
Full Text Views 5165 1062 131
PDF Downloads 3020 741 84
EPUB Downloads 0 0 0