Validation of an automated machine learning algorithm for the detection and analysis of cerebral aneurysms

Marco Colasurdo Department of Radiology, Division of Neuroradiology, The University of Texas Medical Branch, Galveston, Texas;

Search for other papers by Marco Colasurdo in
Current site
Google Scholar
PubMed
Close
 MD
,
Daphna Shalev Viz.ai Inc., San Francisco, California;

Search for other papers by Daphna Shalev in
Current site
Google Scholar
PubMed
Close
 MSc
,
Ariadna Robledo Departments of Neurosurgery and

Search for other papers by Ariadna Robledo in
Current site
Google Scholar
PubMed
Close
 MD
,
Viren Vasandani Departments of Neurosurgery and

Search for other papers by Viren Vasandani in
Current site
Google Scholar
PubMed
Close
 MBA
,
Zean Aaron Luna Departments of Neurosurgery and

Search for other papers by Zean Aaron Luna in
Current site
Google Scholar
PubMed
Close
 BS
,
Abhijit S. Rao Departments of Neurosurgery and

Search for other papers by Abhijit S. Rao in
Current site
Google Scholar
PubMed
Close
 BSA
,
Roberto Garcia Departments of Neurosurgery and

Search for other papers by Roberto Garcia in
Current site
Google Scholar
PubMed
Close
 MD
,
Gautam Edhayan Department of Radiology, Division of Neuroradiology, The University of Texas Medical Branch, Galveston, Texas;

Search for other papers by Gautam Edhayan in
Current site
Google Scholar
PubMed
Close
 MD
,
Visish M. Srinivasan Department of Neurosurgery, Barrow Neurological Institute, St. Joseph’s Hospital and Medical Center, Phoenix, Arizona; and

Search for other papers by Visish M. Srinivasan in
Current site
Google Scholar
PubMed
Close
 MD
,
Sunil A. Sheth Department of Neurology, McGovern Medical School, University of Texas Health Science Center, Houston, Texas

Search for other papers by Sunil A. Sheth in
Current site
Google Scholar
PubMed
Close
 MD
,
Yoni Donner Viz.ai Inc., San Francisco, California;

Search for other papers by Yoni Donner in
Current site
Google Scholar
PubMed
Close
 PhD
,
Orin Bibas Viz.ai Inc., San Francisco, California;

Search for other papers by Orin Bibas in
Current site
Google Scholar
PubMed
Close
 HSD
,
Nicole Limzider Viz.ai Inc., San Francisco, California;

Search for other papers by Nicole Limzider in
Current site
Google Scholar
PubMed
Close
 BMedSc
,
Hashem Shaltoni Neurology, The University of Texas Medical Branch, Galveston, Texas;

Search for other papers by Hashem Shaltoni in
Current site
Google Scholar
PubMed
Close
 MD
, and
Peter Kan Departments of Neurosurgery and

Search for other papers by Peter Kan in
Current site
Google Scholar
PubMed
Close
 MD, MPH
Free access

OBJECTIVE

Machine learning algorithms have shown groundbreaking results in neuroimaging. The authors herein evaluated the performance of a newly developed convolutional neural network (CNN) to detect and analyze intracranial aneurysms (IAs) on CTA.

METHODS

Consecutive patients with CTA studies between January 2015 and July 2021 at a single center were identified. The ground truth determination of cerebral aneurysm presence or absence was made from the neuroradiology report. The primary outcome was the performance of the CNN in detecting IAs in an external validation set, measured using area under the receiver operating characteristic curve statistics. Secondary outcomes included accuracy for location and size measurement.

RESULTS

The independent validation imaging data set consisted of 400 patients with CTA studies, median age 40 years (IQR 34 years) and 141 (35.3%) of whom were male; 193 patients (48.3%) had a diagnosis of IA on neuroradiologist evaluation. The median maximum IA diameter was 3.7 mm (IQR 2.5 mm). In the independent validation imaging data set, the CNN performed well with 93.8% sensitivity (95% CI 0.87–0.98), 94.2% specificity (95% CI 0.90–0.97), and a positive predictive value of 88.2% (95% CI 0.80–0.94) in the subgroup with an IA diameter ≥ 4 mm.

CONCLUSIONS

The described Viz.ai Aneurysm CNN performed well in identifying the presence or absence of IAs in an independent validation imaging set. Further studies are necessary to investigate the impact of the software on detection rates in a real-world setting.

ABBREVIATIONS

AI = artificial intelligence; CNN = convolutional neural network; IA = intracranial aneurysm; PPV = positive predictive value; ROC = receiver operating characteristic.

OBJECTIVE

Machine learning algorithms have shown groundbreaking results in neuroimaging. The authors herein evaluated the performance of a newly developed convolutional neural network (CNN) to detect and analyze intracranial aneurysms (IAs) on CTA.

METHODS

Consecutive patients with CTA studies between January 2015 and July 2021 at a single center were identified. The ground truth determination of cerebral aneurysm presence or absence was made from the neuroradiology report. The primary outcome was the performance of the CNN in detecting IAs in an external validation set, measured using area under the receiver operating characteristic curve statistics. Secondary outcomes included accuracy for location and size measurement.

RESULTS

The independent validation imaging data set consisted of 400 patients with CTA studies, median age 40 years (IQR 34 years) and 141 (35.3%) of whom were male; 193 patients (48.3%) had a diagnosis of IA on neuroradiologist evaluation. The median maximum IA diameter was 3.7 mm (IQR 2.5 mm). In the independent validation imaging data set, the CNN performed well with 93.8% sensitivity (95% CI 0.87–0.98), 94.2% specificity (95% CI 0.90–0.97), and a positive predictive value of 88.2% (95% CI 0.80–0.94) in the subgroup with an IA diameter ≥ 4 mm.

CONCLUSIONS

The described Viz.ai Aneurysm CNN performed well in identifying the presence or absence of IAs in an independent validation imaging set. Further studies are necessary to investigate the impact of the software on detection rates in a real-world setting.

In Brief

Researchers validated a novel automated convolutional neural network (CNN) for the detection and analysis of intracranial aneurysms (IAs). The described Viz.ai Aneurysm CNN performed well in identifying the presence or absence of IAs with a diameter ≥ 4 mm with 93.8% sensitivity, 94.2% specificity, and a positive predictive value of 88.2%. The implementation of artificial intelligence in real-world clinical practice can have a significant effect on patient care, and further studies are necessary to investigate the impact of the software on detection rates in a real-world setting.

Intracranial aneurysms (IAs) represent a pathological dilatation of the vessel wall of a cerebral artery. They can potentially arise from any artery of the circle of Willis or its branches. Over 80% of ruptured and unruptured aneurysms are localized in the anterior circulation.1

Incidental aneurysms have a prevalence ranging from 2% to 8% in the general population.1,2 They can present with dramatic clinical scenarios after rupture with subarachnoid hemorrhage and symptoms related to compression of adjacent brain or cranial nerves and case fatality rates ranging from 8% to 67%.3

The widespread use of advanced imaging techniques, in part related to recent developments in the field of acute ischemic stroke, has led to an increased detection of incidental IAs.4 Despite the continuous improvements in available treatments, the decision to treat unruptured IAs is still under debate with a few analytical tools available to guide management.46

Conventional 2D DSA and 3D rotational DSA7 are considered the gold standard for detecting and characterizing IAs. Nevertheless, CTA with the use of 3D reconstructions has shown similar results with reported sensitivities ranging from 80% to 97%810 even for aneurysms measuring less than 3 mm. Moreover, digital image processing and manipulation techniques are also influenced by the experience of the individual operator.11

Artificial intelligence (AI) and deep learning can assist and streamline image interpretation and scrutiny as witnessed by the detection of large-vessel occlusion and the management of endovascular thrombectomy intervention.1215 As preliminarily shown by Hassan et al.16 and other authors,17,18 the implementation of AI in real-world clinical practice can have a significant impact on patient care.

Viz.ai has developed a deep convolutional neural network (CNN) designed to detect and analyze cerebral aneurysms on CTA. In this work, we outline the CNN and assess its performance in detecting the presence or absence of IAs.

Methods

This Health Insurance Portability and Accountability Act (HIPAA)–compliant study was approved by the IRB at our institution. The requirement for patient consent was waived because of the study’s retrospective nature and the de-identification of patient data. Nonemployee and nonconsultant authors had complete control over the data, information, and analysis.

Cohort Characteristics

CTA studies obtained at the same institution from January 2015 to July 2021 with a final diagnosis of IA were retrospectively collected for 193 consecutive patients. From the same period and institution, 207 additional CTA examinations with no evidence of aneurysm or significant intracranial pathology were randomly selected to reach a total of 400 subjects.

Imaging Settings

The images used to develop and test the algorithm were acquired in the axial plane on six scanners by different manufacturers. The following criteria were inspected to validate technical and clinical adequacy: axial monochromatic head CTA scan and whole-head volume included without excessive motion artifacts. Images were stored on a PACS.

Pipeline for Analysis

The training data set utilized to develop the CNN included CTA scans from patients with and without IAs. Among a total of 4351 scans, the subgroup with IAs included 1796 subjects (369 males [8.5%]) with a mean patient age of 70 years, whereas the subgroup without IAs included 2555 subjects (1119 males [25.7%]) with a mean patient age of 65 years. Both age and gender were statistically significantly different between the two subgroups (p < 0.05). Gender data were missing for 15.9% of the subjects in the training data set. Training cohort demographic details are listed in Table 1.

TABLE 1.

Training and testing data set demographics

Data SetIANo IAp Value*
Training (n = 4351)
 Gender, no. (%)<0.05
  F 892 (20.5)1278 (29.4)
  M369 (8.5)1119 (25.7)
  Missing data693 (15.9) 
 Mean age in yrs (SD)70 (13)65 (16)<0.05
Testing (n = 400)
 Gender, no. (%) 
  F143 (35.7)116 (29.0)
  M50 (12.5)91 (22.8)
 Mean age in yrs (SD)66 (16)35 (13)<0.05

Chi-square and t-test for independent samples. Boldface type indicates statistical significance.

The classification deep CNN was trained for 200,000 steps using stochastic gradient descent to optimize the cross-entropy loss between the network predictions and the segmentation labels created by Viz.ai radiology-trained annotators. The initial learning rate was 0.002 and was annealed to 0 using cosine annealing and weight decay of 10–6. The segmentation deep CNN was trained for 1,200,000 steps using cross-entropy losses for both outputs. The initial learning rate was 0.005 and was annealed to 0 using cosine annealing.

The testing data set utilized 400 unique CTA scans, 193 of which had at least one IA.

Gold-standard ground truth on the training data set was then obtained on the basis of the read of a dedicated radiology-trained annotator aware of the presence of IAs. Real-world ground truth on the testing data set was established on the basis of the final reports provided by experienced neuroradiologists with, respectively, 3, 5, 6, 7, and 10+ years of dedicated neuroradiology experience. A second independent read was then established by a specialized radiology-trained annotator blinded to the prior results. Next, we established agreement between the two obtained reads, and in the case of discordance, a third independent read by an independent board-certified neuroradiologist was used as a discriminating factor.

CNN Pipeline and Analysis of Data

Before receiving data from a CTA, the Viz.ai Aneurysm CNN requires several preprocessing procedures. Then if metal is detected in the relevant regions or if the skull is only partially inspected, the scan will not be processed by the CNN. The data subsequently go through the registration step. Using a deep CNN, the input scan is registered onto a 128-keypoint atlas, and it is transformed so that the total Euclidean distance between the transformed keypoints and the atlas keypoints is minimized. Next, several predetermined regions of interest are cropped from the transformed input scan and taken as input for the classification and segmentation steps.

The scan runs through the algorithm’s classification and segmentation steps, each of which consists of a deep CNN. In the classification step, a probability value is calculated for each input region of interest, representing the probability that the region of interest is part of an aneurysm. This results in a highly sensitive model output. To balance for specificity, the segmentation step then classifies each candidate region of interest as having an aneurysm or not, and a segmentation probability is calculated at the voxel level. The probabilities of all candidate regions of interest are combined into a single score for the entire scan. A threshold is determined by evaluating the model on the Viz.Ai internal test set and choosing the threshold that best optimizes the sensitivity and specificity of the algorithm on this set. If the single final score passes the predetermined threshold, the scan is then labeled "positive," meaning that it is suspected of containing an aneurysm.

A flowchart of the CNN pipeline can be found in Fig. 1.

FIG. 1.
FIG. 1.

CNN pipeline analysis flowchart describing a stepwise approach from input scan to algorithm output. Figure is available in color online only.

Statistical Analysis

Statistical analyses were performed using descriptive data, including ranges, means, medians, and standard deviations for continuous variables and frequencies and percentages for categorical variables. Unadjusted comparisons of two groups were conducted using the two-tailed t-test or chi-square test. Sample size was justified to accept the primary endpoint of both sensitivity and specificity greater than 0.8 on a two-sided Clopper-Pearson confidence interval (95%). System performance methods were examined using sensitivity, specificity, positive predictive value (PPV), and total accuracy. Confidence intervals were calculated using the Wilson procedure with correction for continuity.19 Receiver operating characteristic (ROC) curve analysis was used to determine the characteristics and performance of the automated IA detection for the different subgroups. A p value less than 0.05 was considered statistically significant.

Results

Patient Characteristics

Four hundred patients were identified and included in our study as the testing data set. The median patient age for the whole cohort was 40 years old (IQR 34 years). One hundred forty-one subjects (35.3%) were male. The subgroup with IAs included 50 males and had a mean patient age of 66 years, whereas the subgroup without IAs included 91 males and had a mean patient age of 35 years. Both age and gender were statistically significantly different between the two subgroups (p < 0.05). There were no missing data in the analyzed cohort. Testing cohort demographic details are listed in Table 1. Flowcharts for the training and testing data sets appear in Fig. 2.

FIG. 2.
FIG. 2.

Flowcharts illustrating the breakdown of the analyzed data sets.

A total of 233 IAs were identified in the testing data set; 157 subjects had 1 aneurysm, 36 subjects had 2 aneurysms, and 4 additional subjects had 3 aneurysms each. The median maximum IA diameter was 3.7 mm (IQR 2.5 mm), and the most frequent location was the internal carotid artery, with 109 aneurysms (46.8%). The overall distribution of IAs is detailed in Table 2.

TABLE 2.

Overall distribution of IAs

LocationNo. (average size in mm)
Rt SideLt Side
Internal carotid artery53 (4.7)56 (3.9)
 Cavernous-paraophthalmic artery27 (5.3)27 (3.5)
 Communicating artery16 (3.0)9 (4.2)
 Supra-/paraclinoid artery*10 (6.1)20 (4.2)
Anterior communicating artery37 (3.9)
Anterior cerebral artery9 (5.1)6 (3.4)
Middle cerebral artery40 (4.5)19 (3.8)
Posterior circulation
 Basilar artery9 (4.3)
 Vertebral artery3 (4.2)1 (7.0)

Including internal carotid artery terminus aneurysms.

Including posterior inferior cerebellar artery aneurysms.

Detection of IAs

The algorithm successfully ran on 392 scans, of which 206 were negative and 186 were flagged as positive. The remaining 8 studies were not analyzed because of an incomplete skull representation on the scan (5 studies), metal artifacts (2 studies), and variation in spacing between the scan’s slices beyond the amount manageable for the algorithm (1 study). The mean processing time of the algorithm, including notification, was 114.7 seconds (SD 11.4 seconds).

System performance for the entire group and all subgroups was measured with an algorithm score threshold of 1.5. For the entire population, performance was measured with a sensitivity of 81.7% (95% CI 0.75–0.87, with 152/186 cases), a specificity of 94.2% (95% CI 0.90–0.97, with 194/206 cases), a PPV of 92.7% (95% CI 0.88–0.96, with 152/164 cases), and an accuracy of 88.3% (95% CI 0.85–0.91, with 346/392 cases).

Performance in the Subgroup of Aneurysms ≥ 3 mm

Performance in the subgroup of aneurysms ≥ 3 mm was measured with a sensitivity of 89.4% (95% CI 0.83–0.94, with 126/141 cases), a specificity of 94.2% (95% CI 0.90–0.97, with 194/206 cases), a PPV of 91.3% (95% CI 0.85–0.95, with 126/138 cases), and an accuracy of 92.2% (95% CI 0.89–0.95, with 320/347 cases). Cerebral aneurysms excluded from this subgroup (maximum diameter < 3 mm) are defined by some authors as very small cerebral aneurysms. These are often managed with serial follow-up, as treatment-related mortality and morbidity are often equal to or greater than the risk of spontaneous rupture. Although CTA detection rates for aneurysms smaller than 3 mm are notoriously lower, excellent sensitivities have been reported20 with the use of modern scanners and bone-subtraction technology.

Performance in the Subgroup of Aneurysms ≥ 4 mm

At the algorithm’s FDA-cleared threshold of 4 mm and above, performance was measured with a sensitivity of 93.8% (95% CI 0.87–0.98, with 90/96 cases), a specificity of 94.2% (95% CI 0.90–0.97, with 194/206 cases), a PPV of 88.2% (95% CI 0.80–0.94, with 90/102 cases), and an accuracy of 94.0% (95% CI 0.91–0.96, with 284/302 cases).

While no strict treatment criteria exist, therapy recommendations based on scores and aneurysm properties have been developed, and saccular aneurysms measuring at least 4 mm in diameter are often considered clinically actionable. For reference, the unruptured intracranial aneurysm treatment score (UIATS)21 assigns a value of 0 to aneurysms with a size < 4 mm.

Performance in the Subgroup of Aneurysms ≥ 5 mm

Performance in the subgroup of aneurysms ≥ 5 mm was measured with a sensitivity of 95.3% (95% CI 0.87–0.99, with 61/64 cases), a specificity of 94.2% (95% CI 0.90–0.97, with 194/206 cases), a PPV of 83.6% (95% CI 0.73–0.91, with 61/73 cases), and an accuracy of 94.4% (95% CI 0.91–0.97, with 255/270 cases). Aneurysms excluded from this subgroup, measuring < 5 mm in maximum diameter, are generally defined as small cerebral aneurysms.22

ROC analysis demonstrated an area under the curve of 0.93 for the entire data set, 0.97 for the subgroup ≥ 3 mm, and 0.99 for the subgroups ≥ 4 or ≥ 5 mm, as shown in Fig. 3. Examples of correct flagging and aneurysm identification are provided in Fig. 4. Examples of incorrectly flagged scans (false positives) are provided in Fig. 5.

FIG. 3.
FIG. 3.

ROC analysis for the entire data set and the subgroups of aneurysms ≥ 3, ≥ 4, and ≥ 5 mm. Figure is available in color online only.

FIG. 4.
FIG. 4.

True-positive examples. Aneurysm detection and segmentation in a subject with an anterior communicating artery and a left middle cerebral artery bifurcation aneurysm (A). A second subject demonstrates a left internal carotid artery terminus aneurysm (B), and a third subject demonstrates a small anterior communicating artery aneurysm (C). Figure is available in color online only.

FIG. 5.
FIG. 5.

False-positive examples. The infundibular origin of a right posterior communicating artery is erroneously identified as an aneurysm (A). The communicating segment of the left internal carotid artery is indicated as an aneurysm (B). Atherosclerosis in the right internal carotid artery is misclassified as an aneurysm (C). These are common locations for aneurysms and often require catheter-based angiography to completely rule out abnormalities. Figure is available in color online only.

Discussion

In this study, we found the newly developed Viz.ai Aneurysm CNN to be accurate in the detection of IAs. Its performance in detecting the presence or absence of IAs was extremely accurate with an area under the curve of 0.93 for the entire external validation testing data set.

CNNs are fairly new technologies in the field of AI in which a human-like algorithm is developed and mimics the structure and function of the brain. Similar applications for the detection and segmentation of IAs on time-of-flight MRA23 as well as on DSA24,25 have been reported. Deep learning algorithms have also been used to aid detection and segmentation in different vascular territories such as for aortic aneurysms26 or presurgical aortic evaluations.27

The studied CNN showed not only great performance at its FDA-cleared threshold of 4 mm or greater, but also promising sensitivity and specificity when including aneurysms measuring 3 mm and above. Lesions smaller than 3 mm are often easily overlooked in asymptomatic CTA examinations.

Automated tools for the detection of incidental intracranial pathology are already part of real-world triage processes, and it is easy to foresee the value of automated IA detection tools considering the number of CTA examinations that are acquired in busy tertiary care centers.

When analyzing the cohort of aneurysms 4 mm and above, the CNN encountered 12 false-positive cases, 8 of which were found to be an infundibular dilatation of vessel origin, a common pitfall encountered by even experienced readers and which often requires invasive angiography to be fully understood. An additional case was found to be a fusiform middle cerebral artery dilatation, which at times can also be confused for aneurysm changes. No explanation was found in 3 cases. These results also contained 6 false-negative cases, which consisted of aneurysms measuring 4 mm or smaller in size.

CTA is one of the modalities of choice for the diagnosis of IAs, and together with MRA and DSA, is commonly used for preoperative and follow-up imaging. Interestingly, as recently shown,28 IAs represent the second most common misdiagnosis when examining cerebrovascular pathology, after ischemic lesions. Data derived from CTA reads that will impact management are location, size, presence of other associated abnormalities, and presence of multiple lesions.

In this work, we validated an FDA-approved,29 commercially available CNN to accurately and automatically detect cerebral aneurysms based on CTA. The Viz.ai Aneurysm CNN showed optimal sensitivity, with a PPV of 92.7% for the whole external validation cohort, which makes it a promising diagnostic tool. Minimizing spectrum bias with the use of an external data set, completely blinded to model development, is a known methodological resource for the validation of CNNs for medical diagnostic purposes.30 Furthermore, our validation data set included different populations usually encountered in clinical practice with aneurysms localized in the anterior and posterior circulation, ruptured aneurysms, and lesions located near the skull base or cavernous sinuses. Its reliability with multiple scanners from different manufacturers is another advantage not to be overlooked.

Our study is not without limitations. First, the CNN was assessed using a retrospective single-center cohort and ground truth was established based on CTA examinations; although DSA is considered the gold standard for a definitive diagnosis of IA, both CTA and MRA can be considered cornerstones for screening and first-line diagnosis. Second, 8 examinations were considered technically or clinically inadequate and were excluded from analysis. Third, our data set did not include ruptured aneurysms with large hematomas or extensive subarachnoid hemorrhage, sometimes challenging factors in imaging interpretation. Data regarding analysis of the training data set were managed by Viz.ai employees who are included as authors of this paper. We did not perform a subgroup analysis based on aneurysm location or concomitant intracranial pathology; to the best of our knowledge, this is yet to be explored in the literature and should be investigated in future studies. Additionally, the cohort with no IAs had significantly more males, a predilection known from prior studies;21,31 it also included examinations from a considerably younger data set with no significant pathology or image distortion. Finally, the algorithm in its current state creates a segmentation slice but does not provide information regarding the presence of infundibular dilatation, measurements, or labels regarding the parent vessel.

Future iterations of the Viz.ai Aneurysm CNN should go far beyond aneurysm detection, focusing on aneurysm diameter measurements, identification of the aneurysm neck, detection of branching vessels, and possible exclusion of an infundibular origin of intracranial vessels. Future endeavors should focus on exploring the role of AI-aided diagnosis in real-world scenarios, with potential focus on reducing false negatives and improving reading times.

Conclusions

The described Viz.ai Aneurysm CNN performed well in identifying the presence or absence of IAs in an independent validation imaging set. Further studies are necessary to investigate the impact of the software on detection rates in a real-world setting.

Key Message

Machine learning algorithms have shown groundbreaking results in neuroimaging, and numerous AI applications are in use or development in the neurosciences. IAs represent the second most missed cerebrovascular pathology. In our work, we validated the Viz.ai Aneurysm CNN on an independent, external imaging data set of 233 IAs, showing exceptional results. CNNs for the automated detection and analysis of IAs will have a significant impact on the diagnosis and management of cerebrovascular pathology.

Disclosures

Dr. Sheth reports grant payments to UTHealth from Viz.ai. Ms. Bibas reports personal fees from Viz.ai during the conduct of this study. Dr. Kan reports grants from Viz.ai during the conduct of this study.

Author Contributions

Conception and design: Kan, Colasurdo, Shalev, Garcia, Srinivasan. Acquisition of data: Kan, Colasurdo, Shalev, Robledo, Vasandani, Luna, Rao, Garcia, Srinivasan, Bibas. Analysis and interpretation of data: Kan, Colasurdo, Shalev, Robledo, Vasandani, Luna, Edhayan, Srinivasan, Sheth, Donner, Bibas, Limzider. Drafting the article: Kan, Colasurdo, Srinivasan, Shaltoni. Critically revising the article: Kan, Shalev, Robledo, Edhayan, Srinivasan. Reviewed submitted version of manuscript: Kan, Colasurdo, Shalev, Vasandani, Garcia, Edhayan, Srinivasan, Sheth. Approved the final version of the manuscript on behalf of all authors: Kan. Statistical analysis: Kan, Colasurdo, Shalev, Bibas. Administrative/technical/material support: Kan, Robledo, Rao, Garcia. Study supervision: Kan, Robledo, Garcia.

Supplemental Information

Previous Presentations

This work was presented as an oral presentation at the Congress of Neurological Surgeons Annual Meeting held in San Francisco, California, on October 8–12, 2022.

References

  • 1

    Thien A, See AAQ, Ang SYL, et al. Prevalence of asymptomatic unruptured intracranial aneurysms in a southeast Asian population. World Neurosurg. 2017;97:326332.

  • 2

    Juvela S, Poussa K, Lehto H, Porras M. Natural history of unruptured intracranial aneurysms: a long-term follow-up study. Stroke. 2013;44(9):24142421.

  • 3

    Nieuwkamp DJ, Setz LE, Algra A, Linn FHH, de Rooij NK, Rinkel GJE. Changes in case fatality of aneurysmal subarachnoid haemorrhage over time, according to age, sex, and region: a meta-analysis. Lancet Neurol. 2009;8(7):635642.

    • PubMed
    • Search Google Scholar
    • Export Citation
  • 4

    Brown RD Jr, Broderick JP. Unruptured intracranial aneurysms: epidemiology, natural history, management options, and familial screening. Lancet Neurol. 2014;13(4):393404.

    • PubMed
    • Search Google Scholar
    • Export Citation
  • 5

    Bijlenga P, Gondar R, Schilling S, et al. PHASES score for the management of intracranial aneurysm: a cross-sectional population-based retrospective study. Stroke. 2017;48(8):21052112.

    • PubMed
    • Search Google Scholar
    • Export Citation
  • 6

    Wiebers DO, Whisnant JP, Huston J III, et al. Unruptured intracranial aneurysms: natural history, clinical outcome, and risks of surgical and endovascular treatment. Lancet. 2003;362(9378):103110.

    • PubMed
    • Search Google Scholar
    • Export Citation
  • 7

    Wong SC, Nawawi O, Ramli N, Abd Kadir KA. Benefits of 3D rotational DSA compared with 2D DSA in the evaluation of intracranial aneurysm. Acad Radiol. 2012;19(6):701707.

    • PubMed
    • Search Google Scholar
    • Export Citation
  • 8

    Tomandl BF, Köstner NC, Schempershofe M, et al. CT angiography of intracranial aneurysms: a focus on postprocessing. Radiographics. 2004;24(3):637655.

  • 9

    Lu L, Zhang LJ, Poon CS, et al. Digital subtraction CT angiography for detection of intracranial aneurysms: comparison with three-dimensional digital subtraction angiography. Radiology. 2012;262(2):605612.

    • PubMed
    • Search Google Scholar
    • Export Citation
  • 10

    D’Argento F, Pedicelli A, Ciardi C, et al. Intra- and inter-observer variability in intracranial aneurysm segmentation: comparison between CT angiography (semi-automated segmentation software stroke VCAR) and digital subtraction angiography (3D rotational angiography). Radiol Med (Torino). 2021;126(3):484493.

    • PubMed
    • Search Google Scholar
    • Export Citation
  • 11

    Udupa JK. Three-dimensional visualization and analysis methodologies: a current perspective. Radiographics. 1999;19(3):783806.

  • 12

    Murray NM, Unberath M, Hager GD, Hui FK. Artificial intelligence to diagnose ischemic stroke and identify large vessel occlusions: a systematic review. J Neurointerv Surg. 2020;12(2):156164.

    • PubMed
    • Search Google Scholar
    • Export Citation
  • 13

    Soun JE, Chow DS, Nagamine M, et al. Artificial intelligence and acute stroke imaging. AJNR Am J Neuroradiol. 2021;42(1):211.

  • 14

    Morey J, Fiano E, Yaeger K, Zhang X, Fifi J. Impact of Viz LVO on time-to-treatment and clinical outcomes in large vessel occlusion stroke patients presenting to primary stroke centers. medRxiv. Preprint posted online July 5, 2020. doi:10.1101/2020.07.02.20143834

    • PubMed
    • Search Google Scholar
    • Export Citation
  • 15

    Kalagara R, Chennareddy S, Matsoukas S, et al. Abstract 1122-000033: Automated detection of intracerebral hemorrhage using artificial intelligence: pilot deployment of Viz ICH. Stroke Vasc Interv Neurol. 2021;1(S1):e12079.

    • PubMed
    • Search Google Scholar
    • Export Citation
  • 16

    Hassan AE, Ringheanu VM, Rabah RR, Preston L, Tekle WG, Qureshi AI. Early experience utilizing artificial intelligence shows significant reduction in transfer times and length of stay in a hub and spoke model. Interv Neuroradiol. 2020;26(5):615622.

    • PubMed
    • Search Google Scholar
    • Export Citation
  • 17

    Rodrigues G, Barreira CM, Bouslama M, et al. Automated large artery occlusion detection in stroke: a single-center validation study of an artificial intelligence algorithm. Cerebrovasc Dis. 2022;51(2):259264.

    • PubMed
    • Search Google Scholar
    • Export Citation
  • 18

    Morey JR, Zhang X, Yaeger KA, et al. Real-world experience with artificial intelligence-based triage in transferred large vessel occlusion stroke patients. Cerebrovasc Dis. 2021;50(4):450455.

    • PubMed
    • Search Google Scholar
    • Export Citation
  • 19

    Wilson EB. Probable inference, the law of succession, and statistical inference. J Am Stat Assoc. 1927;22(158):209212.

  • 20

    Yang ZL, Ni QQ, Schoepf UJ, et al. Small intracranial aneurysms: diagnostic accuracy of CT angiography. Radiology. 2017;285(3):941952.

  • 21

    Etminan N, Brown RD Jr, Beseoglu K, et al. The unruptured intracranial aneurysm treatment score: a multidisciplinary consensus. Neurology. 2015;85(10):881889.

  • 22

    Chen W, Wang J, Xin W, Peng Y, Xu Q. Accuracy of 16-row multislice computed tomographic angiography for assessment of small cerebral aneurysms. Neurosurgery. 2008;62(1):113122.

    • PubMed
    • Search Google Scholar
    • Export Citation
  • 23

    Ueda D, Yamamoto A, Nishimori M, et al. Deep learning for MR angiography: automated detection of cerebral aneurysms. Radiology. 2019;290(1):187194.

  • 24

    Duan H, Huang Y, Liu L, Dai H, Chen L, Zhou L. Automatic detection on intracranial aneurysm from digital subtraction angiography with cascade convolutional neural networks. Biomed Eng Online. 2019;18(1):110.

    • PubMed
    • Search Google Scholar
    • Export Citation
  • 25

    Jin H, Geng J, Yin Y, et al. Fully automated intracranial aneurysm detection and segmentation from digital subtraction angiography series using an end-to-end spatiotemporal deep neural network. J Neurointerv Surg. 2020;12(10):10231027.

    • PubMed
    • Search Google Scholar
    • Export Citation
  • 26

    López-Linares K, Aranjuelo N, Kabongo L, et al. Fully automatic detection and segmentation of abdominal aortic thrombus in post-operative CTA images using deep convolutional neural networks. Med Image Anal. 2018;46:202214.

    • PubMed
    • Search Google Scholar
    • Export Citation
  • 27

    Jani VP, Kachenoura N, Redheuil A, et al. Deep learning-based automated aortic area and distensibility assessment: the Multi-Ethnic Study of Atherosclerosis (MESA). J Digit Imaging. 2022;35(3):594604.

    • PubMed
    • Search Google Scholar
    • Export Citation
  • 28

    Biddle G, Assadsangabi R, Broadhead K, Hacein-Bey L, Ivanovic V. Diagnostic errors in cerebrovascular pathology: retrospective analysis of a neuroradiology database at a large tertiary academic medical center. AJNR Am J Neuroradiol. 2022;43(9):12711278.

    • PubMed
    • Search Google Scholar
    • Export Citation
  • 29

    510(k) Premarket Notification. Viz ANEURYSM, Viz ANX. K213319. FDA.gov. Accessed January 30, 2023. https://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfpmn/pmn.cfm?ID=K213319

    • PubMed
    • Search Google Scholar
    • Export Citation
  • 30

    Park SH, Han K. Methodologic guide for evaluating clinical performance and effect of artificial intelligence technology for medical diagnosis and prediction. Radiology. 2018;286(3):800809.

    • PubMed
    • Search Google Scholar
    • Export Citation
  • 31

    Johnsen LH, Herder M, Vangberg T, et al. Prevalence of unruptured intracranial aneurysms: impact of different definitions - the Tromsø Study. J Neurol Neurosurg Psychiatry. 2022;93(8):902907.

    • PubMed
    • Search Google Scholar
    • Export Citation
  • Collapse
  • Expand

Fiber tractography of the dorsal language pathways projecting to the lateral frontal cortex can be reliably used to map the speech output area, verified with intraoperative electrical stimulation. © Junfeng Lu, published with permission. See the article by Zhao et al. (pp 1140–1151).

  • FIG. 1.

    CNN pipeline analysis flowchart describing a stepwise approach from input scan to algorithm output. Figure is available in color online only.

  • FIG. 2.

    Flowcharts illustrating the breakdown of the analyzed data sets.

  • FIG. 3.

    ROC analysis for the entire data set and the subgroups of aneurysms ≥ 3, ≥ 4, and ≥ 5 mm. Figure is available in color online only.

  • FIG. 4.

    True-positive examples. Aneurysm detection and segmentation in a subject with an anterior communicating artery and a left middle cerebral artery bifurcation aneurysm (A). A second subject demonstrates a left internal carotid artery terminus aneurysm (B), and a third subject demonstrates a small anterior communicating artery aneurysm (C). Figure is available in color online only.

  • FIG. 5.

    False-positive examples. The infundibular origin of a right posterior communicating artery is erroneously identified as an aneurysm (A). The communicating segment of the left internal carotid artery is indicated as an aneurysm (B). Atherosclerosis in the right internal carotid artery is misclassified as an aneurysm (C). These are common locations for aneurysms and often require catheter-based angiography to completely rule out abnormalities. Figure is available in color online only.

  • 1

    Thien A, See AAQ, Ang SYL, et al. Prevalence of asymptomatic unruptured intracranial aneurysms in a southeast Asian population. World Neurosurg. 2017;97:326332.

  • 2

    Juvela S, Poussa K, Lehto H, Porras M. Natural history of unruptured intracranial aneurysms: a long-term follow-up study. Stroke. 2013;44(9):24142421.

  • 3

    Nieuwkamp DJ, Setz LE, Algra A, Linn FHH, de Rooij NK, Rinkel GJE. Changes in case fatality of aneurysmal subarachnoid haemorrhage over time, according to age, sex, and region: a meta-analysis. Lancet Neurol. 2009;8(7):635642.

    • PubMed
    • Search Google Scholar
    • Export Citation
  • 4

    Brown RD Jr, Broderick JP. Unruptured intracranial aneurysms: epidemiology, natural history, management options, and familial screening. Lancet Neurol. 2014;13(4):393404.

    • PubMed
    • Search Google Scholar
    • Export Citation
  • 5

    Bijlenga P, Gondar R, Schilling S, et al. PHASES score for the management of intracranial aneurysm: a cross-sectional population-based retrospective study. Stroke. 2017;48(8):21052112.

    • PubMed
    • Search Google Scholar
    • Export Citation
  • 6

    Wiebers DO, Whisnant JP, Huston J III, et al. Unruptured intracranial aneurysms: natural history, clinical outcome, and risks of surgical and endovascular treatment. Lancet. 2003;362(9378):103110.

    • PubMed
    • Search Google Scholar
    • Export Citation
  • 7

    Wong SC, Nawawi O, Ramli N, Abd Kadir KA. Benefits of 3D rotational DSA compared with 2D DSA in the evaluation of intracranial aneurysm. Acad Radiol. 2012;19(6):701707.

    • PubMed
    • Search Google Scholar
    • Export Citation
  • 8

    Tomandl BF, Köstner NC, Schempershofe M, et al. CT angiography of intracranial aneurysms: a focus on postprocessing. Radiographics. 2004;24(3):637655.

  • 9

    Lu L, Zhang LJ, Poon CS, et al. Digital subtraction CT angiography for detection of intracranial aneurysms: comparison with three-dimensional digital subtraction angiography. Radiology. 2012;262(2):605612.

    • PubMed
    • Search Google Scholar
    • Export Citation
  • 10

    D’Argento F, Pedicelli A, Ciardi C, et al. Intra- and inter-observer variability in intracranial aneurysm segmentation: comparison between CT angiography (semi-automated segmentation software stroke VCAR) and digital subtraction angiography (3D rotational angiography). Radiol Med (Torino). 2021;126(3):484493.

    • PubMed
    • Search Google Scholar
    • Export Citation
  • 11

    Udupa JK. Three-dimensional visualization and analysis methodologies: a current perspective. Radiographics. 1999;19(3):783806.

  • 12

    Murray NM, Unberath M, Hager GD, Hui FK. Artificial intelligence to diagnose ischemic stroke and identify large vessel occlusions: a systematic review. J Neurointerv Surg. 2020;12(2):156164.

    • PubMed
    • Search Google Scholar
    • Export Citation
  • 13

    Soun JE, Chow DS, Nagamine M, et al. Artificial intelligence and acute stroke imaging. AJNR Am J Neuroradiol. 2021;42(1):211.

  • 14

    Morey J, Fiano E, Yaeger K, Zhang X, Fifi J. Impact of Viz LVO on time-to-treatment and clinical outcomes in large vessel occlusion stroke patients presenting to primary stroke centers. medRxiv. Preprint posted online July 5, 2020. doi:10.1101/2020.07.02.20143834

    • PubMed
    • Search Google Scholar
    • Export Citation
  • 15

    Kalagara R, Chennareddy S, Matsoukas S, et al. Abstract 1122-000033: Automated detection of intracerebral hemorrhage using artificial intelligence: pilot deployment of Viz ICH. Stroke Vasc Interv Neurol. 2021;1(S1):e12079.

    • PubMed
    • Search Google Scholar
    • Export Citation
  • 16

    Hassan AE, Ringheanu VM, Rabah RR, Preston L, Tekle WG, Qureshi AI. Early experience utilizing artificial intelligence shows significant reduction in transfer times and length of stay in a hub and spoke model. Interv Neuroradiol. 2020;26(5):615622.

    • PubMed
    • Search Google Scholar
    • Export Citation
  • 17

    Rodrigues G, Barreira CM, Bouslama M, et al. Automated large artery occlusion detection in stroke: a single-center validation study of an artificial intelligence algorithm. Cerebrovasc Dis. 2022;51(2):259264.

    • PubMed
    • Search Google Scholar
    • Export Citation
  • 18

    Morey JR, Zhang X, Yaeger KA, et al. Real-world experience with artificial intelligence-based triage in transferred large vessel occlusion stroke patients. Cerebrovasc Dis. 2021;50(4):450455.

    • PubMed
    • Search Google Scholar
    • Export Citation
  • 19

    Wilson EB. Probable inference, the law of succession, and statistical inference. J Am Stat Assoc. 1927;22(158):209212.

  • 20

    Yang ZL, Ni QQ, Schoepf UJ, et al. Small intracranial aneurysms: diagnostic accuracy of CT angiography. Radiology. 2017;285(3):941952.

  • 21

    Etminan N, Brown RD Jr, Beseoglu K, et al. The unruptured intracranial aneurysm treatment score: a multidisciplinary consensus. Neurology. 2015;85(10):881889.

  • 22

    Chen W, Wang J, Xin W, Peng Y, Xu Q. Accuracy of 16-row multislice computed tomographic angiography for assessment of small cerebral aneurysms. Neurosurgery. 2008;62(1):113122.

    • PubMed
    • Search Google Scholar
    • Export Citation
  • 23

    Ueda D, Yamamoto A, Nishimori M, et al. Deep learning for MR angiography: automated detection of cerebral aneurysms. Radiology. 2019;290(1):187194.

  • 24

    Duan H, Huang Y, Liu L, Dai H, Chen L, Zhou L. Automatic detection on intracranial aneurysm from digital subtraction angiography with cascade convolutional neural networks. Biomed Eng Online. 2019;18(1):110.

    • PubMed
    • Search Google Scholar
    • Export Citation
  • 25

    Jin H, Geng J, Yin Y, et al. Fully automated intracranial aneurysm detection and segmentation from digital subtraction angiography series using an end-to-end spatiotemporal deep neural network. J Neurointerv Surg. 2020;12(10):10231027.

    • PubMed
    • Search Google Scholar
    • Export Citation
  • 26

    López-Linares K, Aranjuelo N, Kabongo L, et al. Fully automatic detection and segmentation of abdominal aortic thrombus in post-operative CTA images using deep convolutional neural networks. Med Image Anal. 2018;46:202214.

    • PubMed
    • Search Google Scholar
    • Export Citation
  • 27

    Jani VP, Kachenoura N, Redheuil A, et al. Deep learning-based automated aortic area and distensibility assessment: the Multi-Ethnic Study of Atherosclerosis (MESA). J Digit Imaging. 2022;35(3):594604.

    • PubMed
    • Search Google Scholar
    • Export Citation
  • 28

    Biddle G, Assadsangabi R, Broadhead K, Hacein-Bey L, Ivanovic V. Diagnostic errors in cerebrovascular pathology: retrospective analysis of a neuroradiology database at a large tertiary academic medical center. AJNR Am J Neuroradiol. 2022;43(9):12711278.

    • PubMed
    • Search Google Scholar
    • Export Citation
  • 29

    510(k) Premarket Notification. Viz ANEURYSM, Viz ANX. K213319. FDA.gov. Accessed January 30, 2023. https://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfpmn/pmn.cfm?ID=K213319

    • PubMed
    • Search Google Scholar
    • Export Citation
  • 30

    Park SH, Han K. Methodologic guide for evaluating clinical performance and effect of artificial intelligence technology for medical diagnosis and prediction. Radiology. 2018;286(3):800809.

    • PubMed
    • Search Google Scholar
    • Export Citation
  • 31

    Johnsen LH, Herder M, Vangberg T, et al. Prevalence of unruptured intracranial aneurysms: impact of different definitions - the Tromsø Study. J Neurol Neurosurg Psychiatry. 2022;93(8):902907.

    • PubMed
    • Search Google Scholar
    • Export Citation

Metrics

All Time Past Year Past 30 Days
Abstract Views 1953 750 0
Full Text Views 1097 849 49
PDF Downloads 746 508 43
EPUB Downloads 0 0 0