Boosting phase-contrast MRI performance in idiopathic normal pressure hydrocephalus diagnostics by means of machine learning approach

Aleš VlasákDepartment of Neurosurgery, 2nd Faculty of Medicine, Charles University in Prague and Motol University Hospital, Prague;
Department of Neurosurgery and Neurooncology, 1st Faculty of Medicine, Charles University in Prague and Military University Hospital, Prague;

Search for other papers by Aleš Vlasák in
jns
Google Scholar
PubMed
Close
 MD
,
Václav GerlaDepartment of Cognitive Systems and Neurosciences, Czech Institute of Informatics, Robotics and Cybernetics, Czech Technical University, Prague;

Search for other papers by Václav Gerla in
jns
Google Scholar
PubMed
Close
 MSc, PhD
,
Petr SkalickýDepartment of Neurosurgery, 2nd Faculty of Medicine, Charles University in Prague and Motol University Hospital, Prague;
Department of Neurosurgery and Neurooncology, 1st Faculty of Medicine, Charles University in Prague and Military University Hospital, Prague;

Search for other papers by Petr Skalický in
jns
Google Scholar
PubMed
Close
 MD
,
Arnošt MládekDepartment of Neurosurgery and Neurooncology, 1st Faculty of Medicine, Charles University in Prague and Military University Hospital, Prague;
Department of Cognitive Systems and Neurosciences, Czech Institute of Informatics, Robotics and Cybernetics, Czech Technical University, Prague;

Search for other papers by Arnošt Mládek in
jns
Google Scholar
PubMed
Close
 MSc, PhD
,
Vojtěch SedlákDepartment of Radiology, Military University Hospital, Prague; and

Search for other papers by Vojtěch Sedlák in
jns
Google Scholar
PubMed
Close
 MD
,
Jiří VránaDepartment of Radiology, Military University Hospital, Prague; and

Search for other papers by Jiří Vrána in
jns
Google Scholar
PubMed
Close
 MD, PhD
,
Helen WhitleyDepartment of Neurosurgery, 2nd Faculty of Medicine, Charles University in Prague and Motol University Hospital, Prague;

Search for other papers by Helen Whitley in
jns
Google Scholar
PubMed
Close
 BSc
,
Lenka LhotskáDepartment of Cognitive Systems and Neurosciences, Czech Institute of Informatics, Robotics and Cybernetics, Czech Technical University, Prague;
Department of Natural Sciences, Faculty of Biomedical Engineering, Czech Technical University, Prague, Czech Republic

Search for other papers by Lenka Lhotská in
jns
Google Scholar
PubMed
Close
 MSc, PhD
,
Vladimír Beneš Sr.Department of Neurosurgery and Neurooncology, 1st Faculty of Medicine, Charles University in Prague and Military University Hospital, Prague;

Search for other papers by Vladimír Beneš Sr. in
jns
Google Scholar
PubMed
Close
 MD, PhD
,
Vladimír Beneš Jr.Department of Neurosurgery, 2nd Faculty of Medicine, Charles University in Prague and Motol University Hospital, Prague;

Search for other papers by Vladimír Beneš Jr. in
jns
Google Scholar
PubMed
Close
 MD, PhD
, and
Ondřej BradáčDepartment of Neurosurgery, 2nd Faculty of Medicine, Charles University in Prague and Motol University Hospital, Prague;
Department of Neurosurgery and Neurooncology, 1st Faculty of Medicine, Charles University in Prague and Military University Hospital, Prague;

Search for other papers by Ondřej Bradáč in
jns
Google Scholar
PubMed
Close
 MD, MSc, PhD
View More View Less
Free access

OBJECTIVE

Phase-contrast MRI allows detailed measurements of various parameters of CSF motion. This examination is technically demanding and machine dependent. The literature on this topic is ambiguous. Machine learning (ML) approaches have already been successfully utilized in medical research, but none have yet been applied to enhance the results of CSF flowmetry. The aim of this study was to evaluate the possible contribution of ML algorithms in enhancing the utilization and results of MRI flowmetry in idiopathic normal pressure hydrocephalus (iNPH) diagnostics.

METHODS

The study cohort consisted of 30 iNPH patients and 15 healthy controls examined on one MRI machine. All major phase-contrast parameters were inspected: peak positive, peak negative, and average velocities; peak amplitude; positive, negative, and average flow rates; and aqueductal area. The authors applied ML algorithms to 85 complex features calculated from a phase-contrast study.

RESULTS

The most distinctive parameters with p < 0.005 were the peak negative velocity, peak amplitude, and negative flow. From the ML algorithms, the Adaptive Boosting classifier showed the highest specificity and best discrimination potential overall, with 80.4% ± 2.9% accuracy, 72.0% ± 5.6% sensitivity, 84.7% ± 3.8% specificity, and 0.812 ± 0.047 area under the receiver operating characteristic curve (AUC). The highest sensitivity was 85.7% ± 5.6%, reached by the Gaussian Naive Bayes model, and the best AUC was 0.854 ± 0.028 by the Extra Trees classifier.

CONCLUSIONS

Feature extraction algorithms combined with ML approaches simplify the utilization of phase-contrast MRI. The highest-performing ML algorithm was Adaptive Boosting, which showed good calibration and discrimination on the testing data, with 80.4% accuracy, 72.0% sensitivity, 84.7% specificity, and 0.812 AUC. Phase-contrast MRI boosted by the ML approach can help to determine shunt-responsive iNPH patients.

ABBREVIATIONS

AdaBoost = Adaptive Boosting; AUC = area under the ROC the curve; CV = cross-validation; GaussNB = Gaussian Naive Bayes; GBDT = Gradient Boosting Decision Tree; ICP = intracranial pressure; iNPH = idiopathic NPH; LogReg = Logistic Regression; ML = machine learning; MLP = multilayer perceptron; NPH = normal pressure hydrocephalus; RF = Random Forest; ROC = receiver operating characteristic; XGB = XGBoost.

OBJECTIVE

Phase-contrast MRI allows detailed measurements of various parameters of CSF motion. This examination is technically demanding and machine dependent. The literature on this topic is ambiguous. Machine learning (ML) approaches have already been successfully utilized in medical research, but none have yet been applied to enhance the results of CSF flowmetry. The aim of this study was to evaluate the possible contribution of ML algorithms in enhancing the utilization and results of MRI flowmetry in idiopathic normal pressure hydrocephalus (iNPH) diagnostics.

METHODS

The study cohort consisted of 30 iNPH patients and 15 healthy controls examined on one MRI machine. All major phase-contrast parameters were inspected: peak positive, peak negative, and average velocities; peak amplitude; positive, negative, and average flow rates; and aqueductal area. The authors applied ML algorithms to 85 complex features calculated from a phase-contrast study.

RESULTS

The most distinctive parameters with p < 0.005 were the peak negative velocity, peak amplitude, and negative flow. From the ML algorithms, the Adaptive Boosting classifier showed the highest specificity and best discrimination potential overall, with 80.4% ± 2.9% accuracy, 72.0% ± 5.6% sensitivity, 84.7% ± 3.8% specificity, and 0.812 ± 0.047 area under the receiver operating characteristic curve (AUC). The highest sensitivity was 85.7% ± 5.6%, reached by the Gaussian Naive Bayes model, and the best AUC was 0.854 ± 0.028 by the Extra Trees classifier.

CONCLUSIONS

Feature extraction algorithms combined with ML approaches simplify the utilization of phase-contrast MRI. The highest-performing ML algorithm was Adaptive Boosting, which showed good calibration and discrimination on the testing data, with 80.4% accuracy, 72.0% sensitivity, 84.7% specificity, and 0.812 AUC. Phase-contrast MRI boosted by the ML approach can help to determine shunt-responsive iNPH patients.

Despite being first described more than 55 years ago,1 idiopathic normal pressure hydrocephalus (iNPH) is still without a sufficiently sensitive and specific diagnostic test. The need for such a test is accentuated by the fact that iNPH is one of the treatable forms of dementia, with reported improvement of 60%–80% after shunt insertion.2 In current practice, the diagnosis consists of clinical examination (typically characterized by the triad of gait disturbance, mental deterioration, and urinary incontinence) and functional testing. Functional tests include the spinal tap test, external lumbar drainage, or lumbar infusion test. Although these tests can accurately predict response to treatment,3 they are painful and associated with rare, but potentially serious, complications.4 For these reasons, numerous studies in the past few decades have focused on finding a simple imaging biomarker. Enlarged ventricles are mandatory for iNPH diagnosis, but ventriculomegaly has numerous causes, so the specificity of this sign is naturally very low. Unfortunately, more detailed tests have not resulted in the unambiguous confirmation or exclusion of an iNPH diagnosis. Basic MRI sequences are the source of several significant signs, such as high Evans’ index, dilated sylvian fissures, tight high convexity, acute callosal angle, and focal sulcal dilation. All these measurements are components of the DESH (disproportionately enlarged subarachnoid space hydrocephalus) score, introduced by Shinoda et al.5 Our recent study showed that the DESH score lacks sufficient sensitivity and specificity to be used as a stand-alone diagnostic or prognostic marker for iNPH.6 Our laboratory has also tested the hypothesis that structural volume analysis can reveal specific patterns unique to iNPH patients.6 Despite identifying several interesting differences in structural volumes in iNPH patients, this method did not reveal any signs that differentiate shunt-responsive patients. Diffusion tensor imaging repeatedly showed changes in white matter,7,8 but again, the sensitivity and specificity were not sufficient for this to be used as a stand-alone test.

The CSF flow void phenomenon observed in the cerebral aqueduct of iNPH patients has led to interest in this region. Phase-contrast MRI allows detailed measurement of various parameters of CSF motion.9 These include aqueductal stroke volume and peak velocity measurements, with several studies showing promising results.10–12 However, some authors have pointed out that the examination is technically demanding and machine dependent.13

The aim of this study was to evaluate the possible contribution of machine learning (ML) algorithms in enhancing the results of MRI flowmetry in normal pressure hydrocephalus (NPH) diagnostics. In contrast to the previously published studies, we have looked at the method from a wider perspective of all the available features.

Methods

IRB Statement and Reporting

The study was conducted in accordance with the rules and regulations of the Military University Hospital in Prague, as approved by the institutional ethics board. All patients signed informed consent forms prior to the procedures. The Transparent Reporting of Multivariable Prediction Models for Individual Prognosis or Diagnosis (TRIPOD) checklist was followed in our study.

Patients

In the period between September 2016 and March 2020, 109 patients were referred to the Department of Neurosurgery and Neurooncology, 1st Faculty of Medicine, and Military University Hospital in Prague with suspected NPH. Before being referred to our department, all patients had undergone standard MRI and exhibited ventriculomegaly (Evans’ index > 0.3). All patients had gait disturbance and at least one of the other two typical symptoms: mental deterioration or urinary incontinence. The gait was recorded on camera, and the disturbance was evaluated using the Dutch Gait Scale.14,15 All patients underwent thorough neuropsychological examination including the Wechsler Memory Scale, Third Edition; Montreal Cognitive Assessment; verbal fluency tests; Trail Making Test; Rey-Osterrieth Complex Figure Drawing Test; and Geriatric Depression Scale.16 After completing all of the above tests, 86 remaining patients underwent a lumbar infusion (modified Katzmann’s) test.17 All patients had normal CSF opening pressure (< 20 cm H2O) and normal CSF composition, laboratory values, and cell count. At the end of the test, lumbar drainage was performed using the same needle, and CSF was drained for 120 hours. Shunt insertion was indicated for all patients with a positive lumbar infusion test (resistance to outflow > 9 mm Hg/ml/min) and at least 15% improvement in the Dutch Gait Scale after lumbar drainage. After patients completed all tests, we identified 40 iNPH patients. All of them were indicated for ventriculoperitoneal shunt surgery.

From this cohort, 30 patients completed the study protocol on the same MRI machine. The same protocol was performed on 15 healthy controls. Subjects selected for the control group were age- and sex-related volunteers who showed no symptoms typical of iNPH and for whom no neurological disorder was detected during clinical and imaging examination. The demographic data are presented in Table 1.

TABLE 1.

Demographic data of both iNPH and healthy control groups

Groupp Value
iNPHControl
Male63.3%60.0%0.668
Female36.7%40.0%
Age in yrs, mean ± SD72.8 ± 5.271.4 ± 6.40.044

An algorithm of data acquisition and processing is shown in Fig. 1.

FIG. 1.
FIG. 1.

Diagram summarizing the methodology of the presenting study.

MRI Acquisition

MRI scans were acquired prior to functional testing on the 3T GE Discovery MR750w (GE Healthcare) at the Military University Hospital in Prague. A standard 32-channel head coil was used. The MRI protocol included a phase-contrast CSF flow study with the following parameters: a single oblique axial section perpendicular to the aqueduct with slice thickness 7 mm, FOV 16 cm, matrix size 256 × 224, TR 33 msec, TE 7.4 msec, and flip angle 30°. Cardiac gating was applied using an MR-compatible peripheral pulse transducer attached to the subject’s finger, producing 32 frames evenly spread throughout the cardiac cycle. The location of the acquisition section was determined by a senior neuroradiologist (J.V.) in the middle section of the aqueduct. The default velocity encoding gradient was 20 cm/sec.

Image Interpretation

Phase-contrast images were reconstructed and reviewed using commercial software (FlowAnalysis, GE Healthcare). The region of interest was drawn manually by a senior neuroradiologist (J.V.) and included all voxels with CSF flow signal. The software was then used to extract velocity and flow-time curves for each subject. The results were transferred to an offline workstation for statistical analysis.

Feature Extraction

For each patient, 7 CSF flowmetry vectors were obtained directly from MRI: aqueductal area, peak positive velocity, peak negative velocity, average velocity, positive flow rate, negative flow rate, and average flow rate. Each vector is composed of 32 points evenly distributed along one cardiac cycle. To calculate CSF flow features, 32 points of each vector were interpolated by a cubic spline. Spline interpolation is often preferred over polynomial interpolation because the interpolation error is minimal even when using low-degree polynomials for the spline. The features of a given vector were calculated using the piecewise smooth interpolating function, and these features characterize the spline behavior. Altogether, 85 basic features were identified. These parameters were derived from final splines using the following basic mathematical values: mean value, standard deviation, maximum and minimum values, time delay of found minimum and maximum values, maximum and minimum values of first and second derivatives of signal data, skewness, and kurtosis. This feature extraction procedure expands the possibilities of basic characteristics that can be obtained directly from a phase-contrast MRI test.

Statistical Analysis

Comparisons of continuous variables were made using t-tests for independent samples. Comparisons of categorical variables were done using the chi-square test. In all cases, p < 0.05 was considered significant. Basic computations were performed using Statistica version 13.5 software (TIBCO).

ML Approach

A set of 87 selected complex features were calculated for each patient (85 features plus age and sex). The whole patient data set was divided into training and testing parts by k-fold (k = 5) cross-validation (CV). We improve k-fold CV by using stratified resampling, which ensures that the relative class frequencies (iNPH and control) are approximately preserved in each fold according to the original class frequencies in the full data set. Stratified k-fold CV is useful for small and/or imbalanced data sets (30 iNPH and 15 healthy controls in our case).18 Another improvement was the repetition of the k-fold stratified CV process N times (n = 10), enabling an estimate of the mean and standard deviation in a performance.

The following 8 different state-of-the-art ML models were deployed by using the aforementioned robust CV design: multilayer perceptron (MLP), Gaussian Naive Bayes (GaussNB), Gradient Boosting Decision Tree (GBDT), Logistic Regression (LogReg), Extra Trees (ExtraTrees), Random Forest (RF), XGBoost (XGB) and Adaptive Boosting (AdaBoost). The listed algorithms are implemented and described in the Scikit-Learn Python library19 and were run in Python 3.8. Because of the better repeatability of the proposed solution, default settings of all algorithms were used. Accuracy, sensitivity, specificity, receiver operating characteristic (ROC) curve, and area under the ROC curve (AUC) were used to compare the performance of all ML methods.

Results

The study cohort consisted of 30 iNPH patients consecutively examined on one MRI machine. The mean age in the group was 72.8 ± 5.2 years, and there were 19 men and 11 women. In the control group, there were 9 men and 6 women with an average age of 71.4 ± 6.4 years. The groups were similar in the scope of basic demographic information.

Within the scope of phase-contrast MRI, all major parameters were inspected: peak positive, peak negative, and average velocities; peak amplitude; positive, negative, and average flow rates; and aqueductal area. Using the t-test for a direct comparison, we found significant differences, with p < 0.05 in 47 of 85 tested features. Many of the parameters are not easy to interpret. We selected the major parameters according to their appearance in the literature and present them in Table 2.

TABLE 2.

Results of major phase-contrast MRI parameters and their significance according to t-test

Groupp Value
iNPHControl
Aqueductal area (mm2)17.2693 ± 9.079920.4736 ± 10.01550.297
Peak positive velocity (cm/sec)3.3465 ± 2.04261.7965 ± 0.72320.007
Peak negative velocity (cm/sec)−3.0787 ± 1.9189−1.3570 ± 0.58680.002
Peak amplitude (cm/sec)6.4252 ± 3.81393.1535 ± 1.18110.003
Average velocity (cm/sec)5.0215 ± 4.26015.0106 ± 4.80310.994
Average velocity, skewness2.7707 ± 1.04971.8253 ± 1.16540.010
Average velocity, kurtosis11.2075 ± 6.00857.1157 ± 5.16390.032
Positive flow (ml/min)0.2561 ± 0.22420.1140 ± 0.07070.022
Negative flow (ml/min)−0.1996 ± 0.1561−0.0597 ± 0.03580.002
Average flow (ml/min)0.0564 ± 0.09440.0537 ± 0.06050.919
Highest peak positive velocity (cm/sec)8.5090 ± 6.19654.4040 ± 2.79880.019
Flow (ml/beat)0.0419 ± 0.08060.0403 ± 0.05780.947

Values are presented as mean ± SD. Boldface type indicates statistical significance.

The most distinctive parameters (p < 0.005) were peak negative velocity, peak amplitude, and negative flow. The mean peak negative velocity was −3.0787 ± 1.9189 cm/sec in iNPH patients and −1.3570 ± 0.5868 cm/sec in healthy controls (p = 0.002). The mean peak amplitude was 6.4252 ± 3.8139 cm/sec in iNPH patients and 3.1535 ± 1.1811 cm/sec in healthy controls (p = 0.003). The mean negative flow was −0.1996 ± 0.1561 ml/min in iNPH patients and −0.0597 ± 0.0358 ml/min in healthy controls (p = 0.002). A fourth important feature was mean peak positive velocity, which was −3.3465 ± 2.0426 cm/sec in iNPH patients and 1.7965 ± 0.7232 cm/sec in healthy controls (p = 0.007). The results of these 4 features are presented in Fig. 2.

FIG. 2.
FIG. 2.

Boxplots with mean values (solid lines), first and third quantiles, minimum and maximum values (whiskers), and outliers (circles) shown for the four most significant parameters differentiating the iNPH group from the healthy controls: peak positive velocity (A), peak negative velocity (B), peak amplitude (C), and negative flow (D).

We continued with further computations using 8 different state-of-the-art ML models. The results in terms of accuracy classification of respective ML models are presented in Table 3.

TABLE 3.

Accuracy, sensitivity, specificity, and AUC computed over 10 CV repetitions

ML ApproachAccuracySensitivitySpecificityAUC
MLP72.0 ± 3.454.7 ± 5.680.7 ± 5.50.750 ± 0.048
GaussNB73.8 ± 2.485.7 ± 5.673.3 ± 2.40.770 ± 0.009
GBDT73.8 ± 4.364.0 ± 3.778.7 ± 6.50.747 ± 0.064
LogReg76.9 ± 4.065.3 ± 5.682.7 ± 4.90.808 ± 0.029
ExtraTrees77.3 ± 1.973.3 ± 4.779.3 ± 4.90.854 ± 0.028
RF78.2 ± 4.074.7 ± 5.680.0 ± 4.10.813 ± 0.027
XGB79.6 ± 1.972.0 ± 5.683.3 ± 2.40.840 ± 0.037
AdaBoost80.4 ± 2.972.0 ± 5.684.7 ± 3.80.812 ± 0.047

Values are presented as mean ± SD. Individual models are sorted according to the resulting accuracy (from lowest to highest). The best value for each of the parameters is shown in italics.

Table 3 compares accuracies, sensitivities, specificities, and AUCs for all ML algorithms developed. From these algorithms, the AdaBoost classifier showed the highest specificity and best discrimination potential overall, with 80.4% ± 2.9% accuracy, 72.0% ± 5.6% sensitivity, 84.7% ± 3.8% specificity, and 0.812 ± 0.047 AUC. The highest sensitivity was 85.7% ± 5.6%, reached by the GaussNB model, and the best AUC was 0.854 ± 0.028, by the ExtraTrees classifier. The final ROCs and calibration curves for all ML models are presented in Fig. 3.

FIG. 3.
FIG. 3.

ROC (left) and calibration (right) curves for all individual ML models. The dashed diagonal line represents the performance of an ideal model, where the predicted outcome would correspond perfectly with the actual outcome.

The importance of a feature by AdaBoost was computed as the normalized total reduction of the criterion brought by that feature (the higher the reduction, the more important the feature). It is also known as the Gini importance.19 Feature importance differs a lot from the significance counted with the chi-square test. No “major” feature had any importance for the AdaBoost classifier, and the most significant parameters, regardless of importance, played only a minor role in its computations (Table 4).

TABLE 4.

Feature importance for the AdaBoost classifier

Feature Descriptionp ValueFeature Importance by AdaBoost
Best 4 features according to phase-contrast MRIMean negative flow0.0020
Mean peak negative velocity0.0020
Mean peak amplitude0.0030
Mean peak positive velocity0.0070
Best 4 features according to t-testMinimum peak negative velocity0.00010
Peak negative velocity, amplitude0.00020.002
Maximum negative flow derivation0.00020.014
Peak negative velocity standard deviation0.00020.006
Best 4 features according to ML (AdaBoost)Average velocity derivation, position of maximum value0.0120.016
Negative flow derivation, maximum value0.00020.014
Negative flow derivation, position of minimum value0.0270.013
Maximum negative flow0.2710.008

The four most significant parameters are given for each of phase-contrast MRI, the whole data set, and machine learning.

Discussion

According to the most accepted theory, the development of the major symptoms of iNPH is caused by ventricular dilation leading to mechanical stress on the periventricular white matter. This causes ischemia and hypoxia of axons.20 The severed ependymal layer progressively loses plasticity, and pulsatility is significantly reduced as a consequence.21 This process leads to an impairment of bulk flow through the outlets of the CSF compartments.22 Compressed adjacent white matter loses integrity and becomes stiff, leading to reduced transmission of pulsatile waves.23 This process causes dilation of ventricles, which further slows down the CSF flow.24 The exact pathophysiological mechanism of iNPH remains unclear, but the above theory remains one of the most widely accepted.25 Since MRI has become more broadly available, many scientists have had great hopes in phase-contrast MRI, which seemed to promise the long-anticipated biomarker for selecting patients with shunt-responsive iNPH.26 Unfortunately, it has become clear that phase-contrast MRI is not easy to interpret, as its results vary according to the MRI machine used.13 This led to the fact that not many authors really tested this method, and the literature on this topic is scarce. All present articles have focused on one or a few parameters of phase-contrast MRI. The most studied parameters are peak mean velocity and aqueductal stroke volume. The results are rather controversial. There is a study by Tawfik et al.11 that shows great results for both parameters, with a diagnostic accuracy 92.5%–93.3% for peak mean velocity and even 100% for aqueductal stroke volume. However, these results were not supported by those of other authors. In contrast, aqueductal stroke volume was identified as a poor predictor of shunt responsiveness by Blitz et al.27 This unfavorable result was supported by those of other authors.28

In our study, we have looked at all the available parameters of phase-contrast MRI examination. The flow curve is defined by 7 vectors: aqueductal area, peak positive velocity, peak negative velocity, average velocity, positive flow rate, negative flow rate, and average flow rate. We have identified the three most distinct parameters with a p value < 0.005 with direct comparison using the t-test: peak negative velocity, peak amplitude, and negative flow. These findings may be related to the increased intracranial pressure (ICP) pulsatility in iNPH patients observed from invasive ICP monitoring,29,30 while the altered negative flow and higher peak amplitude observed on phase-contrast MRI in normal conditions could represent increased ICP pulsatility observed in overnight ICP monitoring.31 The results of ICP monitoring on shunt response prediction in the literature vary,31–33 but the role of altered wave characteristics observed in our study with regard to prediction of shunt response has yet to be clarified.

ML approaches have already been successfully utilized in medical research.34 Regarding iNPH, a few models have used this technique, for example, in gait35 or MRI36 analysis. To our knowledge, no ML approach has been applied to enhance the results of CSF flowmetry. We have considered this method beneficial, because some flowmetry features are difficult to interpret as they lack a clear clinical correlation, and their physiological explanation is rather speculative and under further investigation. Furthermore, the importance of individual features does not necessarily correlate with the p values. Features that would have been ignored in standard statistical testing as insignificant in iNPH patient discrimination may turn out to be influential in the scope of ML and vice versa. Using this method, we achieved a sensitivity of up to 86% and specificity of 85%. The best accuracy was 80%. The highest-performing ML algorithm was AdaBoost. This model showed good calibration and discrimination on the testing data, with 80.4% accuracy, 72.0% sensitivity, 84.7% specificity, and 0.812 AUC. With the wide variety of published results of phase-contrast MRI, it is not easy to make a direct comparison. In contrast to the results of some papers,27,28 our results show the benefit of the phase-contrast MRI method in distinguishing iNPH patients from healthy controls, but we cannot confirm the results of Tawfik et al.11

The developed ML models were optimized for highly accurate prediction rather than explanation, and model parameters thus cannot be simply deployed for the purpose of explaining the effect of individual features on the differentiation of iNPH patients and healthy controls. Some of the frequently used ML models (especially ensemble-based algorithms) allow the use of the so-called optimization of hyperparameters. This could further improve the performance of the ML models. This approach could be used in the future if the data set were enlarged. Further external validations on data from multiple neurosurgical centers would be appropriate before using these approaches in clinical practice. In the case of identical scanning parameters (detailed in the MRI Acquisition section), our approach is fully transferable to scanners of the same model (GE Discovery MR750w). In the case of different scanning parameters and/or different scanner models being used, prior calibration on a phantom should yield the same results when using our approach.

Study Limitations

We are aware of several limitations of the present study. First, because of a change of MRI machines during the selected study period, 10 of the 40 iNPH patients had to be removed from the study. It has been well documented that phase-contrast MRI can vary considerably according to the equipment.13 The method is also operator dependent. The region of interest is manually drawn, which emphasizes the importance of an experienced neuroradiologist. Second, several patients may be incorrectly grouped in or erroneously excluded from the iNPH group due to the specificity and sensitivity limitations of the lumbar infusion test and external lumbar drainage test. As these functional tests are current best practice, this problem limits all studies concerning iNPH. Third, despite using an effective stratified fivefold CV repeated 10 times to validate the ML approaches, the credibility of this methodology could be increased by using a larger data set with various types of CVs. Fourth, the accuracy of our protocol is limited mainly by the heterogeneity of iNPH phenotypes, in a situation in which the very nature of this disease is multifactorial. Rather than increasing the number of patients or variables examined, it is proposed to combine results from different MRI modalities in the future.

Conclusions

Developed feature extraction algorithms combined with ML approaches simplify the utilization of phase-contrast MRI. The highest-performing ML algorithm was AdaBoost, which showed good calibration and discrimination on the testing data, with 80.4% accuracy, 72.0% sensitivity, 84.7% specificity, and 0.812 AUC. Phase-contrast MRI boosted by the ML approach can help to determine shunt-responsive iNPH patients.

Acknowledgments

The research reported in this paper was supported by the Charles University Grant Agency (GAUK; no. 1068120) and the Student Grant Competition 2020 of the Czech Technical University (no. SGS21/186/OHK4/3T/37).

Disclosures

The authors report no conflict of interest concerning the materials or methods used in this study or the findings specified in this paper.

Author Contributions

Conception and design: Bradáč, Skalický. Acquisition of data: Skalický, Sedlák, Vrána. Analysis and interpretation of data: Gerla, Mládek, Vrána. Drafting the article: Vlasák. Critically revising the article: Bradáč, Gerla, Skalický, Mládek, Beneš Jr. Reviewed submitted version of manuscript: Bradáč, Gerla, Sedlák, Vrána, Whitley, Lhotská, Beneš Jr. Approved the final version of the manuscript on behalf of all authors: Bradáč. Statistical analysis: Gerla, Mládek. Administrative/technical/material support: Whitley. Study supervision: Bradáč, Lhotská, Beneš Sr.

References

  • 1

    Adams RD, Fisher CM, Hakim S, Ojemann RG, Sweet WH. Symptomatic occult hydrocephalus with “normal” cerebrospinal fluid pressure. A treatable syndrome. N Engl J Med. 1965;273:117126.

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 2

    Malm J. Improving research and care for patients with idiopathic NPH. Lancet Neurol. 2015;14(6):561563.

  • 3

    Skalický P, Mládek A, Vlasák A, De Lacy P, Beneš V, Bradáč O. Normal pressure hydrocephalus—an overview of pathophysiological mechanisms and diagnostic procedures. Neurosurg Rev. 2020;43(6):14511464.

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 4

    El Ahmadieh TY, Wu EM, Kafka B, et al. Lumbar drain trial outcomes of normal pressure hydrocephalus: a single-center experience of 254 patients. J Neurosurg. 2019;132(1):306312.

    • Search Google Scholar
    • Export Citation
  • 5

    Shinoda N, Hirai O, Hori S, et al. Utility of MRI-based disproportionately enlarged subarachnoid space hydrocephalus scoring for predicting prognosis after surgery for idiopathic normal pressure hydrocephalus: clinical research. J Neurosurg. 2017;127(6):14361442.

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 6

    Vlasák A, Skalický P, Mládek A, Vrána J, Beneš V, Bradáč O. Structural volumetry in NPH diagnostics and treatment–future or dead end? Neurosurg Rev. 2021;44(1):503514

    • Crossref
    • Search Google Scholar
    • Export Citation
  • 7

    Keong NC, Pena A, Price SJ, et al. Diffusion tensor imaging profiles reveal specific neural tract distortion in normal pressure hydrocephalus. PLoS One. 2017;12(8):e0181624.

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 8

    Hoza D, Vlasák A, Hořínek D, Sameš M, Alfieri A. DTI-MRI biomarkers in the search for normal pressure hydrocephalus aetiology: a review. Neurosurg Rev. 2015;38(2):239244.

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 9

    Sakhare AR, Barisano G, Pa J. Assessing test-retest reliability of phase contrast MRI for measuring cerebrospinal fluid and cerebral blood flow dynamics. Magn Reson Med. 2019;82(2):658670.

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 10

    Algin O, Hakyemez B, Parlak M. The efficiency of PC-MRI in diagnosis of normal pressure hydrocephalus and prediction of shunt response. Acad Radiol. 2010;17(2):181187.

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 11

    Tawfik AM, Elsorogy L, Abdelghaffar R, Naby AA, Elmenshawi I. Phase-contrast MRI CSF flow measurements for the diagnosis of normal-pressure hydrocephalus: observer agreement of velocity versus volume parameters. AJR Am J Roentgenol. 2017;208(4):838843.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • 12

    Witthiwej T, Sathira-ankul P, Chawalparit O, Chotinaiwattarakul W, Tisavipat N, Charnchaowanish P. MRI study of intracranial hydrodynamics and ventriculoperitoneal shunt responsiveness in patient with normal pressure hydrocephalus. J Med Assoc Thai. 2012;95(12):15561562.

    • PubMed
    • Search Google Scholar
    • Export Citation
  • 13

    Bradley WG Jr. Magnetic resonance imaging of normal pressure hydrocephalus. Semin Ultrasound CT MR. 2016;37(2):120128.

  • 14

    Boon AJ, Tans JT, Delwel EJ, et al. Dutch normal pressure hydrocephalus study: baseline characteristics with emphasis on clinical findings. Eur J Neurol. 1997;4(1):3947.

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 15

    Ravdin LD, Katzen HL, Jackson AE, Tsakanikas D, Assuras S, Relkin NR. Features of gait most responsive to tap test in normal pressure hydrocephalus. Clin Neurol Neurosurg. 2008;110(5):455461.

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 16

    Devito EE, Pickard JD, Salmond CH, Iddon JL, Loveday C, Sahakian BJ. The neuropsychology of normal pressure hydrocephalus (NPH). Br J Neurosurg. 2005;19(3):217224.

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 17

    Katzman R, Hussey F. A simple constant-infusion manometric test for measurement of CSF absorption. I. Rationale and method. Neurology. 1970;20(6):534544.

    • PubMed
    • Search Google Scholar
    • Export Citation
  • 18

    Ojala M, Garriga GC. Permutation tests for studying classifier performance. J Mach Learn Res. 2010;11:18331863.

  • 19

    Pedregosa F, Varoquaux G, Gramfort A, et al. Scikit-learn: machine learning in Python. J Mach Learn Res. 2011;12:28252830.

  • 20

    Akai K, Uchigasaki S, Tanaka U, Komatsu A. Normal pressure hydrocephalus. Neuropathological study. Acta Pathol Jpn. 1987;37(1):97110.

  • 21

    Greitz D. Radiological assessment of hydrocephalus: new theories and implications for therapy. Neurosurg Rev. 2004;27(3):145167.

  • 22

    Rekate HL, Nadkarni TD, Wallace D. The importance of the cortical subarachnoid space in understanding hydrocephalus. J Neurosurg Pediatr. 2008;2(1):111.

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 23

    Preuss M, Hoffmann KT, Reiss-Zimmermann M, et al. Updated physiology and pathophysiology of CSF circulation—the pulsatile vector theory. Childs Nerv Syst. 2013;29(10):18111825.

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 24

    Ammar A, Abbas F, Al Issawi W, et al. Idiopathic normal-pressure hydrocephalus syndrome: is it understood? The comprehensive idiopathic normal-pressure hydrocephalus theory (CiNPHT). In: Ammar A, ed. Hydrocephalus: What Do We Know? And What Do We Still Not Know? Springer International Publishing;2017:6782.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • 25

    Wang Z, Zhang Y, Hu F, Ding J, Wang X. Pathogenesis and pathophysiology of idiopathic normal pressure hydrocephalus. CNS Neurosci Ther. 2020;26(12):12301240.

  • 26

    Bradley WG Jr, Scalzo D, Queralt J, Nitz WN, Atkinson DJ, Wong P. Normal-pressure hydrocephalus: evaluation with cerebrospinal fluid flow measurements at MR imaging. Radiology. 1996;198(2):523529.

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 27

    Blitz AM, Shin J, Balédent O, et al. Does phase-contrast imaging through the cerebral aqueduct predict the outcome of lumbar CSF drainage or shunt surgery in patients with suspected adult hydrocephalus? AJNR Am J Neuroradiol. 2018;39(12):22242230.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • 28

    Shanks J, Markenroth Bloch K, Laurell K, et al. Aqueductal CSF stroke volume is increased in patients with idiopathic normal pressure hydrocephalus and decreases after shunt surgery. AJNR Am J Neuroradiol. 2019;40(3):453459.

    • PubMed
    • Search Google Scholar
    • Export Citation
  • 29

    Eide PK, Sorteberg W. Diagnostic intracranial pressure monitoring and surgical management in idiopathic normal pressure hydrocephalus: a 6-year review of 214 patients. Neurosurgery. 2010;66(1):8091.

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 30

    Eide PK, Brean A. Cerebrospinal fluid pulse pressure amplitude during lumbar infusion in idiopathic normal pressure hydrocephalus can predict response to shunting. Cerebrospinal Fluid Res. 2010;7(1):5.

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 31

    Eide PK, Sorteberg W. Outcome of surgery for idiopathic normal pressure hydrocephalus: role of preoperative static and pulsatile intracranial pressure. World Neurosurg. 2016;86:186193.e1.

    • PubMed
    • Search Google Scholar
    • Export Citation
  • 32

    Nabbanja E, Czosnyka M, Keong NC, et al. Is there a link between ICP-derived infusion test parameters and outcome after shunting in normal pressure hydrocephalus? Acta Neurochir Suppl. 2018;126:229232.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • 33

    Qvarlander S, Lundkvist B, Koskinen LOD, Malm J, Eklund A. Pulsatility in CSF dynamics: pathophysiology of idiopathic normal pressure hydrocephalus. J Neurol Neurosurg Psychiatry. 2013;84(7):735741.

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 34

    Garg A, Mago V. Role of machine learning in medical research: a survey. Comput Sci Rev. 2021;40:100370.

  • 35

    Jeong S, Yu H, Park J, Kang K. Quantitative gait analysis of idiopathic normal pressure hydrocephalus using deep learning algorithms on monocular videos. Sci Rep. 2021;11(1):12368.

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 36

    Irie R, Otsuka Y, Hagiwara A, et al. A novel deep learning approach with a 3D convolutional ladder network for differential diagnosis of idiopathic normal pressure hydrocephalus and Alzheimer’s disease. Magn Reson Med Sci. 2020;19(4):351358.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Collapse
  • Expand
Artwork from Agarwal et al. (E9). Copyright Kenneth X. Probst. Published with permission.
  • View in gallery
    FIG. 1.

    Diagram summarizing the methodology of the presenting study.

  • View in gallery
    FIG. 2.

    Boxplots with mean values (solid lines), first and third quantiles, minimum and maximum values (whiskers), and outliers (circles) shown for the four most significant parameters differentiating the iNPH group from the healthy controls: peak positive velocity (A), peak negative velocity (B), peak amplitude (C), and negative flow (D).

  • View in gallery
    FIG. 3.

    ROC (left) and calibration (right) curves for all individual ML models. The dashed diagonal line represents the performance of an ideal model, where the predicted outcome would correspond perfectly with the actual outcome.

  • 1

    Adams RD, Fisher CM, Hakim S, Ojemann RG, Sweet WH. Symptomatic occult hydrocephalus with “normal” cerebrospinal fluid pressure. A treatable syndrome. N Engl J Med. 1965;273:117126.

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 2

    Malm J. Improving research and care for patients with idiopathic NPH. Lancet Neurol. 2015;14(6):561563.

  • 3

    Skalický P, Mládek A, Vlasák A, De Lacy P, Beneš V, Bradáč O. Normal pressure hydrocephalus—an overview of pathophysiological mechanisms and diagnostic procedures. Neurosurg Rev. 2020;43(6):14511464.

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 4

    El Ahmadieh TY, Wu EM, Kafka B, et al. Lumbar drain trial outcomes of normal pressure hydrocephalus: a single-center experience of 254 patients. J Neurosurg. 2019;132(1):306312.

    • Search Google Scholar
    • Export Citation
  • 5

    Shinoda N, Hirai O, Hori S, et al. Utility of MRI-based disproportionately enlarged subarachnoid space hydrocephalus scoring for predicting prognosis after surgery for idiopathic normal pressure hydrocephalus: clinical research. J Neurosurg. 2017;127(6):14361442.

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 6

    Vlasák A, Skalický P, Mládek A, Vrána J, Beneš V, Bradáč O. Structural volumetry in NPH diagnostics and treatment–future or dead end? Neurosurg Rev. 2021;44(1):503514

    • Crossref
    • Search Google Scholar
    • Export Citation
  • 7

    Keong NC, Pena A, Price SJ, et al. Diffusion tensor imaging profiles reveal specific neural tract distortion in normal pressure hydrocephalus. PLoS One. 2017;12(8):e0181624.

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 8

    Hoza D, Vlasák A, Hořínek D, Sameš M, Alfieri A. DTI-MRI biomarkers in the search for normal pressure hydrocephalus aetiology: a review. Neurosurg Rev. 2015;38(2):239244.

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 9

    Sakhare AR, Barisano G, Pa J. Assessing test-retest reliability of phase contrast MRI for measuring cerebrospinal fluid and cerebral blood flow dynamics. Magn Reson Med. 2019;82(2):658670.

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 10

    Algin O, Hakyemez B, Parlak M. The efficiency of PC-MRI in diagnosis of normal pressure hydrocephalus and prediction of shunt response. Acad Radiol. 2010;17(2):181187.

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 11

    Tawfik AM, Elsorogy L, Abdelghaffar R, Naby AA, Elmenshawi I. Phase-contrast MRI CSF flow measurements for the diagnosis of normal-pressure hydrocephalus: observer agreement of velocity versus volume parameters. AJR Am J Roentgenol. 2017;208(4):838843.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • 12

    Witthiwej T, Sathira-ankul P, Chawalparit O, Chotinaiwattarakul W, Tisavipat N, Charnchaowanish P. MRI study of intracranial hydrodynamics and ventriculoperitoneal shunt responsiveness in patient with normal pressure hydrocephalus. J Med Assoc Thai. 2012;95(12):15561562.

    • PubMed
    • Search Google Scholar
    • Export Citation
  • 13

    Bradley WG Jr. Magnetic resonance imaging of normal pressure hydrocephalus. Semin Ultrasound CT MR. 2016;37(2):120128.

  • 14

    Boon AJ, Tans JT, Delwel EJ, et al. Dutch normal pressure hydrocephalus study: baseline characteristics with emphasis on clinical findings. Eur J Neurol. 1997;4(1):3947.

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 15

    Ravdin LD, Katzen HL, Jackson AE, Tsakanikas D, Assuras S, Relkin NR. Features of gait most responsive to tap test in normal pressure hydrocephalus. Clin Neurol Neurosurg. 2008;110(5):455461.

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 16

    Devito EE, Pickard JD, Salmond CH, Iddon JL, Loveday C, Sahakian BJ. The neuropsychology of normal pressure hydrocephalus (NPH). Br J Neurosurg. 2005;19(3):217224.

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 17

    Katzman R, Hussey F. A simple constant-infusion manometric test for measurement of CSF absorption. I. Rationale and method. Neurology. 1970;20(6):534544.

    • PubMed
    • Search Google Scholar
    • Export Citation
  • 18

    Ojala M, Garriga GC. Permutation tests for studying classifier performance. J Mach Learn Res. 2010;11:18331863.

  • 19

    Pedregosa F, Varoquaux G, Gramfort A, et al. Scikit-learn: machine learning in Python. J Mach Learn Res. 2011;12:28252830.

  • 20

    Akai K, Uchigasaki S, Tanaka U, Komatsu A. Normal pressure hydrocephalus. Neuropathological study. Acta Pathol Jpn. 1987;37(1):97110.

  • 21

    Greitz D. Radiological assessment of hydrocephalus: new theories and implications for therapy. Neurosurg Rev. 2004;27(3):145167.

  • 22

    Rekate HL, Nadkarni TD, Wallace D. The importance of the cortical subarachnoid space in understanding hydrocephalus. J Neurosurg Pediatr. 2008;2(1):111.

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 23

    Preuss M, Hoffmann KT, Reiss-Zimmermann M, et al. Updated physiology and pathophysiology of CSF circulation—the pulsatile vector theory. Childs Nerv Syst. 2013;29(10):18111825.

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 24

    Ammar A, Abbas F, Al Issawi W, et al. Idiopathic normal-pressure hydrocephalus syndrome: is it understood? The comprehensive idiopathic normal-pressure hydrocephalus theory (CiNPHT). In: Ammar A, ed. Hydrocephalus: What Do We Know? And What Do We Still Not Know? Springer International Publishing;2017:6782.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • 25

    Wang Z, Zhang Y, Hu F, Ding J, Wang X. Pathogenesis and pathophysiology of idiopathic normal pressure hydrocephalus. CNS Neurosci Ther. 2020;26(12):12301240.

  • 26

    Bradley WG Jr, Scalzo D, Queralt J, Nitz WN, Atkinson DJ, Wong P. Normal-pressure hydrocephalus: evaluation with cerebrospinal fluid flow measurements at MR imaging. Radiology. 1996;198(2):523529.

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 27

    Blitz AM, Shin J, Balédent O, et al. Does phase-contrast imaging through the cerebral aqueduct predict the outcome of lumbar CSF drainage or shunt surgery in patients with suspected adult hydrocephalus? AJNR Am J Neuroradiol. 2018;39(12):22242230.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • 28

    Shanks J, Markenroth Bloch K, Laurell K, et al. Aqueductal CSF stroke volume is increased in patients with idiopathic normal pressure hydrocephalus and decreases after shunt surgery. AJNR Am J Neuroradiol. 2019;40(3):453459.

    • PubMed
    • Search Google Scholar
    • Export Citation
  • 29

    Eide PK, Sorteberg W. Diagnostic intracranial pressure monitoring and surgical management in idiopathic normal pressure hydrocephalus: a 6-year review of 214 patients. Neurosurgery. 2010;66(1):8091.

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 30

    Eide PK, Brean A. Cerebrospinal fluid pulse pressure amplitude during lumbar infusion in idiopathic normal pressure hydrocephalus can predict response to shunting. Cerebrospinal Fluid Res. 2010;7(1):5.

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 31

    Eide PK, Sorteberg W. Outcome of surgery for idiopathic normal pressure hydrocephalus: role of preoperative static and pulsatile intracranial pressure. World Neurosurg. 2016;86:186193.e1.

    • PubMed
    • Search Google Scholar
    • Export Citation
  • 32

    Nabbanja E, Czosnyka M, Keong NC, et al. Is there a link between ICP-derived infusion test parameters and outcome after shunting in normal pressure hydrocephalus? Acta Neurochir Suppl. 2018;126:229232.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • 33

    Qvarlander S, Lundkvist B, Koskinen LOD, Malm J, Eklund A. Pulsatility in CSF dynamics: pathophysiology of idiopathic normal pressure hydrocephalus. J Neurol Neurosurg Psychiatry. 2013;84(7):735741.

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 34

    Garg A, Mago V. Role of machine learning in medical research: a survey. Comput Sci Rev. 2021;40:100370.

  • 35

    Jeong S, Yu H, Park J, Kang K. Quantitative gait analysis of idiopathic normal pressure hydrocephalus using deep learning algorithms on monocular videos. Sci Rep. 2021;11(1):12368.

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 36

    Irie R, Otsuka Y, Hagiwara A, et al. A novel deep learning approach with a 3D convolutional ladder network for differential diagnosis of idiopathic normal pressure hydrocephalus and Alzheimer’s disease. Magn Reson Med Sci. 2020;19(4):351358.

    • Crossref
    • Search Google Scholar
    • Export Citation

Metrics

All Time Past Year Past 30 Days
Abstract Views 0 0 0
Full Text Views 1187 757 20
PDF Downloads 787 380 24
EPUB Downloads 0 0 0