The effect of augmented reality on the accuracy and learning curve of external ventricular drain placement.

OBJECTIVE
The traditional freehand technique for external ventricular drain (EVD) placement is most frequently used, but remains the primary risk factor for inaccurate drain placement. As this procedure could benefit from image guidance, the authors set forth to demonstrate the impact of augmented-reality (AR) assistance on the accuracy and learning curve of EVD placement compared with the freehand technique.


METHODS
Sixteen medical students performed a total of 128 EVD placements on a custom-made phantom head, both before and after receiving a standardized training session. They were guided by either the freehand technique or by AR, which provided an anatomical overlay and tailored guidance for EVD placement through inside-out infrared tracking. The outcome was quantified by the metric accuracy of EVD placement as well as by its clinical quality.


RESULTS
The mean target error was significantly impacted by either AR (p = 0.003) or training (p = 0.02) in a direct comparison with the untrained freehand performance. Both untrained (11.9 ± 4.5 mm) and trained (12.2 ± 4.7 mm) AR performances were significantly better than the untrained freehand performance (19.9 ± 4.2 mm), which improved after training (13.5 ± 4.7 mm). The quality of EVD placement as assessed by the modified Kakarla scale (mKS) was significantly impacted by AR guidance (p = 0.005) but not by training (p = 0.07). Both untrained and trained AR performances (59.4% mKS grade 1 for both) were significantly better than the untrained freehand performance (25.0% mKS grade 1). Spatial aptitude testing revealed a correlation between perceptual ability and untrained AR-guided performance (r = 0.63).


CONCLUSIONS
Compared with the freehand technique, AR guidance for EVD placement yielded a higher outcome accuracy and quality for procedure novices. With AR, untrained individuals performed as well as trained individuals, which indicates that AR guidance not only improved performance but also positively impacted the learning curve. Future efforts will focus on the translation and evaluation of AR for EVD placement in the clinical setting.

tion of the location and size of the ventricles. Despite it being a frequent procedure, the accuracy rate is only around 80%, with complications (mainly hemorrhage and infection) occurring in up to 40% of the cases. 3,4 Image guidance can help a great deal in improving EVD placement accuracy but remains underused in daily practice. 5,6 Several imaging-based solutions have been proposed, ranging from physical guides to neuronavigational systems. To date, these solutions have failed to find their way into the daily neurosurgical practice, either because they require supplementary material and/or time, or because they are not easily applicable outside of the operating room. 1,[7][8][9] In a recent study, the use of the freehand technique was identified as the primary risk factor for inaccurate placement when compared with image-guided techniques. 10 An augmented-reality (AR) head-mounted device (HMD) that can function as a stand-alone unit, capable of tracking the patient's position while displaying an exact overlay of anatomical structures and surgical plans, has the potential to facilitate the learning process and improve the accuracy of EVD placement in everyday neurosurgical practice. 11,12 To further evaluate the efficacy of EVD placement using AR assistance, a software solution utilizing the Holo-Lens (Microsoft Corp.) HMD was developed to facilitate AR-assisted EVD placements. The software solution provided accurate tracking of both a phantom model and an EVD by means of an infrared (IR) tracking algorithm integral to the HMD (inside-out tracking). The AR solution implemented a user interface (UI) tailored specifically toward EVD placement and directed users through requisite procedural steps.
Our aim was to demonstrate the impact of AR assistance on the accuracy of EVD placement, defined both as the deviation from a predefined target as well as the clinical quality. The impact on the learning curve for this procedure was also evaluated by comparing outcomes of medical students without any surgical experience between traditional freehand techniques and AR-assisted placements both before and after receiving standardized training.

Phantom Model
EVD placement was simulated on a phantom head using lumbar puncture needles (Spinocan 20G, B. Braun Melsungen AG) as a substitute for intracranial drains with a rigid stylet. The phantom head consisted of two parts: a 3D-printed head base with a face that was designed from anonymized brain MR images, and a replaceable skull dome that was created by casting a 2-component polyurethane casting foam (Poly-Pur H45 foam A & B component, Poly-Service BV) in a plaster mold of the skull dome and adding a laser-cut wooden spacer as an interface between the head base and the skull dome. The assembled phantom head was fitted atop a laser-cut plexiglass fixture that was mounted on a tripod. For the AR guidance, additional IR retroreflective marker spheres (Northern Digital Inc.) were fitted in a unique constellation of 4 spheres to both the lumbar puncture needles by means of a SureTrak 2 (Medtronic) large fighter and to the plexiglass fixture by means of a 3D-printed tracking anchor to allow pose estimation for the drain substitute and the phantom head, respectively.
The phantom models with the attached tracking anchor were scanned with CT. The surface models (face, scalp, and skull) were generated from the CT scan using the 3D Slicer version 4.10 (http://www.slicer.org) software package. MRI data were registered to this CT scan and used for further brain and ventricle segmentation. The foramen of Monro was selected as a precise and easily reproducible target point for both guidance techniques since this anatomical landmark is-in extension-already targeted by the currently applied freehand technique. 2 The ipsilateral foramen, as well as Kocher's point on the same side, was marked on the abovementioned MR image to serve as a reference target and entry point, respectively. The distance between them was measured to be 7.1 cm on the right and 7.5 cm on the left. The relative coordinates of the IR-tracked spheres were defined from the CT images, providing known tracked registration, as well as the coordinates of the previously marked foramen of Monro and Kocher's points, which were rigidly transformed to the CT coordinate system along with the MR image ( Fig. 1).

Participants
Medical students without prior experience with EVD placement, or with the use of AR or surgical navigation in general, were eligible for inclusion. Color vision deficiency was an exclusion criterion. Participants were randomly allocated to either the freehand technique group or the AR guidance group. All participants received a short introduction to the study setup and to the task that was expected from them. Participants allocated to the AR guidance group also received a short introduction on the use of the HoloLens HMD. Pre-experiment questionnaires, evaluating sociodemographic characteristics and experience in using AR, were collected from all participants. A standardized questionnaire, adapted from Condino et al., evaluating the personal user experience and attitude toward surgical innovation was obtained post-experiment (Fig. 2). 13 To assess individual geometric comprehension abilities, all participants underwent an evaluation of their ability to interpret and implement 2D to 3D information before the start of the EVD placement task. This consisted of a series of standardized tests, subdivided into 3 phases: 1) 3 visuospatial ability tests (card rotation test, cube comparison test, and map-planning test), 2) a perceptual ability test (PicSOr cube-and-arrow test), and 3) and a psychomotor ability test performed on a box trainer. 14-16

Ventricular Drain Placement
All participants were asked to perform the EVD placement task on the phantom head, first without any instructions or prior training. The freehand group had access to MR images but without any further guidance. The AR group used the AR HMD, which projected an overlay of the virtual phantom head (with skin, skull, brain, and ventricles containing the target structure) registered to the physical phantom head. This was performed 4 times (left and right lateral ventricles on 2 phantom heads) by each participant. After having received a standardized training session on EVD placement given by a certified neurosurgeon (J.D.), the same process was repeated. 2

Augmented Reality
An AR application was developed internally (Unity Technologies) for deployment to a HoloLens HMD (Fig.  3A). The application defined the experimental workflow and provided visualizations to assist in that workflow. The workflow was broken into 3 stages: display calibration, drain calibration, and drain placement.
Display calibration consisted of 2 stages, the first being the determination of the participant's interpupillary distance (IPD), measured with a digital IPD meter (PD-82 II, Rexxam Co. Ltd.), which was used to adjust the HMD's IPD setting via its web portal. The second display calibration was performed to correct misalignment of AR content with reality due to the variable position of the HMD on the participant's head. Validation of stage two calibration was confirmed by the participant when the alignment of feedback visuals and reality showed a strong correlation.
Calibration of the tracked drain was accomplished through a secondary tracked calibration object. This object provided a keyed fit to the tracked drain, allowing the orientation of its axis and the location of a tip to be determined (Fig. 3C).
Placement of the drains was performed on the phantom head affixed to an IR tracker. The participants were required to indicate the approach for drain placement (left or right), after which Kocher's point was indicated as the entry and displayed as a bullseye (Fig. 3D). On lining up the tracked drain's tip within a 2-mm threshold of the bullseye for minimally 2 seconds, the correct retrieval of the physical entry point was confirmed. Next, the bullseye was placed at the foramen of Monro (Fig. 3E). A line protruding from the tracked drain down its long axis visually indicated the current trajectory toward the bullseye. The bullseye transitioned from red to orange if the resulting trajectory would result in target error below 4 mm, and to green within 2 mm. Provided the target was green, the participant could proceed with EVD placement. The system also indicated the distance of the drain tip to the bullseye to prevent overshooting. When the bullseye was reached, the completion was confirmed (Fig. 3F).

Outcome Measurement
After completion of all drain placement tasks, the replaceable skull domes were evaluated through CT analysis (maintaining preplacement scan parameters). The postprocedure scans were registered to the initial MR image containing the coordinates of the entry and target points to bring all data points into the same coordinate system (Fig. 1).
Quantitative performance was characterized as the Euclidean distance between planned and performed endpoints. Ground truth and experimental data were collected by two blinded investigators (F.V.G. and C.V.) based on the imaging data. Corresponding data values obtained by each investigator were compared for agreement (< 2-mm linear displacement) and averaged. Where data were outside the agreement threshold, both investigators remeasured the coordinates and came to a consensus.
Qualitative performance was characterized by drain placement quality using the modified Kakarla scale (mKS) for frontal EVD placement. 4,8 This was determined independently by the two investigators, where disagreements were resolved through consensus. As the task required the participants to advance the drain deeper than indicated by EVD placement guidelines to reach the planned target point, 2,17 the mKS grade was based on the drain's position at a 5-cm depth, as measured from the surface of the skull. The following 2 cm of depth could aid as a further indication of the quality of the drain's position, yet was not used to define the compartment it was in since in practice a drain at this depth would only be advanced without a rigid guide, as defined per the abovementioned guidelines.

Statistical Analysis
Statistical analysis was accomplished using the IBM SPSS Statistics software version 26 (IBM Corp.) and Prism 9.0.0 (GraphPad Software) for graph creation; p < 0.05 was considered statistically significant.

Quantitative and Qualitative Performance
Quantitative performance data were reported as the accuracy of EVD placement. Accuracy was defined as the mean target error (in mm) and compared using a mixed- model ANOVA, accounting for repeated measurements and allowing a random intercept. The target error was the dependent variable, and guidance technique (freehand vs AR) and training (untrained vs trained) were the predictors. In case of any significant difference, pairwise analyses using the Dunn-Šidák's correction for multiple comparisons were applied to identify its source.
Qualitative performance data were reported as the proportion of EVD placements per mKS grade. These were dichotomized into good (grade 1a and 1b) and poor (grade 2 and 3) EVD placement, which allowed analysis with the chi-square test.

Spatial Aptitude Tests and Subjective Experience
The visuospatial ability tests were scored and reported as a decimal fraction; the errors of the perceptual ability test were evaluated with a Pearson correlation between the estimated and actual angles and reported as a decimal fraction; and the psychomotor ability test was reported as a combination of the absolute number of correctly executed actions and the decimal fraction of incorrect actions on the box trainer. The interaction between the quantitative EVD placement performance and the spatial aptitude tests was analyzed by calculating their correlation coefficient for each guidance technique, both before and after training.
Data gathered from the questionnaires assessed the confidence in EVD placement before and after training for both groups, as well as the general user experience (e.g., intuitiveness of experience, discomfort) for participants who used the HoloLens. The confidence levels were analyzed with the same ANOVA as mentioned above.

Participant Characteristics
Sixteen medical students without prior experience with EVD placement, AR, or surgical navigation were recruited. Informed consent was obtained from all participants after which they were randomly allocated to either the freehand technique group (n = 8) or the AR guidance group (n = 8). Table 1 lists participant characteristics.

Quantitative Performance
The overall EVD placement accuracy, represented as the mean deviation (± SD) from the target point, was 16.7 ± 5.4 mm for the freehand group and 12.0 ± 4.5 mm for the AR group. Accuracy results of EVD placement before and after training and per study group are listed in Figure 4. The mixed-model analysis comparing the 4 groups (freehand vs AR, untrained vs trained) indicated that there was a significant source of variation resulting from either AR (p = 0.013) or training (p = 0.004) but also from the presence of an important interaction between both (p = 0.001). The pairwise analyses confirmed the significant impact of either AR (p = 0.003) or training (p = 0.02) in a direct comparison with the untrained freehand performance. Other comparisons between the separate groups showed no significant differences (Fig. 4).

Qualitative Performance
The overall quality of EVD placement, represented as the number of good EVD placements, was significantly higher for the AR group at 59.4% compared with 35.9% for the freehand group (χ 2 [1, n = 128] = 7.05, p = 0.008). Comparing the 4 groups, the chi-square test for the dichotomized mKS grades showed a significantly higher number of good EVD placements (grades 1a and 1b) resulting from AR guidance (χ 2 [1, n = 64] = 7.75, p = 0.005), but not from training, in direct comparison to the untrained freehand performance. There was no significant difference between untrained and trained performance for either technique or between both AR groups and the trained freehand performance (Table 2 and Fig. 5).

Spatial Aptitude Tests
The analysis of the spatial aptitude tests revealed a positive correlation (r) between the perceptual ability test and the performance of untrained individuals using AR guidance (r = 0.63), which disappeared completely after training. Another positive correlation was found between the map-planning test and the untrained freehand performance, yet not for the trained freehand performance (r = 0.66). Lastly, the number of errors during the psychomotor ability test had a negative correlation with both the untrained AR performance (r = −0.56) and the trained freehand performance (r = −0.54).

Subjective Experience
Confidence in EVD placement was low for untrained Values represent the number of participants (%) unless indicated otherwise. * None of the characteristics were significantly different between the freehand and AR groups.
freehand performance and increased significantly through training (p < 0.001). This effect was not seen in the AR group, in which confidence was already high. All participants in the AR group agreed on the intuitiveness of the voice interaction with the HoloLens. All agreed on the intuitiveness of the gesture interaction, aside from 1 participant who remained indifferent. There was a clear presence of physical discomfort (5/8) but not visual fatigue (2/8) (Table 3).

Discussion
Accuracy of AR-guided EVD placement as performed by inexperienced individuals was significantly higher compared with freehand placement. This was especially true for the performance before the training session, where AR guidance virtually eliminated the procedural learning curve. This observation is in line with the assumption proposed by Schneider et al. that AR-assisted EVD placement had a steep learning curve (indicating quick mastery of the technique). 18 Untrained freehand performance showed an important deviation from the target, as one would expect from procedure novices, which decreased significantly after training. The untrained AR group, however, performed as accurately as the trained individuals of both groups, even without any prior experience. This was true for both the quantitative analysis, in which metric deviation from the target was assessed, as well as for the qualitative analysis, where the clinical mKS was used to assess EVD placement. The mKS analyses showed that using AR guidance, untrained students can already reach 59% grade 1 placement, approximating the literature-described accuracy as performed by trained professionals and even matching the accuracy from previously reported AR-assisted EVD placement by trained professionals. 3,4,6,[18][19][20] It is also important to note that results in our study were attained with only single-pass placements instead of multiple passes as described in the literature. Training did not further improve accuracy of AR-guided EVD placement but did improve results in the freehand group. This is to be expected and confirms the usefulness of the applied freehand guidelines for EVD placement. Although this was not statistically significant, the quality of placement even after training was still lower than with AR guidance. Taken together, these results demonstrate that AR guidance can improve the accuracy of EVD placement, especially when performed by individuals with limited or no training.
Interestingly, forgoing previous instructions, the 3D ability appeared to be an advantageous human factor to perform well in AR-supported tasks. This significant correlation between PicSOr scores and untrained AR-assisted performance decreased after training, as the participants relied more on trained guidelines and less on their 3D abilities. [21][22][23][24] The significant correlation between the mapplanning test and the untrained freehand performance, on the other hand, might suggest the ability to correctly determine a certain trajectory in 3D space based on limited information (as there was no AR guidance or any training), although the literature remains divided on the value of these tests. [14][15][16]21,23 Here, too, we saw a decrease in correlation as reliance on training increased. Lastly, the negative correlation between the error fraction on the box trainer and the untrained AR-assisted and trained freehand performance might illustrate the ability to interpret and correctly act on newly gained spatial information (the AR overlay and the EVD placement training, respectively). 25 To our knowledge, this is the first reported trial where tracking was done from the HoloLens using IR tracking without external cameras. This inside-out infrared tracking has several advantages over the more commonly used off-the-shelf red, green, blue (RGB) tracking, such as Vuforia, which relies on large QR-code style markers. 18,26,27 Not only does it apply general hardware-linked benefits, such as a wider field of view when compared with the HMD's  Values represent the number of participants or mean ± SD (range). The subjective experience, gauging confidence level measured on a scale from 0 (very unconfident) to 5 (very confident) for participants in both groups (n = 16) before and after training, as well as intuitiveness and discomforts of the HoloLens as reported by the participants in the AR group (n = 8).
RGB camera, allowing an improved tracking frustum, but it also facilitates intraoperative use since no additional tracking hardware is required and markers have a smaller footprint. In test conditions, our IR tracking showed high accuracy (0.78 ± 0.74 mm) as well as less of a perceived drift compared with RGB tracking (Frantz et al., unpublished data). Others have reported on the use of more portable devices for AR assistance, such as tablets or smartphones. Although these devices allow higher mobility and flexibility than the classic neuronavigational systems, they also impede the surgeon from using both hands and can be affected by an important parallax error. 28,29 These problems are ameliorated by using an HMD, although physical discomfort can be experienced. This was reported during our experiment by 5 of the 8 participants (Table 3). Future versions of these HMDs will improve user comfort. Although the mean error still appears quite large for such an intervention, the circumstances and limitations of the study should also be taken into account. The overarching limitation of our experimental setup was that it mimics the real-life situation only to a certain extent. For example, the loss of resistance that is experienced upon ventricular puncture, which is often used as feedback during actual EVD placement, was not replicated. This lack of feedback during placement might have affected participant performance. In addition, participants were required to deviate from the standard operative guidelines by aiming toward a point that was deeper than one would usually advance a drain. We adapted to this limitation by adopting the qualitative evaluation using the mKS, as described in the text. To maintain objectivity in analysis, measurements and assessments according to the mKS were performed at a later date, and separately, by two investigators who were blinded to each participant's allocated group. For future research, a clinical trial assessing AR-guided EVD placement using our proprietary IR tracking method is planned, as well as clinical trials for evaluation of AR guidance in other neurosurgical procedures such as tumor resection and pedicle screw placement.

Conclusions
Use of an AR HMD to provide tailored guidance for EVD placement yields a high accuracy of EVD placement for procedure novices compared with untrained freehand placement. With this experiment, we not only confirmed the improved performance for this procedure even when performed by novices, but also the hypothesized impact on the learning curve, as an untrained individual with the use of AR was able to perform EVD placement as well as a trained individual. The use of AR especially favors individuals with a certain innate spatial aptitude, a correlation that may apply to surgery in general. Our future efforts will therefore focus on the translation and evaluation of AR for EVD placement in the clinical setting, performed by trained professionals.