Interval assessment using task- and procedure-based simulations: an attempt to supplement neurosurgical residency curriculum

View More View Less
  • 1 Department of Neurosurgery, All India Institute of Medical Sciences, New Delhi, India
Free access

OBJECTIVE

The longer learning curve and smaller margin of error make nontraditional, or "out of operating room" simulation training, essential in neurosurgery. In this study, the authors propose an evaluation system for residents combining both task-based and procedure-based exercises and also present the perception of residents regarding its utility.

METHODS

Residents were evaluated using a combination of task-based and virtual reality (VR)–based exercises. The results were analyzed in terms of the seniority of the residents as well as their laboratory credits. Questionnaire-based feedback was sought from the residents regarding the utility of this evaluation system incorporating the VR-based exercises.

RESULTS

A total of 35 residents were included in this study and were divided into 3 groups according to seniority. There were 11 residents in groups 1 and 3 and 13 residents in group 2. On the overall assessment of microsuturing skills including both 4-0 and 10-0 microsuturing, the suturing skills of groups 2 and 3 were observed to be better than those of group 1 (p = 0.0014). Additionally, it was found that microsuturing scores improved significantly with the increasing laboratory credits (R2 = 0.72, p < 0.001), and this was found to be the most significant for group 1 residents (R2 = 0.85, p < 0.001). Group 3 residents performed significantly better than the other two groups in both straight (p = 0.02) and diagonal (p = 0.042) ring transfer tasks, but there was no significant difference between group 1 and group 2 residents (p = 0.35). Endoscopic evaluation points were also found to be positively correlated with previous laboratory training (p = 0.002); however, for the individual seniority groups, the correlation failed to reach statistical significance. The 3 seniority groups performed similarly in the cranial and spinal VR modules. Group 3 residents showed significant disagreement with the utility of the VR platform for improving surgical dexterity (p = 0.027) and improving the understanding of surgical procedures (p = 0.034). Similarly, there was greater disagreement for VR-based evaluation to identify target areas of improvement among the senior residents (groups 2 and 3), but it did not reach statistical significance (p = 0.194).

CONCLUSIONS

The combination of task- and procedure-based assessment of trainees using physical and VR simulation models can supplement the existing neurosurgery curriculum. The currently available VR-based simulations are useful in the early years of training, but they need significant improvement to offer beneficial learning opportunities to senior trainees.

ABBREVIATIONS

ETV = endoscopic third ventriculostomy; EVD = external ventricular drainage; ICC = intraclass correlation coefficient; NETS = Neurosurgery Education and Training School; OR = operating room; OSATS = Objective Structured Assessment of Technical Skill; PGY = postgraduate year; SAS = Skills Assessment Score; VR = virtual reality.

OBJECTIVE

The longer learning curve and smaller margin of error make nontraditional, or "out of operating room" simulation training, essential in neurosurgery. In this study, the authors propose an evaluation system for residents combining both task-based and procedure-based exercises and also present the perception of residents regarding its utility.

METHODS

Residents were evaluated using a combination of task-based and virtual reality (VR)–based exercises. The results were analyzed in terms of the seniority of the residents as well as their laboratory credits. Questionnaire-based feedback was sought from the residents regarding the utility of this evaluation system incorporating the VR-based exercises.

RESULTS

A total of 35 residents were included in this study and were divided into 3 groups according to seniority. There were 11 residents in groups 1 and 3 and 13 residents in group 2. On the overall assessment of microsuturing skills including both 4-0 and 10-0 microsuturing, the suturing skills of groups 2 and 3 were observed to be better than those of group 1 (p = 0.0014). Additionally, it was found that microsuturing scores improved significantly with the increasing laboratory credits (R2 = 0.72, p < 0.001), and this was found to be the most significant for group 1 residents (R2 = 0.85, p < 0.001). Group 3 residents performed significantly better than the other two groups in both straight (p = 0.02) and diagonal (p = 0.042) ring transfer tasks, but there was no significant difference between group 1 and group 2 residents (p = 0.35). Endoscopic evaluation points were also found to be positively correlated with previous laboratory training (p = 0.002); however, for the individual seniority groups, the correlation failed to reach statistical significance. The 3 seniority groups performed similarly in the cranial and spinal VR modules. Group 3 residents showed significant disagreement with the utility of the VR platform for improving surgical dexterity (p = 0.027) and improving the understanding of surgical procedures (p = 0.034). Similarly, there was greater disagreement for VR-based evaluation to identify target areas of improvement among the senior residents (groups 2 and 3), but it did not reach statistical significance (p = 0.194).

CONCLUSIONS

The combination of task- and procedure-based assessment of trainees using physical and VR simulation models can supplement the existing neurosurgery curriculum. The currently available VR-based simulations are useful in the early years of training, but they need significant improvement to offer beneficial learning opportunities to senior trainees.

Neurosurgery is an especially trying field, as the central nervous system is extremely unforgiving and sensitive to the most minor insults. Traditionally, the surgical trainee acquires factual knowledge primarily via reading or through the teachings of seniors and then would apply it practically, usually in the operating room (OR), on patients. This practice has evolved from the method of Halsted, in which experience was acquired through working and contributing to patient care, while skills and acumen were gained through gradually increasing responsibility in the OR. It has long been acknowledged that the best method of acquiring technical finesse and sound surgical acumen is by performing the procedure under watchful guidance. The longer learning curve in neurosurgery as well as the higher stakes of the most minuscule errors makes the traditional implementation of the Halsted model inefficient.14 Hence, nontraditional, or "out of OR" simulation-based training, is essential for neurosurgery.57 Currently, this need is higher due to multiple factors such as an increase in the number of residents, work hour restrictions, the number of postresidency trainees, financial and ethical constraints, neurosurgical subspecialization, and focus on using technology to improve patient outcomes at the cost of learning opportunities for trainees.810

The COVID-19 pandemic led to the diversion of medical resources and personnel to manage COVID-19 cases as well as the halting of routine procedures. This caused a significant decrease in the practical OR exposure for trainees11,12 and has further highlighted the flaws in the traditional neurosurgical training system and in turn re-emphasized the role of simulation-based training in a complex surgical branch like neurosurgery.

Traditionally, our Neurosurgery Education and Training School (NETS) laboratory has been providing skills training using low- and high-fidelity simulation models over the last few years.5,6 Cadaver-based training is routinely part of our fellowship program, but because of the pandemic, it has suffered a setback in the last 2 years. To overcome this challenge, we switched to virtual reality (VR)–based training using two validated neurosurgery simulation systems, NeuroVR (CAE Healthcare) and ImmersiveView (ImmersiveTouch), to improve the understanding of basic neurosurgical procedures.

In the light of this background, we decided to update the existing neurosurgery skills training curriculum at our institution, combining the benefits of both hands-on task-based and VR-based training with supervision, objective evaluation, and feedback. In this paper, we report the results of the neurosurgical skills evaluation according to this revamped curriculum and insights regarding the performance of neurosurgery residents during these task- and procedure-based simulations. Additionally, we attempt to evaluate the perception of residents regarding the utility and challenges of these simulations.

Methods

In this study, we attempted to compare the results of the evaluation of task-based and procedure-based simulation among neurosurgery residents belonging to 3 different seniority levels. Group 1 included postgraduate year (PGY) 1–2 (6-year course) and PGY1 (3-year course) residents. Group 2 included PGY3–4 (6-year course) and PGY2 (3-year course) residents. Group 3 included the senior-most residents and fellows in the department. All residents performed both task-based and procedure-based activities (Video 1).

VIDEO 1. Video describing the complete protocol for evaluation of residents elaborating all the task- and procedure (VR)–based exercises and their scoring systems. © Ashish Suri, published with permission. Click here to view.

The study was approved by the institution’s ethics committee.

The task-based, hands-on skills activities included silastic sheet microsuturing and neuroendoscopic training using a neuroendoscope box trainer. The performance on microsuturing and endoscopy tasks was evaluated using an expert objective scoring system, the NETS–Skills Assessment Score (SAS), which has been used and validated in a previous publication.6 This scoring system was obtained from the Objective Structured Assessment of Technical Skill (OSATS)13 and excluded 3 components included in the original scoring system (knowledge of instruments, knowledge of operative steps, and use of assistance). These components were not applicable to the assessment of task-based learning using silastic sheet microsurgery and a neuroendoscope box trainer. Instead, we included an additional component, effectualness, which is the reflection of the final outcome of the microsuturing task or the neuroendoscope box trainer task. The details of NETS-SAS are summarized in Supplemental Table 1. Our laboratory has been providing task-based, hands-on training for the last several years to the residents and fellows of the department on a rotation basis, for which they are awarded laboratory credits. The laboratory credit system is related to the number of practice sessions performed by the trainee. The type and complexity of the task performed dictate the number of laboratory credits awarded, as shown in Supplemental Table 2. The total laboratory credits were the arithmetic sum of the individual session laboratory credits of the trainees.

For this study, scoring was done by two fully trained neurosurgeons as independent examiners, and the mean of the two independent scores was used to rate the performance of the residents. For the purpose of evaluation, the residents were evaluated while performing two iterations of 4-0 microsuturing (magnification factor, 0.6, magnification ×4.25) simulating dural suturing and two iterations of 10-0 microsuturing (magnification factor 1.6, magnification ×11.33) simulating nerve and vessel anastomosis.6 The residents’ endoscopic skills were evaluated by having the resident move a ring from one peg to another in a straight direction (straight ring transfer task) and by having the resident move a ring from one peg to another in a diagonal direction (diagonal ring transfer task) using the NETS neuroendoscope box trainer. An auxiliary camera is mounted to record the activity, and videos captured are used for evaluation of the task performed14,15

For procedure-based evaluation, we used the two commercially available VR platforms: NeuroVR and ImmersiveView. The procedure-based evaluation consisted of a composite of cranial and spinal modules from both platforms. In the NeuroVR system, the residents were evaluated on glioma, endoscopic third ventriculostomy (ETV), and hemilaminectomy modules, whereas on ImmersiveView, ventriculostomy (external ventricular drainage [EVD]) and open pedicle screw modules were used. Both VR platforms have an in-built evaluation system that has been used in previous studies, and its details are summarized in Supplemental Tables 3–5.1622 For the sake of convenience in calculating the cumulative scores, the individual scores of VR modules were linearly transformed between zero and the maximum allotted evaluation points for each VR module as summarized in Supplemental Table 6. The relative weightage of evaluation points assigned to each task and procedure was decided by the senior author of this study based on his neurosurgical training experience over the last 20 years. The cumulative evaluation was the arithmetic sum of task-based and procedure-based evaluation points. The details of the maximum evaluation points allotted to each task and procedure are illustrated in Fig. 1.

FIG. 1.
FIG. 1.

Relative contribution of each individual task and procedure to the overall neurosurgical skills evaluation. M = magnification; MF = magnification factor.

The scores of task-based and procedure-based activities were compared among various seniority groups. We also assessed the correlation between previous NETS laboratory skills training experience according to laboratory credits and the performance of the residents in this interval assessment. After interval assessment, the residents were given a Google Forms–based feedback survey to review their experience with the aforementioned task- and procedure-based evaluation activity. The detailed feedback is reported in Supplemental Questionnaire.

Statistical Analysis

The study data were summarized using Microsoft Excel 2016, and analysis was done using RStudio packages dplyr, ggplot2, and tidyverse. The evaluation points for individual groups are reported as the median and IQR and were compared between groups using the Kruskal-Wallis test. Correlation between the scores of the residents and previous laboratory credits was assessed using linear regression. Bland-Altman plots were generated to assess the limits of agreement between two independent examiners for task-based evaluation. The interobserver agreement was reported using the intraclass correlation coefficient (ICC)(3,k) based on the consistency of observations (intraclass correlation). For all statistical tests used in the current study, p < 0.05 was considered statistically significant.

Results

A total of 35 residents were included in this study and were divided into 3 groups according to seniority. There were 11 residents in groups 1 and 3 and 13 residents in group 2. In the task-based evaluation of hands-on skills, residents were first evaluated using NETS-SAS scoring for 4-0 and 10-0 microsuturing on silastic sheets. The interobserver agreement was assessed via ICC(3,k), which showed an excellent interrater reliability of 0.990 (95% CI 0.984–0.994) for 4-0 microsuturing and 0.991 (95% CI 0.986–0.995) for 10-0 microsuturing. The interrater reliability is depicted using Bland-Altman plots in Fig. 2A and B.

FIG. 2.
FIG. 2.

Bland-Altman plots depicting interrater reliability in the evaluation of 4-0 microsuturing (A), 10-0 microsuturing (B), straight ring transfer task (C), and diagonal ring transfer task (D).

The scores of group 2 (median 15, IQR 14.25–15.75) and group 3 (median 14.5, IQR 11.75–15.5) residents were significantly higher than those of their junior counterparts in group 1 (median 7.25, IQR 5.5–10.75) (p = 0.0038) for the two iterations of 4-0 microsuturing. Similarly, in the two iterations of 10-0 microsuturing, group 3 (median score 12.25, IQR 10.75–14.5) and group 2 (median score 12.25, IQR 10.75–14.5) fared significantly better compared with group 1 (median score 6.5, IQR 5–10.25) (p = 0.0011). On the overall assessment of microsuturing skills, the trend of better suturing skills with increasing seniority was maintained (p = 0.0014). However, there was no significant difference in any iteration (4-0, 10-0, or overall) between groups 2 and 3. When evaluating the relationship with previous task-based experience in the NETS laboratory, it was found that microsuturing scores improved significantly with increasing laboratory credits (p < 0.001), and this was found to be most significant in group 1 residents (R2 = 0.85, p < 0.001). Results of the microsuturing evaluation are illustrated in Fig. 3.

FIG. 3.
FIG. 3.

A–C: Box-and-whisker plots for comparison of scores among the seniority groups for 4-0 microsuturing (A), 10-0 microsuturing (B), and microsuturing overall (C). D–F: Histograms showing the scores of the individual candidates in the various seniority groups and the overlapping frequency polygon depicting their respective laboratory credits for 4-0 microsuturing (D), 10-0 microsuturing (E), and microsuturing overall (F). G: Scatterplot showing the correlation between microsuturing scores and laboratory credits among the various seniority groups.

For their endoscopic skills assessment, the interobserver agreement was excellent with an ICC(3,k) of 0.959 (95% CI 0.934–0.975) for the straight ring transfer task and good with ICC(3,k) of 0.758 (95% CI 0.611–0.850) for the diagonal ring transfer task (Fig. 2C and D). In the straight ring transfer task, it was found that group 3 residents (median score 14.5, IQR 13.25–15) performed significantly better compared with group 2 (median score 12.75, IQR 11.75–13.75) and group 1 (median score 12.5, IQR 8.75–12.75) (p = 0.02). A similar trend was seen in diagonal ring transfer tasks as well, with group 3 performing significantly better (p = 0.042) than the other two groups. However, there was no significant difference between group 1 and group 2 in straight as well as diagonal ring transfer tasks (p = 0.35). Endoscopic evaluation points were found to be positively correlated with previous laboratory training (p = 0.002); however, for the individual groups, statistical significance was not reached. The results of the endoscopic evaluation are illustrated in Fig. 4.

FIG. 4.
FIG. 4.

A–C: Box-and-whisker plots for comparison of scores among the seniority groups for straight ring transfer task (A), diagonal ring transfer task (B), and overall ring transfer tasks (C). D–F: Histograms showing the scores of the individual candidates in the various seniority groups and the overlapping frequency polygon depicting their respective laboratory credits for straight ring transfer (D), diagonal ring transfer (E), and overall ring transfer tasks (F). G: Scatterplot showing the correlation between the endoscopy score and laboratory credits among the various seniority groups.

For procedure-based evaluation, the NeuroVR and ImmersiveView VR modules were used with in-built scoring algorithms. All residents were evaluated on 3 cranial (glioma and ETV using NeuroVR and EVD using ImmersiveView) and 2 spinal (hemilaminectomy using NeuroVR and open pedicle screw using ImmersiveView) modules. The entire procedure-based evaluation was subdivided into 5 subtasks based on these cranial modules (VR points: glioma, 15; ETV, 10; EVD, 5; hemilaminectomy, 5; and open pedicle screw, 5). We did not find any statistically significant difference in the performance when comparing the different groups as summarized in Fig. 5.

FIG. 5.
FIG. 5.

Box-and-whisker plots for comparison of the scores among the seniority groups for cranial (A–C) and spinal (D and E) modules of VR-based systems. A: ImmersiveView EVD. B: NeuroVR ETV. C: NeuroVR glioma (GBM). D: NeuroVR hemilaminectomy. E: ImmersiveView open pedicle screw.

Cumulative evaluation was performed using the combination of evaluation points allotted to the task-based and procedure-based evaluations. When comparing the relationship of overall task-based scores (microsuturing + endoscopy), it was found that group 2 (median score 39.75, IQR 37.88–42.75) and group 3 (median score 40, IQR 37.13–43.5) performed significantly better than group 1 (median score 26.63, IQR 20.13–30.13) (p = 0.002), but there was no significant difference between groups 2 and 3 (p = 0.95). There was a significant correlation between overall task-based scores and previous laboratory experience (R2 = 0.72, p < 0.001). There was no statistically significant association between overall procedure-based scores (3 cranial and 2 spinal modules) with seniority (p = 0.32). Additionally, the overall VR-based score did not show any significant correlation with previous laboratory experience (p = 0.085). The summary of the association of overall task- and procedure-based scores with seniority groups and laboratory credits is illustrated in Fig. 6.

FIG. 6.
FIG. 6.

A–C: Raincloud plots for comparisons among the seniority groups for overall task-based score (A), overall procedure (VR)–based score, and overall total score (C). D–F: Scatterplots showing the correlation between laboratory credits and scores among the various seniority groups. Overall task-based score (D), overall procedure (VR)–based score (E), and overall total score (F).

All residents were asked to review the task-based and procedure (VR)–based evaluation exercise using an anonymous survey. The majority of residents agreed that silastic sheet microsuturing (91.43%), the neuroendoscope box trainer ring transfer exercise (85.71%), and VR modules (54.29%) help to improve surgical dexterity. There was no significant difference in the opinion that microsuturing (p = 0.406) and the neuroendoscope box trainer (p = 0.693) improved surgical dexterity among the different groups. However, for the VR modules, group 3 residents showed significant disagreement with the utility of the VR platform for improving surgical dexterity (p = 0.027). Similarly, group 3 residents felt less satisfied with VR modules in improving the understanding of the surgical procedure compared with groups 1 and 2 (p = 0.034). The majority of the residents in all 3 groups felt that task-based evaluation will help them identify their target areas of improvement (p = 0.534). Although the majority of residents felt the same for procedure-based evaluation, there was more disagreement for VR-based evaluation to identify target areas of improvement among the senior residents (groups 2 and 3), but it did not reach statistical significance (p = 0.194) (Fig. 7). The majority of the residents (82.8%) were satisfied with the current combination of task-based and VR-based assessment of neurosurgical skills and would like this assessment to be made part of the curriculum and the final exit examination.

FIG. 7.
FIG. 7.

Feedback of the residents regarding neurosurgical skills evaluation. A: Perception regarding utility of the task- and VR-based simulation training for improving surgical dexterity. B: Perception regarding utility of the task- and VR-based simulation for identifying target areas of improvement. C: Role of VR-based simulation in improving the understanding of the procedure.

Discussion

Task-based training entails the development and refining of basic skills such as dexterity and maneuverability as they are applied in the most basic and common aspects of neurosurgery, such as manipulating instruments, suturing under the microscope, and using endoscopic guidance to perform simple mechanical tasks. For suturing, silastic sheets provide a reasonably good practice tool, and for endoscopy, our in-house neuroendoscope box trainer platform helps in developing good bimanual coordination and spatial awareness using basic tasks such as object manipulation and ring shifting maneuvers.23,24 It offers not only recording of the endoscopic view but also recording through an external camera fixed to the box, which can record both the endoscope and instrument motion from a stationary reference. This requires the trainee to adapt and hone their skills to accomplish the task satisfactorily.15 Objective scoring systems are used to grade trainees while they develop their skills and progress along their individual learning curves. Evaluation of trainees’ progress and feedback is done via an in-house NETS-SAS scoring system for task-based training that is similar to video OSATS.1,25,26 Experts are posted on-site and review video footage of the various training modules. Personalized tips on how to improve, technical demonstrations by experts on various modules, and supervised performance of tasks provide a holistic learning experience to the residents during their routine training.

It is unequivocally recognized that surgical skills are correlated with practice. As the residents advance in their training, they invariably attain more hands-on practice in the skills training laboratory as well as in the OR. However, seniority and the practice hours, especially pertaining to laboratory training, are not perfectly correlated. Also, the stage of training may have an impact, in addition to its contribution via hands-on practice, in the form of greater theoretical knowledge of surgical nuances and a greater time spent in the OR observing experts performing the correct maneuvers. The hands-on practice done in the NETS laboratory as part of our routine resident training was quantifiable and available in terms of laboratory credits. Hence, we evaluated the impact of both seniority and laboratory credits to evaluate their relative importance in determining the surgical skills of the residents.

For microsuturing exercises, the laboratory-based training was found to significantly improve the skills for both 4-0 and 10-0 microsuturing. This relation was most prominently seen among the residents in group 1, signifying a steeper learning curve during the early years of training. Similarly, for endoscopic tasks, performance was found to be positively correlated with previous laboratory training. However, for the individual seniority groups, the correlation failed to reach statistical significance. This might be attributable to the fact that the correlation between endoscopic skills and laboratory credits was of lower magnitude and the sample size of individual groups was not sufficient to capture the significance of the correlation.

However, surgical skills cannot be reduced to the sum of their component tasks, and integrating the manual dexterity required for the individual tasks with the cognitive skills to combine those tasks into a complete procedure is extremely important for training as well as evaluation of the neurosurgical trainees. Procedure-based training should impart complete procedural learning instead of the practice of "part-tasks" or stereotypical practice of a series of simplified component steps that comprise an entire complex procedure. High-fidelity simulation models such as cadavers and live animals provide satisfactory surrogates for many complex neurosurgical procedures and remain the gold standard of procedure-based training despite lacking the flexibility and immersion afforded by VR-based modules. There were significant logistical issues in arranging these high-fidelity simulation models even before the pandemic. However, with the arrival of COVID-19, arranging these modules became even more difficult, resulting in a significant negative impact on resident procedure-based training.11,12 In the absence of cadavers and other physical models of procedure-based training, VR modules have partially filled the lacuna.16,18,20,27 In the current study, we used two separate validated VR platforms—the NeuroVR and the ImmersiveView—to provide procedure-based training.

Considering that greater OR procedural exposure is gained with increasing seniority, it was expected that performance on the VR modules might improve with seniority. However, our findings were contrary, and there was no statistically significant difference in the performance of the 3 seniority groups on the cranial and spinal VR modules. This may result from the inability of the in-built scoring algorithms to capture the nuances learned with OR experience, which may sometimes be counterproductive to scoring better. Additionally, the current VR systems lack overall seriousness when it is known that poor performance will not lead to patient harm as in the OR, and the modules have relatively lower fidelity for experts who use different sensorimotor cues to guide during real neurosurgical procedures.5,18,23

The utility and acceptability of the combination of task- and procedure-based modules were assessed based on feedback obtained from the residents. The task-based exercises, including both microsuturing and endoscopy exercises, were uniformly supported by the residents across all seniority groups as contributing positively to the enhancement of their surgical dexterity and identifying target areas of improvement. On the other hand, the VR-based procedures were found to be more beneficial by residents in groups 1 and 2 in improving surgical dexterity, identifying areas of improvement, and improving the understanding of the surgical procedure. The senior-most residents found the utility of VR-based modules to be equivocal, and this seems to be due to their greater exposure to actual surgical scenarios and intraoperative nuances. This highlights that although VR modules are constantly evolving, they only partially address the unmet needs due to the lack of cadaver-based training. Therefore, the currently available VR-based simulations are useful in the early years of training, but they need significant improvement to offer beneficial learning opportunities to senior trainees.

A drawback of assessing performance and providing training on task-based exercises is the simplification and degeneration of complex procedures into simple repeatable controlled steps that can be graded. The superior performance of individual steps does not automatically translate into expertise in the intended procedure. The feedback obtained in this study implies that there are existing deficits in the commercially available VR platforms and suggests that VR-based platforms provide a suboptimal simulation of real surgical scenarios with insufficient fidelity. Hence, it is not prudent to solely use the VR modules for imparting technical skills in basic operating techniques. Increased focus on only simulator-based training may make a trainee excellent in dealing with "known" and controlled situations but leave them hapless when unexpected and uncontrolled occurrences happen, as in the OR. Hence, simulation-based training is best used as an adjunct or preparatory step that will enable the trainees to better absorb teachings from the OR and provide them with preparation when they have supervised hands-on training in the OR.5,18,27

This study puts forth a method for holistic training as well as assessment of neurosurgical skills using a combination of hands-on tasks and VR-based exercises. The task-based evaluation assesses technical skills, and the VR-based evaluation provides context for combining and applying these technical skills. Combining the two aspects of surgical skill during training and evaluation will improve the existing neurosurgical curriculum and enable the assessment of trainees for their understanding of the procedure as well as psychomotor skills.28,29 The existing pattern of the neurosurgical examination at the majority of institutions in the world does not include such assessment methodologies, and this study will stimulate skills-based evaluation in the neurosurgical curriculum across the globe.

Conclusions

In this study, task- and procedure-based training were combined, and an expert objective interval assessment of trainees was conducted in which both physical and virtual simulators were used. This training combines traditional laboratory-based teaching approaches with recent technological advances to compensate for procedural simulations to expedite the long learning curve in neurosurgical training. The currently available VR-based simulations are useful in the early years of training, but they need significant improvement to offer beneficial learning opportunities to senior trainees. Our assessment model could form the precedent for neurosurgical training curriculum and standardized evaluation, especially in the post–COVID-19 era.

Acknowledgments

This paper is the result of research projects funded by extramural grants from the Department of Biotechnology–DBT, Ministry of Science and Technology, Govt. of India, BT/PR13455/CoE/34/24/2015. PI: Dr. Ashish Suri.

We would like to acknowledge the efforts of neurosurgeons for participation in this study. We would like to thank the technical and application specialists from Neurosurgery Skills Training Facility, NETS, All India Institute of Medical Sciences, New Delhi, India. We thank Mr. Subhas Bora, Mr. Ajab Singh, Mr. Shashi Shekhar, Mr. Trivender Yadav, Mr. Suresh Kothari, Mr. Vikram Singh, Mr. Satish Kumar, and Mr. Prabhav for their valuable support.

Disclosures

The authors report no conflict of interest concerning the materials or methods used in this study or the findings specified in this paper.

Author Contributions

Conception and design: all authors. Acquisition of data: Sharma, Katiyar, Narwal. Analysis and interpretation of data: Sharma, Katiyar. Drafting the article: all authors. Critically revising the article: Suri, Sharma, Katiyar, Kale.

Supplemental Information

Online-Only Content

Supplemental material is available online.

References

  • 1

    Jensen RL, Alzhrani G, Kestle JRW, Brockmeyer DL, Lamb SM, Couldwell WT. Neurosurgeon as educator: a review of principles of adult education and assessment applied to neurosurgery. J Neurosurg. 2017;127(4):949957.

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 2

    Fritz T, Stachel N, Braun BJ. Evidence in surgical training—a review. Innov Surg Sci. 2019;4(1):713.

  • 3

    Suri A, Tripathi M. Letter to the Editor: Neurosurgery skills training laboratories and curriculum: a supplement to Halstedian practice. J Neurosurg. 2016;125(6):16121613.

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 4

    Banerji AK. Neurosurgical training and evaluation—need for a paradigm shift. Neurol India. 2016;64(6):11191124.

  • 5

    Suri A, Patra DP, Meena RK. Simulation in neurosurgery: past, present, and future. Neurol India. 2016;64(3):387395.

  • 6

    Suri A, Tripathi M, Bettag M, Roy TS, Lalwani S. Simulation based skills training in neurosurgery and contemporary surgical practices. Ann Natl Acad Med Sci. 2016;52(1):5675.

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 7

    Kirkman MA, Ahmed M, Albert AF, Wilson MH, Nandi D, Sevdalis N. The use of simulation in neurosurgical education and training. A systematic review. J Neurosurg. 2014;121(2):228246.

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 8

    Raman HS, Limbrick DD, Ray WZ, et al. Prevalence, management, and outcome of problem residents among neurosurgical training programs in the United States. J Neurosurg. 2018;130(1):322326.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • 9

    Jensen RL, Kestle JRW, Brockmeyer DL, Couldwell WT. Principles of remediation for the struggling neurosurgery resident. World Neurosurg. 2021;146:e1118e1125.

    • PubMed
    • Search Google Scholar
    • Export Citation
  • 10

    Lipsman N, Khan O, Kulkarni AV. "The actualized neurosurgeon": a proposed model of surgical resident development. World Neurosurg. 2017;99:381386.

  • 11

    Wittayanakorn N, Nga VDW, Sobana M, Bahuri NFA, Baticulon RE. Impact of COVID-19 on neurosurgical training in Southeast Asia. World Neurosurg. 2020;144:e164e177.

    • PubMed
    • Search Google Scholar
    • Export Citation
  • 12

    Aljuboori ZS, Young CC, Srinivasan VM, et al. Early effects of COVID-19 pandemic on neurosurgical training in the United States: a case volume analysis of 8 programs. World Neurosurg. 2021;145:e202e208.

    • PubMed
    • Search Google Scholar
    • Export Citation
  • 13

    Martin JA, Regehr G, Reznick R, et al. Objective Structured Assessment of Technical Skill (OSATS) for surgical residents. Br J Surg. 1997;84(2):273278.

  • 14

    Singh R, Baby B, Damodaran N, et al. Design and validation of an open-source, partial task trainer for endonasal neuro-endoscopic skills development: Indian experience. World Neurosurg. 2016;86:259269.

    • PubMed
    • Search Google Scholar
    • Export Citation
  • 15

    Singh R, Britty B, Srivastav VK, et al. All India Institute of Medical Sciences, assignee. Neuro-endoscope box trainer. US patent 20170316720A1.November 2, 2017. Accessed June 14, 2022. https://patents.google.com/patent/US20170316720A1/en

    • PubMed
    • Search Google Scholar
    • Export Citation
  • 16

    Lemole GM Jr, Banerjee PP, Luciano C, Neckrysh S, Charbel FT. Virtual reality in neurosurgical education: part-task ventriculostomy simulation with dynamic visual and haptic feedback. Neurosurgery. 2007;61(1):142149.

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 17

    Gasco J, Patel A, Ortega-Barnett J, et al. Virtual reality spine surgery simulation: an empirical study of its usefulness. Neurol Res. 2014;36(11):968973.

  • 18

    Alaraj A, Charbel FT, Birk D, et al. Role of cranial and spinal virtual and augmented reality simulation using immersive touch modules in neurosurgical training. Neurosurgery. 2013;72(suppl 1):115-123.

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 19

    Banerjee PP, Luciano CJ, Lemole GM Jr, Charbel FT, Oh MY. Accuracy of ventriculostomy catheter placement using a head- and hand-tracked high-resolution virtual reality simulator with haptic feedback. J Neurosurg. 2007;107(3):515521.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • 20

    Alotaibi FE, AlZhrani GA, Sabbagh AJ, Azarnoush H, Winkler-Schwartz A, Del Maestro RF. Neurosurgical assessment of metrics including judgment and dexterity using the virtual reality simulator NeuroTouch (NAJD Metrics). Surg Innov. 2015;22(6):636642.

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 21

    Alotaibi FE, AlZhrani GA, Mullah MAS, et al. Assessing bimanual performance in brain tumor resection with NeuroTouch, a virtual reality simulator. Neurosurgery. 2015;11(suppl 2):89-98.

    • PubMed
    • Search Google Scholar
    • Export Citation
  • 22

    Winkler-Schwartz A, Bajunaid K, Mullah MAS, et al. Bimanual psychomotor performance in neurosurgical resident applicants assessed using NeuroTouch, a virtual reality simulator. J Surg Educ. 2016;73(6):942953.

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 23

    Baby B, Singh R, Suri A, et al. A review of virtual reality simulators for neuroendoscopy. Neurosurg Rev. 2020;43(5):12551272.

  • 24

    Singh R, Baby B, Suri A. A virtual repository of neurosurgical instrumentation for neuroengineering research and collaboration. World Neurosurg. 2019;126:e84e93.

    • Search Google Scholar
    • Export Citation
  • 25

    Sarkiss CA, Philemond S, Lee J, et al. Neurosurgical skills assessment: measuring technical proficiency in neurosurgery residents through intraoperative video evaluations. World Neurosurg. 2016;89:18.

    • Search Google Scholar
    • Export Citation
  • 26

    Niveditha M, Sharma R, Suri A. Microsurgical suturing assessment scores: a systematic review. Neurosurg Rev. 2022;45(1):119124.

  • 27

    Gélinas-Phaneuf N, Del Maestro RF. Surgical expertise in neurosurgery: integrating theory into practice. Neurosurgery. 2013;73(suppl 1):3038.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • 28

    Higgins M, Madan CR, Patel R. Deliberate practice in simulation-based surgical skills training: a scoping review. J Surg Educ. 2021;78(4):13281339.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • 29

    Marcus H, Vakharia V, Kirkman MA, Murphy M, Nandi D. Practice makes perfect? The role of simulation-based deliberate practice and script-based mental rehearsal in the acquisition and maintenance of operative neurosurgical skills. Neurosurgery. 2013;72(suppl 1):124-130.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • View in gallery

    Relative contribution of each individual task and procedure to the overall neurosurgical skills evaluation. M = magnification; MF = magnification factor.

  • View in gallery

    Bland-Altman plots depicting interrater reliability in the evaluation of 4-0 microsuturing (A), 10-0 microsuturing (B), straight ring transfer task (C), and diagonal ring transfer task (D).

  • View in gallery

    A–C: Box-and-whisker plots for comparison of scores among the seniority groups for 4-0 microsuturing (A), 10-0 microsuturing (B), and microsuturing overall (C). D–F: Histograms showing the scores of the individual candidates in the various seniority groups and the overlapping frequency polygon depicting their respective laboratory credits for 4-0 microsuturing (D), 10-0 microsuturing (E), and microsuturing overall (F). G: Scatterplot showing the correlation between microsuturing scores and laboratory credits among the various seniority groups.

  • View in gallery

    A–C: Box-and-whisker plots for comparison of scores among the seniority groups for straight ring transfer task (A), diagonal ring transfer task (B), and overall ring transfer tasks (C). D–F: Histograms showing the scores of the individual candidates in the various seniority groups and the overlapping frequency polygon depicting their respective laboratory credits for straight ring transfer (D), diagonal ring transfer (E), and overall ring transfer tasks (F). G: Scatterplot showing the correlation between the endoscopy score and laboratory credits among the various seniority groups.

  • View in gallery

    Box-and-whisker plots for comparison of the scores among the seniority groups for cranial (A–C) and spinal (D and E) modules of VR-based systems. A: ImmersiveView EVD. B: NeuroVR ETV. C: NeuroVR glioma (GBM). D: NeuroVR hemilaminectomy. E: ImmersiveView open pedicle screw.

  • View in gallery

    A–C: Raincloud plots for comparisons among the seniority groups for overall task-based score (A), overall procedure (VR)–based score, and overall total score (C). D–F: Scatterplots showing the correlation between laboratory credits and scores among the various seniority groups. Overall task-based score (D), overall procedure (VR)–based score (E), and overall total score (F).

  • View in gallery

    Feedback of the residents regarding neurosurgical skills evaluation. A: Perception regarding utility of the task- and VR-based simulation training for improving surgical dexterity. B: Perception regarding utility of the task- and VR-based simulation for identifying target areas of improvement. C: Role of VR-based simulation in improving the understanding of the procedure.

  • 1

    Jensen RL, Alzhrani G, Kestle JRW, Brockmeyer DL, Lamb SM, Couldwell WT. Neurosurgeon as educator: a review of principles of adult education and assessment applied to neurosurgery. J Neurosurg. 2017;127(4):949957.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • 2

    Fritz T, Stachel N, Braun BJ. Evidence in surgical training—a review. Innov Surg Sci. 2019;4(1):713.

  • 3

    Suri A, Tripathi M. Letter to the Editor: Neurosurgery skills training laboratories and curriculum: a supplement to Halstedian practice. J Neurosurg. 2016;125(6):16121613.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • 4

    Banerji AK. Neurosurgical training and evaluation—need for a paradigm shift. Neurol India. 2016;64(6):11191124.

  • 5

    Suri A, Patra DP, Meena RK. Simulation in neurosurgery: past, present, and future. Neurol India. 2016;64(3):387395.

  • 6

    Suri A, Tripathi M, Bettag M, Roy TS, Lalwani S. Simulation based skills training in neurosurgery and contemporary surgical practices. Ann Natl Acad Med Sci. 2016;52(1):5675.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • 7

    Kirkman MA, Ahmed M, Albert AF, Wilson MH, Nandi D, Sevdalis N. The use of simulation in neurosurgical education and training. A systematic review. J Neurosurg. 2014;121(2):228246.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • 8

    Raman HS, Limbrick DD, Ray WZ, et al. Prevalence, management, and outcome of problem residents among neurosurgical training programs in the United States. J Neurosurg. 2018;130(1):322326.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • 9

    Jensen RL, Kestle JRW, Brockmeyer DL, Couldwell WT. Principles of remediation for the struggling neurosurgery resident. World Neurosurg. 2021;146:e1118e1125.

    • Search Google Scholar
    • Export Citation
  • 10

    Lipsman N, Khan O, Kulkarni AV. "The actualized neurosurgeon": a proposed model of surgical resident development. World Neurosurg. 2017;99:381386.

    • Search Google Scholar
    • Export Citation
  • 11

    Wittayanakorn N, Nga VDW, Sobana M, Bahuri NFA, Baticulon RE. Impact of COVID-19 on neurosurgical training in Southeast Asia. World Neurosurg. 2020;144:e164e177.

    • Search Google Scholar
    • Export Citation
  • 12

    Aljuboori ZS, Young CC, Srinivasan VM, et al. Early effects of COVID-19 pandemic on neurosurgical training in the United States: a case volume analysis of 8 programs. World Neurosurg. 2021;145:e202e208.

    • Search Google Scholar
    • Export Citation
  • 13

    Martin JA, Regehr G, Reznick R, et al. Objective Structured Assessment of Technical Skill (OSATS) for surgical residents. Br J Surg. 1997;84(2):273278.

    • Search Google Scholar
    • Export Citation
  • 14

    Singh R, Baby B, Damodaran N, et al. Design and validation of an open-source, partial task trainer for endonasal neuro-endoscopic skills development: Indian experience. World Neurosurg. 2016;86:259269.

    • Search Google Scholar
    • Export Citation
  • 15

    Singh R, Britty B, Srivastav VK, et al. All India Institute of Medical Sciences, assignee. Neuro-endoscope box trainer. US patent 20170316720A1.November 2, 2017. Accessed June 14, 2022. https://patents.google.com/patent/US20170316720A1/en

    • Search Google Scholar
    • Export Citation
  • 16

    Lemole GM Jr, Banerjee PP, Luciano C, Neckrysh S, Charbel FT. Virtual reality in neurosurgical education: part-task ventriculostomy simulation with dynamic visual and haptic feedback. Neurosurgery. 2007;61(1):142149.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • 17

    Gasco J, Patel A, Ortega-Barnett J, et al. Virtual reality spine surgery simulation: an empirical study of its usefulness. Neurol Res. 2014;36(11):968973.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • 18

    Alaraj A, Charbel FT, Birk D, et al. Role of cranial and spinal virtual and augmented reality simulation using immersive touch modules in neurosurgical training. Neurosurgery. 2013;72(suppl 1):115-123.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • 19

    Banerjee PP, Luciano CJ, Lemole GM Jr, Charbel FT, Oh MY. Accuracy of ventriculostomy catheter placement using a head- and hand-tracked high-resolution virtual reality simulator with haptic feedback. J Neurosurg. 2007;107(3):515521.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • 20

    Alotaibi FE, AlZhrani GA, Sabbagh AJ, Azarnoush H, Winkler-Schwartz A, Del Maestro RF. Neurosurgical assessment of metrics including judgment and dexterity using the virtual reality simulator NeuroTouch (NAJD Metrics). Surg Innov. 2015;22(6):636642.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • 21

    Alotaibi FE, AlZhrani GA, Mullah MAS, et al. Assessing bimanual performance in brain tumor resection with NeuroTouch, a virtual reality simulator. Neurosurgery. 2015;11(suppl 2):89-98.

    • Search Google Scholar
    • Export Citation
  • 22

    Winkler-Schwartz A, Bajunaid K, Mullah MAS, et al. Bimanual psychomotor performance in neurosurgical resident applicants assessed using NeuroTouch, a virtual reality simulator. J Surg Educ. 2016;73(6):942953.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • 23

    Baby B, Singh R, Suri A, et al. A review of virtual reality simulators for neuroendoscopy. Neurosurg Rev. 2020;43(5):12551272.

  • 24

    Singh R, Baby B, Suri A. A virtual repository of neurosurgical instrumentation for neuroengineering research and collaboration. World Neurosurg. 2019;126:e84e93.

    • Search Google Scholar
    • Export Citation
  • 25

    Sarkiss CA, Philemond S, Lee J, et al. Neurosurgical skills assessment: measuring technical proficiency in neurosurgery residents through intraoperative video evaluations. World Neurosurg. 2016;89:18.

    • Search Google Scholar
    • Export Citation
  • 26

    Niveditha M, Sharma R, Suri A. Microsurgical suturing assessment scores: a systematic review. Neurosurg Rev. 2022;45(1):119124.

  • 27

    Gélinas-Phaneuf N, Del Maestro RF. Surgical expertise in neurosurgery: integrating theory into practice. Neurosurgery. 2013;73(suppl 1):3038.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • 28

    Higgins M, Madan CR, Patel R. Deliberate practice in simulation-based surgical skills training: a scoping review. J Surg Educ. 2021;78(4):13281339.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • 29

    Marcus H, Vakharia V, Kirkman MA, Murphy M, Nandi D. Practice makes perfect? The role of simulation-based deliberate practice and script-based mental rehearsal in the acquisition and maintenance of operative neurosurgical skills. Neurosurgery. 2013;72(suppl 1):124-130.

    • Crossref
    • Search Google Scholar
    • Export Citation

Metrics

All Time Past Year Past 30 Days
Abstract Views 0 0 0
Full Text Views 192 192 192
PDF Downloads 201 201 201
EPUB Downloads 0 0 0