Construct validity of the Surgical Autonomy Program for the training of neurosurgical residents

View More View Less
  • 1 Department of Neurosurgery, Duke University School of Medicine, Durham;
  • | 2 Surgical Autonomy Program Inc., Durham; and
  • | 3 Department of Biomedical Engineering, Duke University, Durham, North Carolina
Free access

OBJECTIVE

There is no standard way in which physicians teach or evaluate surgical residents intraoperatively, and residents are proving to not be fully competent at core surgical procedures upon graduating. The Surgical Autonomy Program (SAP) is a novel educational model that combines a modified version of the Zwisch scale with Vygotsky’s social learning theory. The objective of this study was to establish preliminary validity evidence that SAP is a reliable measure of autonomy and a useful tool for tracking competency over time.

METHODS

The SAP breaks each surgical case into 4 parts, or zones of proximal development (ZPDs). Residents are evaluated on a 4-tier autonomy scale (TAGS scale) for each ZPD in every surgical case. Attendings were provided with a teaching session about SAP and identified appropriate ZPDs for surgical cases under their area of expertise. All neurosurgery residents at Duke University Hospital from July 2017 to July 2021 participated in this study. Chi-square tests and ordinal logistic regression were used for the analyses.

RESULTS

Between 2017 and 2021, there were 4885 cases logged by 27 residents. There were 30 attendings who evaluated residents using SAP. Faculty completed evaluations on 91% of cases. The ZPD of focus directly correlated with year of residency (postgraduate year) (χ2 = 1221.1, df = 15, p < 0.001). The autonomy level increased with year of residency (χ2 = 3553.5, df = 15, p < 0.001). An ordinal regression analysis showed that for every year increase in postgraduate year, the odds of operating at a higher level of independence was 2.16 times greater (95% CI 2.11–2.21, p < 0.001). The odds of residents performing with greater autonomy was lowest for the most complex portion of the case (ZPD3) (OR 0.18, 95% CI 0.17–0.20, p < 0.001). Residents have less autonomy with increased case complexity (χ2 = 160.28, df = 6, p < 0.001). Compared with average cases, residents were more likely to operate with greater autonomy on easy cases (OR 1.44, 95% CI 1.29–1.61, p < 0.001) and less likely to do so on difficult cases (OR 0.72, 95% CI 0.67–0.77, p < 0.001).

CONCLUSIONS

This study demonstrates preliminary evidence supporting the construct validity of the SAP. This tool successfully tracks resident autonomy and progress over time. The authors’ smartphone application was widely used among surgical faculty and residents, supporting integration into the perioperative workflow. Wide implementation of SAP across multiple surgical centers will aid in the movement toward a competency-based residency education system.

ABBREVIATIONS

PGY = postgraduate year; SAP = Surgical Autonomy Program; TAGS = teach and demonstrate, advise and scaffold, guide and monitor, solo and observe; ZPD = zone of proximal development.

OBJECTIVE

There is no standard way in which physicians teach or evaluate surgical residents intraoperatively, and residents are proving to not be fully competent at core surgical procedures upon graduating. The Surgical Autonomy Program (SAP) is a novel educational model that combines a modified version of the Zwisch scale with Vygotsky’s social learning theory. The objective of this study was to establish preliminary validity evidence that SAP is a reliable measure of autonomy and a useful tool for tracking competency over time.

METHODS

The SAP breaks each surgical case into 4 parts, or zones of proximal development (ZPDs). Residents are evaluated on a 4-tier autonomy scale (TAGS scale) for each ZPD in every surgical case. Attendings were provided with a teaching session about SAP and identified appropriate ZPDs for surgical cases under their area of expertise. All neurosurgery residents at Duke University Hospital from July 2017 to July 2021 participated in this study. Chi-square tests and ordinal logistic regression were used for the analyses.

RESULTS

Between 2017 and 2021, there were 4885 cases logged by 27 residents. There were 30 attendings who evaluated residents using SAP. Faculty completed evaluations on 91% of cases. The ZPD of focus directly correlated with year of residency (postgraduate year) (χ2 = 1221.1, df = 15, p < 0.001). The autonomy level increased with year of residency (χ2 = 3553.5, df = 15, p < 0.001). An ordinal regression analysis showed that for every year increase in postgraduate year, the odds of operating at a higher level of independence was 2.16 times greater (95% CI 2.11–2.21, p < 0.001). The odds of residents performing with greater autonomy was lowest for the most complex portion of the case (ZPD3) (OR 0.18, 95% CI 0.17–0.20, p < 0.001). Residents have less autonomy with increased case complexity (χ2 = 160.28, df = 6, p < 0.001). Compared with average cases, residents were more likely to operate with greater autonomy on easy cases (OR 1.44, 95% CI 1.29–1.61, p < 0.001) and less likely to do so on difficult cases (OR 0.72, 95% CI 0.67–0.77, p < 0.001).

CONCLUSIONS

This study demonstrates preliminary evidence supporting the construct validity of the SAP. This tool successfully tracks resident autonomy and progress over time. The authors’ smartphone application was widely used among surgical faculty and residents, supporting integration into the perioperative workflow. Wide implementation of SAP across multiple surgical centers will aid in the movement toward a competency-based residency education system.

Intraoperative teaching is a critical component of surgical residency. In recent years, changes in medical practice, such as demands on faculty productivity and the implementation of residency duty hour restrictions, have made this more challenging.17 Currently, residents are deemed competent at surgical procedures and are allowed to graduate from residency programs by satisfying a certain number of surgical cases as predetermined by national residency review committees.4,5 However, recent studies have shown that some surgical residents are not prepared to operate independently after residency, even when completing such requirements, demonstrating the limitations of this current approach to residency education.813 As such, educators have proposed a competency-based residency curriculum whereby residents gain autonomy in complex surgical cases by demonstrating mastery at easier portions of the case.1417 In this proposed model, the number of index cases that a resident must complete for graduation can be personalized based on the resident’s ability to demonstrate complete autonomy and competency.15,16

Currently, there is no universal way in which surgical educators evaluate resident competency within the operating room. Many rating scales have been developed to attempt to solve this problem, including the Objective Structured Assessment of Technical Skills (OSATS)18 and the Zwisch scale.14,17 The OSATS is an assessment tool for grading technical proficiency in surgery within 4 domains: respect for tissue, time and motion, instrument handling, and flow of operation and forward planning. The Zwisch scale rates resident autonomy level during an entire surgical case. Although excellent for evaluation, both tools view a surgical case as one entity. Additionally, they both function as an assessment tool and do not provide educators a framework for teaching in an effective and timely manner. In a study that used the Zwisch scale to investigate the competency of general surgery residents by the end of residency, only approximately 27% of residents in the last 6 months of training reached complete autonomy on all cases and only approximately 33% reached complete autonomy on core procedures.19 This illustrates the need for the development of a standardized training framework that will not only assist surgical educators in teaching residents within the operating room, but also ensure that the majority of residents are competent surgeons by the time they finish training and move on to independent practice.

The Surgical Autonomy Program (SAP) is a novel education model that was created to guide both attendings and residents through intraoperative teaching. This model uses a modified version of the Zwisch scale, called the TAGS framework, in combination with Vygotsky’s social learning theory, which subdivides a large learning task into discrete, smaller zones of proximal development (ZPDs).20 In order to deploy SAP, our group developed a smartphone application (app) to track residents’ progress and allow for formative, rapid feedback. Our SAP smartphone app was designed to be user-friendly and provide real-time formative feedback for residents. The SAP has successfully been implemented within the neurosurgery residency training program at Duke University.20 In this study, we aimed to establish preliminary validity evidence that SAP is a reliable measure of autonomy and a useful tool for tracking resident competency over time.

Methods

SAP Design

A critical component of SAP is that it breaks each surgical case into 4 ZPDs. The ZPDs are sequential in nature, in which ZPD1 is always positioning and opening for the case (typically being the least complex portion), ZPD2 is the second most complex surgical portion of the case, ZPD3 is the most critical or complex portion of the case, and ZPD4 is the third most complex portion of the case and includes closing. For example, in an anterior cervical discectomy and fusion, ZPD1 is positioning and exposing down to the correct level; ZPD2 is retractor/distractor placement, discectomy, endplate preparation, and sizing of the graft; ZPD3 is removal of osteophyte, opening the posterior longitudinal ligament, and foraminal decompression; and ZPD4 is implanting the grafts and placing hardware, as well as closing. Typically, a resident always starts in ZPD1 and should not advance to the next stage until they achieve full autonomy on the previous stage. One exception is that residents may operate in ZPD4 prior to becoming fully autonomous on ZPD3, given that it is a much less complex portion of the procedure. To rate autonomy, the SAP uses a modified version of the Zwisch scale called the TAGS framework. The first level is teach and demonstrate (T), in which the faculty member is the primary surgeon and the resident simply observes and assists. The next level is advise and scaffold (A), in which the resident is the primary surgeon but is being verbally guided by the faculty. The third level is guide and monitor (G), in which the resident is the primary surgeon and only receives intermittent feedback or guidance from the attending on fine points or dealing with intraoperative complications. The final level is solo and observe (S), in which the resident performs the designated part of the surgery safely on their own, without any faculty guidance, and can teach a junior resident. Once a resident achieves the last stage, they are determined to be fully competent on that ZPD and can move on to the next ZPD level. A resident is fully autonomous, and thus competent, in an entire case once they reach solo and observe (S) on all 4 ZPDs.

Prior to the case, the resident and faculty determine on which ZPD the resident is mainly focusing through discussion and analysis of the resident’s previous performance. While the residents are evaluated using the TAGS scale for each of the ZPDs during a case, the ZPD of focus is the one in which the resident spends the most time operating. This allows for the resident to build on the skills they gained in earlier ZPDs as they progress through an index case. The SAP smartphone app allows the resident and faculty to look back at the resident’s progress across the different ZPDs for a given index case, permitting them to determine which zone the resident should focus on for the next case. Once a case is complete, the resident or attending creates a new entry within SAP by logging the surgical date, attending and resident physician, subspecialty, and case type. The resident rates the difficulty level of the case (easy, average, or difficult), designates on which ZPD the resident was mainly focused, and assigns a TAGS level for all 4 ZPDs. The faculty then enters the same information and is also able to freely dictate feedback in a text box. The free dictation allows for the faculty to specifically comment on areas of improvement, such as tissue handling, technical nuances, or the flow of the surgery. Once completed by both the resident and faculty, the resident receives the faculty feedback to compare with their self-assessment.

Participants and Residency Program Structure

All neurosurgery residents at Duke University Hospital from July 2017 to July 2021 participated in this study, and all attending faculty within the neurosurgery department were encouraged to use SAP for each surgical case. Attendings were provided with a teaching session about SAP and guidance on how to incorporate SAP into their operative teaching. Attendings within each neurosurgery subspecialty were asked to provide feedback for identifying appropriate ZPDs for each index case that fell under their area of expertise. This ensured that faculty had the same understanding of which parts of the case aligned with the 4 ZPDs. Fifty-six index cases were compiled based on Accreditation Council for Graduate Medical Education (ACGME) neurosurgery resident case minimums and divided into their appropriate subspecialty (e.g., lumbar fusion fell within spine surgery).21

Given that 1st-year residents (postgraduate year [PGY]–1) primarily spend time understanding the medical management of neurosurgical patients by rotating in the ICU and related units, cases in which residents were in PGY-1 were excluded from our analysis (n = 23). PGY-2 and PGY-3 are junior residency years. During this time, residents complete 20 months of neurosurgery service time broken into five 4-month blocks (pediatrics, spine junior A, spine junior B, functional/vascular, and tumor). PGY-4 is an academic year in which residents participate in research and have minimal clinical responsibilities. PGY-5 includes being the senior resident at the Veterans Affairs hospital (4 months), Duke regional community hospital (4 months), and pediatric neurosurgery service (4 months). The chief residency year (PGY-6) includes 4 months on spine, cranial, and functional/vascular subspecialties. PGY-7 is dedicated to residents gaining specialized surgical skills in Committee on Advanced Subspecialty Training (CAST) fellowships or additional research initiatives.

Data Collection

Data were extracted for all neurosurgical cases that took place from July 2017 to July 2021 at Duke University Hospital. This study was submitted to the institutional review board of Duke University and determined to be exempt from approval, and all data were de-identified prior to analyses.

Statistical Analysis

Descriptive statistics were used to describe the distribution of cases for each subspecialty, the ZPDs across each postgraduate year, the level of autonomy dictated by the TAGS scale across postgraduate years, and case complexity. Chi-square tests were used to look for differences in ZPD focus over time, level of autonomy over time, and level of autonomy based on case complexity. Further ordinal logistic regression was used to evaluate the odds of increased autonomy given ZPD, postgraduate year, and case complexity while controlling for the attending physician who completed the evaluation. All statistical analyses were run using RStudio statistical software (version 4.1; RStudio Team 2021).

Results

From 2017 to 2021, 4885 cases were logged by 27 different residents training in Duke’s neurosurgical residency training program. There were 30 different attendings at Duke who evaluated residents using SAP. The most common subspecialties that used SAP were spine (n = 1898) and cranial (n = 1494) (Table 1). The top 3 cases performed were craniotomy for supratentorial tumors (n = 660), thoracic and/or lumbar fusion (n = 570), and anterior cervical discectomy and fusion with or without corpectomy (n = 443).

TABLE 1.

Case subspecialties

SubspecialtyNo. of Cases
Spine1898
Cranial1494
Functional669
Pediatrics623
Vascular90
Peripheral nerve87
Endovascular16
Critical care procedures8

Faculty completed evaluations on 91% of cases. Based on faculty evaluations, 9.7% (n = 429) of cases focused on ZPD1, 27.2% (n = 1208) focused on ZPD2, 43.1% (n = 1914) focused on ZPD3, and 20.1% (n = 892) focused on ZPD4. For the zone on which the resident was focusing, attendings assigned level T (teach and demonstrate) in 2.1% (n = 95) of cases, level A (advise and scaffold) in 17.0% (n = 755), level G (guide and monitor) in 39.4% (n = 1750), and level S (solo and observe) in 41.5% (n = 1843).

The SAP is a 2D evaluation tool that incorporates both the ZPD and the level of autonomy using the TAGS scale. The ZPD of focus directly correlates with increasing postgraduate year (Table 2 and Fig. 1) (χ2 = 1221.1, df = 15, p < 0.001). Levels of autonomy also increase with year of residency (χ2 = 3553.5, df = 15, p < 0.001) (Table 3 and Fig. 2). An ordinal regression analysis was run to evaluate the odds of increased autonomy, given the ZPD focus and year of residency. For every year increase in postgraduate year, the odds of operating at a higher level of independence was 2.16 times greater (95% CI 2.11–2.21, p < 0.001) when controlling for ZPD, the attending providing the evaluation, and case difficulty as determined by the attending. The odds of residents performing at a higher level of autonomy as dictated by the TAGS scale was lowest for the most complex portion of the case (ZPD3). Residents were 57% less likely to operate at a higher level of independence in ZPD2 compared with ZPD1 (OR 0.43, 95% CI 0.40–0.47; p < 0.001), 82% less likely to operate at a higher level of independence in ZPD3 compared with ZPD1 (OR 0.18, 95% CI 0.17–0.20; p < 0.001), and 44% less likely to operate at a higher level of independence in ZPD4 compared with ZPD1 (OR 0.56, 95% CI 0.52–0.61; p < 0.001) when controlling for postgraduate year, the attending providing the evaluation, and case difficulty as determined by the attending. Figure 2 is a graphical representation of the proportion of time spent operating within each level of autonomy. Figure 3 demonstrates similar results but split by ZPD focus. When pooling the ZPDs of focus for all procedure instances, residents in PGY-6 reached solo and observe on 48.5% (n = 560) of cases and PGY-7 residents reached solo and observe on 72.9% (n = 86) of cases (Table 3). The most common cases that did not reach solo and observe on all zones were thoracic and lumbar fusion (n = 11), extreme lateral interbody fusion (XLIF)/anterior lumbar interbody fusion (ALIF) (n = 9), and craniotomy for tumor resection (n = 7). In comparing resident self-assessment to attending assessment within the ZPD on which the resident was focusing, the resident self-assessment of autonomy matched the attending’s 51% (n = 2276) of the time. Of the cases that were not concordant, 87% (n = 1886) were within one level on the TAGS scale (i.e., T vs A). The difference between attending or resident rating was calculated by giving each level of the TAGS scale a number (T = 1, A = 2, G = 3, S = 4) and subtracting the resident evaluation from the attending evaluation. The average difference in autonomy score was 0.54, indicating that attendings’ ratings were overall higher than residents’ ratings.

TABLE 2.

ZPD focus over time

ZPD1ZPD2ZPD3ZPD4
PGY-2 (n = 730)39.1 (250)35.3 (226)16.6 (106)9.1 (58)
PGY-3 (n = 1085)10.2 (102)43.9 (439)34.2 (342)11.7 (117)
PGY-4 (n = 298)7.6 (22)24.7 (72)47.8 (139)19.9 (58)
PGY-5 (n = 1348)2.2 (27)19.9 (246)54.6 (676)23.4 (290)
PGY-6 (n = 1297)2.0 (23)17.4 (201)51.0 (589)29.6 (342)
PGY-7 (n = 127)4.2 (5)20.3 (24)52.5 (62)22.9 (27)

Values are given as percent (n). The calculated percentage does not include nonevaluated cases.

FIG. 1.
FIG. 1.

ZPD focus during the year of residency (PGY) is determined by the attending. Proportion is the number of cases spent in each ZPD of the total number of cases for that residency year.

TABLE 3.

TAGS per PGY

TAGS
PGY-2 (n = 730)3.8 (24)17.5 (112)41.6 (266)37.2 (238)
PGY-3 (n = 1085)1.9 (19)20.3 (203)44.0 (440)33.8 (338)
PGY-4 (n = 298)3.1 (9)14.4 (42)47.8 (139)34.7 (101)
PGY-5 (n = 1348)2.0 (25)12.6 (156)43.4 (538)42.0 (520)
PGY-6 (n = 1297)1.6 (18)20.3 (234)29.7 (343)48.5 (560)
PGY-7 (n = 127)0 (0)6.8 (8)20.3 (24)72.9 (86)

Values are given as percent (n). The calculated percentage does not include nonevaluated cases.

FIG. 2.
FIG. 2.

Level of autonomy (TAGS) assigned by the attending across all ZPDs. Proportion is the number of cases given each TAGS designation for each year of residency (PGY).

FIG. 3.
FIG. 3.

Level of autonomy (TAGS) assigned by the attending for each ZPD: ZPD1 (A), ZPD2 (B), ZPD3 (C), and ZPD4 (D). Proportion is the number of cases given each TAGS designation for each year of residency (PGY).

Faculty and residents evaluated case complexity after each procedure. Case difficulty was a subjective metric decided by the attending and resident separately at the end of each case. Per faculty rating, 9.8% (n = 436) of cases were easy, 65.1% (n = 2893) were average, and 25.1% (n = 1114) were hard. There was agreement between attendings’ and residents’ independent assessment of case difficulty in 76% (n = 3376) of cases. For cases that did not match on complexity evaluation, there was one level of difference (e.g., easy vs average) on 97% (n = 1038) of the cases and two levels of difference (easy vs hard) only 3% (n = 27) of the time (2 cases were removed because of missing resident data). Not surprisingly, more guidance is given to residents with increased case complexity (χ2 = 160.28, df = 6, p < 0.001) (Table 4 and Fig. 4). An ordinal regression analysis demonstrated that residents were 44% more likely to operate at a higher level of independence on an easy case (OR 1.44, 95% CI 1.29–1.61; p < 0.001) and 28% less likely to operate at a higher level of independence on a hard case (OR 0.72, 95% CI 0.67–0.77; p < 0.001), when compared with average cases, while controlling for ZPD and attending physician. When pooling the ZPDs of focus for all procedure instances, residents reached solo and observe in their zone of focus on 36.2% (n = 403) of difficult cases (Table 4).

TABLE 4.

TAGS per case difficulty

TAGS
Easy (n = 436)1.6 (7)11.5 (50)32.6 (142)54.4 (237)
Average (n = 2893)1.7 (48)16.7 (483)40.1 (1159)41.6 (1203)
Hard (n = 1114)3.6 (40)19.9 (222)40.3 (449)36.2 (403)

Values are given as percent (n). The calculated percentage does not include nonevaluated cases.

FIG. 4.
FIG. 4.

Level of autonomy (TAGS) assigned by the attending by case difficulty. Proportion is the number of cases given each TAGS designation for each case complexity.

Discussion

To our knowledge, the SAP is the first tool developed that incorporates a resident evaluation scale within an educational framework. Our study supports the construct validity of the SAP for intraoperative teaching and evaluation. We demonstrated learning curves that nicely depict where resident focus is targeted during surgical procedures over the course of residency training. Residents primarily focus on ZPD1, the initial stage of a surgical case, during PGY-2, which is their 1st true operative year of residency after internship. ZPD2 is the major area of focus for junior residents in PGY-3, whereas residents mainly focus on ZPD3 and ZPD4 during their senior and chief residency years. Residents spend the most time focusing on ZPD3, the most complex and critical portion of the surgical case, from PGY-4 onward. This provides strong evidence that residents are learning and gaining skill in a stepwise manner, as they progress from ZPD1 to ZPD4, by the time they finish residency.

Residents are operating as the primary surgeon (A, G, or S) within their zone of focus nearly 98% of the time and operate either with minimal feedback or completely independently (either G or S) 81% of the time, supporting the SAP’s goal of increasing resident autonomy within the OR. Furthermore, resident autonomy increases with year of residency. By the end of the chief residency year (PGY-6), residents had reached solo and observe on their zone of focus almost 50% of the time, and by the end of residency (PGY-7), residents were fully autonomous within their zone of focus more than 70% of the time. Given that PGY-6 and PGY-7 residents spend the majority of their time focusing on ZPD3, this gives us confidence that residents are becoming competent in performing the most complex part of a surgical case. Additionally, residents in their final year of residency operated as the lead surgeon (either G or S on the TAGS scale) on all ZPDs in more than 80% of the cases, and in more than 90% of cases PGY-7 residents operated in the highest two levels of autonomy during ZPD3. Interestingly, general surgery residents who were evaluated using the System for Improving and Measuring Procedural Learning (SIMPL) reached complete independence on only 33.3% of the core procedures in their last 6 months of training.19 We attribute gains in resident autonomy within the operating room to the clear method of intraoperative teaching provided by SAP. Specifically, SAP breaks down each surgical case into 4 components instead of viewing the case as one entity. By agreeing on which ZPD the resident will focus, the attending and trainee are able to direct their efforts to an appropriate part of a surgical case, rather than an operative portion beyond the trainee’s reach, avoiding cognitive overload.22 Furthermore, previous research has shown that better learning occurs when the teacher and trainee agree on learning objectives.23 The ZPDs were modeled after Lev Vygotsky’s social learning theory, in which a more knowledgeable other (a teacher) guides a learner progressively through a task.2426 The SAP is the first to successfully incorporate this educational theory into surgical education.

Furthermore, resident autonomy is lowest during the most critical part of the case (ZPD3) and decreases with case complexity. These results match what one would predict for intraoperative teaching. This shows that both residents and faculty are using SAP in an appropriate manner. Residents overall rated themselves lower than attendings using the TAGS scale. While the residents and attendings matched in their autonomy assessment in approximately 50% of the cases, the large majority of nonconcordant cases were different by only one autonomy level. Future research is needed to understand if resident self-evaluation changes during year of residency and by gender. Finally, given that faculty completed 91% of case evaluations, our smartphone-based application can be seamlessly integrated into the teaching and assessment of resident operative skill and autonomy. This is a reliable and consistent tool to track resident autonomy and competency across their journey within surgical training.

There are some limitations to our study. First, we did not use an alternative and previously validated measure of performance to compare with SAP. Nonetheless, the TAGS framework has a similar framework to the Zwisch scale, which has been widely accepted as a valid measure of autonomy in surgical education.14 Additionally, we did not have a measure of interrater reliability within our analysis; however, we controlled for the attending physician providing the evaluation within our ordinal regression models. Another limitation is that the determination of case difficulty was subjective in nature by both residents and faculty. A future attempt to validate that assessment with an objective comparator is worth future investigation. Lastly, this study is limited by being a single-center analysis. SAP has been implemented in a number of surgical departments across the United States, and future studies will include results from these institutions.

The SAP has the potential to transform surgical education into a competency-based system. The ability to track resident progress, along with a stepwise approach for helping residents gain autonomy in the operating room, makes the SAP a novel tool. Additionally, the ability to view the ZPD on which the resident focused their attention and the level of autonomy they were granted during the previous case of a given procedure allows the attending to know exactly where the resident is in the progression to autonomy on a specific portion of a case. Given the ability to target teaching to the appropriate level, we believe that this tool will allow residents to become fully competent at a faster pace than with traditional intraoperative teaching methods. Prior work has demonstrated that residents progress in autonomy at different rates on a core index procedure.20 Once a sufficient number of surgical cases across multiple institutions are logged, future studies reporting the identification of fast versus slow learners using SAP are warranted. Further research will also include evaluating the number of cases required to reach competency compared with the standard case minimums dictated by residency review committees.21 Additionally, future investigative efforts should compare the rates of resident competency across all neurosurgical index cases by the time training ends among programs that use SAP and those that do not.

Conclusions

This study demonstrates preliminary evidence to support the construct validity of the SAP, a novel, multidimensional surgical education and assessment tool. We have shown that this tool is able to successfully track resident autonomy and progress over time. Additionally, our smartphone-based application was widely used among surgical faculty and residents, supporting the integration into neurosurgical training and evaluation. We hypothesize that larger implementation of SAP across multiple surgical centers will aid in the movement toward a competency-based residency education system.

Disclosures

This work was funded through departmental support by the Duke University Department of Neurosurgery. Dr. Haglund, Mr. Dharmapurikar, and Dr. Lad have financial equity in Surgical Autonomy Program Inc. Dr. Lad: consultant for Surgical Autonomy Program Inc. Dr. Haglund: ownership in Surgical Autonomy Program Inc. and Alphatec; consultant for Alphatec, NuVasive, and Provident Medical; direct stock ownership in Alphatec; and support of non–study-related clinical or research effort overseen by the author: NuVasive, Medtronic, UCB Pharma, and Alphatec.

Author Contributions

Conception and design: Haglund, Suarez, McDaniel. Acquisition of data: Haglund, Suarez, Dharmapurikar. Analysis and interpretation of data: Haglund, Kirsch, Dunn. Drafting the article: Kirsch. Critically revising the article: Haglund, Kirsch, Suarez, McDaniel. Reviewed submitted version of manuscript: Haglund, Kirsch, Suarez, McDaniel, Dunn, Lad. Approved the final version of the manuscript on behalf of all authors: Haglund. Statistical analysis: Kirsch, Dunn. Administrative/technical/material support: Haglund, McDaniel, Dharmapurikar, Lad. Study supervision: Haglund, McDaniel, Lad.

Supplemental Information

Videos

Video Abstract. https://vimeo.com/729628676.

References

  • 1

    Whang EE, Mello MM, Ashley SW, Zinner MJ. Implementing resident work hour limitations: lessons from the New York State experience. Ann Surg. 2003;237(4):449455.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • 2

    Cohen-Gadol AA, Piepgras DG, Krishnamurthy S, Fessler RD. Resident duty hours reform: results of a national survey of the program directors and residents in neurosurgery training programs. Neurosurgery. 2005;56(2):398403.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • 3

    Basu CB, Chen LM, Hollier LH Jr, Shenaq SM. The effect of the Accreditation Council for Graduate Medical Education Duty Hours Policy on plastic surgery resident education and patient care: an outcomes study. Plast Reconstr Surg. 2004;114(7):18781886.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • 4

    Accreditation Council for Graduate Medical Education. Duty Hours Subcommittee Report. Accreditation Council for Graduate Medical Education; 2009.

  • 5

    Philibert I, Amis S, Vasiliou E. The ACGME 2011 Duty Hour Standard. Enhancing Quality of Care, Supervision and Resident Professional Development. Accreditation Council for Graduate Medical Education;2011.https://www.citizen.org/wp-content/uploads/jgme-monograph.pdf

    • Search Google Scholar
    • Export Citation
  • 6

    Abecassis IJ, Sen RD, Ellenbogen RG, Sekhar LN. Developing microsurgical milestones for psychomotor skills in neurological surgery residents as an adjunct to operative training: the home microsurgery laboratory. J Neurosurg. 2021;135(1):194204.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • 7

    Morse CR, Mathisen DJ. Educational challenges of the operating room. Thorac Surg Clin. 2019;29(3):269277.

  • 8

    Kasten SJ. Empiric Evidence for Revision of Minimum Index Case Numbers for Accreditation of Plastic Surgery Training. Thesis. University of Illinois at Chicago; 2012.Accessed June 8, 2022. https://hdl.handle.net/10027/8941

    • Search Google Scholar
    • Export Citation
  • 9

    Jeray KJ, Frick SL. A survey of resident perspectives on surgical case minimums and the impact on milestones, graduation, credentialing, and preparation for practice: AOA critical issues. J Bone Joint Surg Am. 2014;96(23):e195.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • 10

    Stride HP, George BC, Williams RG, et al. Relationship of procedural numbers with meaningful procedural autonomy in general surgery residents. Surgery. 2018;163(3):488494.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • 11

    Williams RG, Swanson DB, Fryer JP, et al. How many observations are needed to assess a surgical trainee’s state of operative competency? Ann Surg. 2019;269(2):377382.

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 12

    Abbott KL, Chen X, Clark M, Bibler Zaidi NL, Swanson DB, George BC. Number of operative performance ratings needed to reliably assess the difficulty of surgical procedures. J Surg Educ. 2019;76(6):e189e192.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • 13

    Mattar SG, Alseidi AA, Jones DB, et al. General surgery residency inadequately prepares trainees for fellowship: results of a survey of fellowship program directors. Ann Surg. 2013;258(3):440449.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • 14

    George BC, Teitelbaum EN, Meyerson SL, et al. Reliability, validity, and feasibility of the Zwisch scale for the assessment of intraoperative performance. J Surg Educ. 2014;71(6):e90e96.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • 15

    Knox ADC, Gilardino MS, Kasten SJ, Warren RJ, Anastakis DJ. Competency-based medical education for plastic surgery: where do we begin? Plast Reconstr Surg. 2014;133(5):702e710e.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • 16

    Frank JR, Snell LS, Cate OT, et al. Competency-based medical education: theory to practice. Med Teach. 2010;32(8):638645.

  • 17

    DaRosa DA, Zwischenberger JB, Meyerson SL, et al. A theory-based model for teaching and assessing residents in the operating room. J Surg Educ. 2013;70(1):2430.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • 18

    Martin JA, Regehr G, Reznick R, et al. Objective structured assessment of technical skill (OSATS) for surgical residents. Br J Surg. 1997;84(2):273278.

    • Search Google Scholar
    • Export Citation
  • 19

    George BC, Bohnen JD, Williams RG, et al. Readiness of US General Surgery Residents for Independent Practice. Ann Surg. 2017;266(4):582594.

  • 20

    Haglund MMCA, Suarez A, Dharmapurikar R, Lad SP, McDaniel KE. The Surgical Autonomy Program (SAP): a pilot study of social learning theory applied to competency based neurosurgical education. Neurosurgery. 2021;88(4):E345E350.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • 21

    Review Committee for Neurological Surgery. Neurological surgery case log defined case categories and required minimum numbers. Accreditation Council for Graduate Medical Education; June 2019. Accessed June 8, 2022. https://www.acgme.org/globalassets/pfassets/programresources/160_neurosurgery_case_categories_and_minimum_numbers_effective_7_1_2019.pdf

    • Search Google Scholar
    • Export Citation
  • 22

    van Merriënboer JJ, Sweller J. Cognitive load theory in health professional education: design principles and strategies. Med Educ. 2010;44(1):8593.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • 23

    Roberts NK, Williams RG, Kim MJ, Dunnington GL. The briefing, intraoperative teaching, debriefing model for teaching in the operating room. J Am Coll Surg. 2009;208(2):299303.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • 24

    Sadideen H, Plonczak A, Saadeddin M, Kneebone R. How educational theory can inform the training and practice of plastic surgeons. Plast Reconstr Surg Glob Open. 2018;6(12):e2042.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • 25

    McLeod SA. Vygotsky’s sociocultural theory of cognitive development. Simply Psychology. August 5, 2018. Accessed June 8, 2022. https://www.simplypsychology.org/vygotsky.html

    • Search Google Scholar
    • Export Citation
  • 26

    Vygotskiĭ LS, Cole M. Mind in Society: The Development of Higher Psychological Processes. Harvard University Press;1978.

  • View in gallery

    ZPD focus during the year of residency (PGY) is determined by the attending. Proportion is the number of cases spent in each ZPD of the total number of cases for that residency year.

  • View in gallery

    Level of autonomy (TAGS) assigned by the attending across all ZPDs. Proportion is the number of cases given each TAGS designation for each year of residency (PGY).

  • View in gallery

    Level of autonomy (TAGS) assigned by the attending for each ZPD: ZPD1 (A), ZPD2 (B), ZPD3 (C), and ZPD4 (D). Proportion is the number of cases given each TAGS designation for each year of residency (PGY).

  • View in gallery

    Level of autonomy (TAGS) assigned by the attending by case difficulty. Proportion is the number of cases given each TAGS designation for each case complexity.

  • 1

    Whang EE, Mello MM, Ashley SW, Zinner MJ. Implementing resident work hour limitations: lessons from the New York State experience. Ann Surg. 2003;237(4):449455.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • 2

    Cohen-Gadol AA, Piepgras DG, Krishnamurthy S, Fessler RD. Resident duty hours reform: results of a national survey of the program directors and residents in neurosurgery training programs. Neurosurgery. 2005;56(2):398403.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • 3

    Basu CB, Chen LM, Hollier LH Jr, Shenaq SM. The effect of the Accreditation Council for Graduate Medical Education Duty Hours Policy on plastic surgery resident education and patient care: an outcomes study. Plast Reconstr Surg. 2004;114(7):18781886.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • 4

    Accreditation Council for Graduate Medical Education. Duty Hours Subcommittee Report. Accreditation Council for Graduate Medical Education; 2009.

    • Search Google Scholar
    • Export Citation
  • 5

    Philibert I, Amis S, Vasiliou E. The ACGME 2011 Duty Hour Standard. Enhancing Quality of Care, Supervision and Resident Professional Development. Accreditation Council for Graduate Medical Education;2011.https://www.citizen.org/wp-content/uploads/jgme-monograph.pdf

    • Search Google Scholar
    • Export Citation
  • 6

    Abecassis IJ, Sen RD, Ellenbogen RG, Sekhar LN. Developing microsurgical milestones for psychomotor skills in neurological surgery residents as an adjunct to operative training: the home microsurgery laboratory. J Neurosurg. 2021;135(1):194204.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • 7

    Morse CR, Mathisen DJ. Educational challenges of the operating room. Thorac Surg Clin. 2019;29(3):269277.

  • 8

    Kasten SJ. Empiric Evidence for Revision of Minimum Index Case Numbers for Accreditation of Plastic Surgery Training. Thesis. University of Illinois at Chicago; 2012.Accessed June 8, 2022. https://hdl.handle.net/10027/8941

    • Search Google Scholar
    • Export Citation
  • 9

    Jeray KJ, Frick SL. A survey of resident perspectives on surgical case minimums and the impact on milestones, graduation, credentialing, and preparation for practice: AOA critical issues. J Bone Joint Surg Am. 2014;96(23):e195.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • 10

    Stride HP, George BC, Williams RG, et al. Relationship of procedural numbers with meaningful procedural autonomy in general surgery residents. Surgery. 2018;163(3):488494.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • 11

    Williams RG, Swanson DB, Fryer JP, et al. How many observations are needed to assess a surgical trainee’s state of operative competency? Ann Surg. 2019;269(2):377382.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • 12

    Abbott KL, Chen X, Clark M, Bibler Zaidi NL, Swanson DB, George BC. Number of operative performance ratings needed to reliably assess the difficulty of surgical procedures. J Surg Educ. 2019;76(6):e189e192.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • 13

    Mattar SG, Alseidi AA, Jones DB, et al. General surgery residency inadequately prepares trainees for fellowship: results of a survey of fellowship program directors. Ann Surg. 2013;258(3):440449.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • 14

    George BC, Teitelbaum EN, Meyerson SL, et al. Reliability, validity, and feasibility of the Zwisch scale for the assessment of intraoperative performance. J Surg Educ. 2014;71(6):e90e96.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • 15

    Knox ADC, Gilardino MS, Kasten SJ, Warren RJ, Anastakis DJ. Competency-based medical education for plastic surgery: where do we begin? Plast Reconstr Surg. 2014;133(5):702e710e.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • 16

    Frank JR, Snell LS, Cate OT, et al. Competency-based medical education: theory to practice. Med Teach. 2010;32(8):638645.

  • 17

    DaRosa DA, Zwischenberger JB, Meyerson SL, et al. A theory-based model for teaching and assessing residents in the operating room. J Surg Educ. 2013;70(1):2430.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • 18

    Martin JA, Regehr G, Reznick R, et al. Objective structured assessment of technical skill (OSATS) for surgical residents. Br J Surg. 1997;84(2):273278.

    • Search Google Scholar
    • Export Citation
  • 19

    George BC, Bohnen JD, Williams RG, et al. Readiness of US General Surgery Residents for Independent Practice. Ann Surg. 2017;266(4):582594.

  • 20

    Haglund MMCA, Suarez A, Dharmapurikar R, Lad SP, McDaniel KE. The Surgical Autonomy Program (SAP): a pilot study of social learning theory applied to competency based neurosurgical education. Neurosurgery. 2021;88(4):E345E350.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • 21

    Review Committee for Neurological Surgery. Neurological surgery case log defined case categories and required minimum numbers. Accreditation Council for Graduate Medical Education; June 2019. Accessed June 8, 2022. https://www.acgme.org/globalassets/pfassets/programresources/160_neurosurgery_case_categories_and_minimum_numbers_effective_7_1_2019.pdf

    • Search Google Scholar
    • Export Citation
  • 22

    van Merriënboer JJ, Sweller J. Cognitive load theory in health professional education: design principles and strategies. Med Educ. 2010;44(1):8593.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • 23

    Roberts NK, Williams RG, Kim MJ, Dunnington GL. The briefing, intraoperative teaching, debriefing model for teaching in the operating room. J Am Coll Surg. 2009;208(2):299303.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • 24

    Sadideen H, Plonczak A, Saadeddin M, Kneebone R. How educational theory can inform the training and practice of plastic surgeons. Plast Reconstr Surg Glob Open. 2018;6(12):e2042.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • 25

    McLeod SA. Vygotsky’s sociocultural theory of cognitive development. Simply Psychology. August 5, 2018. Accessed June 8, 2022. https://www.simplypsychology.org/vygotsky.html

    • Search Google Scholar
    • Export Citation
  • 26

    Vygotskiĭ LS, Cole M. Mind in Society: The Development of Higher Psychological Processes. Harvard University Press;1978.

Metrics

All Time Past Year Past 30 Days
Abstract Views 0 0 0
Full Text Views 282 282 282
PDF Downloads 257 257 257
EPUB Downloads 0 0 0