Traditionally, neurosurgical education is carried out by academically inclined neurosurgeons at teaching institutions. For the most part, these neurosurgeons are excellent role models who teach by example, but who lack formal instruction in adult educational principles; however, increasing pressure for more formalized educational methods in resident training exists.17,81 This pressure to increase the efficacy of neurosurgical education has developed for several reasons, including resident duty-hour restraints, public concerns about supervision of trainees, and the realization that the ever-increasing fund of knowledge required by a practicing neurosurgeon mandates acquiring lifelong learning skills.8,41,43,44,58,72 Furthermore, the contemporary neurosurgeon is expected to be the leader of a medical team that includes advanced practice clinicians, operating room nurses and technicians, and staff in the intensive care and surgical wards. Part of this leadership role involves the instruction, supervision, and assessment of those involved in caring for patients with neurological illnesses. Furthermore, a neurosurgeon is responsible for the education of patients and patient families before treatment plans can be formulated. The goal of this paper is to review the literature regarding education in neurosurgery and to identify principles of adult learning and assessment that should be woven into the framework of neurosurgery education.
Principles of Adult Learning
A complete and comprehensive review of how adults learn can be found elsewhere.85 This report aims to focus on those elements most relevant to neurosurgical education, emphasizing basic tenets that should be understood by all neurosurgery educators. Learning includes the acquisition of 3 domains: knowledge, skills, and attitudes. Becoming a member of the neurosurgical profession not only demands the acquisition of knowledge and skills but also involves a process of growing into the professional community.85 The literature contains a number of models of learning, including 4 that appear to be particularly relevant to neurosurgical education: constructionism, scaffolding, the traditional Miller’s pyramid, and cognitive apprenticeship. A constructionist viewpoint, in which learning is the process of constructing new knowledge on the foundations of what the learner already knows,85,88 seems to be most relevant for neurosurgical education. For example, when a neurosurgeon sees a patient with a new disease process or hears a lecture on that disease, he or she compares this with what is already known and reflects on the difference. This can be exemplified by the experiential learning model proposed by Kolb.51,85 In Kolb’s scheme (Fig. 1), the learner has a concrete experience (observation of a new procedure) upon which he/she reflects (reads and studies the details of the procedure). Through reflection, the learner formulates abstract concepts and makes appropriate generalizations (begins to perform the procedure under direct supervision). The learner then consolidates his/her understanding by testing the implications of this knowledge in new situations (performs this procedure and similar procedures with decreasing levels of supervision), which provides a concrete experience, and the cycle continues.85
The Kolb cycle adapted to neurosurgical training. In Kolb’s scheme, the learner observes a new procedure to obtain a concrete experience upon which to reflect by reading and studying about that procedure. Through reflection, the trainee is able to progress to performing the observed procedure under direct supervision (formulate abstract concepts and make appropriate generalizations in Kolb parlance). The trainee then consolidates his or her understanding by testing the implications of the knowledge in new situations (for the neurosurgical learner, this means being able to perform this procedure and similar procedures unsupervised). This skill development then provides the learner with a concrete experience upon which to build on, and the cycle continues. Adapted from Kolb, David A., Experiential Learning: Experience as a Source of Learning and Development, ©1984. Published with permission of Pearson Education, Inc., New York, NY. Figure is available in color online only.
The term “scaffolding” refers to an educational structure that guides learners thorough the process of understanding how new information is part of the whole.85 Scaffolding includes curriculum organization, specific reading assignments, topic lectures, and planned experiential learning in the clinic and operating room, and it makes explicit the overall objectives and intended learning outcomes for the neurosurgical learner. Probably the best example of this within organized neurosurgery is the Matrix Curriculum developed by educational and academic leaders in the field aiming to establish a “core curriculum” for resident training that will be discussed later in this review.
A third relevant model is Miller’s pyramid (Fig. 2).56 In this model, the learner (1st- or 2nd-year resident) obtains factual knowledge as a prerequisite for future experiences—this is the “knows” level of the pyramid. In the neurosurgical setting, this includes the basics of neuroanatomy, neurophysiology, etc., and is measured by written neurosurgical board examinations and with basic questions on rounds. The next step on the pyramid is the “knows how” level, which is manifested by the ability of the learner (midlevel resident) to manipulate or apply this knowledge to a specific patient’s disease process. As one transcends to the “shows how” stage (senior resident), all acquired knowledge and skill are put to test in a somewhat independent yet supervised environment. Finally, the highest level of Miller’s pyramid is the “does” stage, which corresponds to the practicing neurosurgeon. To the traditional scheme of Miller could be added the capstone “does better,” in which, with further experience, innovation, and research, the “life-long learner” improves practice and moves the field forward.
Miller’s pyramid adapted to the neurosurgical learner. The 1st- or 2nd-year resident learner first obtains factual knowledge in the form of neuroanatomy or neurophysiology, which is usually assessed by written neurosurgical board examinations and as basic questions on rounds as a prerequisite for future experiences (“knows”). The next higher step on the pyramid is the “knows how” level, which is manifest by a midlevel resident’s ability to manipulate or apply this knowledge to a specific patient’s disease process. The next step on the pyramid is to that of a senior resident, the “shows how” stage, where all acquired knowledge and skill is put to test in a somewhat independent yet supervised environment. Finally, the highest level of Miller’s pyramid is the “does” stage. The “does” stage involves the neurosurgical practitioner who is independent and requires no further supervision. To the traditional scheme of Miller is added the capstone, “does better” stage, in which with further experience, innovation, and research the “learner” improves practice and moves the field forward. Modified from Figure 1 from Miller, G: The assessment of clinical skills/competence/performance. Academic Medicine 65(9):S63–S67, 1990. http://journals.lww.com/academicmedicine. Published with permission from Wolters Kluwer Health, Inc. Figure is available in color online only.
Finally, there is the concept of the cognitive apprenticeship model of Collins et al.19 In this model, the novice engages with more experienced individuals performing a given task or solving a problem. The model involves 5 elements: demonstration of the thinking process (Modeling), assisting and supporting learner activities by methods including scaffolding (Coaching), self-analysis and assessment (Reflection), verbalizing the result of the reflection (Articulation), and formation and testing of own theory (Exploration). The fundamental premise in this model is making the thinking process visible to the learner rather than observational.10,19
Principles of Assessment
In general, assessment of medical and surgical trainees is competency based. This means that the trainee achieves a set of predefined criteria during his/her training before moving to the next level of instruction.4,6,9,27,32 The assessment of skills at each level is very important. In neurosurgery, this is particularly challenging because trainees need to be evaluated in several different environments (e.g., operating room, emergency room, clinic, and patient wards) on a broad range of skills and knowledge (e.g., operative, interpersonal, cognitive, organizational, leadership, and ethics).
An effective assessment tool should fulfill 5 important criteria. The assessment process and tool must be 1) appropriate for goal of the assessment; 2) feasible for the rater to use and easily understandable; 3) economical and cost effective; 4) objective with very limited subjectivity of the rating; and 5) reliable and valid.36 Reliability refers to whether a test is consistent in its outcome (inter-item consistency, inter-rater reliability, and inter-test reliability).14,87 If an instrument is not reliable it cannot be valid (Table 1).1,46,55,86 Validity is defined as the property of being true, correct, and in conformity with reality.87 Validity answers the question “Does our assessment measure what we think it measures?” Construct validity is the ability of a measurement scale to score as predicted compared with other measurement scales. Predictive validity, which refers to the extent to which a test predicts future performance,14,28,87 would be the most desired type of validity for surgical practice. For example, predictive validity would indicate what score on the oral neurosurgical board examination predicts a neurological surgeon in practice who makes good clinical decisions and performs safe surgical procedures. No published examples of predictive validity are found in the current neurosurgical literature.
Educational assessment definitions
| Parameter | Definition | Outcome Measure/Example |
|---|---|---|
| Validity | ||
| Face | Extent to which the examination resembles the situation in the real world. | Workplace vs laboratory environment, human vs animal vs synthetic tissue. |
| User’s opinion about functionality and realism of test. | ||
| Content | Extent to which the intended content domain is being measured by the assessment exercise. | Task components of the assessment procedure. Content of test is suited to measure what it is supposed to measure. |
| Written test blueprints to actual course material. | ||
| Construct | Extent to which the test is able to differentiate between a good and bad performer; the test measures the trait it is supposed to measure. | Significance of difference between 2 groups of performers. |
| Experienced vs inexperienced subjects demonstrate different scores on a given task novel to the inexperienced testee. | ||
| Concurrent | Extent to which the result of the test correlates with gold-standard tests known to measure the same thing being tested. | Correlation analysis with other assessment methods. |
| New oral board format demonstrates similar outcome to prior board format. | ||
| Predictive | Extent to which this assessment will predict future performance | Follow-up assessments, proficiency gain curves. |
| Good score on MCAT translates to good Step 1 USMLE scores. | ||
| Reliability | ||
| Inter-rater | Extent of agreement between the scores of 2 or more raters testing the same subject. | Correlation between 2 blinded/nonblinded assessors. |
| Inter-item | Extent to which different components of a test correlate (internal consistency). | Correlation between different items of a test and how these contribute to the outcome of the test. |
| Inter-test | Ability of a test to generate similar results when applied at 2 different time points or agreement of scores when the same test is taken twice. | Correlations between test and retest. |
| Acceptability | Extent to which an assessment procedure is accepted by the subjects involved in the assessment. | Survey results. |
| Educational impact | Extent to which test results and feedback contribute to improve the learning strategy on behalf of the trainer and the trainee. | Survey results, proficiency gain curves. |
| Assessment | ||
| Formative | Aims at development by monitoring a trainee’s progress and giving structured feedback. | Low stake, informal feedback. |
| Midrotation feedback, postprocedure discussion. | ||
| Summative | Used for selection, needs predefined levels of outcome. | High stakes, higher standards for construct validity and reliability are required. |
| MCAT, written board test, credentialing. |
Although it is clear that validity is important, the question remains: “How does one measure or verify validity?” Sources of validity evidence and distraction55 are presented in Table 2.62 The various sources of evidence listed in this table are used to satisfy that validity exists for a certain assessment tool. On the other hand, demonstration of any of the various threats to validity would question a particular assessment method. For example, internal structure provides evidence of validity when there is good test item discrimination and test/retest reproducibility. The ability of a given test question on the written neurosurgical board examination to separate high-scoring from low-scoring examinees on a year-to-year basis provides validity evidence of this assessment tool. Relationship to other variables might be exemplified by comparing outcomes of a new oral board examination format against a prior or simpler version of this examination to verify validity of the new format. Demonstration of construct underrepresentation would question validity of an assessment if it includes too few observations of a surgical procedure or low inter-rater reliability of the faculty observers for a given surgical skill.
Definition of evidence and threats to validity
| Evidence of Validity | |
| Content | Blueprint and specifications of a test as determined by expert review and discussion. |
| Response process | Quality control of format, scoring, and reporting of the assessment. It is defined as evidence of data integrity such that all sources of error associated with the test administration are controlled or eliminated to the maximum extent possible. |
| Internal structure | Examines the score scale reliability, item discrimination, test/retest reproducibility, and generalizability of the evaluation. Statistical (G-study) and psychometric characteristics of examination items or performance prompts are used to score and scale the assessment. |
| Relation to other variables | Refers to the relationship of the assessment to external and independent evidence as well as the generalizability of the assessment. |
| Consequences | Considers the impact of the test outcome on the learners and society and could lead to future changes in curriculum or methods. |
| Threats to Validity | |
| Construct underrepresentation | Too few test items on an examination, too few observations of a surgical procedure, or low inter-rater reliability of the faculty observers for a given surgical skill. This is an assessment that does not test all of the important aspects of a given skill or body of knowledge. The test or observation misses things that should be assessed. |
| Construct-irrelevant variance | Assessing irrelevant items that are not part of what is being tested. This can include poorly constructed questions that make it difficult to answer correctly, which can lead to guessing or to test-wise students using the flaws to arrive at the correct answer when they normally would not. Biased questions can favor one group of students over another. |
Both formative assessment and summative assessment are relevant for the evaluation of training in neurosurgery. The former aims to promote the development of the trainee through close monitoring of progress and through the provision of structured feedback, whereas the latter is used for selection or credentialing, can be passed or failed, and has a preset threshold that has to be reached. Higher standards for construct validity and reliability are required for summative assessment since the stakes are higher and outcomes more definitive.87 Formative assessment is essentially a progress report, much like biannual meetings between a program director and a trainee to confirm steady progression in the residency program. Conversely, examples of summative assessment would include the neurosurgical written board examination, oral board examination, or maintenance of certification tests. In these situations, a person either passes or fails, and the future of the testee is determined by the outcome.
Teaching, Learning, and Assessment in the Operating Room
Supervised, interpersonal mentorships developed through direct apprenticeship, coaching, and training in the operating room are a key component of neurosurgical training.18,37,53 Unfortunately, opportunities to operationalize this type of teaching experience are increasingly difficult because of Accreditation Council for Graduate Medical Education (ACGME) resident duty-hour limitations,41,44,58 financial pressures to be efficient in the operating room,8,43,72 and the understandable emphasis on high-quality care with low complication rates and short hospital stays. It is therefore imperative that surgical education also evolve to reflect an equal degree of efficiency and that effective methods for teaching skills outside the operative setting be employed.18 A complete review of the teaching methods and assessment tools used in medical education is outside the scope of this paper but can be found elsewhere.1,87 Below is a summary of what has been or could be envisioned for neurosurgical training.
Operative Note–Stimulated Recall
Many primary care specialties use a teaching technique called “chart-stimulated recall,” which is a hybrid teaching and assessment format that combines chart review and oral communication based on a documented patient encounter.68 For example, a trainee independently interviews and examines a new patient. They do a preliminary write-up of the encounter and then present this to their instructor/evaluator. The learning and assessment take place during this interaction when the written patient note is critically evaluated. In this scenario, using the learner’s own clinical chart notes adds value, authenticity, and a realistic context for this learning experience.13 Questions about the learner’s clinical decision-making based on the patient chart can be used for education and assessment. A similar approach could be taken with the resident-dictated operative note shortly after the operative experience. The attending surgeon could use this report to help the learner recall and identify areas of emphasis for future skill or cognitive improvement for a given procedure. This would provide a low-risk, safe environment for discussion of resident performance and opportunities for improvement. A slight variation of this technique would be the use of resident operative portfolios for assessing operative procedure outcome. This is done to some extent with case logs and biannual operative data review with residency program directors. A more granular use of detailed operative portfolios could involve discussion of specific operative procedures for learning and assessment purposes. One potential way this could be accomplished would be to compare operative reports from a number of similar procedures to determine whether there are variations or progression of technique over time. Critical evaluation of the operative report from a ventriculoperitoneal shunt procedure done early in training compared with the report from the 50th performance of the same procedure could be instructive and provide a dialog to identify lessons learned and patterns of how competency in a given procedure is obtained.
Operative Mini-Clinical Evaluation Exercise
Another popular teaching tool in medical education is the clinical evaluation exercise (CEX) or the more feasible, shortened mini-CEX.59 The mini-CEX requires an attending physician to observe a resident encounter with a patient and fill out a rating form for that encounter.52,59 This concept was first extended to the operating room in the form of the operative performance rating system (OPRS) in a study by Larson et al.52 In this study, the general surgery training program used 2 “sentinel operative procedures” for each postgraduate year graded by an attending surgeon with a 10-item evaluation instrument unique to each procedure. A follow-up of the OPRS by the same group demonstrated that resident operative performance can be rated by surgical faculty in a consistent manner; however, many barriers to truly accurate and reliable data remain, such as the delay to completion of ratings and controlling for sources of potential rater bias.49 With adaptation and understanding of these limitations, the mini-CEX/OPRS could be used in neurosurgical training programs to enhance operative educational experiences.
Objective Structured Assessment of Technical Skills
Direct supervision and feedback is the traditional method of teaching and assessment in the operating room, but there are concerns that it can be too subjective and lack validity.20,87 There have been many attempts to add objective measures to this process using various instructional and measurement tools, predominantly in general surgery.34,52,54 The Objective Structured Assessment of Technical Skills (OSATS), Objective Structured Clinical Examination (OSCE), Global Rating Scale of Operative Performance, McGill Inanimate System for Laparoscopic Skills (MISTELS), and Imperial College Surgical Assessment Device (ICSAD) are examples of such assessment tools. The method that seems to have become the most feasible and refined is the OSATS, a 5-point rating scale that accounts for various measures of knowledge and skill.23,54,57,70,71,87,91
The use of OSATS in neurosurgery was piloted during a 6-month pediatric neurosurgery rotation. Operative performance was evaluated using the Global Rating Scale of Operative Performance54,70—a 5-point Likert-type scale with 7 categories: respect for tissue, time and motion, instrument handling, knowledge of instruments, flow of operation, use of assistants, and knowledge of specific procedure. This grading scale has been validated in bench models24,33,69,70,90 and an operating room setting.23,57 After a procedure, the residents evaluated their performance using a written form (self-assessment), and their performance was also evaluated by the attending surgeon using a separate copy of the same form. Data were then stratified by faculty versus resident, postgraduate year level, and difficulty of procedure.37 Although there were questions regarding the construct validity and reliability of this method,48 the study did demonstrate the feasibility of using OSATS for neurosurgical resident evaluation.
Video OSATS
A further variation of direct observation and feedback for teaching and assessment includes intraoperative video evaluations. Sarkiss et al.74 described a study in which neurosurgery residents were fitted with a head-mounted video camera while performing craniotomies under attending supervision. Videos were anonymized and scored by attending surgeons, residents, and nonsurgeons using a grading scale similar to those described above to assess the following skills: incision, efficiency of instrument use, cauterization, tissue handling, drilling/craniotomy, confidence, and training level. Not surprisingly, the authors found a strong correlation between skills score and training level. There was high inter-rater reliability regardless of scorer background, suggesting that this method provided a greater objectivity in distinguishing trainees, more accurate tracking of progress in acquiring new skills, and a better way to demonstrate effectiveness of different teaching methods. Even more importantly, this technique may augment coaching as a method for neurosurgical training and education, similar to how a football coach would analyze film from a previous game.74 Similar work in spinal surgery cases has been shown to minimize errors, improve quality, and augment surgical efficiency compared with lectures or no video coaching.40 The obvious limitations to this approach is the need for recording equipment and the significant time commitment required for both review and rating of a given procedure. The fact that the scoring and feedback is not done in “real time” could also be viewed as either a disadvantage or advantage, depending on one’s perspective of this type of educational interaction.
Neurosurgical Simulation
Simulation has been deemed to play an important role in neurosurgical education and is reviewed elsewhere.18,29,30,38,39,50 Successful simulation programs have been reported to aid neurosurgical training in ventriculostomy placement, brain tumor resection, pedicle screw placement, endoscopic transsphenoidal surgery, and otologic skull base procedures.4,5,7,31,42,73,75,76,83 Many of these programs have included virtual reality simulators.2,4,15,16,35,45,66,84 These simulation programs can be improved significantly by game-based simulation training including scoring and timed challenges.17,47,65
Teaching, Learning, and Assessment Outside of the Operating Room
Neurosurgical Curriculum
A critical and first step in neurosurgical education is defining a core set of knowledge deemed imperative for the safe practice of neurosurgery. As described previously, this is the scaffolding upon which our residents’ education is built. Contemporary neurosurgical training is a competency-based training anchored to core competency levels known as “milestones.”17,81 Beginning in July 2013, the Neurological Surgery Milestones Project was launched as a joint initiative between the ACGME and the American Board of Neurological Surgery (ABNS).77 These milestones are intended to be metrics by which residency programs can evaluate resident performance on a semiannual basis.37 The milestones do not provide a standardized approach to evaluation of operative skills, nor is there any attempt to formulate how the learner acquires this knowledge base.37,77 In response to these needs, the American Association of Neurological Surgeons (AANS) and the Congress of Neurological Surgeons (CNS) developed a Matrix Curriculum that defines the objective knowledge or technical skills for a given competency to determine the best teaching methods and assessment tools and sets a standard expectation for proficiency.12 Other specialties, such as orthopedic trauma surgery, have made similar efforts. Importantly, they have noted that to support such a competency-based teaching program, residency programs need to improve the quality of assessment and improve on current formative and summative feedback to their trainees.25,60,61 These observations have led to the development of a modular, competency-based curriculum that has been implemented in orthopedic surgery.3,25 This curriculum is structured in a manner similar to our Matrix Curriculum and could be used in a similar competency-based manner once implemented.
In July 2010, ACGME-accredited neurosurgical training programs assumed direct responsibility for teaching fundamental skills and knowledge related to professionalism, supervision, communication, safety, and basic intensive care and operative procedures to 1st-year residents entering training. The Society of Neurosurgical Surgeons, comprising residency program directors, department chairs, and other educational leaders, created a uniform national curriculum for boot-camp courses taught at 6 regional centers.78–80 In these courses, which are designed for entry-level residents, topics of professionalism, communication, hierarchical supervision, and basic procedural skills relevant to early resident training are taught using didactic lectures and hands-on laboratory sessions. The boot-camp experience has been a productive forum for fostering and assessing professionalism and communication skills in neurosurgical education.26 With the recent emphasis on value-based outcomes, attempts have been made to develop a similar national didactic and hands-on outcomes and quality improvement curriculum for residents.63
Teaching in the Clinic, Intensive Care Unit, and Neurosurgical Ward
The neurosurgical environment is clinically demanding, with a high volume of sick patients and frequent emergencies; these can pose various barriers to learning for training physicians and midlevel providers.64 Although many neurosurgical educators believe that the clinic, intensive care unit, or ward round is a good place for providing opportunistic and relevant teaching, they share concerns about the increase in time necessary to make this possible. Consequently, they often do not take time during rounds or in clinic to teach, while residents feel too pressured by the workload to ask for teaching or pose questions related to the clinical work.64,67 It has been suggested that teaching can be improved in these settings by structuring the rounds for education. This could be achieved by planning the order of patients to be seen so that both clinical issues and teaching points are sufficiently addressed before time pressure becomes a problem.89 The teaching should be in a structured manner, and learning points should be made explicit.82 Before starting rounds, it may be beneficial to ask the residents their learning objectives, for example, what they want to focus on or gain from the ward round.64 Evidence suggests that peer-led teaching and learning is highly effective.11,64 Thus, at least some aspects of induction and training of new and lower-level residents and midlevel providers may be completed by senior residents rather than senior clinicians.63
Assessment in the Clinic, Intensive Care Unit, and Neurosurgical Wards
Resident evaluation traditionally involves global assessments including clinical performance, professional behavior, technical skill, and number of procedures performed.52 Much of that comes from faculty evaluation and feedback. Feedback is more than just information; rather, it is information whose explicit purpose is to promote improvement in learner performance.21,68 Feedback provides the learner with insight into the consequences of specific actions with the opportunity to change any differences between intended and actual outcomes. There are 2 key influences on how and why feedback becomes meaningful. The first is the individual learner’s perception of and response to feedback. The second is the learning culture within which feedback is exchanged.68 For maximum benefit, the feedback should be specific, descriptive, formative (on-going), and limited to the aspect being examined. The learning culture is best when feedback is provided at a quiet time of the day, in a private setting, and when there is time to freely discuss the feedback.
Conclusions
In the modern era, the neurosurgeon is often called upon to be a leader of the healthcare team. Neurosurgeons are responsible for the education of nurses, midlevel providers, medical students, residents, and junior partners. Broad understanding of adult education and assessment principles among neurosurgery educators has the potential to improve the future of neurosurgical practice. The neurosurgical community must find ways to improve our educational efforts. Beyond the topics discussed here, this could be achieved by expanding opportunities for faculty development in education at institutions training medical students, residents, and fellows. Maintenance of Certification (MOC) credit for education-based quality improvement projects and/or formal courses in educational principles at our national neurosurgical meetings could also help in the promotion of the neurosurgeon as educator.
Acknowledgments
Dr. Jensen is currently enrolled as a student in the Masters of Health Education Program at the University of Illinois, Chicago. This work is based on course work in that program. We would like to thank faculty members Ilene Harris, PhD, Ara Tekian, PhD, MHPE, and Yoon Soo Park, PhD, for their excellent instruction on the topics presented in this paper. We would also like to thank Dr. Danielle Roussel for her critical evaluation of this paper. We thank Kristin Kraus, MSc, for her editorial assistance in the preparation of this paper.
Disclosures
Dr. Jensen reports a consultant relationship with Medtronic.
References
- 1↑
Ahmed K, Miskovic D, Darzi A, Athanasiou T, Hanna GB: Observational tools for assessment of procedural skills: a systematic review. Am J Surg 202:469–480, 480.e1–480.e6, 2011
- 2↑
Alaraj A, Charbel FT, Birk D, Tobin M, Luciano C, Banerjee PP, et al.: Role of cranial and spinal virtual and augmented reality simulation using immersive touch modules in neurosurgical training. Neurosurgery 72 (Suppl 1):115–123, 2013 (Erratum in Neurosurgery 73:E913, 2013)
- 3↑
Alman BA, Ferguson P, Kraemer W, Nousiainen MT, Reznick RK: Competency-based education: a new model for teaching orthopaedics. Instr Course Lect 62:565–569, 2013
- 4↑
AlZhrani G, Alotaibi F, Azarnoush H, Winkler-Schwartz A, Sabbagh A, Bajunaid K, et al.: Proficiency performance benchmarks for removal of simulated brain tumors using a virtual reality simulator NeuroTouch. J Surg Educ 72:685–696, 2015
- 5↑
Azarnoush H, Alzhrani G, Winkler-Schwartz A, Alotaibi F, Gelinas-Phaneuf N, Pazos V, et al.: Neurosurgical virtual reality simulation metrics to assess psychomotor skills during brain tumor resection. Int J CARS 10:603–618, 2015
- 6↑
Bhatti NI, Cummings CW: Competency in surgical residency training: defining and raising the bar. Acad Med 82:569–573, 2007
- 7↑
Bova FJ, Rajon DA, Friedman WA, Murad GJ, Hoh DJ, Jacob RP, et al.: Mixed-reality simulation for neurosurgical procedures. Neurosurgery 73 (Suppl 1):138–145, 2013
- 8↑
Bridges M, Diamond DL: The financial impact of teaching surgical residents in the operating room. Am J Surg 177:28–32, 1999
- 10↑
Brown JS, Collins A, Duguid P: Situated cognition and the culture of learning. Educ Res 18:32–42, 1989
- 11↑
Buckley S, Zamora J: Effects of participation in a cross year peer tutoring programme in clinical examination skills on volunteer tutors’ skills and attitudes towards teachers and teaching. BMC Med Educ 7:20, 2007
- 12↑
Burchiel KJ: Neurosurgical education: a new paradigm for curriculum, core, and subspecialty training. Neurosurgery 63 (Suppl 1):88–90, 2016
- 13↑
Carraccio C, Burke AE: Beyond competencies and milestones: adding meaning through context. J Grad Med Educ 2:419–422, 2010
- 14↑
Carter FJ, Schijven MP, Aggarwal R, Grantcharov T, Francis NK, Hanna GB, et al.: Consensus guidelines for validation of virtual reality surgical simulators. Surg Endosc 19:1523–1532, 2005
- 15↑
Chitale R, Ghobrial GM, Lobel D, Harrop J: Simulated lumbar minimally invasive surgery educational model with didactic and technical components. Neurosurgery 73 (Suppl 1):107–110, 2013
- 16↑
Chugh AJ, Pace JR, Singer J, Tatsuoka C, Hoffer A, Selman WR, et al.: Use of a surgical rehearsal platform and improvement in aneurysm clipping measures: results of a prospective, randomized trial. J Neurosurg 126:838–844, 2017
- 17↑
Clarke DB, Kureshi N, Hong M, Sadeghi M, D’Arcy RC: Simulation-based training for burr hole surgery instrument recognition. BMC Med Educ 16:153, 2016
- 18↑
Cobb MI, Taekman JM, Zomorodi AR, Gonzalez LF, Turner DA: Simulation in neurosurgery—a brief review and commentary. World Neurosurg 89:583–586, 2016
- 19↑
Collins A, Brown JS, Newman S: Cognitive apprenticeship: teaching the craft of reading, writing and mathematics, in Resnick LB (ed): Knowing, Learning, and Instruction: Essays in Honor of Robert Glaser. Hillsdale, NJ: Erlbaum, 1989
- 20↑
Darzi A, Smith S, Taffinder N: Assessing operative skill. Needs to become more objective. BMJ 318:887–888, 1999
- 21↑
Donnon T, Al Ansari A, Al Alawi S, Violato C: The reliability, validity, and feasibility of multisource feedback physician assessment: a systematic review. Acad Med 89:511–516, 2014
- 22↑
Downing SM, Haladyna TM: Validity and its threats, in Downing SM, Yudkowsky R (eds): Assessment in Health Professions Education. New York: Routledge, 2009, pp 21–56
- 23↑
Doyle JD, Webber EM, Sidhu RS: A universal global rating scale for the evaluation of technical skills in the operating room. Am J Surg 193:551–555, 2007
- 24↑
Faulkner H, Regehr G, Martin J, Reznick R: Validation of an objective structured assessment of technical skill for surgical residents. Acad Med 71:1363–1365, 1996
- 25↑
Ferguson PC, Kraemer W, Nousiainen M, Safir O, Sonnadara R, Alman B, et al.: Three-year experience with an innovative, modular competency-based curriculum for orthopaedic training. J Bone Joint Surg Am 95:e166, 2013
- 26↑
Fontes RB, Selden NR, Byrne RW: Fostering and assessing professionalism and communication skills in neurosurgical education. J Surg Educ 71:e83–e89, 2014
- 27↑
Frank JR, Snell LS, Cate OT, Holmboe ES, Carraccio C, Swing SR, et al.: Competency-based medical education: theory to practice. Med Teach 32:638–645, 2010
- 28↑
Gallagher AG, Ritter EM, Satava RM: Fundamental principles of validation, and reliability: rigorous science for the assessment of surgical education and training. Surg Endosc 17:1525–1529, 2003
- 29↑
Ganju A, Aoun SG, Daou MR, El Ahmadieh TY, Chang A, Wang L, et al.: The role of simulation in neurosurgical education: a survey of 99 United States neurosurgery program directors. World Neurosurg 80:e1–e8, 2013
- 30↑
Gasco J, Holbrook TJ, Patel A, Smith A, Paulson D, Muns A, et al.: Neurosurgery simulation in residency training: feasibility, cost, and educational benefit. Neurosurgery 73 (Suppl 1):39–45, 2013
- 31↑
Gasco J, Patel A, Ortega-Barnett J, Branch D, Desai S, Kuo YF, et al.: Virtual reality spine surgery simulation: an empirical study of its usefulness. Neurol Res 36:968–973, 2014
- 32↑
Gélinas-Phaneuf N, Del Maestro RF: Surgical expertise in neurosurgery: integrating theory into practice. Neurosurgery 73 (Suppl 1):30–38, 2013
- 33↑
Glarner CE, McDonald RJ, Smith AB, Leverson GE, Peyre S, Pugh CM, et al.: Utilizing a novel tool for the comprehensive assessment of resident operative performance. J Surg Educ 70:813–820, 2013
- 34↑
Gofton WT, Dudek NL, Wood TJ, Balaa F, Hamstra SJ: The Ottawa Surgical Competency Operating Room Evaluation (O-SCORE): a tool to assess surgical competence. Acad Med 87:1401–1407, 2012
- 35↑
Grantcharov TP, Rosenberg J, Pahle E, Funch-Jensen P: Virtual reality computer simulation. Surg Endosc 15:242–244, 2001
- 37↑
Hadley C, Lam SK, Briceño V, Luerssen TG, Jea A: Use of a formal assessment instrument for evaluation of resident operative skills in pediatric neurosurgery. J Neurosurg Pediatr 16:497–504, 2015
- 38↑
Harrop J, Lobel DA, Bendok B, Sharan A, Rezai AR: Developing a neurosurgical simulation-based educational curriculum: an overview. Neurosurgery 73 (Suppl 1):25–29, 2013
- 39↑
Harrop J, Rezai AR, Hoh DJ, Ghobrial GM, Sharan A: Neurosurgical training with a novel cervical spine simulator: posterior foraminotomy and laminectomy. Neurosurgery 73 (Suppl 1):94–99, 2013
- 40↑
Heiland DH, Petridis AK, Maslehaty H, Thissen J, Kinzel A, Scholz M, et al.: Efficacy of a new video-based training model in spinal surgery. Surg Neurol Int 5:1, 2014
- 41↑
Hoh BL, Neal DW, Kleinhenz DT, Hoh DJ, Mocco J, Barker FG II: Higher complications and no improvement in mortality in the ACGME resident duty-hour restriction era: an analysis of more than 107,000 neurosurgical trauma patients in the Nationwide Inpatient Sample database. Neurosurgery 70:1369–1382, 2012
- 42↑
Hooten KG, Lister JR, Lombard G, Lizdas DE, Lampotang S, Rajon DA, et al.: Mixed reality ventriculostomy simulation: experience in neurosurgical residency. Neurosurgery 10 (Suppl 4):576–581, 2014
- 43↑
Hwang CS, Pagano CR, Wichterman KA, Dunnington GL, Alfrey EJ: Resident versus no resident: a single institutional study on operative complications, mortality, and cost. Surgery 144:339–344, 2008
- 44↑
Jagannathan J, Vates GE, Pouratian N, Sheehan JP, Patrie J, Grady MS, et al.: Impact of the Accreditation Council for Graduate Medical Education work-hour regulations on neurosurgical resident education and productivity. J Neurosurg 110:820–827, 2009
- 45↑
Jordan JA, Gallagher AG, McGuigan J, McGlade K, McClure N: A comparison between randomly alternating imaging, normal laparoscopic imaging, and virtual reality training in laparoscopic psychomotor skill acquisition. Am J Surg 180:208–211, 2000
- 47↑
Kerfoot BP, Kissane N: The use of gamification to boost residents’ engagement in simulation training. JAMA Surg 149:1208–1209, 2014
- 49↑
Kim MJ, Williams RG, Boehler ML, Ketchum JK, Dunnington GL: Refining the evaluation of operating room performance. J Surg Educ 66:352–356, 2009
- 50↑
Kirkman MA, Ahmed M, Albert AF, Wilson MH, Nandi D, Sevdalis N: The use of simulation in neurosurgical education and training. A systematic review. J Neurosurg 121:228–246, 2014
- 52↑
Larson JL, Williams RG, Ketchum J, Boehler ML, Dunnington GL: Feasibility, reliability and validity of an operative performance rating system for evaluating surgery residents. Surgery 138:640–649, 2005
- 53↑
Long DM: Competency-based residency training: the next advance in graduate medical education. Acad Med 75:1178–1183, 2000
- 54↑
Martin JA, Regehr G, Reznick R, MacRae H, Murnaghan J, Hutchison C, et al.: Objective structured assessment of technical skill (OSATS) for surgical residents. Br J Surg 84:273–278, 1997
- 55↑
Messick S: Standards of validity and the validity of standards in performance assessment. Educ Meas 14:5–8, 1995
- 56↑
Miller GE: The assessment of clinical skills/competence/performance. Acad Med 65 (9 Suppl):S63–S67, 1990
- 57↑
Niitsu H, Hirabayashi N, Yoshimitsu M, Mimura T, Taomoto J, Sugiyama Y, et al.: Using the Objective Structured Assessment of Technical Skills (OSATS) global rating scale to evaluate the skills of surgical trainees in the operating room. Surg Today 43:271–275, 2013
- 58↑
Norby K, Siddiq F, Adil MM, Haines SJ: The effect of duty hour regulations on outcomes of neurological surgery in training hospitals in the United States: duty hour regulations and patient outcomes. J Neurosurg 121:247–261, 2014
- 59↑
Norcini JJ, Blank LL, Duffy FD, Fortna GS: The mini-CEX: a method for assessing clinical skills. Ann Intern Med 138:476–481
- 60↑
Nousiainen MT, McQueen SA, Ferguson P, Alman B, Kraemer W, Safir O, et al.: Simulation for teaching orthopaedic residents in a competency-based curriculum: do the benefits justify the increased costs? Clin Orthop Relat Res 474:935–944, 2016
- 61↑
Nousiainen MT, McQueen SA, Hall J, Kraemer W, Ferguson P, Marsh JL, et al.: Resident education in orthopaedic trauma: the future role of competency-based medical education. Bone Joint J 98-B:1320–1325, 2016
- 62↑
Park YS, Zar FA, Norcini JJ, Tekian A: Competency evaluations in the next accreditation system: contributing to guidelines and implications. Teach Learn Med 28:135–145, 2016
- 63↑
Parker SL, McGirt MJ, Asher AL, Selden NR: Quality improvement in neurological surgery graduate medical education. Neurosurg Clin N Am 26:231–238, ix, 2015
- 64↑
Phan PN, Patel K, Bhavsar A, Acharya V: Do we need to overcome barriers to learning in the workplace for foundation trainees rotating in neurosurgery in order to improve training satisfaction? Adv Med Educ Pract 7:211–217, 2016
- 65↑
Pitt MB, Borman-Shoap EC, Eppich WJ: Twelve tips for maximizing the effectiveness of game-based learning. Med Teach 37:1013–1017, 2015
- 66↑
Prystowsky JB, Regehr G, Rogers DA, Loan JP, Hiemenz LL, Smith KM: A virtual reality module for intravenous catheter placement. Am J Surg 177:171–175, 1999
- 67↑
Qureshi NS, Swamy NN: Postgraduate trainees’ assessment of the educational value of ward rounds in obstetrics and gynaecology. J Obstet Gynaecol 28:671–675, 2008
- 68↑
Reddy ST, Endo J, Gupta S, Tekian A, Park YS: A case for caution: chart-stimulated recall. J Grad Med Educ 7:531–535, 2015
- 69↑
Regehr G, MacRae H, Reznick RK, Szalay D: Comparing the psychometric properties of checklists and global rating scales for assessing performance on an OSCE-format examination. Acad Med 73:993–997, 1998
- 70↑
Reznick R, Regehr G, MacRae H, Martin J, McCulloch W: Testing technical skill via an innovative “bench station” examination. Am J Surg 173:226–230, 1997
- 71↑
Reznick RK, MacRae H: Teaching surgical skills—changes in the wind. N Engl J Med 355:2664–2669, 2006
- 72↑
Riffaud L, Neumuth T, Morandi X, Trantakis C, Meixensberger J, Burgert O, et al.: Recording of surgical processes: a study comparing senior and junior neurosurgeons during lumbar disc herniation surgery. Neurosurgery 67 (2 Suppl Operative):325–332, 2010
- 73↑
Rosseau G, Bailes J, del Maestro R, Cabral A, Choudhury N, Comas O, et al.: The development of a virtual simulator for training neurosurgeons to perform and perfect endoscopic endonasal transsphenoidal surgery. Neurosurgery 73 (Suppl 1):85–93, 2013
- 74↑
Sarkiss CA, Philemond S, Lee J, Sobotka S, Holloway TD, Moore MM, et al.: Neurosurgical skills assessment: measuring technical proficiency in neurosurgery residents through intraoperative video evaluations. World Neurosurg 89:1–8, 2016
- 75↑
Schirmer CM, Elder JB, Roitberg B, Lobel DA: Virtual reality-based simulation training for ventriculostomy: an evidence-based approach. Neurosurgery 73 (Suppl 1):66–73, 2013
- 76↑
Schirmer CM, Mocco J, Elder JB: Evolving virtual reality simulation in neurosurgery. Neurosurgery 73 (Suppl 1):127–137, 2013
- 77↑
Selden NR, Abosch A, Byrne RW, Harbaugh RE, Krauss WE, Mapstone TB, et al.: Neurological surgery milestones. J Grad Med Educ 5 (1 Suppl 1):24–35, 2013
- 78
Selden NR, Anderson VC, McCartney S, Origitano TC, Burchiel KJ, Barbaro NM: Society of Neurological Surgeons boot camp courses: knowledge retention and relevance of hands-on learning after 6 months of postgraduate year 1 training. J Neurosurg 119:796–802, 2013
- 79
Selden NR, Barbaro N, Origitano TC, Burchiel KJ: Fundamental skills for entering neurosurgery residents: report of a Pacific region “boot camp” pilot course, 2009. Neurosurgery 68:759–764, 2011
- 80
Selden NR, Origitano TC, Burchiel KJ, Getch CC, Anderson VC, McCartney S, et al.: A national fundamentals curriculum for neurosurgery PGY1 residents: the 2010 Society of Neurological Surgeons boot camp courses. Neurosurgery 70:971–981, 2012
- 81↑
Sonnadara RR, Mui C, McQueen S, Mironova P, Nousiainen M, Safir O, et al.: Reflections on competency-based education and training for surgical residents. J Surg Educ 71:151–158, 2014
- 82↑
Stanley P: Structuring ward rounds for learning: can opportunities be created? Med Educ 32:239–243, 1998
- 83↑
Stredney D, Rezai AR, Prevedello DM, Elder JB, Kerwin T, Hittle B, et al.: Translating the simulation of procedural drilling techniques for interactive neurosurgical training. Neurosurgery 73 (Suppl 1):74–80, 2013
- 84↑
Sundar SJ, Healy AT, Kshettry VR, Mroz TE, Schlenk R, Benzel EC: A pilot study of the utility of a laboratory-based spinal fixation training program for neurosurgical residents. J Neurosurg Spine 24:850–856, 2016
- 85↑
Taylor DC, Hamdy H: Adult learning theories: implications for learning and teaching in medical education: AMEE Guide No. 83. Med Teach 35:e1561–e1572, 2013
- 86↑
Van Der Vleuten CP: The assessment of professional competence: developments, research and practical implications. Adv Health Sci Educ Theory Pract 1:41–67, 1996
- 87↑
van Hove PD, Tuijthof GJ, Verdaasdonk EG, Stassen LP, Dankelman J: Objective assessment of technical surgical skills. Br J Surg 97:972–987, 2010
- 88↑
Vygotsky LS: Interaction between learning and development, in Gauvin M, Cole M (eds): Mind and Society. Readings on the Development of Children. Cambridge, MA: Harvard University Press, 1978, pp 79–91
- 89↑
Walton JM, Steinert Y: Patterns of interaction during rounds: implications for work-based learning. Med Educ 44:550–558, 2010
- 90↑
Winckel CP, Reznick RK, Cohen R, Taylor B: Reliability and construct validity of a structured technical skills assessment form. Am J Surg 167:423–427, 1994
- 91↑
Zevin B, Bonrath EM, Aggarwal R, Dedy NJ, Ahmed N, Grantcharov TP, et al.: Development, feasibility, validity, and reliability of a scale for objective assessment of operative performance in laparoscopic gastric bypass surgery. J Am Coll Surg 216:955–965, 965.e1–965.e8, 1029–1031, 1033, 2013
