Assessing residents' operative skills for external ventricular drain placement and shunt surgery in pediatric neurosurgery

Full access

OBJECTIVE

The authors previously demonstrated the use of a validated Objective Structured Assessment of Technical Skills (OSATS) tool for evaluating residents' operative skills in pediatric neurosurgery. However, no benchmarks have been established for specific pediatric procedures despite an increased need for meaningful assessments that can either allow for early intervention for underperforming trainees or allow for proficient residents to progress to conducting operations independently with more passive supervision. This validated methodology and tool for assessment of operative skills for common pediatric neurosurgical procedures—external ventricular drain (EVD) placement and shunt surgery— was applied to establish its procedure-based feasibility and reliability, and to document the effect of repetition on achieving surgical skill proficiency in pediatric EVD placement and shunt surgery.

METHODS

A procedure-based technical skills assessment for EVD placements and shunt surgeries in pediatric neurosurgery was established through the use of task analysis. The authors enrolled all residents from 3 training programs (Baylor College of Medicine, Houston Methodist Hospital, and University of Texas–Medical Branch) who rotated through pediatric neurosurgery at Texas Children's Hospital over a 26-month period. For each EVD placement or shunt procedure performed with a resident, the faculty and resident (for self-assessment) completed an evaluation form (OSATS) based on a 5-point Likert scale with 7 categories. Data forms were then grouped according to faculty versus resident (self) assessment, length of pediatric neurosurgery rotation, postgraduate year level, and date of evaluation (“beginning of rotation,” within 1 month of start date; “end of rotation,” within 1 month of completion date; or “middle of rotation”). Descriptive statistical analyses were performed with the commercially available SPSS statistical software package. A p value < 0.05 was considered statistically significant.

RESULTS

Five attending evaluators (including 2 fellows who acted as attending surgeons) completed 260 evaluations. Twenty house staff completed 269 evaluations for self-assessment. Evaluations were completed in 562 EVD and shunt procedures before the surgeons left the operating room. There were statistically significant differences (p < 0.05) between overall attending (mean 4.3) and junior resident (self; mean 3.6) assessments, and between overall attending (mean 4.8) and senior resident (self; mean 4.6) assessment scores on general performance and technical skills. The learning curves produced for the residents demonstrate a stereotypical U- or V-shaped curve for acquiring skills, with a significant improvement in overall scores at the end of the rotation compared with the beginning. The improvement for junior residents (Δ score = 0.5; p = 0.002) was larger than for senior residents (Δ score = 0.2; p = 0.018).

CONCLUSIONS

The OSATS is an effective assessment tool as part of a comprehensive evaluation of neurosurgery residents' performance for specific pediatric procedures. The authors observed a U-shaped learning curve, contradicting the idea that developing one's surgical technique and learning a procedure represents a monotonic, cumulative process of repetitions and improvement.

ABBREVIATIONSEVD = external ventricular drain; OR = operating room; OSATS = Objective Structured Assessment of Technical Skills; PGY = postgraduate year.

OBJECTIVE

The authors previously demonstrated the use of a validated Objective Structured Assessment of Technical Skills (OSATS) tool for evaluating residents' operative skills in pediatric neurosurgery. However, no benchmarks have been established for specific pediatric procedures despite an increased need for meaningful assessments that can either allow for early intervention for underperforming trainees or allow for proficient residents to progress to conducting operations independently with more passive supervision. This validated methodology and tool for assessment of operative skills for common pediatric neurosurgical procedures—external ventricular drain (EVD) placement and shunt surgery— was applied to establish its procedure-based feasibility and reliability, and to document the effect of repetition on achieving surgical skill proficiency in pediatric EVD placement and shunt surgery.

METHODS

A procedure-based technical skills assessment for EVD placements and shunt surgeries in pediatric neurosurgery was established through the use of task analysis. The authors enrolled all residents from 3 training programs (Baylor College of Medicine, Houston Methodist Hospital, and University of Texas–Medical Branch) who rotated through pediatric neurosurgery at Texas Children's Hospital over a 26-month period. For each EVD placement or shunt procedure performed with a resident, the faculty and resident (for self-assessment) completed an evaluation form (OSATS) based on a 5-point Likert scale with 7 categories. Data forms were then grouped according to faculty versus resident (self) assessment, length of pediatric neurosurgery rotation, postgraduate year level, and date of evaluation (“beginning of rotation,” within 1 month of start date; “end of rotation,” within 1 month of completion date; or “middle of rotation”). Descriptive statistical analyses were performed with the commercially available SPSS statistical software package. A p value < 0.05 was considered statistically significant.

RESULTS

Five attending evaluators (including 2 fellows who acted as attending surgeons) completed 260 evaluations. Twenty house staff completed 269 evaluations for self-assessment. Evaluations were completed in 562 EVD and shunt procedures before the surgeons left the operating room. There were statistically significant differences (p < 0.05) between overall attending (mean 4.3) and junior resident (self; mean 3.6) assessments, and between overall attending (mean 4.8) and senior resident (self; mean 4.6) assessment scores on general performance and technical skills. The learning curves produced for the residents demonstrate a stereotypical U- or V-shaped curve for acquiring skills, with a significant improvement in overall scores at the end of the rotation compared with the beginning. The improvement for junior residents (Δ score = 0.5; p = 0.002) was larger than for senior residents (Δ score = 0.2; p = 0.018).

CONCLUSIONS

The OSATS is an effective assessment tool as part of a comprehensive evaluation of neurosurgery residents' performance for specific pediatric procedures. The authors observed a U-shaped learning curve, contradicting the idea that developing one's surgical technique and learning a procedure represents a monotonic, cumulative process of repetitions and improvement.

Development of residents' operative skills is a key component of neurosurgical training. Currently, little is understood about residents' performance in the operating room (OR), how this develops over the course of training, and how to quantify operative skills.1,10,24,28 We previously published a first foray into this dilemma of how to evaluate residents' operative skills in pediatric neurosurgery11 by using the Objective Structured Assessment of Technical Skills (OSATS) instrument.9 This tool was selected because it is well known, widely used, and has been studied in the laboratory and validated for use in the clinical setting.19,20,35,38

Our aims in this study were to implement a tool that comprehensively assesses neurosurgical residents' technical performance in the OR for external ventricular drain (EVD) placement and shunt surgery—common pediatric neurosurgical procedures—and to determine the feasibility and reliability of such a tool as it relates to self-assessment, postgraduate year (PGY) level, and motor learning, skill acquisition, and repetition through a pediatric neurosurgical resident's rotation.

Methods

Cases and Research Subjects

We included all pediatric shunt procedures for CSF diversion performed during a 26-month period from January 1, 2014, to February 28, 2016 (Table 1). At Texas Children's Hospital, these procedures are posted under 1 of 5 attending neurosurgeons with board certification in pediatric neurosurgery, or a fellow who holds a faculty appointment as a clinical instructor and has independent OR privileges. Most cases are performed with a junior (PGY 1–3) or senior (PGY 4–6) neurosurgical resident during his or her 3- to 6-month rotation. Residents who rotated for less than 3 months on the pediatric neurosurgery service were excluded. Faculty and residents completed the evaluation immediately after the operation. Evaluations completed after leaving the OR were not considered valid for the purposes of this study, and consequently were excluded.

TABLE 1.

Number of EVD placements and shunt procedures in our study

ProcedureNo. of ProceduresNo. of Assessments
EVD placement12744
New VP shunt placement146150
New VA shunt placement1225
New subgaleal shunts37
Shunt revision274303
Overall562529

VA = ventriculoatrial; VP = ventriculoperitoneal.

Our hospital's institutional review board granted us an educational exemption for this project. All neurosurgical residents were notified of the study and allowed to decline participation.

Assessment Tool

Our evaluation form is based on the OSATS global rating scale (Fig. 1). The validated Global Rating Scale of Operative Performance includes a 5-point Likert scale with 7 categories analyzed: respect for tissue, time and motion, instrument handling, knowledge of instruments, flow of operation, use of assistants, and knowledge of specific procedure. Faculty evaluators agreed on operative performance interpretation and evaluation in the context of this form.

FIG. 1.
FIG. 1.

Our evaluation form or data collection sheet for evaluation of a resident's operative skills.

We averaged assessments in the 1st month of the rotation and labeled them as “first.” Likewise, we averaged assessments in the final month of the rotation and labeled them as “last.”

Statistical Analysis

All assessments that were included in the study were pooled and analyzed. There were some surgeries that only had a resident's or attending's assessment, but not both. We evaluated procedure-based construct validity by measuring the difference in scores between junior and senior residents' performance overall for EVD placements and shunt surgeries, and after faculty assessment and self-assessment. We followed their scores over the course of their 3- to 6-month pediatric neurosurgery rotation to evaluate the effect of repetition and motor learning in the acquisition of surgical skills proficiency. All statistical analyses were performed using the SPSS program, version 20.0 (IBM, Inc.).

Results

Feasibility of the Assessment

Over the course of 26 months, 529 distributed assessments were returned for a 55.75% return rate. Each of the 5 faculty-level surgeons completed between 18 and 64 evaluations; 172 evaluations were completed for the junior residents, and 97 evaluations were completed for the senior residents. Each of the 20 neurosurgery residents who rotated through pediatric neurosurgery completed between 0 and 22 evaluations after EVD placement and shunt surgery.

Construct Validity

Overall, for EVD placements and shunt surgeries, there was a statistically significant difference between junior and senior residents' scores at the beginning of the rotation for both attending evaluations (p = 0.001) and self-assessments (p = 0.002) (Table 2). All domains were statistically significant between junior and senior residents when evaluated by the attending. For the self-assessment evaluation there were no significant differences in the following domains: flow of operation (p = 0.052), and use of assistants (p = 0.108).

TABLE 2.

The mean scores of junior and senior residents overall and for each domain of the OSATS for shunt surgeries in pediatric neurosurgery

DomainAttending Surgeon's EvaluationSelf-AssessmentSelf-Assessment vs Attending Surgeon's Evaluation (p value)
PGY-3PGY-5PGY-3 vs PGY-5 (p value)PGY-3PGY-5PGY-3PGY-5
FirstLastp ValueFirstLastp ValueFirstLastFirstLastFirstLastFirstLastFirstLast
Respect for tissue4.44.50.7224.94.90.2080.0020.3163.43.84.64.90.0010.0010.1840.414
Time and motion3.94.50.2434.64.80.7270.0110.4463.33.84.24.80.005<0.0010.0650.867
Instrument handling3.64.50.0674.44.90.1100.0360.1823.33.84.34.90.0040.0250.1610.414
Knowledge of instruments4.24.50.4344.95.00.2080.0010.1933.64.04.54.90.0010.0220.0730.495
Flow of operation3.84.50.1554.95.00.266<0.0010.1703.43.84.34.80.0520.0010.0400.299
Use of assistants3.94.40.1474.64.90.1400.0010.0733.53.84.24.80.1080.0040.0860.414
Knowledge of specific procedure3.94.50.2064.85.00.1600.0010.2013.43.94.44.80.0160.0060.0350.299
Overall4.04.50.0024.74.90.018<0.001<0.0013.43.84.44.80.001<0.001<0.0010.045

First = 1st month of rotation; Last = last month of rotation.

Boldface type indicates statistical significance.

Interrater reliability between the attending surgeon and the resident for each procedure was low, especially for junior residents (Tau-b Kendall coefficient 0.244, p = 0.433). The scores for each domain from the attending physicians' evaluations differed from junior residents' self-assessments, and this difference maintained statistical significance throughout the rotation. For senior residents, the differences in scores for each domain between the attending evaluation and the self-assessment did not reach significance, except for the domain “flow of operation” at the beginning of the rotation. Residents, particularly the junior residents, consistently rated themselves lower than did the attending evaluators.

Pattern of Acquiring Surgical Skills Proficiency

Over the course of the rotation, a stereotypical U- or V-shaped learning curve was found in the residents' scores, with a dip in the 3rd month of the rotation (Fig. 2). There is a steep portion of the learning curve between the beginning and 2nd month of the resident rotation. There is then an apparent decline in the learning curve between the 2nd and 3rd months of the rotation. Finally, the learning curve seems to recover between the 3rd and 4th months of the rotation. Junior residents had a larger improvement in overall scores between the start (4.0) and end (4.5) of the rotation that reached statistical significance (p = 0.002), compared with the senior residents' overall scores at the start (4.7) and end (4.9) of the rotation (p = 0.018) (Table 2).

FIG. 2.
FIG. 2.

Graphs showing score average rated by the attending evaluators and the residents' self-assessment during the months of rotation. Each category of the OSATS rating scale and the overall score are represented. The numbers on the y axis represent the skill grade. Eval = evaluator; NS = attending neurosurgeon. Figure is available in color online only.

Duration of Rotation

Junior and senior residents who spent a greater portion of time on the pediatric neurosurgery service (≥ 4 months) had higher overall scores compared with those junior residents who rotated for < 4 months (p < 0.001).

Discussion

Even for a straightforward, oft-performed procedure such as placement of a ventriculoperitoneal shunt, there are infinite iterations and combinations of operative steps, depending on the neurosurgeon. Furthermore, the surgery is delicate, with postoperative outcomes highly dependent on exquisite and careful surgical technique.15,22,37 How do you objectively judge intangible metrics such as “time and motion” and “flow of operation”? Currently, there is no gold standard with which to compare the trainee. The assessment and evaluation of operative skill is, to a large degree, subjective. More importantly, we currently have no feedback on how effective we have been as educators in the OR. There is significant interest in the objective assessment of these skills.8

We have previously attempted to quantify residents' operative skills in pediatric neurosurgery by using a validated instrument—the OSATS.11 Despite the comprehensive nature of this tool, we were able to show that it was feasible and easily incorporated into the OR workflow. Until now, however, there has been no procedure-based validation for OSATS in pediatric neurosurgery.

Procedure-based construct validity for EVD placements and shunt surgeries in pediatric neurosurgery was demonstrated by the statistically significant differences between the junior and senior residents overall, and for select domains of the tool. In addition, there was an improvement over the course of the rotation, adding to the validity of the construct. Residents are expected to improve their performance through motor learning of surgical procedures over a subspecialty-focused rotation.10 Motor learning refers to improvement of motor skills through practice, and is composed of motor adaptation and skill acquisition elements.16

U-Shaped Learning

Interestingly, the learning curves produced for the residents (Fig. 1) demonstrate a stereotypical U- or V-shaped curve for acquiring skills. The U-shaped curves seem to contradict the idea that improved performance and learning have a direct relationship to time and the accumulation of experience and knowledge. The regression shown by U-shaped curves has become a challenge for cognitive development theories to explain.5,21

Thus, on the one hand, a U-shaped course of development is initially characterized by correct performances based on a lack of knowledge, followed by incorrect performances as a result of incomplete knowledge. These phases are then replaced by correct performances based on advanced knowledge and experience.29,30

However, U-shaped learning could also be explained by the adoption of novel processing strategies. This adoption may initially result in cognitive overload and in a temporary loss of cognitive processing efficiency. Once novel processing strategies have been completely adopted, they may lead to more efficient cognitive processing.3,36 This theory seems to explain the pattern of quantified operative skills scores in our study. The process of adoption and inculcation of new surgical procedures and techniques by residents takes time, an evolution from rote memorization to understanding surgical concepts. Once the concept is understood and ingrained, however, the result is a more efficient performance of the procedure. We expect to find the same U-shaped pattern of learning for other surgical procedures.

Interestingly, U-shaped learning has been observed by cognitive and developmental psychologists during the development of select physical, artistic, and cognitive skills.5,21 These patterns have been found for perception-related abilities,18,27,36 motor coordination,4 eye-hand coordination,12,33,34 verbal skills,14,32 and even the development of creativity.6 Most of them are skills necessary for the development and learning of a surgical procedure.

During training for complex procedures, the time needed to process the new strategies to attain proficiency remains controversial. Several authors have suggested that plateaus derived from learning curves can be used to determine appropriate training durations.16,26 Our study seems to suggest that resident rotations in pediatric neurosurgery of ≥ 4 months provided statistically significantly more skills proficiency than rotations of lesser duration.

In addition to fulfilling the requirements of the governing bodies of neurosurgical resident education, we plan to use this tool to improve resident feedback. By providing timely and structured feedback to residents, deliberate goal-oriented practice, as described in the acquisition of expertise in other disciplines, may be used to accelerate the learning curve.7,10,17 Use of deliberate practice to achieve expertise involves more than routine practice; it also requires constant feedback to continuously create well-defined performance goals set beyond learners' current abilities so that they can continue to be challenged and avoid arrested development.7,10 While in residency, trainees are challenged with increasing performance goals, although many times their goals are not well defined, or associated with detailed feedback.10,25,31 By providing more timely and structured feedback through the use of assessment tools such as the one described in this study, teaching faculty can help residents move toward mastery.

Clinical Implications

We do not know how the absolute scores for junior and senior residents translate into neurosurgical clinical outcomes, nor do we know the clinical significance of a change in assessment score on the order of 0.2–0.5 between the first and last evaluations for a given resident. To the best of our knowledge, there has only been scant literature addressing this provocative topic.

Birkmeyer et al.2 found that surgeons with the highest skill ratings had fewer complications in bariatric surgery compared with surgeons with the lowest skill ratings. This suggests that efforts to improve surgeon proficiency are an important but understudied aspect of outcomes research.

Similarly, Hogg et al.13 showed that technical scoring of a surgeon's performance independently predicts patient outcomes in pancreatic surgery. The authors in that study modified the global rating scale to create a more procedure-specific assessment tool. Therefore, it would be difficult to compare their results directly with our data.

Limitations of the Study

This study was performed assessing a single procedure type at a single institution. However, it should be noted that the case mix that we used in our analysis was somewhat heterogeneous, requiring similar but not identical skill sets—EVD placement; first-time ventriculoperitoneal, ventriculoatrial, and subgaleal shunt insertions; and shunt revision surgery. Moreover, it is possible that this assessment tool may produce different results if it is used to evaluate residents' operative skills for other procedures or at another institution. The low response rate of 55.75% for distributed evaluations is also a major flaw of our study, with the possibility of introducing significant bias. This study was also limited by the number of residents who rotate through the pediatric neurosurgery service and faculty members on the pediatric neurosurgery staff, and the duration of the resident rotation. It is very likely that motor learning and surgical skill acquisition continues past training and into independent practice, which is not measured by our study. Moreover, the evaluators were the surgeons who operated with the resident, and they were not blinded to residents' identities, which may affect the results. Future directions for research may lead us to a blinded video-based assessment that can probably negate this type of bias.

The so-called Holy Grail of surgical education is represented by the question: does improving how we teach surgeons lead to better technical skill and improved patient outcomes? Unfortunately, answering this question requires much preliminary work to first establish a validated method of assessing operative skills for specific procedures, and then correlating with clinical metrics such as OR time, estimated blood loss, or complication rates. Our work is but a small step toward this penultimate goal. The answer to the Holy Grail is beyond the scope of our present study.

Also, applying the assessment to attending physicians themselves may serve to prove or disprove the U-shaped learning theories. Attending physicians would not be expected to have any significant learning curve for these procedures, given their much larger experience. However, one caveat for the studies that lie ahead is that the OSATS was evaluated and validated for learners rather than for experts.

Conclusions

The OSATS tool has been previously validated for evaluation of general surgery residents' operative performance.23 We recently used this scale for evaluation of neurosurgery residents' performance during the pediatric neurosurgery rotation.11 The present study serves to further research into neurosurgery resident education by validating the OSATS tool for specific and common pediatric neurosurgical procedures—EVD placement and shunt surgery. For the first time, we have quantitatively documented the stereotypical course of acquiring surgical proficiency for EVD placement and shunt surgery. Interestingly, we observed a U-shaped learning curve, contradicting the idea that developing a performance and learning a surgical procedure represent a monotonic, cumulative process of repetitions and improvement. Although further work is warranted to confirm the validity of this tool as a standardized procedure-based assessment of resident skill in neurosurgery, based on these findings it seems that this tool can provide meaningful information about residents' competence and progress, and furnish feedback to guide resident education.

References

  • 1

    Ahmed KMiskovic DDarzi AAthanasiou THanna GB: Observational tools for assessment of procedural skills: a systematic review.. Am J Surg 202:469480.e62011

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 2

    Birkmeyer JDFinks JFO'Reilly AOerline MCarlin AMNunn AR: Surgical skill and complication rates after bariatric surgery. N Engl J Med 369:143414422013

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 3

    Bjorklund DFMiller PHCoyle TRSlawinski JL: Instructing children to use memory strategies: evidence of utilization deficiencies in memory training studies. Dev Rev 17:4114411997

    • Crossref
    • Search Google Scholar
    • Export Citation
  • 4

    Butterworth GMorisette P: Onset of pointing and the acquisition of language in infancy. J Reprod Infant Psychol 14:2192311996

  • 5

    Carlucci LCase J: On the necessity of U-shaped learning. Top Cogn Sci 5:56882013

  • 6

    Davis J: Drawing's demise: U-shaped development in graphic symbolization. Stud Art Educ 38:1321571997

  • 7

    Ericsson KA: Deliberate practice and acquisition of expert performance: a general overview. Acad Emerg Med 15:9889942008

  • 8

    Evans AW: Assessing competence in surgical dentistry. Br Dent J 190:3433462001

  • 9

    Faulkner HRegehr GMartin JReznick R: Validation of an objective structured assessment of technical skill for surgical residents. Acad Med 71:136313651996

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 10

    Glarner CEMcDonald RJSmith ABLeverson GEPeyre SPugh CM: Utilizing a novel tool for the comprehensive assessment of resident operative performance. J Surg Educ 70:8138202013

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 11

    Hadley CLam SKBriceño VLuerssen TGJea A: Use of a formal assessment instrument for evaluation of resident operative skills in pediatric neurosurgery. J Neurosurg Pediatr 16:4975042015

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 12

    Hay LBard CFleury MTeasdale N: Kinematics of aiming in direction and amplitude: a developmental study. Acta Psychol (Amst) 77:2032151991

  • 13

    Hogg MEZenati MNovak SChen YJun YSteve J: Grading of surgeon technical performance predicts postoperative pancreatic fistula for pancreaticoduodenectomy independent of patient-related variables. Ann Surg 264:4824912016

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 14

    Jackson DCottrell GW: Attention and U-shaped learning in the acquisition of the past tense.. Presented at the Proceedings of the Nineteenth Annual Conference of the Cognitive Science Society Stanford, California1997. (http://cseweb.ucsd.edu/~gary/pubs/dan-jackson-cogsci97.pdf) [Accessed November 30 2016]

    • Search Google Scholar
    • Export Citation
  • 15

    Jea AAl-Otibi MBonnard ADrake JM: Laparoscopy-assisted ventriculoperitoneal shunt surgery in children: a series of 11 cases. J Neurosurg 106:6 Suppl4214252007

    • PubMed
    • Search Google Scholar
    • Export Citation
  • 16

    Kang SGRyu BJYang KSKo YHCho SKang SH: An effective repetitive training schedule to achieve skill proficiency using a novel robotic virtual reality simulator. J Surg Educ 72:3693762015

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 17

    Kim MJWilliams RGBoehler MLKetchum JKDunnington GL: Refining the evaluation of operating room performance. J Surg Educ 66:3523562009

  • 18

    Kuhl PKStevens EHayashi ADeguchi TKiritani SIverson P: Infants show a facilitation effect for native language phonetic perception between 6 and 12 months. Dev Sci 9:F13F212006

    • Search Google Scholar
    • Export Citation
  • 19

    Martin JARegehr GReznick RMacRae HMurnaghan JHutchison C: Objective structured assessment of technical skill (OSATS) for surgical residents. Br J Surg 84:2732781997

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 20

    Niitsu HHirabayashi NYoshimitsu MMimura TTaomoto JSugiyama Y: Using the Objective Structured Assessment of Technical Skills (OSATS) global rating scale to evaluate the skills of surgical trainees in the operating room. Surg Today 43:2712752013

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 21

    Pauls FMacha TPetermann F: U-shaped development: an old but unsolved problem. Front Psychol 4:3012013

  • 22

    Phan SLiao JJia FMaharaj MReddy RMobbs RJ: Laparotomy vs minimally invasive laparoscopic ventriculoperitoneal shunt placement for hydrocephalus: A systematic review and meta-analysis. Clin Neurol Neurosurg 140:26322016

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 23

    Regehr GMacRae HReznick RKSzalay D: Comparing the psychometric properties of checklists and global rating scales for assessing performance on an OSCE-format examination. Acad Med 73:9939971998

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 24

    Reznick RRegehr GMacRae HMartin JMcCulloch W: Testing technical skill via an innovative “bench station” examination. Am J Surg 173:2262301997

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 25

    Rose JSWaibel BHSchenarts PJ: Disparity between resident and faculty surgeons' perceptions of preoperative preparation, intraoperative teaching, and postoperative feedback. J Surg Educ 68:4594642011

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 26

    Scott DJYoung WNTesfay STFrawley WHRege RVJones DB: Laparoscopic skills training. Am J Surg 182:1371422001

  • 27

    Sebastián-Gallés NBosch L: Developmental shift in the discrimination of vowel contrasts in bilingual infants: is the distributional account all there is to it?. Dev Sci 12:8748872009

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 28

    Seldin N: The Neurological Surgery Milestone Project. ChicagoAccreditation Council for Graduate Medical Education and the American Board of Neurological Surgery2015

    • Search Google Scholar
    • Export Citation
  • 29

    Siegler RS: Developmental sequences within and between concepts. Monogr Soc Res Child Dev 46:1841981

  • 30

    Siegler RS: Five generalizations about cognitive development. Am Psychol 38:2632771983

  • 31

    Snyder RATarpley MJTarpley JLDavidson MBrophy CDattilo JB: Teaching in the operating room: results of a national survey. J Surg Educ 69:6436492012

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 32

    Swingley D: Contributions of infant word learning to language development. Philos Trans R Soc Lond B Biol Sci 364:361736322009

  • 33

    von Hofsten C: Developmental changes in the organization of pre reaching movements. Dev Psychol 20:3783881984

  • 34

    von Hofsten C: Eye–hand coordination in the newborn. Dev Psychol 18:4504611982

  • 35

    van Hove PDTuijthof GJVerdaasdonk EGStassen LPDankelman J: Objective assessment of technical surgical skills. Br J Surg 97:9729872010

  • 36

    Werker JFFennell CCorcoran KStager C: Infants' ability to learn phonetically similar words. Effects of age and vocabulary size. Infancy 3:1302002

    • Crossref
    • Search Google Scholar
    • Export Citation
  • 37

    Whitehead WEJea AVachhrajani SKulkarni AVDrake JM: Accurate placement of cerebrospinal fluid shunt ventricular catheters with real-time ultrasound guidance in older children without patent fontanelles. J Neurosurg 107:5 Suppl4064102007

    • PubMed
    • Search Google Scholar
    • Export Citation
  • 38

    Winckel CPReznick RKCohen RTaylor B: Reliability and construct validity of a structured technical skills assessment form. Am J Surg 167:4234271994

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation

Disclosures

The authors report no conflict of interest concerning the materials or methods used in this study or the findings specified in this paper.

Author Contributions

Conception and design: all authors. Acquisition of data: all authors. Analysis and interpretation of data: all authors. Drafting the article: Jea, Aldave, Briceño, Luerssen. Critically revising the article: Jea, Aldave, Hansen, Briceño. Reviewed submitted version of manuscript: all authors. Approved the final version of the manuscript on behalf of all authors: Jea. Statistical analysis: Jea, Aldave, Hansen, Luerssen. Administrative/technical/material support: Luerssen. Study supervision: Jea.

If the inline PDF is not rendering correctly, you can download the PDF file here.

Article Information

INCLUDE WHEN CITING Published online January 27, 2017; DOI: 10.3171/2016.10.PEDS16471.

Correspondence Andrew Jea, Section of Pediatric Neurosurgery, Indiana University School of Medicine, 705 Riley Hospital Dr., Indianapolis, IN 46202. email: ajea@goodmancampbell.com.

© AANS, except where prohibited by US copyright law.

Headings

Figures

  • View in gallery

    Our evaluation form or data collection sheet for evaluation of a resident's operative skills.

  • View in gallery

    Graphs showing score average rated by the attending evaluators and the residents' self-assessment during the months of rotation. Each category of the OSATS rating scale and the overall score are represented. The numbers on the y axis represent the skill grade. Eval = evaluator; NS = attending neurosurgeon. Figure is available in color online only.

References

  • 1

    Ahmed KMiskovic DDarzi AAthanasiou THanna GB: Observational tools for assessment of procedural skills: a systematic review.. Am J Surg 202:469480.e62011

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 2

    Birkmeyer JDFinks JFO'Reilly AOerline MCarlin AMNunn AR: Surgical skill and complication rates after bariatric surgery. N Engl J Med 369:143414422013

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 3

    Bjorklund DFMiller PHCoyle TRSlawinski JL: Instructing children to use memory strategies: evidence of utilization deficiencies in memory training studies. Dev Rev 17:4114411997

    • Crossref
    • Search Google Scholar
    • Export Citation
  • 4

    Butterworth GMorisette P: Onset of pointing and the acquisition of language in infancy. J Reprod Infant Psychol 14:2192311996

  • 5

    Carlucci LCase J: On the necessity of U-shaped learning. Top Cogn Sci 5:56882013

  • 6

    Davis J: Drawing's demise: U-shaped development in graphic symbolization. Stud Art Educ 38:1321571997

  • 7

    Ericsson KA: Deliberate practice and acquisition of expert performance: a general overview. Acad Emerg Med 15:9889942008

  • 8

    Evans AW: Assessing competence in surgical dentistry. Br Dent J 190:3433462001

  • 9

    Faulkner HRegehr GMartin JReznick R: Validation of an objective structured assessment of technical skill for surgical residents. Acad Med 71:136313651996

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 10

    Glarner CEMcDonald RJSmith ABLeverson GEPeyre SPugh CM: Utilizing a novel tool for the comprehensive assessment of resident operative performance. J Surg Educ 70:8138202013

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 11

    Hadley CLam SKBriceño VLuerssen TGJea A: Use of a formal assessment instrument for evaluation of resident operative skills in pediatric neurosurgery. J Neurosurg Pediatr 16:4975042015

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 12

    Hay LBard CFleury MTeasdale N: Kinematics of aiming in direction and amplitude: a developmental study. Acta Psychol (Amst) 77:2032151991

  • 13

    Hogg MEZenati MNovak SChen YJun YSteve J: Grading of surgeon technical performance predicts postoperative pancreatic fistula for pancreaticoduodenectomy independent of patient-related variables. Ann Surg 264:4824912016

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 14

    Jackson DCottrell GW: Attention and U-shaped learning in the acquisition of the past tense.. Presented at the Proceedings of the Nineteenth Annual Conference of the Cognitive Science Society Stanford, California1997. (http://cseweb.ucsd.edu/~gary/pubs/dan-jackson-cogsci97.pdf) [Accessed November 30 2016]

    • Search Google Scholar
    • Export Citation
  • 15

    Jea AAl-Otibi MBonnard ADrake JM: Laparoscopy-assisted ventriculoperitoneal shunt surgery in children: a series of 11 cases. J Neurosurg 106:6 Suppl4214252007

    • PubMed
    • Search Google Scholar
    • Export Citation
  • 16

    Kang SGRyu BJYang KSKo YHCho SKang SH: An effective repetitive training schedule to achieve skill proficiency using a novel robotic virtual reality simulator. J Surg Educ 72:3693762015

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 17

    Kim MJWilliams RGBoehler MLKetchum JKDunnington GL: Refining the evaluation of operating room performance. J Surg Educ 66:3523562009

  • 18

    Kuhl PKStevens EHayashi ADeguchi TKiritani SIverson P: Infants show a facilitation effect for native language phonetic perception between 6 and 12 months. Dev Sci 9:F13F212006

    • Search Google Scholar
    • Export Citation
  • 19

    Martin JARegehr GReznick RMacRae HMurnaghan JHutchison C: Objective structured assessment of technical skill (OSATS) for surgical residents. Br J Surg 84:2732781997

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 20

    Niitsu HHirabayashi NYoshimitsu MMimura TTaomoto JSugiyama Y: Using the Objective Structured Assessment of Technical Skills (OSATS) global rating scale to evaluate the skills of surgical trainees in the operating room. Surg Today 43:2712752013

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 21

    Pauls FMacha TPetermann F: U-shaped development: an old but unsolved problem. Front Psychol 4:3012013

  • 22

    Phan SLiao JJia FMaharaj MReddy RMobbs RJ: Laparotomy vs minimally invasive laparoscopic ventriculoperitoneal shunt placement for hydrocephalus: A systematic review and meta-analysis. Clin Neurol Neurosurg 140:26322016

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 23

    Regehr GMacRae HReznick RKSzalay D: Comparing the psychometric properties of checklists and global rating scales for assessing performance on an OSCE-format examination. Acad Med 73:9939971998

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 24

    Reznick RRegehr GMacRae HMartin JMcCulloch W: Testing technical skill via an innovative “bench station” examination. Am J Surg 173:2262301997

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 25

    Rose JSWaibel BHSchenarts PJ: Disparity between resident and faculty surgeons' perceptions of preoperative preparation, intraoperative teaching, and postoperative feedback. J Surg Educ 68:4594642011

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 26

    Scott DJYoung WNTesfay STFrawley WHRege RVJones DB: Laparoscopic skills training. Am J Surg 182:1371422001

  • 27

    Sebastián-Gallés NBosch L: Developmental shift in the discrimination of vowel contrasts in bilingual infants: is the distributional account all there is to it?. Dev Sci 12:8748872009

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 28

    Seldin N: The Neurological Surgery Milestone Project. ChicagoAccreditation Council for Graduate Medical Education and the American Board of Neurological Surgery2015

    • Search Google Scholar
    • Export Citation
  • 29

    Siegler RS: Developmental sequences within and between concepts. Monogr Soc Res Child Dev 46:1841981

  • 30

    Siegler RS: Five generalizations about cognitive development. Am Psychol 38:2632771983

  • 31

    Snyder RATarpley MJTarpley JLDavidson MBrophy CDattilo JB: Teaching in the operating room: results of a national survey. J Surg Educ 69:6436492012

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 32

    Swingley D: Contributions of infant word learning to language development. Philos Trans R Soc Lond B Biol Sci 364:361736322009

  • 33

    von Hofsten C: Developmental changes in the organization of pre reaching movements. Dev Psychol 20:3783881984

  • 34

    von Hofsten C: Eye–hand coordination in the newborn. Dev Psychol 18:4504611982

  • 35

    van Hove PDTuijthof GJVerdaasdonk EGStassen LPDankelman J: Objective assessment of technical surgical skills. Br J Surg 97:9729872010

  • 36

    Werker JFFennell CCorcoran KStager C: Infants' ability to learn phonetically similar words. Effects of age and vocabulary size. Infancy 3:1302002

    • Crossref
    • Search Google Scholar
    • Export Citation
  • 37

    Whitehead WEJea AVachhrajani SKulkarni AVDrake JM: Accurate placement of cerebrospinal fluid shunt ventricular catheters with real-time ultrasound guidance in older children without patent fontanelles. J Neurosurg 107:5 Suppl4064102007

    • PubMed
    • Search Google Scholar
    • Export Citation
  • 38

    Winckel CPReznick RKCohen RTaylor B: Reliability and construct validity of a structured technical skills assessment form. Am J Surg 167:4234271994

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation

TrendMD

Metrics

Metrics

All Time Past Year Past 30 Days
Abstract Views 64 64 0
Full Text Views 449 449 25
PDF Downloads 281 281 7
EPUB Downloads 0 0 0

PubMed

Google Scholar