Development and validation of the Skills Assessment in Microsurgery for Brain Aneurysms (SAMBA) instrument for predicting proficiency in aneurysm surgery

Restricted access

OBJECTIVE

Surgical performance evaluation was first described with the OSATS (Objective Structured Assessment of Technical Skills) and modified for aneurysm microsurgery simulation with the OSAACS (Objective Structured Assessment of Aneurysm Clipping Skills). These methods rely on the subjective opinions of evaluators, however, and there is a lack of objective evaluation for proficiency in the microsurgical treatment of brain aneurysms. The authors present a new instrument, the Skill Assessment in Microsurgery for Brain Aneurysms (SAMBA) scale, which can be used similarly in a simulation model and in the treatment of unruptured middle cerebral artery (MCA) aneurysms to predict surgical performance; the authors also report on its validation.

METHODS

The SAMBA scale was created by consensus among 5 vascular neurosurgeons from 2 different neurosurgical departments. SAMBA results were analyzed using descriptive statistics, Cronbach’s alpha indexes, and multivariate ANOVA analyses (p < 0.05).

RESULTS

Expert, intermediate-level, and novice surgeons scored, respectively, an average of 33.9, 27.1, and 16.4 points in the real surgery and 33.3, 27.3, and 19.4 points in the simulation. The SAMBA interrater reliability index was 0.995 for the real surgery and 0.996 for the simulated surgery; the intrarater reliability was 0.983 (Cronbach’s alpha). In both the simulation and the real surgery settings, the average scores achieved by members of each group (expert, intermediate level, and novice) were significantly different (p < 0.001). Scores among novice surgeons were more diverse (coefficient of variation = 12.4).

CONCLUSIONS

Predictive validation of the placenta brain aneurysm model has been previously reported, but the SAMBA scale adds an objective scoring system to verify microsurgical ability in this complex operation, stratifying proficiency by points. The SAMBA scale can be used as an interface between learning and practicing, as it can be applied in a safe and controlled environment, such as is provided by a placenta model, with similar results obtained in real surgery, predicting real surgical performance.

ABBREVIATIONS CV = coefficient of variation; IQR = interquartile range; MANOVA = multivariate analysis of variance; MCA = middle cerebral artery; OSAACS = Objective Structured Assessment of Aneurysm Clipping Skills; SAMBA = Skill Assessment in Microsurgery for Brain Aneurysms; SD = standard deviation.

Article Information

Correspondence Marcelo Magaldi Ribeiro de Oliveira: Federal University of Minas Gerais, Belo Horizonte, Brazil. mmagaldi@hotmail.com.

INCLUDE WHEN CITING Published online June 14, 2019; DOI: 10.3171/2018.7.JNS173007.

Disclosures The authors report no conflict of interest concerning the materials or methods used in this study or the findings specified in this paper.

© AANS, except where prohibited by US copyright law.

Headings

Figures

  • View in gallery

    Flowchart showing SAMBA methodology. Figure is available in color online only.

References

  • 1

    Alaraj ALuciano CJBailey DPElsenousi ARoitberg BZBernardo A: Virtual reality cerebral aneurysm clipping simulation with real-time haptic feedback. Neurosurgery 11 (Suppl 2):52582015

    • PubMed
    • Search Google Scholar
    • Export Citation
  • 2

    Aoun SGEl Ahmadieh TYEl Tecle NEDaou MRAdel JGPark CS: A pilot study to assess the construct and face validity of the Northwestern Objective Microanastomosis Assessment Tool. J Neurosurg 123:1031092015

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 3

    Belykh EMiller EJLei TChapple KByvaltsev VASpetzler RF: Face, content, and construct validity of an aneurysm clipping model using human placenta. World Neurosurg 105:952960960.e1–960.e2 2017

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 4

    de Montbrun SRoberts PLSatterthwaite LMacRae H: Implementing and evaluating a national certification technical skills examination: the Colorectal Objective Structured Assessment Of Technical Skill. Ann Surg 264:162016

    • Crossref
    • Search Google Scholar
    • Export Citation
  • 5

    de Oliveira MMRFerrarez CERamos TMMalheiros JANicolato AMachado CJ: Learning brain aneurysm microsurgical skills in a human placenta model: predictive validity. J Neurosurg 128:8468522018

    • Crossref
    • Search Google Scholar
    • Export Citation
  • 6

    Fan XThompson B: Confidence intervals about score reliability coefficients, please: an EPM guidelines editorial. Educ Psychol Meas 61:5175312001

    • Search Google Scholar
    • Export Citation
  • 7

    Gasco JHolbrook TJPatel ASmith APaulson DMuns A: Neurosurgery simulation in residency training: feasibility, cost, and educational benefit. Neurosurgery 73 (Suppl 1):39452013

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 8

    Ilgen JSMa IWHatala RCook DA: A systematic review of validity evidence for checklists versus global rating scales in simulation-based assessment. Med Educ 49:1611732015

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 9

    Jacobsen MEAndersen MJHansen COKonge L: Testing basic competency in knee arthroscopy using a virtual reality simulator: exploring validity and reliability. J Bone Joint Surg Am 97:7757812015

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 10

    Likert R: A technique for the measurement of attitudes. Arch Psychol 140:1551932

  • 11

    Martin JARegehr GReznick RMacRae HMurnaghan JHutchison C: Objective structured assessment of technical skill (OSATS) for surgical residents. Br J Surg 84:2732781997

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 12

    McHugh ML: Multiple comparison analysis testing in ANOVA. Biochem Med (Zagreb) 21:2032092011

  • 13

    Oliveira MMAraujo ABNicolato AProsdocimi AGodinho JVValle AL: Face, content, and construct validity of brain tumor microsurgery simulation using a human placenta model. Oper Neurosurg 12:61672016

    • Crossref
    • Search Google Scholar
    • Export Citation
  • 14

    Oliveira Magaldi MNicolato AGodinho JVSantos MProsdocimi AMalheiros JA: Human placenta aneurysm model for training neurosurgeons in vascular microsurgery. Neurosurgery 10 (Suppl 4):5926012014

    • Crossref
    • Search Google Scholar
    • Export Citation
  • 15

    Ribeiro de Oliveira MMNicolato ASantos MGodinho JVBrito RAlvarenga A: Face, content, and construct validity of human placenta as a haptic training tool in neurointerventional surgery. J Neurosurg 124:123812442016

    • Crossref
    • Search Google Scholar
    • Export Citation
  • 16

    Terry LKelley K: Sample size planning for composite reliability coefficients: accuracy in parameter estimation via narrow confidence intervals. Br J Math Stat Psychol 65:3714012012

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation

TrendMD

Metrics

Metrics

All Time Past Year Past 30 Days
Abstract Views 208 208 69
Full Text Views 74 74 9
PDF Downloads 56 56 7
EPUB Downloads 0 0 0

PubMed

Google Scholar