Use of surgical video–based automated performance metrics to predict blood loss and success of simulated vascular injury control in neurosurgery: a pilot study

View More View Less
  • 1 Department of Neurosurgery, Keck School of Medicine of the University of Southern California, Los Angeles, California;
  • | 2 Center for Robotic Simulation and Education, USC Institute of Urology, Keck School of Medicine of the University of Southern California, Los Angeles, California;
  • | 3 Computing + Mathematical Sciences, California Institute of Technology, Pasadena, California; and
  • | 4 Division of Neurosurgery, Center for Neuroscience, Children’s National Medical Center, Washington, DC
Restricted access

Purchase Now

USD  $45.00

JNS + Pediatrics - 1 year subscription bundle (Individuals Only)

USD  $515.00

JNS + Pediatrics + Spine - 1 year subscription bundle (Individuals Only)

USD  $612.00
USD  $45.00
USD  $515.00
USD  $612.00
Print or Print + Online Sign in

OBJECTIVE

Experts can assess surgeon skill using surgical video, but a limited number of expert surgeons are available. Automated performance metrics (APMs) are a promising alternative but have not been created from operative videos in neurosurgery to date. The authors aimed to evaluate whether video-based APMs can predict task success and blood loss during endonasal endoscopic surgery in a validated cadaveric simulator of vascular injury of the internal carotid artery.

METHODS

Videos of cadaveric simulation trials by 73 neurosurgeons and otorhinolaryngologists were analyzed and manually annotated with bounding boxes to identify the surgical instruments in the frame. APMs in five domains were defined—instrument usage, time-to-phase, instrument disappearance, instrument movement, and instrument interactions—on the basis of expert analysis and task-specific surgical progressions. Bounding-box data of instrument position were then used to generate APMs for each trial. Multivariate linear regression was used to test for the associations between APMs and blood loss and task success (hemorrhage control in less than 5 minutes). The APMs of 93 successful trials were compared with the APMs of 49 unsuccessful trials.

RESULTS

In total, 29,151 frames of surgical video were annotated. Successful simulation trials had superior APMs in each domain, including proportionately more time spent with the key instruments in view (p < 0.001) and less time without hemorrhage control (p = 0.002). APMs in all domains improved in subsequent trials after the participants received personalized expert instruction. Attending surgeons had superior instrument usage, time-to-phase, and instrument disappearance metrics compared with resident surgeons (p < 0.01). APMs predicted surgeon performance better than surgeon training level or prior experience. A regression model that included APMs predicted blood loss with an R2 value of 0.87 (p < 0.001).

CONCLUSIONS

Video-based APMs were superior predictors of simulation trial success and blood loss than surgeon characteristics such as case volume and attending status. Surgeon educators can use APMs to assess competency, quantify performance, and provide actionable, structured feedback in order to improve patient outcomes. Validation of APMs provides a benchmark for further development of fully automated video assessment pipelines that utilize machine learning and computer vision.

ABBREVIATIONS

APM = automated performance metric; ICAI = internal carotid artery injury; SOCAL = Simulated Outcomes following Carotid Artery Laceration.

Supplementary Materials

    • Supplemental Tables and Figure (PDF 2,031 KB)

Schematics of transseptal interforniceal resection of a superiorly recessed colloid cyst. ©Mark Souweidane, published with permission. See the article by Tosi et al. (pp 813–819).

JNS + Pediatrics - 1 year subscription bundle (Individuals Only)

USD  $515.00

JNS + Pediatrics + Spine - 1 year subscription bundle (Individuals Only)

USD  $612.00
USD  $515.00
USD  $612.00
  • 1

    Birkmeyer JD, Finks JF, O’Reilly A, Oerline M, Carlin AM, Nunn AR, et al. Surgical skill and complication rates after bariatric surgery. N Engl J Med. 2013;369(15):14341442.

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 2

    Brajcich BC, Stulberg JJ, Palis BE, Chung JW, Huang R, Nelson H, Bilimoria KY. Association between surgical technical skill and long-term survival for colon cancer. JAMA Oncol. 2021;7(1):127129.

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 3

    Fecso AB, Bhatti JA, Stotland PK, Quereshy FA, Grantcharov TP. Technical performance as a predictor of clinical outcomes in laparoscopic gastric cancer surgery. Ann Surg. 2019;270(1):115120.

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 4

    Acton RD, Chipman JG, Lunden M, Schmitz CC. Unanticipated teaching demands rise with simulation training: strategies for managing faculty workload. J Surg Educ. 2015;72(3):522529.

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 5

    Hung AJ, Chen J, Gill IS. Automated performance metrics and machine learning algorithms to measure surgeon performance and anticipate clinical outcomes in robotic surgery. JAMA Surg. 2018;153(8):770771.

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 6

    Ghodoussipour S, Reddy SS, Ma R, Huang D, Nguyen J, Hung AJ. An objective assessment of performance during robotic partial nephrectomy: validation and correlation of automated performance metrics with intraoperative outcomes. J Urol. 2021;205(5):12941302.

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 7

    Taylor LK, Thomas GW, Karam MD, Kreiter CD, Anderson DD. Developing an objective assessment of surgical performance from operating room video and surgical imagery. IISE Trans Healthc Syst Eng. 2018;88(2):110116.

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 8

    Chen J, Oh PJ, Cheng N, Shah A, Montez J, Jarc A, et al. Use of automated performance metrics to measure surgeon performance during robotic vesicourethral anastomosis and methodical development of a training tutorial. J Urol. 2018;200(4):895902.

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 9

    Ciric I, Ragin A, Baumgartner C, Pierce D. Complications of transsphenoidal surgery: results of a national survey, review of the literature, and personal experience. Neurosurgery. 1997;40(2):225237.

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 10

    Laws ER Jr. Vascular complications of transsphenoidal surgery. Pituitary. 1999;2(2):163170.

  • 11

    Rowan NR, Turner MT, Valappil B, Fernandez-Miranda JC, Wang EW, Gardner PA, Snyderman CH. Injury of the carotid artery during endoscopic endonasal surgery: surveys of skull base surgeons. J Neurol Surg B Skull Base. 2018;79(3):302308.

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 12

    Donoho DA, Johnson CE, Hur KT, Buchanan IA, Fredrickson VL, Minneti M, et al. Costs and training results of an objectively validated cadaveric perfusion-based internal carotid artery injury simulation during endoscopic skull base surgery. Int Forum Allergy Rhinol. 2019;9(7):787794.

    • PubMed
    • Search Google Scholar
    • Export Citation
  • 13

    Pham M, Kale A, Marquez Y, Winer J, Lee B, Harris B, et al. A perfusion-based human cadaveric model for management of carotid artery injury during endoscopic endonasal skull base surgery. J Neurol Surg B Skull Base. 2014;75(5):309313.

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 14

    Shen J, Hur K, Zhang Z, Minneti M, Pham M, Wrobel B, Zada G. Objective validation of perfusion-based human cadaveric simulation training model for management of internal carotid artery injury in endoscopic endonasal sinus and skull base surgery. Oper Neurosurg (Hagerstown). 2018;15(2):231238.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • 15

    Kugener G, Pangal D, Zada G, Donoho D. Simulated outcomes following carotid artery laceration. figshare. Dataset. Published online August 10, 2021.Accessed October 21, 2021. 10.6084/m9.figshare.15132468.v1

    • Search Google Scholar
    • Export Citation
  • 16

    Zada G, Bakhsheshian J, Pham M, Minneti M, Christian E, Winer J, et al. Development of a perfusion-based cadaveric simulation model integrated into neurosurgical training: feasibility based on reconstitution of vascular and cerebrospinal fluid systems. Oper Neurosurg (Hagerstown). 2018;14(1):7280.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • 17

    Donoho DA, Pangal DJ, Kugener G, Rutkowski M, Micko A, Shahrestani S, et al. Improved surgeon performance following cadaveric simulation of internal carotid artery injury during endoscopic endonasal surgery: training outcomes of a nationwide prospective educational intervention. J Neurosurg. 2021;35(5):13471355.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • 18

    Microsoft/VoTT. 2020.Accessed October 21, 2021. https://github.com/microsoft/VoTT

  • 19

    Pangal DJ, Kugener G, Shahrestani S, Attenello F, Zada G, Donoho DA. A guide to annotation of neurosurgical intraoperative video for machine learning analysis and computer vision. World Neurosurg. 2021;150:2630.

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 20

    Hoerl AE, Kennard RW. Ridge regression: biased estimation for nonorthogonal problems. Technometrics. 1970;12(1):5567.

  • 21

    Tibshirani R. regression shrinkage and selection via the lasso. J R Stat Soc Series B Stat Methodol. 1996;58(1):267288.

  • 22

    Zou H, Hastie T. Regularization and variable selection via the elastic net. J R Stat Soc Series B Stat Methodol. 2005;67(2):301320.

  • 23

    Bunn A, Korpela M. Crossdating in dplR. Accessed October 21, 2021. https://cran.r-project.org/web/packages/dplR/vignettes/xdate-dplR.pdf

    • Search Google Scholar
    • Export Citation
  • 24

    Luongo F, Hakim R, Nguyen JH, Anandkumar A, Hung AJ. Deep learning-based computer vision to recognize and classify suturing gestures in robot-assisted surgery. Surgery. 2021;169(5):12401244.

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 25

    Eijkemans MJC, van Houdenhoven M, Nguyen T, Boersma E, Steyerberg EW, Kazemier G. Predicting the unpredictable: a new prediction model for operating room times using individual characteristics and the surgeon’s estimate. Anesthesiology. 2010;112(1):4149.

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 26

    Kitaguchi D, Takeshita N, Matsuzaki H, Takano H, Owada Y, Enomoto T, et al. Real-time automatic surgical phase recognition in laparoscopic sigmoidectomy using the convolutional neural network-based deep learning approach. Surg Endosc. 2020;34(11):49244931.

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 27

    Meeuwsen FC, van Luyn F, Blikkendaal MD, Jansen FW, van den Dobbelsteen JJ. Surgical phase modelling in minimal invasive surgery. Surg Endosc. 2019;33(5):14261432.

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 28

    Bissonnette V, Mirchi N, Ledwos N, Alsidieri G, Winkler-Schwartz A, Del Maestro RF. Artificial intelligence distinguishes surgical training levels in a virtual reality spinal task. J Bone Joint Surg Am. 2019;101(23):e127.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • 29

    Mirchi N, Bissonnette V, Ledwos N, Winkler-Schwartz A, Yilmaz R, Karlik B, Del Maestro RF. Artificial neural networks to assess virtual reality anterior cervical discectomy performance. Oper Neurosurg (Hagerstown). 2020;19(1):6575.

    • Crossref
    • Search Google Scholar
    • Export Citation

Metrics

All Time Past Year Past 30 Days
Abstract Views 1530 1529 95
Full Text Views 156 156 29
PDF Downloads 222 222 47
EPUB Downloads 0 0 0