Real-time control of a prosthetic hand using human electrocorticography signals

Technical note

Takufumi Yanagisawa M.D., Ph.D.1,2, Masayuki Hirata M.D., Ph.D.1, Youichi Saitoh M.D., Ph.D.1, Tetsu Goto M.D., Ph.D.1, Haruhiko Kishima M.D., Ph.D.1, Ryohei Fukuma M.S.2,3, Hiroshi Yokoi Ph.D.4, Yukiyasu Kamitani Ph.D.2,3, and Toshiki Yoshimine M.D., Ph.D.1
View More View Less
  • 1 Department of Neurosurgery, Osaka University Medical School, Osaka;
  • | 2 ATR Computational Neuroscience Laboratories, Kyoto;
  • | 3 Nara Institute of Science and Technology; and
  • | 4 Department of Precision Engineering, University of Tokyo, Japan
Restricted access

Purchase Now

USD  $45.00

JNS + Pediatrics - 1 year subscription bundle (Individuals Only)

USD  $515.00

JNS + Pediatrics + Spine - 1 year subscription bundle (Individuals Only)

USD  $612.00
Print or Print + Online

Object

A brain-machine interface (BMI) offers patients with severe motor disabilities greater independence by controlling external devices such as prosthetic arms. Among the available signal sources for the BMI, electrocorticography (ECoG) provides a clinically feasible signal with long-term stability and low clinical risk. Although ECoG signals have been used to infer arm movements, no study has examined its use to control a prosthetic arm in real time. The authors present an integrated BMI system for the control of a prosthetic hand using ECoG signals in a patient who had suffered a stroke. This system used the power modulations of the ECoG signal that are characteristic during movements of the patient's hand and enabled control of the prosthetic hand with movements that mimicked the patient's hand movements.

Methods

A poststroke patient with subdural electrodes placed over his sensorimotor cortex performed 3 types of simple hand movements following a sound cue (calibration period). Time-frequency analysis was performed with the ECoG signals to select 3 frequency bands (1–8, 25–40, and 80–150 Hz) that revealed characteristic power modulation during the movements. Using these selected features, 2 classifiers (decoders) were trained to predict the movement state—that is, whether the patient was moving his hand or not—and the movement type based on a linear support vector machine. The decoding accuracy was compared among the 3 frequency bands to identify the most informative features. With the trained decoders, novel ECoG signals were decoded online while the patient performed the same task without cues (free-run period). According to the results of the real-time decoding, the prosthetic hand mimicked the patient's hand movements.

Results

Offline cross-validation analysis of the ECoG data measured during the calibration period revealed that the state and movement type of the patient's hand were predicted with an accuracy of 79.6% (chance 50%) and 68.3% (chance 33.3%), respectively. Using the trained decoders, the onset of the hand movement was detected within 0.37 ± 0.29 seconds of the actual movement. At the detected onset timing, the type of movement was inferred with an accuracy of 69.2%. In the free-run period, the patient's hand movements were faithfully mimicked by the prosthetic hand in real time.

Conclusions

The present integrated BMI system successfully decoded the hand movements of a poststroke patient and controlled a prosthetic hand in real time. This success paves the way for the restoration of the patient's motor function using a prosthetic arm controlled by a BMI using ECoG signals.

Abbreviations used in this paper:

BMI = brain-machine interface; ECoG = electrocorticography; EEG = electroencephalography; EMG = electromyography; FFT = fast Fourier transform; MEG = magnetoencephalography; SVM = support vector machine.

JNS + Pediatrics - 1 year subscription bundle (Individuals Only)

USD  $515.00

JNS + Pediatrics + Spine - 1 year subscription bundle (Individuals Only)

USD  $612.00
  • 1

    Andersen RA, , Musallam S, & Pesaran B: Selecting the signals for a brain-machine interface. Curr Opin Neurobiol 14:720726, 2004

  • 2

    Bengio Y, & Grandvalet Y: No unbiased estimator of the variance of K-fold cross-validation. J Mach Learn Res 5:10891105, 2004

  • 3

    Breiman L: Heuristics of instability and stabilization in model selection. Ann Stat 24:23502383, 1996

  • 4

    Chao ZC, , Nagasaka Y, & Fujii N: Long-term asynchronous decoding of arm motion using electrocorticographic signals in monkeys. Front Neuroengineering 3:3, 2010

    • Search Google Scholar
    • Export Citation
  • 5

    Delorme A, & Makeig S: EEGLAB: an open source toolbox for analysis of single-trial EEG dynamics including independent component analysis. J Neurosci Methods 134:921, 2004

    • Search Google Scholar
    • Export Citation
  • 6

    Donoghue JP, , Nurmikko A, , Friehs G, & Black M: Development of neuromotor prostheses for humans. Suppl Clin Neurophysiol 57:592606, 2004

  • 7

    Fujiwara Y, , Yamashita O, , Kawawaki D, , Doya K, , Kawato M, & Toyama K, et al.: A hierarchical Bayesian method to resolve an inverse problem of MEG contaminated with eye movement artifacts. Neuroimage 45:393409, 2009

    • Search Google Scholar
    • Export Citation
  • 8

    Georgopoulos AP, , Schwartz AB, & Kettner RE: Neuronal population coding of movement direction. Science 233:14161419, 1986

  • 9

    Hochberg LR, , Serruya MD, , Friehs GM, , Mukand JA, , Saleh M, & Caplan AH, et al.: Neuronal ensemble control of prosthetic devices by a human with tetraplegia. Nature 442:164171, 2006

    • Search Google Scholar
    • Export Citation
  • 10

    Hosomi K, , Saitoh Y, , Kishima H, , Oshino S, , Hirata M, & Tani N, et al.: Electrical stimulation of primary motor cortex within the central sulcus for intractable neuropathic pain. Clin Neurophysiol 119:9931001, 2008

    • Search Google Scholar
    • Export Citation
  • 11

    Kamitani Y, & Tong F: Decoding the visual and subjective contents of the human brain. Nat Neurosci 8:679685, 2005

  • 12

    Kato R, , Yokoi H, , Arieta AH, , Yu W, & Arai T: Mutual adaptation among man and machine by using f-MRI analysis. Robot Auton Syst 57:161166, 2009

    • Search Google Scholar
    • Export Citation
  • 13

    Leuthardt EC, , Schalk G, , Moran D, & Ojemann JG: The emerging world of motor neuroprosthetics: a neurosurgical perspective. Neurosurgery 59:114, 2006

    • Search Google Scholar
    • Export Citation
  • 14

    Leuthardt EC, , Schalk G, , Wolpaw JR, , Ojemann JG, & Moran DW: A brain-computer interface using electrocorticographic signals in humans. J Neural Eng 1:6371, 2004

    • Search Google Scholar
    • Export Citation
  • 15

    Mehring C, , Rickert J, , Vaadia E, , Cardosa de Oliveira S, , Aertsen A, & Rotter S: Inference of hand movements from local field potentials in monkey motor cortex. Nat Neurosci 6:12531254, 2003

    • Search Google Scholar
    • Export Citation
  • 16

    Miller KJ, , Zanos S, , Fetz EE, , den Nijs M, & Ojemann JG: Decoupling the cortical power spectrum reveals real-time representation of individual finger movements in humans. J Neurosci 29:31323137, 2009

    • Search Google Scholar
    • Export Citation
  • 17

    Nakamura T, , Kita K, , Kato R, , Matsushita K, & Hiroshi Y: Control strategy for a myoelectric hand: measuring acceptable time delay in human intention discrimination. Conf Proc IEEE Eng Med Biol Soc 2009. 50445047, 2009

    • Search Google Scholar
    • Export Citation
  • 18

    Pistohl T, , Ball T, , Schulze-Bonhage A, , Aertsen A, & Mehring C: Prediction of arm movement trajectories from ECoG-recordings in humans. J Neurosci Methods 167:105114, 2008

    • Search Google Scholar
    • Export Citation
  • 19

    Quian Quiroga R, & Panzeri S: Extracting information from neuronal populations: information theory and decoding approaches. Nat Rev Neurosci 10:173185, 2009

    • Search Google Scholar
    • Export Citation
  • 20

    Ray S, , Crone NE, , Niebur E, , Franaszczuk PJ, & Hsiao SS: Neural correlates of high-gamma oscillations (60–200 Hz) in macaque local field potentials and their potential implications in electrocorticography. J Neurosci 28:1152611536, 2008

    • Search Google Scholar
    • Export Citation
  • 21

    Schalk G, , Miller KJ, , Anderson NR, , Wilson JA, , Smyth MD, & Ojemann JG, et al.: Two-dimensional movement control using electrocorticographic signals in humans. J Neural Eng 5:7584, 2008

    • Search Google Scholar
    • Export Citation
  • 22

    Shain W, , Spataro L, , Dilgen J, , Haverstick K, , Retterer S, & Isaacson M, et al.: Controlling cellular reactive responses around neural prosthetic devices using peripheral and local intervention strategies. IEEE Trans Neural Syst Rehabil Eng 11:186188, 2003

    • Search Google Scholar
    • Export Citation
  • 23

    Thakur PH, , Bastian AJ, & Hsiao SS: Multidigit movement synergies of the human hand in an unconstrained haptic exploration task. J Neurosci 28:12711281, 2008

    • Search Google Scholar
    • Export Citation
  • 24

    Truccolo W, , Friehs GM, , Donoghue JP, & Hochberg LR: Primary motor cortex tuning to intended movement kinematics in humans with tetraplegia. J Neurosci 28:11631178, 2008

    • Search Google Scholar
    • Export Citation
  • 25

    Vapnik VN: Statistical Learning Theory New York, Wiley, 1998

  • 26

    Velliste M, , Perel S, , Spalding MC, , Whitford AS, & Schwartz AB: Cortical control of a prosthetic arm for self-feeding. Nature 453:10981101, 2008

    • Search Google Scholar
    • Export Citation
  • 27

    Waldert S, , Preissl H, , Demandt E, , Braun C, , Birbaumer N, & Aertsen A, et al.: Hand movement direction decoded from MEG and EEG. J Neurosci 28:10001008, 2008

    • Search Google Scholar
    • Export Citation
  • 28

    Wessberg J, , Stambaugh CR, , Kralik JD, , Beck PD, , Laubach M, & Chapin JK, et al.: Real-time prediction of hand trajectory by ensembles of cortical neurons in primates. Nature 408:361365, 2000

    • Search Google Scholar
    • Export Citation
  • 29

    Wolpaw JR, , Birbaumer N, , McFarland DJ, , Pfurtscheller G, & Vaughan TM: Brain-computer interfaces for communication and control. Clin Neurophysiol 113:767791, 2002

    • Search Google Scholar
    • Export Citation
  • 30

    Wolpaw JR, & McFarland DJ: Control of a two-dimensional movement signal by a noninvasive brain-computer interface in humans. Proc Natl Acad Sci U S A 101:1784917854, 2004

    • Search Google Scholar
    • Export Citation
  • 31

    Yanagisawa T, , Hirata M, , Saitoh Y, , Kato A, , Shibuya D, & Kamitani Y, et al.: Neural decoding using gyral and intrasulcal electrocorticograms. Neuroimage 45:10991106, 2009

    • Search Google Scholar
    • Export Citation
  • 32

    Yokoi H, , Kita K, , Nakamura T, , Kato R, , Hernandez A, & Arai T: Mutually adaptable EMG devices for prosthetic hand. The International Journal of Factory Automation, Robotics and Soft Computing 1:7483, 2009

    • Search Google Scholar
    • Export Citation

Metrics

All Time Past Year Past 30 Days
Abstract Views 2064 727 56
Full Text Views 472 62 1
PDF Downloads 281 60 1
EPUB Downloads 0 0 0