Speech neuroprostheses can decode speech based on brain activity

Researchers from HSE University and Moscow Medical and Dental University have developed a machine learning model that can predict the words a subject is about to say based on neural activity recorded with a small set of minimally invasive electrodes. The paper “Speech decoding from a small set of spatially separated minimally invasive intracranial EEG electrodes using a compact and interpretable neural network” Journal of Neuroengineering This research was funded by a grant from the Russian government as part of the “Science and Universities” national project.

Millions of people around the world are affected by language disorders that limit their ability to communicate. Speech disorders have many causes, including stroke and certain birth defects.

Techniques are now available to restore communication function in such patients. This includes a “silent speech” interface that recognizes speech by tracking the movements of articulatory muscles when a person speaks words without making a sound. However, such devices help some patients but not others, such as those with facial paralysis.

Speech neuroprostheses – brain-computer interfaces that can decode speech based on brain activity – could provide an accessible and reliable solution for restoring communication in such patients.

Unlike personal computers, devices with a Brain Computer Interface (BCI) are directly controlled by the brain without the need for a keyboard or microphone.

A major barrier to the widespread use of BCI in voice prostheses is that this technique requires highly invasive surgery to implant electrodes into brain tissue.

The most accurate speech recognition is achieved by neural prostheses with electrodes covering large areas of the cortical surface. However, these solutions for reading brain activity are not intended for long-term use and pose significant risks to patients.

Researchers from the HSE Center for Bioelectrical Interfaces and the Moscow Medical and Dental University have created a working neuroprosthesis that can decode speech with acceptable accuracy by reading brain activity from a small set of electrodes implanted in a limited cortical area. I researched the possibility of creating. The authors suggest that in the future, it may even be possible to perform this minimally invasive procedure under local anesthesia. In the present study, researchers collected data from two epileptic patients who already had intracranial electrodes implanted for the purpose of preoperative mapping to localize the seizure onset zone.

The first patient had a total of 5 sEEG shafts implanted bilaterally with 6 contacts each, and the second patient had 9 cortical recording (ECoG) strips with 8 contacts each. rice field. Unlike ECoG, electrodes for sEEG can be implanted without a full craniotomy through a drilled hole in the skull. In this study, only her 6 contacts on one sEEG shaft in one patient and her 8 contacts on her ECoG strip in one other patient were used to decipher neural activity. bottom.

Subjects were asked to read six sentences aloud, each presented 30 to 60 times in random order. The sentence structure varied and most of the words in a sentence started with the same letter. The sentence contained a total of 26 different words. As subjects read, the electrodes recorded brain activity.

This data was then combined with the audio signal to form 27 classes, including 26 words and 1 silence class. The resulting training dataset (containing signals recorded during the first 40 minutes of the experiment) was fed into a machine learning model with a neural network-based architecture. The neural network’s learning task was to predict the next uttered word (class) based on the neural activity data that preceded the utterance.

When designing the neural network architecture, researchers wanted it to be simple, compact, and easy to interpret. They came up with a two-step architecture that first extracts internal speech representations from recorded brain activity data, generates log-mel spectral coefficients, and then predicts specific classes: words or silences.

A neural network trained in this way achieved an accuracy of 55% using only 6-channel data recorded by one sEEG electrode of the first patient and one ECoG strip of the second patient. We achieved 70% accuracy using only 8 channels of data that were analyzed. Such accuracy is comparable to that demonstrated in other studies using devices that require electrodes to be implanted across the cortical surface.

The resulting interpretable model allows us to neurophysiologically describe which neural information contributes most to the prediction of words about to be uttered. Researchers examined signals from different neuronal populations to identify signals important for downstream tasks. Their findings are consistent with their audio mapping results, suggesting that the model uses pivotal neural signals and thus can be used to decode imaginary speech.

Another advantage of this solution is that it does not require manual feature engineering. The model learned to extract speech representations directly from brain activity data. The interpretability of the results also indicates that the network decodes signals from the brain rather than from accompanying activity, such as electrical signals from articulatory muscles or produced by microphone effects.

Researchers emphasize that predictions were always based on neural activity data that preceded speech. This, they argue, allows decision rules to ensure that they are not using the auditory cortex’s response to utterances that have already been uttered.

Use of such an interface minimizes risk to the patient. If all goes well, it may be possible to decipher fictitious speech from neural activity recorded by a small number of minimally invasive electrodes implanted in outpatients under local anesthesia. “

Alexey Ossadtchi, lead author of the study and director of the Center for Bioelectric Interfaces at the HSE Institute for Cognitive Neuroscience, said: