This proof-of-concept study from UCSF demonstrates a brain-computer interface that decodes combined speech-and-gesture signals to animate full-body virtual avatars. Using ECoG sensors and machine-learning decoders, three implanted participants conveyed synchronized words and upper-body gestures to observers via on-screen avatars. Researchers say the approach better captures natural communication and aim to develop a fully implantable, wireless device for long-term use.
Brain Implant Lets Paralyzed People Speak and Gesture Simultaneously via Virtual Avatars
FILE - Neural activity recorded from a participant with paralysis is decoded in real time to control a personalized virtual avatar, enabling speech and gesture communication. (Chang Lab, UCSF)
A new brain-computer interface (BCI) developed by researchers at the University of California, San Francisco enables some people with paralysis to express speech and upper-body gestures at the same time by controlling full-body virtual avatars.
How It Works
The team used electrocorticography (ECoG) — a thin sensor strip placed on the motor cortex — to record neural activity associated with speaking and moving. Machine-learning decoders were trained to recognize the distinct brain patterns produced when participants intended to speak and to make upper-body gestures simultaneously. Those decoded signals were translated into synchronized speech and avatar motion displayed on a screen.
Study Details and Findings
In experiments described by the National Institutes of Health, three implanted participants produced unique combined speech-and-motion signals that the models could decode. Using the BCI, the participants successfully conveyed their intended words and gestures to two other observers via on-screen avatars, demonstrating more natural, coordinated communication than decoding speech or movement separately.
"Conversation is about much more than the words being spoken. It's a multilayered, dynamic process involving the whole motor cortex," said Edward Chang, M.D., professor of neurological surgery at UCSF. "This proof-of-concept shows it's possible for a BCI to restore some of this freedom and flexibility."
Implications and Limitations
The researchers found that the brain encodes simultaneous speech-and-gesture differently than isolated actions; training decoders on combined patterns produced more lifelike, synchronized avatar outputs. While promising, the study is an early proof-of-concept with a small sample size and used implanted, wired sensors. Real-world communication, long-term safety and usability will require further testing.
What's Next
Scientists plan to develop and test a fully implantable, wireless version of the device for long-term use, with the goal of restoring more natural, expressive communication for people with severe paralysis.
Source: National Institutes of Health news release, published Sept. 14, 2026. Reporting from San Jose.
Help us improve.
