CMU, Meta announce research collaboration aimed at making computer-based tasks and gaming accessible to people with different motor abilities via wearable sensing technology

PITTSBURGH – As part of a larger commitment to developing equitable technology, Carnegie Mellon University and Meta announce a collaborative project to make computer-based tasks accessible to more people. This project focuses on using wearable sensing technology to enable people with different motor abilities to perform everyday tasks and enjoy gaming in digital and mixed reality environments.

Meta’s research in electromyography uses sensors placed on the skin to measure the electrical signals the user generates through muscles in their wrist, which are translated into input signals for various devices. While Meta has already demonstrated that this technology could replace keyboards and joysticks, the team continues to invest and support different projects to confirm that this technology can be used by a wide range of people.

Douglas Weber, a professor in the Department of Mechanical Engineering and the Neuroscience Institute at Carnegie Mellon University, has shown previously that people with complete hand paralysis retain the ability to control muscles in their forearm, even muscles that are too weak to produce movement. His team found that some individuals with spinal cord injury still exhibit unique muscle activity patterns when attempting to move specific fingers, which could be used for human computer interactions.

“This research evaluates bypassing physical motion and relying instead on muscle signals. If successful, this approach could make computers and other digital devices more accessible for people with physical disabilities,” said Weber.

Working with Meta, Weber’s team seeks to build upon their initial results to assess whether and to what extent people with spinal cord injury can interact with digital devices, such as computers and mixed reality systems by using Meta’s surface electromyography (sEMG) research prototype and related software.

The project centers on interactive computing tasks. Approved by the Institutional Review Board, study participants begin by performing a series of adaptive mini games. Once their proficiency is benchmarked, the CMU team creates new games and other activities in mixed reality that are tailored to the abilities and interests of the participant.

“In the digital world, people with full or limited physical ability can be empowered to act virtually, using signals from their motor system,” explained Dailyn Despradel Rumaldo, PhD candidate at Carnegie Mellon University. “In the case of mixed reality technology, we are creating simulated environments where users interact with objects and other users, regardless of motor abilities.”

The project comes as an ongoing research investment by Meta to support the development of equitable and accessible interfaces to help people do more, together.

###

About the College of Engineering: The College of Engineering at Carnegie Mellon University is a top-ranked, engineering college that is known for our intentional focus on cross-disciplinary collaboration in research. The College is well known for working on problems of both scientific and practical importance. Our acclaimed faculty have a focus on innovation management and engineering to yield transformative results that will drive the intellectual and economic vitality of our community, nation and world. The College offers graduate and undergraduate degree programs in biomedical engineering, chemical engineering, civil and environmental engineering, electrical and computer engineering, engineering and public policy, information networking, materials science and engineering and mechanical engineering. Our “maker” culture is ingrained in all that we do, leading to novel approaches and transformative results.