By Trista Moxley
A UNT professor is working to help people who are unable to speak and with severe motor disabilities that impact their ability to sign or type communicate better.
The project started after Department of Computer Science and Engineering Assistant Professor Mark Albert was approached by an acquaintance who uses a wheelchair and is unable to speak due to cerebral palsy. She was searching for a better way to communicate, and he noticed during their conversation that she struggled to accurately touch the buttons she wanted on her tablet, even with the help of a key guard to keep her fingers from sliding off the screen. He describes it as trying to hit the right button on a tablet with one hand tied behind your back and someone constantly shaking the screen.
“The woman who originally inspired this communicates through a tablet, but it’s very slow and time consuming,” Albert says. “It’s hard for her to express emotion in conversation because she’s looking down to type and it takes some concentration. In a social context, long delays and visual distractions are very limiting. If she can simply sign custom movements, and then naturally tie them together and use that in conversation, it’s a game changer.”
Albert, along with his Ph.D. student Riyad Bin Rafiq, created a prototype of an application called TalkMotion. The app works with a wearable device like a smart watch to preprogram common phrases to match movements of the device. The wearer can then use the device in conversation rather than having to tap out messages on a tablet. The goal is to eventually allow the user to train the system using movements they can perform or tailor it to the way they prefer to move.
The prototype of the app is available in the Google store, but Albert and his lab are working to polish and refine it. Currently, the app has preprogrammed gestures and common phrases as well as an option for inputting custom gestures and phrases.
“As I talked to the woman that inspired this, you could just see the excitement on her face,” Albert says. “I always emphasize to her, these things take time, a long time. But even just the slightest possibility is exciting.”