Sign-Speak’s AI sign language translator is transforming accessibility
Sign-Speak has created an AI platform that bidirectionally translates American Sign Language into English in real time.
Sign-Speak, an AI startup co-founded by Yamillet Payano and Nikolas Kelly, is working to close communication gaps for deaf and hard-of-hearing individuals. The platform utilises AI to bidirectionally translate between American Sign Language (ASL) and spoken English in real-time.
During a live demo at AWS re:Invent, the platform showcased its translation capabilities. Kelly used the system to communicate seamlessly – his ASL-signed messages were converted into spoken English for the audience and spoken responses were translated back into ASL.
Sign languages are not the same as English
Earlier this year Elon Musk tweeted:
“What’s the point of sign language in a video if you have subtitles? Am I missing something?”
This was in response to a video that contained both an ASL as well as captions.
One of the challenges Sign-Speak tackles is the misconception that sign languages like ASL and Auslan are signed versions of spoken languages.
“Sign Language and English are not the same language. They have two different grammatical structures,” Payano said during the presentation.
“In English, we would say, ‘I’m going to the store to buy bread,’ and in sign language, you sign, ‘I go store buy bread.’”
Payano emphasised that sign languages incorporate elements like facial expressions and body language to convey context and meaning, which AI systems must account for to ensure accurate translation.
“It’s so much more than just the hands moving. It’s the face. It’s a lot of conviction,” Payano said.
How the Sign-Speak technology works
The foundation of Sign-Speak’s technology lies in its multimodal approach to understanding sign language.
Using cameras, the system captures the movement of hands, arms, and facial expressions in real-time. This allows it to process not only individual signs but also the context in which they are used.
This data is fed into advanced AI models trained on one of the largest datasets of ASL, enabling the system to interpret the unique grammar and structure of the language.
For translating spoken words into sign language, Sign-Speak employs an AI-generated avatar that performs human-like signing gestures.
The avatar ensures accessibility for deaf users who rely on visual communication.
Payano highlighted the use of community-driven feedback to improve the system’s accuracy.
“A lot of our R&D time was spent talking to the community and getting them to partake in our data collection and making sure that they felt safe and they trusted our brand to give us that kind of data,” Payano said.
“We then rebuilt our infrastructure in AWS, because we didn’t have that much resource as any startup, so we had to really stay lean.”
According to the company, the more you use the platform the more comfortable it becomes with an individual’s way of signing resulting in a more tailored and personalised service.
This is particularly important when it comes to considerations like slang and new uses for words in younger generations.
For example, there is a big difference between the traditional use of the term “slay” and how it is used colloquially in 2024. Sign-Speak wants to accommodate these linguistic evolutions.
Social media push for deaf content creators
Sign-Speak is also exploring how its tools handle slang and emerging vocabulary, which present unique challenges for sign language translation.
“If we don’t have a word, we tend to fingerspell it. We’ve tried very hard to reduce that fingerspelling, because the deaf consumer doesn’t just want to be seeing fingerspelling,” Payano said.
Social media has been a proving ground for the company’s innovations. CaptionASL, one of Sign-Speak’s features, allows users to sign videos that are then automatically captioned.
“We created this quick and dirty sign language captioning system where the deaf content creator can sign, and their words would pop up on the video,” Payano said.
According to Sign-Speak, these creators now don’t have to spend hours editing short videos.
“Many deaf content creators spend hours slicing and typing captions manually. Many of them don’t feel comfortable with English because it’s their second language,” Payano said. “Now they have a system that automatically does it.”
This feature has already been tested by over 1,000 users, with requests pouring in for integrations with major platforms like Instagram and TikTok.
Beyond ASL: Expanding Sign-Speak to Auslan and other languages
Sign-Speak’s mission extends beyond ASL, with plans to expand to other sign languages, including Auslan, which is used in Australia.
While ASL has its own distinct grammar and cultural nuances, Auslan and other regional sign languages introduce additional complexities, such as incorporating inflections, local slang and culturally specific expressions.
“Historically there’s not been a lot of research in ASL linguistics because people did not view ASL as a language,” Kelly said.
“So when we go to other countries, we need to look at their language and work with knowledgeable people in that field… and understand their language right now.”
Payano added that Sign-Speak wants to ensure that if they were to expand into other sign languages, it would positively impact deaf and hard-of-hearing people in those countries.
“So right now, we are establishing connections in other countries to see how data collection and coordination can go.”
The founders acknowledged the scale of the challenge, noting that many sign languages lack sufficient datasets.
“We spent nine years building this, about six of that was collecting data. We have one of the largest datasets in American Sign Language. So it is very intensive,” Payano said.
Kelly also noted the community wants to see this happen and has been supportive as well as trusting of Sign-Speak with their data. In turn, that has allowed the team to create an accurate model for ASL.
“A similar process that we have repeated in the US will have to be repeated in other countries. Now that we have learned from our mistakes, hopefully we could do it a little faster,” Payano said.
The author travelled to re:Invent in Las Vegas as a guest of AWS.
Never miss a story: sign up to SmartCompany’s free daily newsletter and find our best stories on LinkedIn.
Comments