+
Sign up for
The Upworthiest
Our top stories of the day delivered straight to your inbox.
upworthy

hard of hearing

Identity

Someone has created a tool that makes captions for deaf people in real world conversations

Company develops an app that transcribes speech into text that follows the speaker.

Canva

Live-captioning technology is gaining ground, an exciting development for the hard of hearing.

Captions are a daily part of most people’s lives even if you don't always realize it. On social media, captions are sometimes automatically generated for videos or you can simply go into your settings to turn them on. These options are designed to be more inclusive, and people that are hard of hearing or deaf need them on videos to understand what's being said. The fact that they’re more widely available is great news.

One developer wanted to take captions a step further, into the real world. Captions on televisions or videos playing on your device's screen are great, but wouldn’t it be helpful to have the ability to turn them on while you’re having an actual conversation? A video posted online by Paul Mealy, product design leadership at Meta, shows a new augmented reality (AR) tool that could be a game changer for people who are deaf or hard of hearing. Augmented reality is something that people are accustomed to through video games like Halo, the first-person game that has a pop-up display showing information about the player's surroundings.

Like the video game, the real-time live-captioning technology uses augmented reality to display captions on top of the person speaking via wearables. The captions can even follow the speaker around and work in crowded spaces, focusing only on the identified speaker's voice. Mealy says the prototype uses directional microphones, where speech is recorded and converted to live text and attached to the person speaking via body tracking in AR. “One of the issues the deaf and hard of hearing have had to deal with during the pandemic is masks obscuring facial expressions, which often provide cues to what a person is saying," he says. "By utilizing captioning, we can help facilitate communication regardless of if the user's face is covered.” According to Mealy, the prototype runs on mobile platforms, but performs best on wearable computing devices such as Microsoft’s HaloLens and other hands-free wearable devices.

The post showing Mealy testing out the prototype has been shared several times across social media, with Pascal Bornet, the chief data officer of Aera Technology, sharing it with his more than 40,000 Twitter followers. Comments on the posts are positive in nature and filled with excitement about the potential benefits. Several commenters said the technology is “amazing” while some had handy tips like allowing the user to make the text stationary instead of staying with the person. Another advised that the text be placed at the side of the screen with an arrow prompting the user to look toward the text. The excitement from the commenters can be felt through the screen.

It’s not clear from the post when this technology will be available commercially, but it’s clear that when it is available it will give deaf and hard of hearing people much more accommodations to navigate hearing spaces. If people could learn ASL throughout their school years as a standard education course as well as having access to this new technology, the world would be so much more accessible to deaf and hard of hearing people. It's wonderful that people like Mealy are working to do their part.

From Your Site Articles
Related Articles Around the Web
closed captioning
More

Chris Pratt apologized to deaf fans online then had harsh words for Instagram itself.

In a recent video on social media, mega-celebrity Chris Pratt made what seemed like a reasonable request.

Photo by Stuart C. Wilson/Getty Images.

The "Guardians of the Galaxy" star asked his fans scrolling through their feed to turn up the volume and listen instead of, as he put it, "just reading the subtitles."

In doing so, he unintentionally alienated a ton of fans who depend on those very subtitles.

Almost 38 million adults living in America have some form of hearing loss, and implying that a subtitled video was somehow a lesser experience was, well, a bit insulting.

So Pratt took to Instagram to issue a powerful and heartfelt apology.

"I have people in my life who are hearing-impaired, and the last thing in the world I would want to do is offend them or anybody who suffers from hearing loss or any other disability," he wrote.

"So truly from the bottom of my heart I apologize. Thanks for pointing this out to me. In the future I'll try to be a little less ignorant about it."

In addition to his writing, Pratt used his hands to apologize in the 100% silent video.

Instagram does this thing where it mutes all the videos it shows and forces you to turn on the volume in order to hear them. (maybe because most people are watching those videos at work when they should be working and don't want to get caught. I know that's when I do it. 😬) So when I made a video recently with subtitles, and requested that people turn up the volume and not just "read the subtitles" it was so people wouldn't scroll past the video on mute, thus watching and digesting the information in the video. HOWEVER, I realize now doing so was incredibly insensitive to the many folks out there who depend on subtitles. More than 38 million Americans live with some sort of hearing disability. So I want to apologize. I have people in my life who are hearing-impaired, and the last thing in the world I would want to do is offend them or anybody who suffers from hearing loss or any other disability. So truly from the bottom of my heart I apologize. Thanks for pointing this out to me. In the future I'll try to be a little less ignorant about it. Now... I know some of you are going to say, "Hey! Chris only apologized because his publicist made him!" Well. That is not the case. As always I control my social media. Nobody else. And I am doing this because I'm actually really sorry. Apologies are powerful. I don't dole them out Willy-Nilly. This is one of those moments where I screwed up and here's me begging your pardon. I hope you accept my apology. And on that note. Why doesn't Instagram have some kind of technology to automatically add subtitles to its videos? Or at least the option. I did a little exploring and it seems lacking in that area. Shouldn't there be an option for closed captioning or something? I've made them lord knows how much money with my videos and pictures. Essentially sharing myself for free. I know they profit. So... GET ON IT INSTAGRAM!!! Put closed captioning on your app. #CCinstaNow

A post shared by chris pratt (@prattprattpratt) on

His willingness to reach out and make amends sparked an outpouring of support for the actor.

Though a few commenters noticed his signing wasn't completely accurate (though not bad for a newbie), hundreds of comments poured in to thank Pratt for taking the time to better educate himself.

While the apology was genuine and moving, it was perhaps the very end of Pratt's message that held the most weight.

"Why doesn't Instagram have some kind of technology to automatically add subtitles to its videos?" he asked. "Shouldn't there be an option for closed captioning or something?"

He's got a good point. While other platforms like Facebook have added auto-captioning tools to make it easy on publishers to create more accessible videos, Instagram seems to be lagging behind.

For a platform with about 700 million monthly users, Pratt doesn't think that's acceptable.

"I've made them lord knows how much money with my videos and pictures," he wrote. "Essentially sharing myself for free. I know they profit. So... GET ON IT INSTAGRAM!!! Put closed captioning on your app. #CCinstaNow"

With well over a million views on the video so far, it's obvious that Pratt's support means a lot to the Deaf and hard of hearing community.

Hopefully, having this kind of star power behind an important cause will lead to some much-needed change.

From Your Site Articles
More

These Irish girls performed a sign-language Christmas classic to prove a powerful point.

Everyone knows the best thing about Christmas is the Christmas songs.

Image via iStock.

"Jingle Bells"? Classic. "Frosty the Snowman"? Timeless. "Feliz Navidad"? You know you secretly love it.

And this holiday season is a perfect time to think about how we can make our favorite traditions accessible to everyone, even people who may not be able to hear all of those festive classics.

That's why three students from Ireland recently made a sign-language sing-a-long of Santa Claus is Coming to Town.

Abbie O'Neill, Amy Durkin, and Joanne O'Donnell, who are students at the Centre for Deaf Studies at Trinity College, donned their finest winter hats and proceeded to jam out to one of the catchiest Christmas classics ever created.

Using only their hands, of course.

The video started off as a class assignment but soon spread far and wide via social media, both inside and beyond the deaf community.

"We've got such supportive responses and messages from hearing and deaf people from all around the world," the three friends wrote in an email.

The video currently has over 200,000 views on Facebook, where hundreds of commenters have shared their admiration for the girls' efforts.

Check the video out below:

It's a month till Christmas yo!! Irish Sign Language performance of Santa Claus is Coming to Town :)

Posted by Abbie O'Neill on Friday, November 25, 2016

But this wasn't just a holly-jolly gesture of inclusion. It was part of a much bigger statement.

Ireland has a deaf and hard-of-hearing population of over 90,000 people, many of whom rely on Irish Sign Language (ISL) to communicate and live their lives. But ISL isn't currently recognized as an official language by the Republic of Ireland. That means many in the deaf community face tremendous difficulties in schools that don't cater to their needs or accessing government programs that aren't required to provide interpreters.

In a broader sense, not recognizing ISL as an official language leads to a much bigger lack of awareness and consideration for the deaf community, too. O'Neill, Durkin, and O'Donnell recall going to see a subtitled film recently. The theater staff forgot to turn on the subtitles until several minutes into the movie. After the movie, they spoke to a deaf couple in the audience about it. The couple "simply laughed — they're all used to this lack of awareness and indifference from the wider hearing community."

In a broader sense, not recognizing ISL leads to a much bigger lack of awareness and consideration for the deaf community.

But there is hope on the horizon, in the form of the Irish Sign Language Bill, sponsored by Sen. Mark Daly, that would designate ISL as a native and independent language. The bill is quickly gaining momentum.

"The Irish deaf community have been denied equal rights and opportunities for years," Daly told The Journal. "The impact of this piece of legislation would be truly transformative."

If and when the bill passes, it will be due in large part to the efforts of Ireland's deaf community in bringing more attention to the conflict.

So while this viral video might look like just an adorable sing-a-long, it's actually a lot more.

Thanks to three students, it's also part of an important mission: improving the lives of thousands and thousands of deaf Irish men and women.

"The Deaf Community in Ireland has been fighting for years to have ISL recognised," the friends wrote. "We're just happy that we could bring some awareness to it."

That alone is a heck of a gift this holiday season.

More

The cool reason you won't hear baristas shouting drink orders at this Starbucks.

A new Starbucks in Malaysia is like no other.

Mohammad Aizad Bin Ariffin just got a promotion, and it means he's "well on his way to achieving [his] dream."

He wants to become the first deaf Starbucks store manager in Malaysia, and now — as the new shift manager at a location in Kuala Lumpur — that goal is in sight.

Photo courtesy of Starbucks.

Although Ariffin — who's worked for the coffee chain for three years — certainly deserves a pat on the back for movin' on up, his deafness isn't all that unique at the Starbucks location. It's actually the norm.

10 of the 13 employees at Ariffin's store are deaf. And that's a big first for Starbucks.

The store, which just recently opened in a shopping center, is the first Starbucks in the world dedicated to hiring deaf employees.

Photo courtesy of Starbucks.

The coffee chain partnered with the Society of Interpreters for the Deaf (SID) to hire, train, and coach new workers, as well as teach sign language to employees who are not deaf or hard of hearing, according to Starbucks.

So is this Starbucks only for customers who are deaf?

Certainly not. Anyone who is deaf or hard of hearing will find the store especially accessible, but a customer who doesn't know sign language can write their order on a menu card.

Each customer is given an order number, which will flash on a screen to let them know their drinks or food is ready.

Photo by Ben Pruchnie/Getty Images.

The biggest difference between this Starbucks and others might just be the fact that you won't hear workers yelling drink orders at one another — they'll be signing instead.

The new store is part of Starbucks' mission to expand opportunities for marginalized groups.

For the deaf community — a population that faces high rates of job discrimination and unemployment — it's a cause worth fighting for.

Three out of four people living with hearing loss reported having more limited employment opportunities than their hearing peers, according to a 2014 study by U.K.-based group Action on Hearing Loss.

Figures like that show why we need far more companies following in Starbucks' footsteps.

To Ariffin, his promotion isn't just about himself, either — now, he's able to help other workers in his shoes aim higher.

"It’s an incredible feeling to share my journey and help develop other deaf [workers],” he notes.

Incredible, indeed.

Cheers on taking a big step forward toward your dream, Mohammad Aizad BinAriffin.

Photo by Ben Pruchnie/Getty Images.


Trending Stories