What Is Live Caption on Samsung Phones, and Does It Work? March 5, 2021 – Posted in: Smartphone news
Live caption was introduced in Android 10 for a few Google Pixel phones, but Samsung has the led the way by introducing it on some of its flagship devices. This feature automatically generates captions when it detects speech from your device, but it isn’t always the better alternative to existing caption services.
Let’s see how to use Live Caption, and test how well it works on podcasts, music, videos, and video or voice chats.
How to Enable Live Caption on Samsung Phones
Live Caption is available on high end Samsung devices running One UI 2.5 and above, including the Galaxy S20 and S21 ranges. To turn on Live Caption on Samsung phones:
- Go to Settings > Accessibility.
- Select Hearing enhancements.
- Select Live Caption, then toggle it on to enable it.
The feature will now automatically detect speech in various media files. So how well does it work?
Live Caption for Podcasts
Live Caption performs well for podcasts and adds accessibility to this previously audio-only platform. The only other option for hearing-impaired podcast listeners are transcripts, which aren’t available for every podcast and are usually written as a supplement rather than a substitute.
It does make some common spelling mistakes. You can see it misspell LeVar’s name in LeVar Burton Reads, and struggle to translate the beloved-but-awkward catchphrase from I Hate It But I Love It: “IHIBILI.” In most cases, the spelling errors are slight enough that the content is still simple to follow.
Where Live Caption starts to struggle more is with multiple speakers, especially if they talk over each other. When this happens, as in the Finish It! screenshot above, the live caption simply transcribes as if it were one person speaking. This can make the text seem confused. However, it isn’t an issue if you can find some podcasts to follow that feature a single speaker.
Use Live Caption With Music
Live Caption does not fully support music yet, and translates most songs as simply “(Music).” It does a little better with slow songs whose lyrics sound similar to spoken words.
But it can only handle slight differences between spoken words and sung words, like a word being made a little longer, for example. It struggles a lot more if the word is broken into distinct syllables, its pronunciation is distorted, or it is stretched by more than a second or two.
For spoken-word songs and poems, it does well so long as the musical accompaniment is quiet. When the music stands out as more than a light background, it confuses the speech detection, and the captions become more inaccurate, or default back to “(Music).”
In general, Live Caption only works when music is so quiet that it’s barely noticeable. When it does pick up the lyrics, it suffers from the same errors of any auto-generated caption service: spelling errors, homonym mistakes, and missed words. So, you’re probably better off searching for a karaoke or “with lyrics” video on YouTube, instead.
Live Caption for Videos
Streaming services like Netflix don’t use automatic captions, only closed captions. Closed captions are professionally prepared and edited, so it’s never recommended to use Live Caption on those platforms. On the other hand, YouTube defaults to its own automatically-generated captions, which Live Caption improves over in some ways.
Live Caption competes closely with YouTube’s automatically generated captions. It has the same struggles with accents, names, and homonyms. Sometimes Live Caption makes an error while YouTube’s captions get it right, as in the screenshot above, and sometimes the reverse happens.
However, Live Caption struggles more with language and translation issues because it only captures English words right now. This means that Android users who need captions in other languages must use a third-party translation app or stick with YouTube’s captions, which are offered in multiple languages.
Additionally, Live Caption always attempts to translate the foreign language into an English sound-alike word, such as turning “gracias” into “grassy is,” but YouTube does this, too.
One advantage Live Caption has is self-correction. While it may make as many errors as other auto-captions, it doesn’t necessarily keep those errors. That is, it may think someone said “Bar bee” until it hears the speaker follow that word with “dolls,” at which point it will go back and correct “bar bee” into “Barbie.”
Because of this correction, live caption is a little slower than YouTube’s captions, so you can sometimes stack them to get four lines at once.
Another advantage is the ability to easily toggle the censorship settings. YouTube defaults to hiding offensive words because its generator sometimes detects words that the speaker did not say, and since YouTube is replacing television for young kids, they need to be careful. Still, it’s inconvenient for the grown-ups to log in to YouTube Studio on a desktop device to change the setting.
Live Caption has the same chance of accidentally interpreting innocent language as offensive but like Android’s Speech-to-Text Censorship Settings, it’s easier to toggle. To access Live Caption settings, go to Settings > Accessibility > Hearing enhancements > Live Caption. There, you will see an option to turn Hide profanity on or off.
Live Captions for Video and Voice Chats
At this time, Live Caption doesn’t work for voice calls. Live Caption works by detecting speech in audio produced from the device, but it isn’t activated by speech that originates from phone or voice calling. Unfortunately, it doesn’t seem to matter whether or not the voice call is also a video call.
Built-in software within voice-chat apps are likely going to be the better bet for a long time. Speech recognition struggles against common voice chat issues such as background noise, echo, and overlapping speech.
Services made specifically with these issues in mind will likely perform much better than those made primarily for professionally-produced content like videos and podcasts.
Live Captions Have Room to Improve
As speech-to-text technology develops, software like Live Caption will hopefully improve. Just the fact that a captioning service is now included as a built-in feature of Samsung and a growing number of other Android phones is good news for accessibility.
For now, automatic captioning services like Live Caption are still a poor substitute for properly-prepared Closed Captions, and the technology will likely take a long time to improve before it can compete more closely.
In the meantime, it’s best to continue pushing for professional captioning in audio media, and use some of the already available sites for downloading subtitles for TV and movies.