Try Live Demo

Live Caption automatically captions speech on your device. You can use it on media like live streams, videos, podcasts, phone calls, video calls, and audio messages. Even in some usage scenarios, Live captioning is a must.

Live captioning is an accessibility feature for the over 466 million people around the world who are deaf or hard of hearing. It is a feature that helps us a lot not only in these situations but also in our normal lives. It is a very useful solution that allows you to watch videos or listen to podcasts, for example, while on a noisy journey or trying not to wake a baby.

This article will help you understand how to implement live captioning for a live video stream using the Ant Media WebRTC server and Amazon Transcribe. You can download the demo project source code from here.

Prerequisite live captioning setup

1. Create AWS Key and Secret

Please follow the instructions provided here to generate the AWS key and secret.

2. Create AWS Transcribe Key

Please follow the instructions provided here to generate AWS transcribe key.

3. Setting up the AWS Transcribe with Web-socket

Set up transcribe streaming permission in your AWS account using the below sample config. In order to learn more about configuring WebSocket with AWS transcribe service please follow this AWS documentation.

{
 "Version": "2012-10-17",
 "Statement": [
 {
 "Sid": "transcribestreaming",
 "Effect": "Allow",
 "Action":"transcribe:StartStreamTranscriptionWebSocket",
 "Resource": "*"
 }
 ]
}

Clone the sample project, setup, and run

Clone project from Git repository:

https://github.com/5exceptions/Live-Captioning-demo-with-Ant-Media-and-AWS-transcribe

2. Install dependencies in the root Folder

Run the below command to install dependencies in the root folder

npm i

3. Installing the AWS SDK for JavaScript on the root folder

npm install aws-sdk

4. Add AWS credentials

Navigate to file src\main\webapp\lib\main.js and add AWS Transcribe key and secret as shown in the below screenshot

function createPresignedUrl() {
let endpoint = "transcribestreaming." + region + ".amazonaws.com:8443";// get a preauthenticated URL that we can use to establish our WebSocket
return v4.createPresignedURL(
'GET',
endpoint,
'/stream-transcription-websocket',
'transcribe',
crypto.createHash('sha256').update('', 'utf8').digest('hex'), {
'key':'AWS Access KeyId',
'secret': 'AWS Secret Key',
'protocol': 'wss',
'expires': 15,
'region': region,
'query': "language-code=" + languageCode + "&media-encoding=pcm&sample-rate=" + sampleRate +"&show-speaker-label=true"
}
);
}

5. Add WebSocket credentials

Navigate to the following files and replace the websocketURL variable with your WebSocket URL i.e. wss://server.com:5443/WebRTCAppEE/websocket

src\main\webapp\index.htmlsrc\main\webapp\play.htmlvar websocketURL = “web socket URL”;

6. Generate transcribe build and run

npm run-script build

Example live captioning screenshots

Take a look at some of the live captions captured:

Publisher

live caption setup

An embedded player with the live caption

Live Captioning with Ant Media Server

This file is located on the below location

/src/main/webapp/dist/main.js

This file functionalities are described below:

  1. Start / Stop Publishing and captioning.
0*Mmf1hzARCTtE5oHI

Start Publishing Button

When you click on the start publishing button on the publishing demo, it will start streaming and also start a speech-to-text conversion from AWS Transcribe. It will trigger the #start_publish_button on the main.js file, when we click on the start publishing button in the publisher end, it will start a speech to text.

$('#start_publish_button').click(function () {
    $('#error').hide(); // hide any existing errors
    toggleStartStop(true); // disable start and enable stop button
    // set the language and region from the dropdowns
    setLanguage();
    setRegion();
    const video = document.getElementById('localVideo');
    const astream =video.captureStream();
    streamAudioToWebSocket(astream)
 });

Stop Publishing Button

As we see a button with the name stop publishing is shown, as disabled on the publishing demo page. It will be active after clicking the start publishing button, then the stop publishing button is now enabled. Click on the button, it will stop video streaming and also stop AWS from transcribing speech to text.

$('#stop_publish_button').click(function () {
    closeSocket();
});

2. Set Language

Set your language in the main.js file. Find the set language function and set your languageCode according to your caption’s language. By default, we set en-US(English, US) language.

function setLanguage() {
    //languageCode = $('#language').find(':selected').val();
    languageCode = 'en-US';
    if (languageCode == "en-US" || languageCode == "es-US")
        sampleRate = 44100;
    else
        sampleRate = 8000;
}

There are many ways to integrate this example into your current project but we are explaining the Iframe way.

For integrating the Publisher page in your project add an iframe tag on the page and use the index.html file URL in the src attribute in the iframe as shown below.

<iframe src=”http://{{server.com}}/stream/Live-Captioning-demo-with-Ant-Media-and-AWS-transcribe/src/main/webapp/" title=”captions demo” style=”width:100%;height:100%;”></iframe>
0*g3rDAUBjWzh58R83

For integrating the Player page in your project add an iframe tag on the page with the blank src, need to update the src dynamically as shown below:

<iframe id=”streamIdurl” src=”” title=”captions player demo” style=”width:100%;height:100%;”></iframe><script>
document.getElementById(“streamIdurl”).src = “http://{{server.com}}/stream/Live-Captioning-demo-with-Ant-Media-and-AWS-transcribe/src/main/webapp/play.html"+window.location.search;</script>src/main/webapp/iframe-player.html?id=stream122
0*JD8nU58502Q agih
That’s how easy it is to add live captioning to your project. While this feature makes the lives of more than 466 million people easier, it provides a competitive advantage, especially to companies in the streaming sector. As we mentioned at the beginning, live captioning has become a mandatory feature in some usage scenarios.

Use Ant Media Server for free for 100 hours every month

You don’t need to pay anything in advance. Just get started and only pay what you use. Ant Media provides 100 hours of free usage every month and you’ll have Ant Media Server WebRTC SDKs, and everything by signing up with zero upfront cost. In addition to the live captioning feature, you will have access to all the features.

You can start for fee now!

Useful links for streaming heroes:

How to Enable SSL?
Developer Community

Check out some of our success stories

Free RTMP Server Installation
WebRTC Guide
RTMP Guide
RTSP Guide
HLS Guide
CMAF Guide
SRT Guide
Android SDK
iOS SDK
Flutter SDK
React Native SDK
Unity SDK
WebRTC to HLS & DASH  
IP Camera Streaming Made Easy with Ant Media Server
How to Set up an IP camera
Ultra-Low Latency Streaming Uses Cases

 

Categories: Tutorial

Hamit Demir

As a B2B tech marketer, Hamit Demir works as a solution expert at Ant Media. He has experience in SEO, Demand Generation, Paid Search & Paid Social, and Content Marketing. He spends his free time learning new things. He loves to talk about streaming and especially WebRTC.

0 Comments

Leave a Reply

Avatar placeholder

Your email address will not be published.