Webrtc video streaming with augmented reality using DeepAR

DeepAR Integration

Augmented Reality (AR) technology has the potential to revolutionize the way we interact with the world around us. By overlaying digital content onto the physical environment, AR allows us to see and interact with information in a whole new way and can enhance our experiences using technology such as webrtc video streaming.

The DeepAR JavaScript SDK is a powerful tool that allows developers to easily add AR capabilities to their projects. With just a few lines of code, developers can create immersive and engaging AR experiences for their users.

We have integrated DeepAR into Ant Media Server so that we can provide more realistic and engaging experiences for users. By leveraging DeepAR’s SDK, we have implemented advanced facial AR filters and enabled interactive experiences.

How it Works ?

The DeepAR libraries make use of web assembly technology, a low-level programming language that is designed to be fast and efficient, to perform various media processing tasks, such as applying different video filters and effects to video streams in real-time.

Additionally, the use of web assembly technology enables the libraries to perform these tasks quickly and efficiently, without consuming large amounts of system resources.

Getting an API key:

1. Signup to DeepAR by navigating to https://developer.deepar.ai/signup

2 Click on create a project

3. Enter a project name (anything)

4. enter the domain name where your ant media server is hosted (Important)

5.Click on copy SDK key

Now we have an API key for the DeepAR SDK, we can use this key in our Ant Media Server.

Using DeepAR API Key in Antmedia Server

Publish video to AMS with DeepAR Effect

Navigate to the following Ant Media Server URL and add the copied API key as the value for the APIkey parametre:

https://domain:5443:LiveApp/publish_webrtc_deep_ar_effects_frame.html?Apikey=REPLACE_YOUR_API_KEY_HERE

With DeepAR now enabled, a drop down menu is available so you can select different kinds of effects. Select an effect and click on the publish button.

How to Use DeepAR with JS SDK

The full code example is available here

Get the API key for DeepAR SDK as described in the above steps under the getting API key section
If required, you can fetch all the available effects using VideoEffect.deepARModelList
After receiving all the available effects, you can apply an effect by calling the enableEffect method on your webrtcAdapter and passing the API key that you generated from the previous section.

The full method call should look like this:

webRTCAdaptor.enableEffect(VideoEffect.DEEPAR, "APIKey", VideoEffect.deepARModelList[0])

It expects an API key and the name of the effect. In the example, the first effect from the effect list is being used
```
VideoEffect.deepARModelList[0].
```

set the path of the direcotry which contains teh deepar files

VideoEffect.DEEP_AR_EFFECTS_URL=“/WebRTCAppEE/js/external/deepar-effects/”;

It is also possible to list all the available effects by using the JavaScript console.log method:

console.log(VideoEffect.deepARModelList[0])

Full code example

<!DOCTYPE html>
<html lang=“en”>
<head>
<meta charset=“UTF-8”>
<meta http-equiv=“X-UA-Compatible” content=“IE=edge”>
<meta name=“viewport” content=“width=device-width, initial-scale=1.0”>
<title>Document</title>
<script type=“text/javascript” src=“https://cdn.jsdelivr.net/npm/deepar@4.0.3/js/deepar.js”></script>
</head>
<video id=“myvideo” autoplay muted></video>
<body>
Select Effects :
<select id=“select_effect”>
<option>No Effect</option>
</select>
<script type=“module”>
import { WebRTCAdaptor } from ‘./js/webrtc_adaptor.js’;
import {VideoEffect} from ‘./js/video-effect.js’;
VideoEffect.DEEP_AR_EFFECTS_URL=“/WebRTCAppEE/js/external/deepar-effects/”;
const webRTCAdaptor = new WebRTCAdaptor({
websocket_url: “ws://localhost:5080/WebRTCAppEE/websocket”,
mediaConstraints: {
video: true,
audio: true,
},
peerconnection_config: {
‘iceServers’: [{‘urls’: ‘stun:stun1.l.google.com:19302’}]
},
sdp_constraints: {
OfferToReceiveAudio : false,
OfferToReceiveVideo : false,
},
localVideoId: “myvideo”, // <video id=”id-of-video-element” autoplay muted></video>
bandwidth: 25000, // default is 900 kbps, string can be ‘unlimited’
dataChannelEnabled: true|false, // enable or disable data channel
callback: (info, obj) => {
if(info==“initialized”){
webRTCAdaptor.publish(“stream1”, null, null, null, “abcd”);
}
}, // check info callbacks bellow
callbackError: function(error, message) {
}, // check error callbacks bellow
});
let select_effect = document.getElementById(“select_effect”);
let APIKey = “Enter Your API KEY HERE”;
window.a=APIKey;
var effectName = VideoEffect.NO_EFFECT;
for (let i = 0; i < VideoEffect.deepARModelList.length; i++) {
let option = document.createElement(“option”);
option.value = VideoEffect.deepARModelList[i];
option.text = VideoEffect.deepARModelList[i];
select_effect.appendChild(option);
}
select_effect.onchange = () => {
let selected_option = select_effect.options[select_effect.selectedIndex].text;
if( selected_option === “No Effect”){
effectName = VideoEffect.NO_EFFECT;
} else {
effectName = VideoEffect.DEEPAR;
}
console.log(“effect changed”, selected_option);
webRTCAdaptor.enableEffect(effectName, APIKey, selected_option).then(() => {
console.log(“Effect: “+ effectName+” is enabled”);
}).catch(err => {
console.error(“Effect: “+ effectName+” is not enabled. Error is “ + err);
alert(err.name);
});
}
</script>
</body>
</html>

Using your own .deepar Files

If you have your own DeepAR files that you want to in Integrate, please follow these steps.

1. Copy the files to your Ant Media Server at the following location:

/usr/local/antmedia/webapp/{APP_NAME}/js/external/deepar-effects

2. Append the name of the effect file in the array or you can add the file name in video-effect.js file located at /usr/local/antmedia/webapps/WebRTCAppEE/js/video-effect.js

VideoEffect.deepARModelList.push("filename")

3. Now you can use the new Effect.

Categories: Tutorial

Tags: Augmented RealityDeepARWebRTCWebRTC Video Streaming