Seamlessly Integrate Any AI Model with Ant Media Server Python Plugin

Integrating AI workflows with media streaming has never been easier! With my newly developed Python plugin for Ant Media Server, you can effortlessly integrate your Python-based AI models, whether for real-time object detection, speech recognition, or any other AI-driven media processing task. This blog post will guide you through setting up and using this plugin to enhance your streaming experience.this tutorial is targeted towards developers who have python and AI experience and want to have an easy integration with media streaming uses cases, such as conferencing, surveillance, assistance, monitoring and assistance.

Agenda

1. We will first install Python Plugin for Ant Media Server.
   • What this plugin does ?
   • When a stream is published to application name with LiveApp , this plugin will start to receive frame by frame video in a python callback looking something like this.
   • onVideoFrame(streamId , videoFrame)
   • By default the plugin draws a Rectangle on the Video and re stream the video back to Ant Media Server.
2. In This Blog post we will modify the plugin to run face detection AI model on the video stream, then the video will be annotated and the face highlighted stream will be available in WebRTCAppEE.

Prerequisites

Before diving into the building src code, make sure you have the following:

• Ant Media Should be installed on your local development environment , You can install Ant Media Server referring this document.
• Ensure Python 3.8+ & pip3 is installed on your system.

Download The Python Plugin project , install_dev_dependencies.sh file downloads the development dependencies that are required to build the src code of the project.

git clone https://github.com/ant-media/Plugins.git
git checkout addNewPythonPlugin
cd Plugins/PythonPlugin
sudo ./install_dev_dependencies.sh

Add Adaptive Bitrate

1. Go to Ant Media Server Web panel
2. Go to LiveApp applicaiton.
3. Go to Settings.
4. In Adaptive Streaming Section add new bitrate of 720p
5. Save the by clinking save button at bottom of the page
6. it should look like below screenshot.

Building From SRC

sudo ./redeploy.sh

it should now download dependencies, build the src code and start Ant Media Server on the terminal. Ant Media Server will now start make sure that there are no errors when Ant Media starts.

See the plugin in action.

1. got to sample page and publish a WebRTC stream. http://IP_ADDRESS:5080/LiveApp/
2. go to web panel in WebRTCAppEE application.
3. now you should see a new video stream in WebRTCAppEE application. (if you don’t see troubleshooting section)

You should See a Rectangle drawn on top of the Video when Playing the stream from WebRTCAppEE. (if you don’t see troubleshooting section)

Troubleshooting

1 .make sure that ant media is not running already, if its running stop the server and try redeploy command again.

 sudo systemctl status antmedia
 sudo systemctl stop antmedia

2. make sure that jna-version.jar file is preset in /usr/local/antmedia/lib/

3. make sure that libPythonWrapper.so file is present at /usr/local/antmedia/lib/native/

4. make sure that PythonPlugin.jar file is present in /usr/local/antmedia/plugins/

5. if you have ant media in a different directory other then default /usr/local/antmedia set the location of the directory in redeploy script

6. Make sure that you don’t have any adaptive bitrate in WebRTCAppEE & one 720P abr should be present in LiveApp

7. Make Sure OpenCV is build with Gstreamer support. it should say YES when running below command.

sudo python3 -c "import cv2; print(cv2.getBuildInformation())" | grep GSt

if any of the above condition does not match you should try redeploy again and see if there is any errors.

Python Plugin Apis

open libpythonWrapper.pyx file present under PythonPlugin/src/main/python

streamStarted(streamId)

Integrating Face Detection

Now we know the basic APIs Lets start modifying the src code to integrate Face Detection.We will use below code to integrate Face Detection opencv library.

        gray_image = cv2.cvtColor(rgb_image, cv2.COLOR_RGB2GRAY)
        face_classifier = cv2.CascadeClassifier(
            "haarcascade_frontalface_default.xml"
        )
        face = face_classifier.detectMultiScale(
            gray_image, scaleFactor=1.1, minNeighbors=5, minSize=(40, 40)
        )
        for (x, y, w, h) in face:
            cv2.rectangle(rgb_image, (x, y), (x + w, y + h), (0, 255, 0), 4)

As you might have guessed where should we place the above code ? you are right, it should be in onVideoFrame callback.

in libpythonWrapper.pyx under onVideoFrame function you can see the line in the code where we a are drawing a static rectangle on the Video some thing similar to below , lets replace that line with our face detection code above.

cv2.rectangle(rgb_image, (20, 20), (240, 240), (255, 0, 0), 2)

Python OpenCV library is installed by default when you install the plugin.

⚠️warning: any extra python libraries that you might need to install you should install with sudo pip install package_name , it will not work without sudo (libraries will not be visible in python plugin if sudo is not used)

Download model xml file and place it in ant media directory by default its /usr/local/antmedia

cd /usr/local/antmedia

sudo wget https://raw.githubusercontent.com/kipr/opencv/refs/heads/master/data/haarcascades/haarcascade_frontalface_default.xml

now lets rebuild the src code with the same command.

sudo ./redeploy.sh

now it will compile our changes for python code and restart Ant Media Server.

now publish a new stream with sample page to server in LiveApp and play the stream from WebRTCAppEE , like we did before and you will see the face detection in action.

Bonus

When developing the plugin, we have to run the redeploy script every time we make changes to either the Python or Java code. This script rebuilds the Python and Java source code, copies the binaries to the Ant Media installation directory, and restarts the server. Then, we have to wait for the server to initialize before we can send a video stream to LiveApp and see the modified frames in WebRTCAppEE.

Sounds like a long and tedious process, right? It really slows down development.

So, I created something I call a Plugin Simulator. What does it do? Just as the name suggests, it simulates the plugin’s behavior. Instead of going through the whole redeployment process, it takes a test.mp4 file, decodes it, and passes the frames to our Python program—essentially mimicking how the plugin would behave.

The simulator automatically calls streamStarted and onVideoFrame functions in Python and then sends the processed video to Ant Media in WebRTCAppEE. The best part? When using the Plugin Simulator, only the Python source code is recompiled when you make changes—there’s no need to recompile Java or restart Ant Media Server every time.

This significantly speeds up development and makes life much easier. 🚀

running plugin simulator.

plugin simulator is available in PythonPlugin/src/main/python/PluginSimulator to run plugin simulator cd into PluginSimulator directory and run the redeploy script.

make sure that ant media is running on you machine , sudo systemctl status antmedia

cd src/main/python/PluginSimulator
sudo ./redeploy.sh 

you should now see a new stream in WebRTCAppEE.

Conclusion

With this Python plugin, integrating AI workflows with Ant Media Server has never been easier. You can now apply real-time AI processing to your media streams, including face recognition, speech-to-text, background removal, and security monitoring & analysis.

Give it a try and let me know your thoughts! 🚀

💡 Get Involved

Check out the GitHub Repository for more information or to contribute!