We are starting a series of blog posts where we will share how Ant Media Server is scaled up to 100K viewers step by step. Let’s start with 10K as first step 🙂 In some use cases such as trivia quiz apps, webinars, conference room applications; you may need Ultra-Low Latency using WebRTC with a large number of viewers and publishers and therefore you may need to scale up your solution to support 10K viewers which could be achieved with Ant Media Server scaling solution.
Today I’m going to show you how easy to perform a load test with 10K viewers which is just the beginning of our new blog series. In the future, we are going to publish new blogs about our load tests supporting an increased number of viewers that will go up to 100K.
In order to monitor the test, you need to have some tools as a prerequisite. We are going to explain how to use those tools step by step. By performing these steps you can also perform the test by yourself.
What do we need?
- 1 x Load Balancer (Nginx)
- 1 x Database server (MongoDB)
- 10 x Load Servers
- 20 x Ant Media Server Edge Cluster
- 1 x Ant Media Server Origin
- 1 x Kafka Server
- 1 x Monitoring Tool
- Test Tool
Now let’s see each item in detail.
Since Nginx doesn’t require much from the system resources so we have used a server with 4 Cores. We are going to utilize Nginx for load balancing and use its round-robin mechanism to distribute the load to Ant Media Edge servers.
The configuration is as follows.
In order to make Ant Media Server to work as in cluster mode, we need to install MongoDB and apply necessary configuration. You can access the setup guide from this link.
Load servers are configured to act as the viewers eventually the load itself. All the load will be created by using these instances and we will distribute the load to edge servers using load balancer. Based on our design, load servers are configured to get the stream from Edge Servers with the bit rate of 600 KBps and 480p where the adaptive bit rate by default is off and eventually, each load server is going to act as 1000 viewers to get 1000 streams.
Ant Media Server Cluster on AWS
After finishing the installation on AWS for a single instance, we took the image of that instance and using same image, we have set up other instances all at once. In addition, we preferred to use C5.2xlarge instances. Check this post for a detailed guide about setting up scalable ultra low latency streaming cluster on AWS.
How the test tool works
We run this test tool on Load instances. We send the broadcast via ffmpeg tool, usage is as follows:
ffmpeg -re -i test.flv -codec copy -f flv rtmp://Ant_Media_Origin_IP/WebRTCAppEE/stream1
You can use the link below to download and install the test tool.
wget https://antmedia.io/webrtctest-release.zip apt update apt install unzip openjdk-8-jre unzip webrtctest-release.zip cd webrtctest-release
Test tool usage
bash run.sh -m player -n 1 -i stream1 -s nginx_load_balancer_ip -u false -k kafka_server_ip:9092
Kafka collects the information from each server and sends them to the monitoring server. Before taking the instance image on AWS, configuring Kafka will ease the work.
You can review the document for Kafka installation from here.
Kafka settings are as follows.
vim /usr/local/antmedia/conf/red5.properties server.kafka_brokers=your_kafka_server_ip:9092 service antmedia restart
We have used the most suitable monitoring tool for 10K testing, Grafana. On the backend, there is also Elasticsearch and Logstash working with Grafana.
Please refer to the this link for installation.
Based on our observation, Ant Media Servers handled the capacity 10K WebRTC viewers without any quality degradation and average CPU load was around 80% and Memory usage was around 50%.
Our next blog will be about how Ant Media Server will handle 30K, 50K and 100K. Stay tuned!
If you have any questions, please just drop a line to contact (at) antmedia.io