Challenge Description
Real-time video applications have never played a more critical role in our lives as they enable us to live and work remotely while staying connected with the rest of the world. However, the rapid increase in the use of real-time video also poses an unprecedented challenge for consistently delivering high quality of experience (QoE) — such as high video and audio quality, low delay and few stalls — to all users.
A pivotal algorithm to optimize the QoE for real-time video communications is bandwidth estimation. It runs on the endpoint of a real-time video application and aims at adapting the video bitrate dynamically to stay within the available network capacity. To this end, it generally collects packet statistics from the network path and regularly computes a bandwidth estimate for the future. It then passes the estimate into a video codec as a target bitrate, requesting the codec to encode video frames in an average bitrate approximately equivalent to the target. As a result, the bandwidth estimator avoids network oversubscription by controlling the sending rate of video indirectly through the codec.
Although we have only focused on video so far, a bandwidth estimator is required to take audio into account too and deliver high quality audio. In this challenge, we call for a novel bandwidth estimation scheme implemented in the provided framework, such that it is able to attain superior overall QoE on a real-world testbed we built for real-time communications (RTC) of video and audio.
Considerations
- Relation to congestion control
From the network's perspective, bandwidth estimation for RTC effectively functions as congestion control, deciding how many packets to send and when to send them without causing congestion-induced packet delay or loss. Although bandwidth estimation and congestion control may be used interchangeably in this context, we opt for bandwidth estimation to emphasize application-level performance rather than packet-level behavior. - Video and audio quality
We adopt VMAF as the quality metric for video and DNSMOS as the quality metric for audio. Evidence has shown that they reflect the user's perceptual quality of video and audio better, respectively. We refer the reader to the above links for details. - Frame loss and delay
A straightforward approach to achieve the highest video and audio quality is to have the bandwidth estimator output substantially overestimated bandwidth. However, it will render the network congested and incur persistent packet losses. Depending on how the RTC application handles lost packets, this may further lead to either a high frame drop rate or long frame delays, eventually resulting in poor QoE. - Coarse-grained rate control
Although the bandwidth estimator effectively performs rate control via the video codec, today's widely used codecs are not fine-grained enough to control the exact output bitrate due to their own technical artifacts. In other words, given a target bitrate suggested by the bandwidth estimator, the video encoder might undershoot or overshoot its target. Accordingly, all bandwidth estimation schemes proposed in this challenge will need to tolerate this coarse-grained rate control. - "Unpredictable" bandwidth
Real-world bandwidth often fluctuates over time, especially on cellular networks, and might appear "unpredictable," i.e., the bandwidth does not exhibit high regularity. It is worth noting that even so, an optimal bandwidth estimator always exists — given a particular QoE function, it is simply the one that empirically attains the highest QoE over the specified network paths.
Task, Dataset and Test Environment
This challenge invites you to design and implement a bandwidth estimator on the receiver side of a provided RTC system named AlphaRTC. AlphaRTC is a fork of Google's WebRTC codebase, except that to facilitate your development, we have replaced WebRTC's default bandwidth estimator with a Python file template, where you are only required to implement two placeholder functions as described in our documentation. During evaluation, the completed Python file will be executed in a runtime environment we created. Also see an example workflow for your submission.
In order to create a customized bandwidth estimator, you may leverage any available dataset such as network traces, as well as any simulated or real-world environments if applicable. In the meantime, we provide a "gym" — a simulation environment based on ns-3 — for training ML-based bandwidth estimation algorithms. We will also release a limited number of network traces collected in the test environment as examples; stay tuned for further updates on OpenNetLab’s website.
OpenNetLab is a real-world testbed we built for RTC, consisting of a handful of real network nodes. It serves as the ultimate test environment of this challenge. During the final evaluation, we will have the testbed nodes make automated video calls to each other using AlphaRTC and then rank the evaluated bandwidth estimators according to the criteria described below.
Evaluation Criteria
The sole performance indicator of our interest is the user's QoE. Although the exact QoE function is still to be finalized, it will comprise the following metrics:
- Perceptual video quality metric based on the VMAF model
- Perceptual audio quality metric based on the DNSMOS model
- Average frame drop rate
- Average frame delay
- Variance of frame delay
- Network performance metrics (e.g., packet loss rate)
Future updates and more details will be posted on OpenNetLab’s website.
Important Dates
- Submissions and notifications
- Presentations and demos: During the MMSys week
Submission Guidelines
Submissions should provide enough details for the implemented algorithm in a short technical paper, prepared according to the guidelines provided for the Open Dataset and Software Track. A link to the code (preferably a GitHub project) must also be included in the paper. Complete your submission here.
The authors will be notified after a review process and the authors of the accepted papers need to prepare a camera-ready version, so that their papers can be published by ACM DL.
Winners are to be chosen by a committee appointed by the challenge organizers and results will be final. If contributions of sufficient quality are not received, then some or all of the awards may not be granted. The challenge is open to any individual, commercial or academic institution.
Awards
The winner will be awarded 5,000 USD and the runner-up will be awarded 2,500 USD.
Questions
If you have any questions, please post them in the Google Group of the challenge or send an email to opennetlab-challenge@googlegroups.com.