Source-linked AI summary
Automated Mobile Video Objective Testing System
Eric Petajan, Jonathan Lynam, Morey Antebi, Hessam Moeini, David Lindero, Lars Ernstrom, Gyanesh Patra, Szilveszter Nadas
TL;DR
Cellular spectrum is constrained while demand and application diversity make cross-application QoE measurement difficult. The paper presents AMVOTS, which uses captured Reference and Distorted feeds for flexible VMAF-based scoring, alignment, and rolling QoE aggregation. It also describes feeding these metrics to a QoE controller to prototype radio-resource allocation.
Problem
Mobile operators need QoE measurement across diverse applications and network conditions to optimize cellular-spectrum usage, but closed-source mobile applications are difficult to measure.
Method
AMVOTS captures Reference and Distorted video, aligns their frames, computes VMAF and interaction scores, and reports rolling Total QoE to a controller.
Results
AMVOTS supports QoE scoring for closed-source mobile applications using only HDMI video capture, including interactive applications and varied timing conditions.
Takeaways & Limitations
AMVOTS enables QoE-in-the-Loop experimentation and prototyping of QoE-aware radio-resource allocation using real-time metrics.
Abstract
from arXiv · showhide
Applying QoE analysis to optimize usage of cellular spectrum is of high interest to mobile network operators. A key challenge is to be able to perform QoE measurement across very different types of apps, from DASH VoD to interactive applications such as Video Conferencing and Cloud Gaming. This paper presents AMVOTS, a QoE measurement system developed by AT&T, which is flexible enough to support a large range of application types and network conditions. We also discuss using AMVOTS as part of a closed loop to prototype QoE-aware radio resource allocation.
I. INTRODUCTION
Cellular throughput demand is growing while radio propagation limits bandwidth supply, motivating user-centric measurement of experience across applications.
- Cellular throughput demand is projected to exceed 200 exabytes per month by 2025 and approach 500 exabytes by 2030.
- Radio signal propagation limits bandwidth supply despite largely unconstrained demand, making licensed-spectrum value a key priority.
- The paper frames this user-centric objective as quantifying and improving end-user experience across applications, or “Happiness per Hertz”.
II. AMVOTS
AMVOTS is AT&T’s lab system for measuring video QoE across closed-source mobile applications and varied network conditions using reference and distorted video feeds.
- AMVOTS receives Reference and Distorted video feeds, aligns corresponding frames, and computes VMAF scores for lab-based QoE estimation.Feeds may come from saved clips or live HDMI capture from computers or mobile phones.
- Because AMVOTS requires only captured video and no application integration, it can score closed-source applications such as remote desktop streaming.A server can provide the Reference feed while a network-impaired phone client supplies the Distorted feed.
- Saved reference files support repeatable testing of applications including DASH VoD and video conferencing.For VoD, the reference can be a mezzanine file or the service’s top representation; conferencing can replay a saved camera input.
- Live HDMI Reference and Distorted captures allow interactive applications to be tested, including with human participants.File-based inputs additionally support automatic frame alignment and video corrections before scoring.
- A Dell R740 system with an AJA Corvid 44 can capture and score 1080p@60fps feed pairs, while 4K processing may fluctuate and incur delays.The system can capture up to four total feeds in more complex tests.
A. Normalizing and Correcting the Video Feeds
AMVOTS normalizes captured feeds to handle video embedded in complex interfaces and corrects overlays or processing differences before VMAF scoring.
- Cropping, scaling, rotation, and other normalization align video subwindows between Reference and Distorted feeds.Scaling is the most common correction, especially when applications use complex interfaces such as conferencing layouts.
- Overlays absent from the Reference are masked by copying content from another user-defined region, optionally with mirroring, rather than black filling.This approach yields more accurate results than black fill or ignoring the affected region.
B. Frame Alignment and Score Aggregation
AMVOTS aligns imperfect video streams with a fast binary frame-difference method, then aggregates spatial and temporal QoE into a rolling Total QoE score.
- Frames are converted to RGB24, downscaled to 160x90, binarized per color channel around each frame’s mean intensity, and compared by mismatching-pixel counts.
- Each Distorted frame is compared with a window of Reference frames when the feeds appear out of sync, with heuristics optimizing synchronized cases.
- Figure 2 provides binary per-channel comparisons and an alternating-eighths QA view for visually checking alignment correctness.
- Before alignment, successive frames are de-duplicated using MAD, while source freezes are distinguished from stalls caused by duplicated Distorted frames.Alignment addresses delayed, missing, or duplicated frames arising from rebuffering, jitter, and packet loss.
- The alignment algorithm is fast and effective, remaining accurate with very low video quality through downscaling and 1-bit color-field matching.Downscaling reduces resolution mismatches, while color-field quantization compensates for color-space distortions.
- Every second, AMVOTS computes a Total QoE score for the preceding 10 seconds as the minimum of harmonic-mean video QoE and interaction QoE.Interaction QoE incorporates temporal factors such as frame freezes and delays, and the result is sent to the QoE controller.
III. QOE-IN-THE-LOOP AND QOE-AWARE RESOURCE ALLOCATION
AMVOTS supports near-realtime QoE reporting and enables QoE-in-the-Loop experiments that connect application behavior and network controls to video quality. These measurements can prototype QoE-aware radio allocation, with prior results indicating both quality and scalability gains.
- QoE-in-the-Loop: AMVOTS reports frame alignment, timing, and VMAF data through REST, typically in batches of 30 frames for lower execution overhead.Batching improves execution overheads but adds reporting latency.
- QoE-in-the-Loop: AMVOTS enables QoE-in-the-Loop testing by dynamically changing network conditions or traffic-management parameters and measuring resulting application behavior and QoE.Examples include reconfiguring analog signal attenuation or RBS traffic-management parameters.
- QoE-Aware Resource Allocation: QoE-aware allocation can support approximately 3 times more video flows at acceptable Spatial QoE.The reported gains come largely from redirecting bandwidth away from flows already achieving near-perfect quality, such as VMAF 95+, toward flows needing more resources.
- QoE-Aware Resource Allocation: Real-time QoE metrics can feed a QoE Controller that intelligently allocates radio resources across video flows in an RAN testbed.The controller can modify traffic-management parameters at the RBS; production deployment would require QoE reporting from applications or phone chipsets.
IV. CONCLUSION
AMVOTS provides AT&T with a laboratory system for measuring mobile video QoE, including closed-source applications through HDMI capture and full-reference VMAF. The authors position such tools as valuable for operator and vendor decisions aimed at increasing end-user value as video and QoE awareness grows.
- IV. CONCLUSION: AMVOTS measures mobile video QoE in laboratory radio-network tests, including closed-source applications using only HDMI video output from the phone.It uses VMAF, a full-reference pixel-based spatial QoE model.
- IV. CONCLUSION: Tools like AMVOTS can help vendors and operators choose products and network approaches that bring more value to end users.The authors connect this role to growing operator awareness of video and QoE.