Source-linked AI summary
MAV-C: A Framework for the Joint Objective Estimation of Audio-Visual Complexity in Immersive Virtual Environments
Luca Resti, Amelia Gully, Michael McLoughlin, Gavin Kearney, Alena Denisova
TL;DR
Existing objective metrics focus on signal distortion, and no framework jointly models audio-visual complexity over time for video. MAV-C combines entropy-based audio and visual features through motion-mediated fusion; synthetic and gameplay results support modality separation and show that visual complexity varies across gameplay categories while audio complexity does not.
Problem
Existing objective quality metrics often focus on compression or transmission distortions, while video lacks a framework jointly modeling audio and visual complexity over time.
Source: §1 · pp. 1–2Method
MAV-C combines entropy-based audio features with visual complexity features mediated by temporal motion to estimate audio-visual complexity.
Source: §1 · p. 2Results
On SYN, modality sub-scores have CoV < 0.003; on GAM, CV varies by category (KW p = 0.021) while CA does not (p = 0.437), and CA/CV correlation is r = 0.03.
Source: §5 · p. 7Takeaways & Limitations
Sensitivity analysis identifies wa and ρv as primary parameter levers, with subjective calibration planned for future work.
Source: §5 · p. 7, §6 · p. 8MAV-C has not been subjectively validated, and GAM contains only violent videogame scenes, limiting the scope of its observed parameter hierarchy.
Source: §6 · p. 8
Abstract
from arXiv · showhide
We present the Motion-Aware Audio-Visual Complexity Metric (MAV-C), a reference-free framework for the joint objective estimation of audio-visual complexity. The metric combines entropy-based audio features (temporal, spectral, and spatial) with visual features (Sobel gradient magnitude, chromatic uniqueness, and optical flow) via a parametric fusion stage, producing a continuous joint complexity score CAV (t) [0,1]. We validate MAV-C on two datasets: a controlled synthetic corpus (SYN) of stimuli with known signal characteristics and a naturalistic gameplay corpus (GAM) of 60 clips drawn from the SAFEPLAY-X dataset. On SYN, the metric exhibits strong validity: the audio score CA and visual score CV are each insensitive to changes in the opposite modality (CoV < 0.003), the joint score CAV spans [0.00,0.90] across all parameter combinations, and single-axis feature sweeps produce monotone trajectories (Spearman up to 0.995). On GAM, CV differs significantly across content categories (Kruskal-Wallis p = 0.021) while CA does not, and the two sub-scores are uncorrelated (r = 0.03), confirming they operate on independent signal dimensions. OFAT sensitivity analysis identifies a two-tier parameter hierarchy, with modality balance (wa) and visual regularization (v) as most significant tunable parameters. Full subjective calibration is planned as future work.
1 Introduction
Audio-visual complexity requires joint assessment because perceptual impact depends on both modalities, while existing objective metrics often measure signal distortion rather than content demand. MAV-C addresses the lack of a framework for video by combining audio and visual complexity features with motion, though subjective validation awaits a joint rated dataset.
1 Introduction
- Many multimedia applications depend on the joint interaction of audio and visual signals, but distortion-focused metrics can miss content demand in locally rendered environments.Dense visual signals and multisource acoustics can saturate cognitive capacity and produce cross-modal interference.Source: §1 · p. 1
- Prior objective complexity features address modalities separately: audio work uses spectro-temporal and spatial sound properties, while visual work uses structural and chromatic variety.Existing subjective datasets and derived models focus on a single modality.Source: §1 · pp. 1–2
- Video complexity lacks an annotated perceptual dataset and a framework that jointly models audio and visual complexity over time.The gap is especially relevant to video games, where audio and visual scene content jointly influence perception [4].Source: §1 · p. 2
- MAV-C is a reference-free metric that combines entropy-based audio features with visual complexity features mediated by temporal motion.Its component weights are designed for calibration against human complexity ratings, but subjective validation is deferred because no joint audiovisual dataset with video ratings exists.Source: §1 · p. 2
2 The MAV-C Architecture
MAV-C estimates synchronized audio-visual complexity through parallel modality pipelines, compresses extracted features, forms separate modality scores, and fuses them into a continuous joint score. Its design combines entropy-based audio features, visual structure, color and motion features, and tunable fusion choices.
2 The MAV-C Architecture
- MAV-C produces a continuous joint complexity score from synchronized audio-visual stimuli through feature extraction, logarithmic range compression, and late fusion.The score is normalized to [0, 1] using feature-specific physical upper bounds rather than dataset statistics.Source: §2 · p. 2
2.1 Feature Extraction
- Audio features capture temporal, spectral, and optional spatial entropy, while visual features represent structure, color uniqueness, and optical-flow motion.Audio and visual features are aligned on a shared time grid; video features are merged onto the 100 Hz audio grid.Source: §2.1 · pp. 2–3
2.2 Logarithmic Range Compression
- Log-Theoretical Scaling maps raw features into [0, 1]; γj = 10 is fixed for most features, while optical flow uses a separate γflow.The separate flow curvature spreads its low-valued, frame-diagonal-normalized motion range across the output interval.Source: §2.2 · pp. 3–4
2.3 Modality Sub-Scores
- Audio complexity aggregates log-scaled, amplitude-weighted entropy features, while visual complexity combines MSG and MUC scores and scales spatial complexity by motion.Amplitude weighting makes the audio score zero for silent content; feature weights use within-clip variance with regularization toward a uniform prior.Source: §2.3 · p. 4
2.4 Feature Fusion and Cognitive Integration
- The modality scores are fused by weighted Lp pooling, which increasingly favors the dominant modality as p approaches infinity, then temporally integrated with a causal EWMA filter.The EWMA coefficient depends on the sampling interval and a time constant representing perceptual integration time.Source: §2.4 · pp. 4–5
3 Stimuli
The study uses controlled synthetic stimuli to verify metric behavior and naturalistic gameplay clips for real-world evaluation. The two datasets cover distinct audio-visual conditions and content settings.
SYN stimuli
- SYN comprises 18 audiovisual stimuli, each 15 seconds at 30 fps and 1280×720 with 48 kHz audio, spanning floor/ceiling anchors, a midpoint, boundary cases, and feature sweeps.The stimuli were processed in spatial mode with K = 3, including SASE, and all frames sampled.Source: §3 · p. 5
- SYN varies audio spectral content, temporal density, and spatial distribution, alongside video texture density and motion speed.Audio spatial conditions varied source count from 1 to 64 and angular velocity from 0° to 90°/s; video texture blocks ranged from 1 to 64 pixels and motion from 0 to 15 pixels per frame.Source: §3 · p. 5
GAM stimuli
- GAM contains 60 five-second SAFEPLAY-X gameplay clips, evenly split among threat, assault, and lethal narrative categories, processed in stereo because trajectory metadata were unavailable.The category labels describe narrative content type and do not imply a perceptual-complexity ordering.Source: §3 · p. 5
4 Results
MAV-C was evaluated with one-at-a-time parameter sweeps on synthetic stimuli and with gameplay clips. The synthetic tests show modality isolation and monotone responses, while gameplay results show distinct visual and audio patterns across content categories.
4.1 Default Parameters and Sensitivity
- The seven tunable fusion parameters were initialized to stated midpoints, and OFAT sweeps varied one parameter at a time while holding the others at default values.Figure 2 shows primary-fusion, temporal-integration, and adaptive-weight-regularization panels, with full-range and IQR bands across 18 SYN conditions.Source: §4.1 · p. 5, Figure 2 · p. 6
4.2 Synthetic Dataset
- Across SYN conditions, CAV spanned [0.00, 0.90], and single-axis sweeps produced monotone trajectories for texture (ρ = 0.995), audio (ρ = 0.899), and motion (ρ = 0.896).Each reported Spearman correlation had p < 0.001.Source: §4.2 · p. 6
- SYN independence tests found CoV < 0.003 and less than 0.010 variation in the opposite-modality sub-score; the joint condition reached a higher CAV ceiling than either unimodal condition.For SASE boundary conditions, CA rose from 0.624 for one localized source to 0.786 for a fully diffuse 64-source field.Source: §4.2 · p. 6
4.3 Gameplay Dataset
- At default parameters, GAM CAV values spanned [0.35, 0.68] (median 0.60), within the SYN anchor interval [0.19, 0.73] and neither saturated nor floored.Across all parameter combinations, GAM CAV spanned [0.00, 0.94].Source: §4.3 · p. 6
- On GAM, CA and CV were uncorrelated (r = 0.03), while CV differed across threat, assault, and lethal categories (H = 7.70, p = 0.021) and CA did not (p = 0.437).The reported pattern accompanies increasing optical flow and texture density with content intensity while the audio mix remains broadly constant.Source: §4.3 · pp. 6–7
5 Discussion
SYN tests support modality isolation and monotonic responses, while GAM shows that visual complexity varies across content categories independently of audio complexity. Sensitivity analysis identifies the parameters with the greatest influence and underscores the need for perceptual calibration.
5 Discussion
- On SYN, modality sub-scores are isolated (CoV < 0.003) and respond monotonically to targeted feature changes (Spearman ρ ≥0.88).These results are reported as confirmation that the architecture functions as intended.Source: §5 · p. 7
- On GAM, CV differs across categories (KW p = 0.021) while CA does not (p = 0.437), and their correlation is near zero (r = 0.03).The figure reports the per-clip CA–CV relationship across 60 clips (r = 0.03, p=0.824).Source: §5 · p. 7, Figure 4 · p. 7
- OFAT identifies wa and ρv as primary levers (spans: SYN 0.33 and 0.17; GAM 0.35 and 0.15), while τ has negligible effect on both (<0.01).The reported GAM spans for p, αv, and γflow are 0.09, 0.03, and 0.01, respectively.Source: §5 · p. 7
- αa spans 0.06 on SYN but only 0.01 on GAM, with stronger influence on stimuli featuring extreme spectral variation.Source: §5 · p. 7
- Across GAM parameter combinations, CAV spans [0.00, 0.94]; the equal-weight default wa = 0.5 is a conservative starting point pending subjective ratings.The appropriate modality balance is described as varying with content type, task, and observer.Source: §5 · p. 7
6 Limitations and Future Work
MAV-C has not been subjectively validated, and its evaluation and implementation have practical scope constraints. Future work includes obtaining perceptual ratings, broadening corpus coverage, and assessing alternatives to computationally intensive dense optical flow.
6 Limitations and Future Work
- MAV-C has no subjective validation because no joint audiovisual complexity dataset with video and perceptual ratings currently exists.A study to obtain such a dataset and tune parameters, including γj, is planned.Source: §6 · p. 8
- GAM contains only violent videogame scenes, so the observed parameter hierarchy may shift for other content types.Source: §6 · p. 8
- The spatial audio mode requires directional source metadata, which may be unavailable in production pipelines.Approximating source trajectories is proposed as a way to broaden applicability.Source: §6 · p. 8
- Dense optical flow is computationally intensive and may miss high-level motion salience, motivating evaluation of lightweight alternatives.Source: §6 · p. 8