STAR-Bench: Probing Deep Spatio-Temporal Reasoning as Audio 4D Intelligence

1 INTRODUCTION

• Prior benchmarks mainly assess coarse-grained semantics recoverable from text, whereas human auditory intelligence also draws on dynamic cues that are hard to describe linguistically.Figure 1 compares humans and Gemini 2.5 Pro with and without captions across audio benchmarks; the text gives examples such as inferring water level from pouring sounds and a vehicle’s trajectory from its sound.Source: §1 · pp. 1–2, Figure 1 · p. 2
• The paper defines audio 4D intelligence as reasoning over sound-source dynamics in time and three-dimensional space, grounded in physical-world understanding.STAR-BENCH is introduced to evaluate this ability.Source: §1 · p. 2
• STAR-BENCH evaluates six acoustic attributes through foundational perception tasks and holistic reasoning over temporal order and 3D scene structure.Temporal tasks cover continuous processes and discrete event scripts; spatial tasks cover static localization, multi-source relations, and dynamic trajectories.Source: §1 · p. 2
• The evaluation of 19 models finds that leading closed-source models excel in knowledge and reasoning but are bottlenecked by fine-grained perception, while open-source models are weak across all three core capabilities.The authors also identify averaging multi-channel audio into mono as a major bottleneck for spatial reasoning.Source: §1 · pp. 2–3
• The authors formalize audio 4D intelligence, introduce STAR-BENCH with expert-validated curation, and provide an evaluation of 19 audio-language and omni-language models.They frame the benchmark’s analyses and standardized protocols as baselines and testbeds for future research.Source: §1 · p. 3

2 RELATED WORK

• Earlier benchmarks emphasized transcription, captioning, and simple question answering, while recent work has begun to investigate logical audio reasoning.The passage cites earlier work (Wang et al., 2024; Yang et al., 2024) and recent benchmarks (Sakshi et al., 2025; Ma et al., 2025; Kumar et al., 2025).Source: §2 · p. 3
• Prior benchmarks’ temporal questions often test event timing or order, while spatial tasks often stop at single-source localization; STAR-BENCH instead targets physical or causal dynamics and stereo-cue reasoning in complex scenes.The cited examples include MMAU (Sakshi et al., 2025), MMAU-Pro (Kumar et al., 2025), and MMAR (Ma et al., 2025).Source: §2 · p. 3
• STAR-BENCH requires physical or causal knowledge, multi-step reasoning, and integration across clips or events, alongside hierarchical task design, curated samples, and robust evaluation.The passage contrasts this scope with prior benchmarks in Table 1.Source: §2 · p. 3

3 STAR-BENCH

• STAR-BENCH combines procedurally synthesized foundational perception tasks with real-world holistic temporal and spatial reasoning tasks.The benchmark is designed to evaluate dynamic sound sources across time and 3D space.Source: §3 · p. 3, Figure 2 · p. 4

3.1 FOUNDATIONAL ACOUSTIC PERCEPTION

• Foundational perception probes six acoustic attributes through absolute-range tests and relative discrimination tasks, using synthesized signals and physics-based spatial simulations.Absolute tests measure sensory limits; relative tests assess detection of small attribute changes across difficulty levels.Source: §3.1 · pp. 4–5

3.2.1 TEMPORAL REASONING TASKS

• Temporal reasoning shuffles three audio segments and asks models to restore their order across continuous processes and discrete event sequences.Continuous tasks track source motion or state evolution; discrete tasks assess functional, conventional, and causal event sequences.Source: §3.2.1 · pp. 5–6
• Continuous-process tasks cover moving-source trajectories and stationary-object state changes, including fluid, thermal, decay, and biological dynamics.Models must interpret cues such as Doppler shifts and loudness changes to reconstruct motion.Source: §3.2.1 · p. 5

3.2.2 SPATIAL REASONING TASKS

• Spatial reasoning spans static localization, relative placement of simultaneous sources, and tracking moving-source trajectories.These tasks extend beyond static-source localization to integrate spatial and temporal cues in complex acoustic scenes.Source: §3.2.2 · p. 6
• Because mono conversion can cancel or discard binaural cues, evaluation compares native stereo input with separately presented left and right channels.The channel-wise setting tests whether models can use spatial information when it is preserved at input.Source: §3.2.2 · p. 6, Figure 3 · p. 6

3.3 DATA CURATION PIPELINE

• The curation pipeline combines controlled synthesis and simulation with real-world sourcing, automated filtering, human annotation, expert review, and a human-solvability check.Items are retained only if at least two-thirds of domain experts solve them independently and correctly.Source: §3.3 · pp. 6–7

4 EVALUATION

• The evaluation covers 19 models and uses multiple-choice accuracy, repeated under minor prompt perturbations to report Average Accuracy and All-Correct Rate.Table 2 reports Average Accuracy; All-Correct Rate results are in Appendix D.Source: §4 · pp. 7–8, Table 2 · p. 8

4.1 MAIN RESULT ANALYSIS

• Humans reach 75.6% perception, 88.0% temporal, and 73.7% spatial accuracy, while Gemini 2.5 Pro leads models at 49.59% average accuracy.Most open-source models perform near random guessing, and model Average Accuracy exceeds All-Correct Rate, indicating low reliability.Source: §4.1 · p. 8
• Gemini 2.5 Pro leads on foundational perception and temporal tasks, but nearly all models perform poorly on spatial tasks.The authors suggest that most models discard multichannel information during preprocessing; open-source reasoning modes can also underperform their no-thinking counterparts.Source: §4.1 · p. 9

4.2 DISCUSSION: WHY DO EXISTING MODELS STRUGGLE ON STAR-BENCH?

• In 200 failed predictions, Qwen-2.5-Omni shows a 54% knowledge gap, while Gemini 2.5 Pro's primary bottleneck is fine-grained perception (84%).The analysis also finds weak native spatial awareness and vision-centric hallucinations across models.Source: §4.2 · p. 9
• Gemini 2.5 Pro reaches 99% accuracy with uncut audio, while open-source models show minimal to no improvement across the simplified temporal settings.The settings add a global caption and then a complete audio track as a reference; the authors identify cross-input comparison, grounding, and integration as open-source weaknesses.Source: Figure 9 · p. 10, §4.2 · p. 10
• Fig. 8 contrasts Gemini's broader pitch–loudness coverage with humans' near-full range; all models trail humans, especially on fine loudness differences.Gemini's accuracy falls as perceptual granularity increases, but its duration perception is an exception, outperforming other models on segment lengths.Source: §4.2 · pp. 9–10, Figure 8 · p. 10

5 CONCLUSION

• STAR-BENCH evaluates audio 4D intelligence over time and 3D space, using human annotation, consensus review, and expert validation to curate samples.Source: §5 · p. 10
• The benchmark establishes standardized tasks and protocols and offers actionable diagnostics for model developers.Source: §5 · p. 10
• The authors expect STAR-BENCH to accelerate progress on advanced audio models and training with spatialized corpora, capabilities they describe as crucial for embodied agents.Source: §5 · p. 10

THE USE OF LARGE LANGUAGE MODELS

• Gemini 2.5 Pro assisted with expanding and consolidating the benchmark taxonomy, while DeepSeek-V3 and Gemini 2.5 Pro pre-screened candidate data.Source: THE USE OF LARGE LANGUAGE MODELS · p. 15
• Humans verified final task definitions and data samples; GPT-4o generated some illustrative figures, and GPT-5 polished the manuscript text.Only human-verified revisions were included in the final version.Source: THE USE OF LARGE LANGUAGE MODELS · p. 15

A.1 AUDIO LANGUAGE MODELS

• The related-work overview describes LALMs as audio-encoder and LLM systems and OLMs as extending the paradigm to unified multimodal understanding.It lists ASR, spoken question answering, and audio captioning among LALM tasks, and names examples from both model groups.Source: §A.1 · p. 15

A.2 AUDIO BENCHMARKS

• AudioBench and AIR-Bench focus largely on transcription, spoken question answering, and captioning, while MMAU and MMAR results suggest caption-equipped LLMs can match advanced LALMs.Source: §A.2 · p. 15
• Existing spatio-temporal benchmarks are limited in scale and depth, often reducing temporal analysis to event order and spatial analysis to single-source localization.Video and embodied-AI benchmarks address these dimensions primarily through vision, while audio benchmarks scarcely capture multi-stream cues and subtle acoustic changes.Source: §A.2 · p. 15
• STAR-BENCH targets cross-audio comparison and integration, plus fine-grained spatio-temporal reasoning about physical changes using raw audio cues.Table 3 presents task examples for foundational acoustic perception.Source: §A.2 · p. 16, Table 3 · p. 16

B.1 PROMPTS FOR AUDIO CAPTIONING

• The Gemini 2.5 Pro captioning prompt requests detailed descriptions of speech, music, environmental sounds, and other noticeable elements.It asks for the description to be as specific as possible.Source: §B.1 · p. 16

B.2 DETAIL INFORMATION FOR FOUNDATIONAL ACOUSTIC PERCEPTION

• Tab. 3 presents the ranges and levels used for each acoustic attribute alongside illustrative foundational-perception examples.Source: §B.2 · p. 16

B.2.1 BINAURAL AUDIO SYNTHESIS

• Binaural audio was simulated in three room sizes with listener and source positions varied, using SOFA KEMAR HRTFs and BRIR convolution.Rendering preserved the first 80 ms and attenuated the late tail by 0.5 to emphasize distance cues, then normalized peaks across the batch.Source: §B.2.1 · pp. 16–17
• Fixed partitions yielded 96 absolute-azimuth, 72 absolute-elevation, and 51 absolute-distance utterances per room-listener set, plus 120 relative-azimuth, 64 relative-elevation, and 45 relative-distance utterances.Source: §B.2.1 · p. 17

B.3.1 PROMPT USED FOR AI-ASSISTED FILTERING OF TEMPORAL TASK DATA

• Gemini evaluates temporal candidates for reordering suitability, process type, rationale, and quality, discarding only samples marked “not applicable” before professional verification.The criteria include sequence uniqueness, semantic clarity, and logical universality.Source: §B.3.1 · p. 17

B.3.2 DETAILS OF HUMAN ANNOTATION AND QUALITY CONTROL

• Annotators receive guidelines and trial training, then use initial annotation, independent review, negotiation, and expert arbitration when consensus fails.Samples still unresolved by experts are discarded; experts also spot-check a random 10% after cross-validation.Source: §B.3.2 · pp. 17–18
• Review criteria target temporal boundary and ordering ambiguity, while spatial checks assess cue-answer alignment, question clarity, event mapping, and distractor plausibility.Source: §B.3.2 · pp. 17–18

• Questions use classification accuracy, with string matching to identify the selected option label or full option text.Source: C ROBUST EVALUATION · p. 19
• Perception and spatial questions use CircularEval with cyclically rotated options across N runs, while temporal questions use three segment-order runs; AA averages runs and ACR requires every run correct.GPT-4o Audio was evaluated only once per question because of API costs.Source: C ROBUST EVALUATION · pp. 19–20
• Audio Flamingo 3 models receive concatenated audios separated by two seconds of silence; models supporting multi-audio receive clips sequentially with textual indices.Source: C ROBUST EVALUATION · p. 20
• Ten university students evaluated a random subset of approximately 10% of each task to establish a general, non-expert human baseline.Source: C ROBUST EVALUATION · p. 20

D BREAKDOWN RESULTS

• The breakdown section presents detailed results for perception, temporal reasoning, and spatial reasoning in Tables 4, 5, and 6.Source: D BREAKDOWN RESULTS · p. 20

E.1 HIGH OUTPUT INSTABILITY AND CONCENTRATED PREDICTIONS

• Gemini 2.5 Pro’s All-Correct Rate is 25.01 percentage points below its Average Accuracy, while most open-source models have ACR near zero.The passage characterizes the open-source models’ results as failure to maintain consistent predictions under minor input perturbations.Source: §E.1 · p. 20

E.2 ABLATION STUDY ON SPATIAL REASONING.

• Native audio preprocessing discards multichannel information needed for fine-grained spatial reasoning; channel-wise input mitigates some loss, but models still struggle to align channels and use interaural differences.The human–model performance gap remains pronounced, motivating audio encoders that natively support multichannel input.Source: §E.2 · p. 20, Table 6 · p. 21

F CASE STUDY

• The case studies examine errors in temporal reasoning (Figs. 12–17) and spatial reasoning (Fig. 18).Source: F CASE STUDY · p. 20