PaperlayerView exact revision on arXiv

Source-linked AI summary

Sonic-AMG: Streaming Human Motion Generation from Spatial Audio

Jingcheng Ni, Zekun Li, Zhiyang Dou, Xiaoyan Cong, Ziyao Zeng, Hongyang Du, Randall Balestriero, Taku Komura, Srinath Sridhar

arXiv:2610.13300v1cs.CVcs.GRcs.LGcs.ROcs.SD

TL;DR

Prior work synthesizes complete motion sequences from fully observed acoustic and spatial inputs, while streaming generation must respond as those inputs evolve. Sonic-AMG generates motion autoregressively from separate acoustic and spatial tokens, using rehearsal training and spatial feedback. On SAM, it achieves state-of-the-art FID and R-precision and supports real-time causal generation from streaming spatial audio.

  • Problem

    Prior work synthesizes complete motion sequences from fully observed acoustic and spatial inputs, limiting its ability to respond causally as incoming audio evolves.

    Source: §1 · p. 2

  • Method

    Sonic-AMG generates streaming motion from separate acoustic and spatial tokens, using rehearsal training and spatial feedback to address history and conditioning mismatches.

    Source: §1 · p. 3

  • Results

    On SAM, Sonic-AMG achieves state-of-the-art FID and R-precision and supports real-time causal online generation of coherent reactions across successive sound events and varying source locations.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Despite training on individual audio–motion clips, Sonic-AMG generates coherent motion across successive sound events with varying content and source locations.

    Source: §1 · p. 3

    Evaluation is limited to SAM, so generalization to broader acoustic environments and human reactions remains to be assessed.

    Source: A LIMITATIONS AND FUTURE WORK · p. 16

Abstract

from arXiv · show

Modeling human reactions to spatial audio requires understanding both what is heard and where it comes from. Prior work generates motion offline from complete acoustic and spatial input sequences. Humans, however, react to sounds without knowing what they will hear next. We present Sonic-AMG, to our knowledge the first framework for streaming human motion generation from spatial audio. The model encodes acoustic content and body-relative source geometry as separate tokens. Conditioned on these tokens, a causal transformer with a flow head generates continuous motion latents, from which a causal decoder reconstructs motion segments. Standard teacher-forced training derives motion histories and body-relative source locations from ground-truth motion, whereas streaming inference relies on generated motion for both, resulting in a training-inference mismatch. To reduce this history discrepancy, we introduce rehearsal training, which combines full-sequence teacher-forced supervision with supervised prediction along short autoregressive self-rollouts. To keep spatial conditioning consistent with generated motion, we propose spatial feedback, which recomputes body-relative source locations from generated root poses during both rehearsal training and inference. On SAM, Sonic-AMG achieves state-of-the-art FID and R-precision. It further supports real-time online generation of coherent reactions across sound events and source locations. Project page: https://jingchengni.com/sonic-amg/.

1 INTRODUCTION

Sound-driven reactions depend on both acoustic content and source location relative to the body, which changes as a listener moves (Chen et al., 2020; 2021); prior complete-window synthesis cannot respond causally to incoming audio streams.

1 INTRODUCTION

  • Sonic-AMG generates streaming motion by conditioning a causal transformer and flow-matching head on incoming audio, spatial tokens, and preceding motion latents.A pretrained causal temporal VAE represents motion latents, and a causal decoder reconstructs each segment without requiring the full audio in advance or revising generated frames.Source: §1 · pp. 2–3
  • Dedicated what and where tokens separate acoustic content from spatial cues, including an 11-dimensional spatial event representation for body-relative source geometry.Simple log-mel features yield better results than MOSPA’s 2,272-dimensional audio features; the spatial representation draws on studies of auditory space, source motion, and looming.Source: §1 · pp. 2–3
  • Rehearsal training addresses motion-history mismatch by combining full-sequence teacher-forced supervision with supervised prediction on short autoregressive self-rollouts.Unlike two-forward training, later rollout predictions depend on earlier sampled outputs rather than ground-truth histories (Mihaylova & Martins, 2019; Xiao et al., 2025).Source: §1 · pp. 2–3
  • Spatial feedback addresses conditioning mismatch by recomputing body-relative source coordinates from generated root poses during rehearsal training and inference.This avoids using complete source-location sequences precomputed from recorded motion, which encode future ground-truth root motion and are incompatible with causal streaming.Source: §1 · pp. 2–3
  • On SAM, Sonic-AMG achieves state-of-the-art FID and R-precision and supports real-time causal generation of coherent motion across successive sound events and source locations.The model is trained on individual audio–motion clips but can generate across events with varying content and source locations.Source: §1 · p. 3

2 RELATED WORK

Prior work spans audio-conditioned motion synthesis and causal generation, but Sonic-AMG targets real-time causal reactions to spatial-audio streams. It also addresses the mismatch between teacher-forced histories and generated histories through rehearsal training.

Audio-conditioned motion generation

  • Sonic-AMG enables real-time causal motion generation from incoming spatial audio, unlike MOSPA’s future-dependent full-sequence conditioning and MoSAT’s non-streaming setup (Xu et al., 2026).It also updates body-relative source geometry through spatial feedback.Source: §2 · p. 3

Causal motion generation

  • Causal decoding enables online motion generation, but teacher forcing still creates exposure bias because training uses recorded histories while inference uses model predictions.Prior systems combine causal decoding with transformer-conditioned generative heads for online text control, two-character interaction, and co-speech gestures (Xiao et al., 2025; Cen et al., 2025; Li et al., 2026; Liu et al., 2026).Source: §2 · p. 3
  • Unlike two-forward training, which uses replacements from a preliminary teacher-forced pass, rehearsal training combines teacher forcing with short autoregressive self-rollouts.This captures error propagation across generated segments while retaining sequence-wide supervision (Mihaylova & Martins, 2019; Xiao et al., 2025).Source: §2 · p. 4

3 METHOD

Sonic-AMG generates motion autoregressively from binaural audio and a known source position, conditioning on separate acoustic and body-relative spatial tokens. Rehearsal training supervises predictions along sampled histories, while spatial feedback updates source geometry from generated poses.

3.1 MOTION AND CONDITIONING REPRESENTATIONS

  • Motion is encoded at 30 Hz into 16-dimensional latents, each covering four frames, by a pretrained causal temporal VAE whose encoder and decoder stay frozen during generator training.Each frame represents global joint positions, local 6D rotations, and joint velocities for 25 joints.Source: §3.1 · p. 4
  • The semantic branch maps shared causal-CNN features from both ears’ 80-dimensional log-mel spectra into aligned acoustic tokens, adding the annotated reaction-style embedding.Acoustic content and source geometry use separate tokens because the modalities may act as noise for one another when fused.Source: §3.1 · pp. 4–5
  • The spatial branch encodes body-relative source direction, distance, their increments, inverse distance, bearing change, and looming in an 11-dimensional descriptor.The descriptor captures proximity, relative motion, and approach.Source: §3.1 · p. 5

3.2 CAUSAL MOTION GENERATION

  • A causal transformer interleaves acoustic, spatial, and motion tokens, while a rectified-flow head predicts each next latent from history and a causal decoder appends its motion segment.At inference, the flow sampler uses ten Euler steps, and the decoder does not revise earlier frames.Source: §3.2 · pp. 5–6

3.3 REHEARSAL TRAINING

  • Rehearsal training combines full-sequence teacher-forced supervision with next-latent supervision across short recursively sampled autoregressive histories (H = 16 by default).The two objectives are equally weighted, and gradients do not propagate through sampled latents or the sampling procedure.Source: §3.3 · p. 6

3.4 SPATIAL FEEDBACK

  • Spatial feedback recomputes body-relative source geometry from decoded generated root poses during training rollouts and inference, conditioning the next block on updated geometry.During training rollouts, the recomputed geometry is detached before spatial encoding.Source: §3.4 · p. 6

4 EXPERIMENTS

Experiments evaluate Sonic-AMG on SAM using FID, R-precision, and feature-space diversity, under both ground-truth and Online spatial-conditioning protocols.

4.1 COMPARISON WITH EXISTING METHODS

  • Under GT SSL, Sonic-AMG achieves the best distributional fidelity and conditioning alignment among compared methods, while maintaining strong performance under Online SSL.Online conditioning recomputes spatial conditions from generated root poses, unlike GT SSL, which uses recorded motion.Source: §4.1 · p. 7
  • Under Online evaluation, Sonic-AMG produces recognizable, source-appropriate reactions and coherent sequences across 10–12 s clips, unlike MOSPA’s fixed 8 s output window.Examples include covering the ears for crowd noise and moving away from an explosion while lowering posture.Source: §4.1 · pp. 7–8, Figure 3 · p. 8, Figure 4 · p. 8

4.2 STREAMING EFFICIENCY AND LATENCY

  • At 30 Hz, Sonic-AMG’s 95th-percentile four-frame step time stays below the 133.3 ms budget on both GPUs, enabling causal real-time Online generation.MOSPA is noncausal despite an RTF below one on RTX A6000, and exceeds RTF one on TITAN RTX.Source: §4.2 · p. 8, Table 3 · p. 9

4.3 REHEARSAL TRAINING

  • Rehearsal training improves fidelity and conditioning alignment over all two-forward schedules under both SSL protocols, and full-sequence supervision remains essential alongside self-rollouts.Rollout Only with History improves over removing supervision but does not recover rehearsal’s fidelity or top-rank retrieval.Source: Table 4 · p. 9, §4.3 · pp. 9–10

4.2 STREAMING EFFICIENCY AND LATENCY

  • On RTX A6000, Sonic-AMG reduces estimated initial response latency from 10,857 ms to approximately 193 ms, about a 56-fold reduction versus MOSPA.Causal blockwise generation begins as audio arrives, while MOSPA waits for its complete 8 s input window.Source: §4.2 · p. 9

4.4 EFFECT OF SPATIAL FEEDBACK

  • Spatial feedback substantially improves Online fidelity and alignment by updating body-relative SSL during rehearsal so spatial conditions match the generated motion trajectory.A GT-trained model degrades when evaluated with Online SSL because its training did not expose it to spatial-condition changes caused by motion prediction errors.Source: Table 5 · p. 10, §4.4 · p. 10

4.5 ACOUSTIC AND SPATIAL REPRESENTATION

  • With Online SSL, separate acoustic and spatial tokens improve fidelity and alignment over fused tokens, while learned log-mel features outperform MOSPA’s aggregated audio descriptors.Separate tokens let the transformer attend to acoustic and spatial cues independently; the learned features are consistent with reducing redundant or task-irrelevant audio variation.Source: Table 6 · p. 10, §4.5 · p. 10

5 CONCLUSION

Sonic-AMG generates motion autoregressively from streaming spatial audio using rehearsal training and spatial feedback. It achieves state-of-the-art FID and R-precision on SAM and generates coherent reactions across sound events and source locations.

5 CONCLUSION

  • Sonic-AMG achieves state-of-the-art FID and R-precision on SAM and generates coherent reactions across sound events and source locations.It supports real-time autoregressive motion generation from streaming spatial audio using separate acoustic and spatial representations; rehearsal training combines teacher-forced supervision with short self-rollouts, and spatial feedback keeps source coordinates consistent with generated motion.Source: §5 · p. 10

AI USE STATEMENT

  • Generative AI tools polished the manuscript and assisted with code development, and the authors take full responsibility for the final manuscript and code.Source: AI USE STATEMENT · p. 11

REPRODUCIBILITY STATEMENT

  • The authors state that code and the model will be released upon acceptance.The statement also points to sections and appendices for model, evaluation, implementation, training, and runtime details.Source: REPRODUCIBILITY STATEMENT · p. 11

A LIMITATIONS AND FUTURE WORK

Sonic-AMG’s evaluation leaves open how spatial inference, generalization beyond SAM, and long-horizon coherence should be assessed.

A LIMITATIONS AND FUTURE WORK

  • Main experiments assume known world-space source positions at inference; No-SSL initially demonstrates binaural-audio spatial conditioning, but systematic evaluation of alternative spatial inference methods remains future work.Their effects on motion generation also remain to be assessed.Source: A LIMITATIONS AND FUTURE WORK · p. 16
  • Evaluation is limited to SAM, leaving generalization to broader acoustic environments and human reactions unassessed.Source: A LIMITATIONS AND FUTURE WORK · p. 16
  • Long-horizon coherence and event transitions are supported primarily by qualitative evidence; future work should establish quantitative benchmarks for longer streams with more diverse transitions.Source: A LIMITATIONS AND FUTURE WORK · p. 16

B ADDITIONAL VISUAL RESULTS AND PHYSICS-BASED SIMULATION · B.1 REACTIONS TO DIFFERENT SOUNDS

The supplementary results present audio-synchronized human-motion and physics-simulation videos, including a continuous sequence across three sound events and ten single-sound examples. The single-sound reactions combine recognizable gestures with changes in body orientation and position, generated with online spatial feedback.

B ADDITIONAL VISUAL RESULTS AND PHYSICS-BASED SIMULATION

  • The supplementary package includes four MOSPA comparisons, ten single-sound examples, and a continuous sequence spanning three sound events.The examples are available as HTML and videos.Source: B ADDITIONAL VISUAL RESULTS AND… · p. 16
  • The single-sound and continuous examples pair human-motion visualizations with humanoid physics simulations synchronized to stereo audio.Source: B ADDITIONAL VISUAL RESULTS AND… · p. 16
  • Figure 6 aligns human motion with G1 simulation snapshots across the continuous Music → Explosion → Birdsong sequence.Source: Figure 6 · p. 16

B.1 REACTIONS TO DIFFERENT SOUNDS

  • Across diverse sounds, Sonic-AMG combines recognizable gestures with changes in body orientation and position, using a feedback-trained model with online spatial updates.Source: §B.1 · p. 16
  • The ten single-sound examples include hands raised toward the ears, an arm raised in response to a call, and a hand brought toward the mouth.Figures 7 and 8 show the examples; Figure 7 presents six sounds, with poses ordered chronologically.Source: §B.1 · p. 16, Figure 7 · p. 17, Figure 8 · p. 18
  • The full videos show gestures developing alongside stepping and body turns.Source: §B.1 · p. 16

B.2 CONTINUOUS REACTIONS ACROSS SOUND EVENTS · B.3 EXECUTION IN A HUMANOID PHYSICS SIMULATION

Sonic-AMG demonstrates continuous, spatially conditioned reactions across changing sound events and source locations, then shows those reactions tracked by a humanoid in an uninterrupted simulation.

B.2 CONTINUOUS REACTIONS ACROSS SOUND EVENTS

  • Across 30.13 seconds, generation combines Music, Explosion, and Birdsong at different source locations, beginning with four ground-truth motion frames and continuing from generated body state.Table 7 describes the continuous demonstration setup, including source-location changes at event boundaries and a generated 30 Hz sequence.Source: §B.2 · p. 18, Table 7 · p. 18
  • Sonic-AMG sustains coherent reactions across event transitions without requiring new ground-truth motion initialization.Later events continue from the generated body state after the initial four ground-truth frames.Source: §B.2 · p. 18
  • At each transition, generation resumes from generated motion history in a root-aligned frame, updates source coordinates through spatial feedback, and keeps previously emitted frames fixed.Source: §B.2 · p. 18

B.3 EXECUTION IN A HUMANOID PHYSICS SIMULATION

  • A pretrained humanoid controller tracks the generated reactions without additional policy training after human motions are retargeted to a Unitree G1.The motions are retargeted using General Motion Retargeting (GMR) and tracked with the released pretrained GEAR-SONIC controller.Source: §B.3 · pp. 18–19
  • The simulation interpolates references from 30 to 50 Hz, integrates physics at 1,000 Hz, and uses gravity, ground contact, and actuator-limited PD torques.After initialization, the humanoid’s free root evolves through physics integration.Source: §B.3 · p. 19
  • The humanoid tracks the references through all three sound events in one uninterrupted simulation.Figure 6 pairs human reactions with their simulated humanoid execution.Source: §B.3 · p. 19

C STREAMING TIMING PROTOCOLS

The appendix specifies how streaming-generation timing and response latency are measured, then compares online generation with MOSPA, whose full-sequence protocol does not support causal blockwise streaming.

C STREAMING TIMING PROTOCOLS

  • Timing uses one 700-step run per GPU and protocol, discards 20 warm-up steps, starts from a zero motion latent, and measures 93.3 s of added motion.The KV cache retains up to 4,096 token positions; measurements use FP32 with TF32 where supported and GPU synchronization. GT and Online use GT-SSL and feedback-trained rehearsal models, respectively.Source: C STREAMING TIMING PROTOCOLS · p. 19
  • At batch size one, timing covers both guidance branches, ten Euler steps, and KV-cache updates; Online also decodes generated latents and recomputes spatial tokens from the last two poses.Timing starts from precomputed acoustic tokens and excludes audio preprocessing and encoding, initialization, rendering, and output delivery.Source: C STREAMING TIMING PROTOCOLS · pp. 19–20
  • Online feedback raised mean step time from 41.2 to 50.2 ms on RTX A6000 and from 102.5 to 118.0 ms on TITAN RTX.The corresponding online RTFs are 0.376 and 0.885, calculated as mean step time divided by 4/30 s.Source: C STREAMING TIMING PROTOCOLS · p. 19
  • MOSPA jointly denoises a full 240-frame sequence from the full audio window, so it does not support causal blockwise streaming.MOSPA evaluates its network 125 times over the 240-token sequence, whereas online generation takes 7.5 autoregressive steps, each extending the KV cache by three tokens and using ten Euler steps; moving to TITAN RTX multiplies MOSPA block time by 3.99 and online step time by 2.35.Source: C STREAMING TIMING PROTOCOLS · p. 19
  • RTFs were 0.357 (MOSPA) vs. 0.376 (online) on RTX A6000 and 1.427 vs. 0.885 on TITAN RTX, favoring online on TITAN RTX.One 240-frame block spans 8.0 s of motion; RTF measures computational throughput independently of input waiting time.Source: C STREAMING TIMING PROTOCOLS · p. 19
  • Estimated response latency was 192.9 ms for online generation on RTX A6000 versus 10,857 ms for MOSPA’s 8.0 s input window.The online estimate adds 142.7 ms for the first acoustic token to become available to 50.2 ms mean generation time; MOSPA’s estimate includes 2,857 ms generation.Source: C STREAMING TIMING PROTOCOLS · p. 19

D GENERATION WITHOUT SSL INPUTS

No-SSL replaces inference-time source-location inputs with spatial features inferred from binaural audio, while retaining geometric supervision during training. On SAM, it surpasses EDGE in motion fidelity and achieves conditioning alignment and diversity comparable to MOSPA.

D GENERATION WITHOUT SSL INPUTS

  • No-SSL removes source-location inputs at inference, deriving conditioning from audio while sampled motion latents extend autoregressive history; it omits root-pose spatial feedback.Its spatial conditions follow the recorded binaural stream rather than an explicit source position updated relative to the generated body.Source: D GENERATION WITHOUT SSL INPUTS · pp. 20–21
  • The spatial branch encodes 192-dimensional ILD/IPD features with a causal CNN and four-frame pooling; the semantic branch derives acoustic tokens from log-mel features.The variant keeps the interleaved [ak, sk, mk] sequence, causal transformer, rectified-flow head, frozen causal motion autoencoder, and teacher-forced plus 16-step rehearsal objectives.Source: D GENERATION WITHOUT SSL INPUTS · p. 20, Figure 9 · p. 20
  • A training-only auxiliary head predicts normalized 11-dimensional geometry from spatial features; inference uses learned features directly, while semantic features also support event and class losses.The predicted geometric descriptor is not passed to the motion generator.Source: D GENERATION WITHOUT SSL INPUTS · p. 21
  • No-SSL surpasses EDGE in motion fidelity and achieves conditioning alignment and diversity comparable to MOSPA on SAM.The comparison concerns streaming generation without source-location inputs or access to future audio.Source: Table 8 · p. 21, D GENERATION WITHOUT SSL INPUTS · p. 21

E IMPLEMENTATION DETAILS

Sonic-AMG uses a causal transformer and flow head to generate motion from acoustic and spatial tokens, with a pretrained causal motion autoencoder and causal audio features. Its implementation specifies training settings, feature delay, spatial conditioning, and inference guidance.

E IMPLEMENTATION DETAILS

  • SAM pairs binaural recordings with human reactions across 27 sound scenarios and three reaction styles.Source: E IMPLEMENTATION DETAILS · p. 22
  • The generator uses a 12-layer, 12-head transformer of width 768 and a nine-block flow head, trained with AdamW; the H = 16 rehearsal model takes about 63 hours for 150,000 updates on one L40S GPU.The 130,000-update checkpoint is reached after approximately 54 hours; reported times include validation and checkpointing but exclude motion-autoencoder pretraining.Source: E IMPLEMENTATION DETAILS · p. 22
  • The causal motion autoencoder is pretrained with Gaussian reconstruction, KL regularization, and root-position reconstruction, then fixed during generator training.Motion channels are standardized using training-set statistics, decoded channels are returned to physical coordinates before source-geometry computation, and the initial ground-truth latent encodes four frames.Source: E IMPLEMENTATION DETAILS · p. 22
  • Causal audio tokens are derived from 48 kHz audio using 80-dimensional log-mel features and causal convolutional networks, with a 9.33 ms delay beyond the nominal four-frame boundary.The feature pipeline uses a 2048-sample Hann window, hop 1600, noncentered windows, and four-frame mean pooling; the final analysis window for block k ends at sample (4k + 3)1600 + 2048.Source: E IMPLEMENTATION DETAILS · p. 22
  • Spatial descriptors are channel-normalized and passed through an 11 → 768 → 768 SiLU MLP; evaluation updates egocentric source coordinates from the generated root.Direction and distance increments are measured per latent step, with both set to zero in the first block.Source: E IMPLEMENTATION DETAILS · p. 22
  • Training jointly drops current acoustic and spatial conditions with probability 0.1, while inference uses guided conditional and null branches and ten uniform Euler steps per latent.Training rollouts use ten unguided steps; only the conditional inference branch updates the persistent key–value cache.Source: E IMPLEMENTATION DETAILS · p. 22

F TRAINING SEQUENCE CONSTRUCTION

This section specifies two-pass motion-input replacement and autoregressive rehearsal rollouts, including how spatial conditions and sequence context are handled.

Two-forward training

  • The two-forward procedure retains the first motion latent, replaces later inputs with detached first-pass predictions, and decodes the updated sequence before the causally masked second pass.Second-pass outputs do not feed later inputs within that pass; replacement schedules change motion-input replacement, not the spatial-conditioning protocol.Source: F TRAINING SEQUENCE CONSTRUCTION · p. 23

Rehearsal initialization and recursion

  • Rehearsal samples a rollout start, seeds it with a stop-gradient prediction, then recursively appends sampled latents while supervising each transition against the next ground-truth latent.The decoder uses the ground-truth prefix followed by generated latents; online spatial updates derive each next spatial descriptor from the decoded root, whereas GT-SSL retains recorded conditions.Source: F TRAINING SEQUENCE CONSTRUCTION · p. 23
  • Each rollout spans H transitions and 4H frames; acoustic features are re-encoded at its start, acoustic and transformer contexts reset, and the decoder retains its ground-truth prefix.Gradients stop through sampled latents and decoded geometry, but remain through transformer states and conditioning encoders in the supervised objective.Source: F TRAINING SEQUENCE CONSTRUCTION · p. 23

Rollout-only ablations

  • Rollout-only ablations train on rollout loss alone: one resets contexts like rehearsal, while the history variant builds a ground-truth prefix cache and propagates rollout gradients through it.Both use H = 16 and ten sampling steps per latent.Source: F TRAINING SEQUENCE CONSTRUCTION · p. 23
Loading 2610.13300v1…