Source-linked AI summary
Long-Horizon Audio-Visual Generation for Persistent Stories and Interactive Worlds
Nan Duan, Haoyang Huang, Weiyang Jin, Haoran Li, Yaowei Li, Yuming Li, Yijun Liu, Xin Lu, Xiaoxiao Ma, Yanwen Ma, Yaofeng Su, Yilang Sun, Haoyu Wang, Zeyue Xue, Songchun Zhang, Junhao Zhuang
TL;DR
Long-horizon audio-visual generation must preserve identities, follow user controls, and remain stable across extended rollouts. JoyAI-Echo-1.5 combines composable cross-shot memory, calibrated metric 6-DoF control, and causal rollout-aware training, achieving improved long-video metrics and leading world-model performance. The system therefore supports coherent persistent stories and continuously evolving interactive worlds within the reported evaluation scope.
Problem
Existing video models are limited in preserving identities, following heterogeneous controls, and remaining stable when generating from extended self-generated histories.
Method
JoyAI-Echo-1.5 uses cross-shot audio-visual memory, calibrated metric 6-DoF trajectories, and causal few-step training with teacher forcing and Self-Gradient Forcing.
Results
JoyAI-Echo-1.5 achieves the best performance on six of seven long-video metrics, including highest ViCLIP 0.8264, Self-CIDS 0.7937, and Voice 0.8524 scores.
Takeaways & Limitations
The results support memory, geometric control, and rollout-aware training as a foundation for persistent stories and interactive worlds.
Takeaways & Limitations
The causal-memory construction uses a leak-free prefix in which every context index satisfies t < tstart.
Abstract
from arXiv · showhide
Video generation is progressing beyond isolated clips toward long-form narratives and interactive worlds, requiring models to preserve identities, follow user controls, and remain stable over extended rollouts. We present JoyAI-Echo-1.5, a unified audio-visual generation system with two purpose-built variants. The long-video variant introduces composable cross-shot memory that aggregates visual evidence across multiple prior shots and speaker cues derived from speech-filtered full-shot audio, enabling persistent character appearance and voice identity across flexible combinations of text, image, and memory conditioning. The world-model variant converts heterogeneous navigation inputs into calibrated metric 6-DoF camera trajectories and injects them through a geometry-aware conditioning pathway, enabling controller-agnostic interaction across flexible viewpoints. To support efficient long-horizon generation, we transform a bidirectional audio-visual backbone into a causal few-step generator using progressive teacher forcing and short- and long-horizon Self-Gradient Forcing on self-generated rollouts. Experiments demonstrate strong performance in both settings. JoyAI-Echo-1.5 achieves improvements over existing long-video baselines in cross-shot consistency, visual quality, text alignment, and speech fidelity. Its world-model variant ranks first on WBench, with an average score of 81.7, and achieves leading visual quality and long-horizon persistence on SANA-WM-Bench. Together, these results indicate that memory, geometric control, and rollout-aware training provide a practical foundation for generating coherent stories and continuously evolving interactive worlds. Project page: https://echo-team-joy-future-academy-jd.github.io/Echo-1.5-Page/.
1 Introduction
JoyAI-Echo-1.5 addresses long-horizon audio-visual generation through two purpose-built systems: one for persistent stories and one for interactive worlds. It combines cross-shot memory, geometric camera control, and rollout-aware causal training to maintain identity, responsiveness, and stability beyond fixed temporal windows.
- Motivation: Existing video models struggle with identity persistence, self-generated-history drift, and consistent motion under heterogeneous action inputs.These limitations arise because many models are designed around fixed temporal windows.
- Long-video system: The long-video system aggregates visual evidence across multiple prior shots and speech-filtered audio to preserve character appearance and speaker identity.It supports composable text, image, and memory conditioning.
- World-model system: The world-model system converts heterogeneous navigation inputs into calibrated metric 6-DoF trajectories for precise, controller-agnostic camera control.This supports interaction across diverse data sources, environments, and viewpoints.
- Efficient generation: The causal few-step training pipeline combines audio-visual teacher forcing with short- and long-horizon Self-Gradient Forcing on self-generated rollouts.Training on generated histories targets accumulated errors while preserving synchronized audio-visual dynamics within a bounded computational budget.
2 Data
JoyAI-Echo-1.5 uses a two-stage, quality-controlled data pipeline that combines identity-centric memory supervision with broader high-quality audio-visual coverage. The resulting corpus supports cross-shot identity learning while preserving general generation, language, resolution, and multi-shot diversity.
- Two-stage curriculum: Stage I constructs identity-centric memory–target pairs from scene-disjoint clips featuring recurring characters.Stage II adds high-quality single- and multi-shot videos with broader resolution, aspect-ratio, language, and transition coverage.
- Stage-I memory corpus: Memory groups preserve recurring-character diversity across scenes, clothing, viewpoints, poses, motion, illumination, expressions, dialogue, and acoustic environments.This diversity encourages memory representations to capture persistent character evidence rather than scene-specific appearance.
- Conditioning configurations: Each training sample independently varies memory and first-frame availability, yielding T2AV, I2AV, MT2AV, and MTI2AV conditioning configurations.Memory length is also varied so the model remains effective with limited character evidence.
- Audio-visual memory: Visual memory uses one sampled frame per shot, while audio memory retains complete speech-filtered shot audio to describe speaker characteristics and reduce background leakage.The visual and audio branches therefore use different temporal granularities for memory construction.
3 Long-Horizon Video Generation
The long-horizon video system builds composable audio-visual memories from prior shots and trains a unified interface for text, image, and memory conditioning. Distribution matching distillation then converts the generator into an efficient few-step model while addressing memory-distribution sensitivity.
- Unified memory conditioning: JoyAI-Echo-1.5 improves on JoyAI-Echo-1.0 by combining multi-shot visual memory, speech-filtered full-shot audio memory, separated RoPE regions, and joint conditioning.The framework supports multiple combinations of text, image, and memory inputs.
- Long-form rollout: Long-form generation updates a memory bank after each shot by extracting audio-visual information from generated history.The current shot may additionally receive a first-frame image condition without consuming a memory slot.
- Memory construction: Historical memory stores one encoded frame from each selected prior shot and complete speech-filtered audio from that shot.Multiple historical shots provide compact visual observations, while speech filtering limits music and environmental contamination in speaker cues.
- Positional conditioning: Separated temporal RoPE regions distinguish the current target, historical memory, and memory supplied before generation while preserving audio-event structure.The regions allow memory and image conditions to coexist without placing memory inside the target-shot timeline.
- Joint training: The unified objective supports T2AV, I2AV, MT2AV, and MTI2AV without changing the model interface or introducing separate objectives.Memory tokens remain clean conditions and are excluded from the prediction loss.
- Few-step acceleration: An 8-step student generator is produced with distribution matching distillation for lower-latency long-form audio-visual generation.Task-adaptive audio weighting and visual-memory calibration improve robustness across single-shot and memory-conditioned settings.
4 Action-Conditioned World Modeling
The world-model variant unifies heterogeneous navigation inputs as calibrated relative 6-DoF camera trajectories and injects their geometry into audio-visual generation. A progressive curriculum separates audio-visual prior acquisition, control alignment, and joint consolidation.
- Overview: The system formulates synchronized video and audio generation under text, optional media context, and user-specified camera intent.Media context may be empty, a reference image, or an audio-visual prefix; observed prefixes are clamped while only the unobserved suffix is generated.
- Unified Camera-Intent Interface: Heterogeneous controllers are unified by mapping navigation inputs to subject-agnostic metric 6-DoF trajectories and calibrating relative translations with a global scale.This decouples user-facing controls from model-level conditions and addresses controller incompatibility and motion-scale mismatch.
- Unified Camera-Intent Interface: Relative camera motion uses SE(3) poses and converts discrete controls through Fctrl into step-wise camera increments, while continuous trajectories directly provide relative poses.The shared geometric interface supports both raw actions and estimated poses.
- Scale Calibration: Relative translations are divided by a fixed global dataset-level scale while rotation matrices remain unchanged, preserving consistent velocity semantics across data sources.The calibration is designed to avoid boundary artifacts in multi-turn rollouts.
- Geometry-Aware Conditioning: UCPE injects camera geometry into parallel attention while remaining invariant to global coordinate shifts and preserving calibrated relative translation magnitudes.The pathway derives ray-local transformations from calibrated poses and camera intrinsics.
- Training Curriculum: A three-stage curriculum first adapts audio-visual priors, then trains the UCPE control pathway, and finally jointly fine-tunes the backbone and trajectory pathway.The stages respectively disable trajectory conditioning, freeze the backbone, and unfreeze both components for end-to-end co-adaptation.
5 Autoregressive Audio-Visual Generation
The model is converted from bidirectional diffusion to causal few-step streaming generation through teacher forcing and short- and long-horizon Self-Gradient Forcing. Differentiable reconstruction of self-generated histories preserves audio-visual and camera-pathway interactions while enabling parallel training computation.
- Causal Initialization: Teacher forcing initializes a causal streaming generator from a pretrained bidirectional multi-step audio-visual diffusion model.The broader transition targets high-throughput, low-latency autoregressive generation with few-step sampling under self-generated histories.
- Audio-Visual Teacher Forcing: Aligned audio and video macro-chunks share a noise level, placing paired modalities at a consistent diffusion stage.The streams are divided into temporally aligned macro-chunks, although their token counts may differ.
- Audio-Visual Teacher Forcing: Each noisy chunk attends to preceding clean chunks and its current noisy tokens, while the clean target and future chunks remain masked.The same causal constraint is applied to video self-attention, audio self-attention, and both audio-visual cross-attention directions.
- Audio-Visual Teacher Forcing: LoRA-based fine-tuning adapts the pretrained backbone to causal chunk-wise generation while preserving its original flow-matching objective.The pretrained backbone remains fixed while trainable LoRA parameters learn causal audio-visual behavior.
- Self-Gradient Forcing: Self-Gradient Forcing trains on the generator’s own autoregressive histories while restoring gradients through the computation that encodes generated contexts.Its joint audio-visual DMD objective supervises modality quality and consistency through A2V and V2A pathways.
- Self-Gradient Forcing: Forward 1 reproduces inference-time few-step rollouts without gradients, and Forward 2 reconstructs the recorded rollout states in one differentiable parallel pass.Generated clean latents serve as stop-gradient context while sampled noisy states retain their exit timestep.
- Long-Horizon SGF: The causal mask is applied across video, audio, A2V, V2A, and UCPE pathways so reconstructed supervision covers cross-modal history exchange and camera conditioning.Long-horizon reconstruction further restricts context with sink-plus-FIFO masks while retaining the long trajectory as a parallel sparse computation graph.
- Causal Memory: The generator samples the next synchronized chunk from variant-specific conditions, including text with memory or image for long video and trajectory for world modeling.A compact recurrent state is introduced to retain structure beyond the sliding window without changing the causal mask or few-step sampler.
6 Agentic Planning and Control
The Director Agent turns open-ended requests into persistent, shot-wise production workflows for short and long videos. It combines screenplay planning, prompt enhancement, continuity-aware conditioning, generation, review, and memory updates.
- Review and Generation: The workflow supports autonomous completion and human-in-the-loop checkpoints, including shot generation, intermediate review, continuity-memory updates, and revision.The same planning and generation procedure supports single-shot short-video requests with T = 1.
- Persistent Workspace: A persistent workspace stores the request, screenplay, story profile, shot plan, generated artifacts, memory banks, review decisions, and neighboring-shot relations.Later shots use accepted visual evidence and current user decisions, and the workflow can resume from an intermediate shot.
- Planning and Prompt Enhancement: The Director Agent expands a brief into a complete screenplay with narrative order, shot-level dramatic jobs, and editable shot descriptions.It resolves events, characters, locations, temporal progression, and intended ending before generation begins.
- Continuity Control: Shot planning marks adjacent pairs as scene transitions, ordinary edits, or continuous actions to control how scene context and motion carry across shots.Continuous actions preserve the previous shot’s terminal state and motion direction.
- Planning and Prompt Enhancement: Prompt enhancement organizes each shot around story, visual, camera, audio, and temporal fields while preserving request invariants and adding observable execution detail.The temporal structure expands each beat from initial state through trigger, action, completion, reaction, and final hold.
7 Arbitrary-Step Super-Resolution
The arbitrary-step super-resolution stage restores delivery-resolution detail while allowing inference-time control over computation. MeanFlow distillation turns interval prediction into a solver-free sampler whose step count can vary without retraining.
- SR Design: The SR stage restores fine texture and suppresses generation artifacts after synchronized audio-visual generation at the training corpus resolution.It targets delivery-resolution output and addresses artifacts produced by the upstream generator.
- SR Design: AIGC-oriented degradation models temporally structured failures such as flicker, motion jitter, and color shift rather than treating artifacts as independent per-frame noise.Degradation strength is annealed toward a weak setting matched to the upstream generator.
- MeanFlow Distillation: MeanFlow replaces instantaneous velocity with average velocity over an interval, so one network evaluation directly performs the corresponding interval update.The update is xr = xt − (t − r)u(xt, r, t), making the model an interval integrator rather than an ODE-solver step approximator.
- MeanFlow Distillation: The MeanFlow objective uses a local regression target corrected by a stop-gradient Jacobian–vector product, with weighted updates and a gradual transition from flow matching.The weight down-weights high-noise samples where the JVP estimate is least stable.
- Arbitrary-Step Sampling: The distilled sampler supports any K network evaluations without a scheduler, solver, or per-budget retraining.K = 1 enables one-step generation, while larger K trades computation for fidelity using the same checkpoint.
8 Results
JoyAI-Echo-1.5 shows improved long-horizon audio-visual consistency and strong interactive-world performance across quantitative, qualitative, and human evaluations.
- Long-Horizon Audio-Visual Generation: JoyAI-Echo-1.5 achieves the best performance on six of seven long-video metrics, including ViCLIP 0.8264, Self-CIDS 0.7937, and Voice 0.8524.It also leads Imaging at 0.7467, CLIP at 0.2868, and speech Recall at 0.9674.
- Long-Horizon Audio-Visual Generation: JoyAI-Echo-1.5 receives higher preference than HappyOyster across all five human-evaluation dimensions, led by audio-visual synchronization at 48.4% versus 20.3%.It also leads story-instruction following at 38.6% versus 17.7% and dialogue/background-audio following at 39.2% versus 23.5%.
- Long-Horizon Audio-Visual Generation: JoyAI-Echo-1.5 is preferred for memory/identity consistency in 24.8% of comparisons versus 15.7% for HappyOyster, with higher audio-quality preference at 44.4% versus 33.3%.The user study reports perceptible gains in instruction following, audio-visual synchronization, and long-range consistency.
- Long-Horizon Audio-Visual Generation: The 10-minute case study maintains recognizable character identities, coherent scene evolution, and stable visual quality across distant timestamps and frequent shot transitions.Qualitative comparisons also report less character, scene, and color drift over extended generation.
- Interactive World Modeling: Both WBench variants rank at the top, with undistilled JoyAI-Echo-1.5 achieving the highest Average score of 81.7 and the causal model ranking second at 81.0.The causal model achieves the highest Interaction score at 87.9, while the undistilled model reaches Consistency 89.8.
- Interactive World Modeling: Across qualitative world-model evaluations, JoyAI-Echo-1.5-Causal better preserves scene layout, subject identity, visual style, and geometric structure while following successive controls.This stability persists over approximately 60 seconds of continuous causal streaming.
- Interactive World Modeling: At 961 frames, the undistilled model reaches VBench Overall 81.36 and revisit PSNR 15.10 dB on Simple, while rotation error rises to 12.05° on Hard.The results show strong visual persistence alongside accumulated pose drift on difficult long-horizon trajectories.
- Interactive World Modeling: On SANA-WM-Bench, JoyAI-Echo-1.5-Causal achieves the highest VBench scores on both 961-frame splits: 80.13 on Simple and 81.06 on Hard.The model retains strong 60-second autoregressive rollout capability, although distillation moderately degrades trajectory accuracy and revisit consistency relative to the undistilled model.
9 Conclusion
The paper presents JoyAI-Echo-1.5 as a framework for persistent stories and interactive worlds, combining memory, geometric control, and rollout-aware training.
- Long-Video Generation: The long-video variant uses composable cross-shot audio-visual memory to preserve character appearance, speaker identity, and narrative continuity.It supports flexible text, image, and memory conditioning.
- Interactive World Modeling: The world-model variant translates heterogeneous navigation inputs into calibrated metric 6-DoF trajectories for controller-agnostic interaction across environments and viewpoints.The framework also uses few-step distillation, causal teacher forcing, and Self-Gradient Forcing for efficient synchronized generation.
- Overall Findings: Experiments report improved cross-shot consistency, instruction following, speech fidelity, trajectory control, and long-horizon world persistence.The paper reports leading results on long-form generation evaluations and public world-model benchmarks.