Source-linked AI summary
Hy-Nomas: Towards Free-form Interactive World Generation
Qiaoling Chen, Jiaxiang Cheng, Xueqing Deng, Tiankai Hang, Ji Hou, Jiaqi Li, Yifan Liu, Tao Lu, Junshu Tang, Yu Tang, Yao Teng, Yu Tian, Heng Wang, Weizhi Wang, Xiyao Wang, Xilun Wu, Mingyu Yang, Yangbin Yu, Rui Zhao, Longjun Cao, Linghao Chen, Haoyi Duan, Jiaming Han, Mingji Han, Enhua Jiang, Shunkai Li, Yanzhe Liang, Qinwei Ma, Cong Wang, Lei Wang, Bing Wu, Ke Xing, Ray Ying, Zhentao Yu, Key Zhang, Yuan Zhou, Zhengguang Zhou, Kai Zou
TL;DR
Interactive world generation must follow successive commands while retaining relevant history and action consequences. Hy-Nomas combines an autoregressive world model with structured captions, action-aware causal attention, and a Prompt Enhancer; it reports advantages over evaluated world models across interactive tasks.
Problem
Interactive world models must respond to commands during generation, model action-dependent dynamics, and preserve consequences of earlier actions, unlike conventional fixed-prompt video generation.
Source: §1 · p. 3Method
Hy-Nomas uses an autoregressive 8B model with structured captions and action-aware causal attention, while a Prompt Enhancer turns requests and observations into structured conditions for successive segments.
Source: §1 · pp. 3–4Results
Hy-Nomas shows advantages over evaluated world models across interactive tasks, with stronger text-driven action control than the larger models in its benchmark.
Source: §1 · p. 4, §9 · p. 30Takeaways & Limitations
The released weights, code, benchmarks, and lightweight adaptation workflow support reproducible research on interactive world generation.
Source: §1 · p. 4For robotics, plausible responses alone are insufficient; predicted outcomes must match the effects of executable robot actions.
Source: §9 · p. 29
Abstract
from arXiv · showhide
Interactive world generation requires models to follow successive commands while preserving relevant history and action consequences. We present Hy-Nomas, a video world model with an 8B diffusion backbone that supports free-form text instructions and continuous camera control across multiple interaction turns. It generates variable-length segments conditioned on current inputs and cached history. Our central design aligns structured video captions with action-aware causal attention: descriptive context remains accessible across segments, while action instructions are restricted to their intended temporal windows. A vision-language Prompt Enhancer (PE) translates user requests and multimodal interaction histories into structured conditions for the next segment. A multi-stage curriculum progressively introduces causal generation, text interaction, camera control, and training on model-generated histories, followed by diffusion distillation. In human evaluation, Hy-Nomas shows superior performance over existing world models. We release model weights, code and benchmarks, together with a lightweight adaptation workflow, to support reproducible research on interactive world generation.
1 Introduction
Interactive world models must respond to commands arriving during generation while preserving earlier action consequences, unlike fixed-prompt video generation. Hy-Nomas combines multi-turn text and camera control with structured captions, action-aware causal attention, and progressive training; the paper reports advantages over existing world models across interactive tasks.
1 Introduction
- A vision-language Prompt Enhancer converts user requests and current observations into structured prompts of suitable duration for the next segment.It carries forward dialogue and generated-video history across turns.Source: §1 · p. 3
- The training strategy progressively adds segment causality, localized action conditioning, and camera control, then trains on model-generated histories before diffusion distillation.The broader recipe also includes high-quality annealing and segment forcing.Source: §1 · pp. 3–4
- Hy-Nomas supports multi-turn free-form instructions and continuous camera navigation within one autoregressive 8B model.Users can direct both events in the world and how they explore it.Source: §1 · pp. 3–4
- Structured captions and action-aware causal attention keep descriptive context accessible while restricting action instructions to their intended segment and time window.The design aims to retain action consequences without repeatedly reissuing the action.Source: §1 · pp. 3–4
- The authors release model weights and code, with bounded-cache inference and a lightweight workflow for downstream adaptation.They also release benchmarks to facilitate reproducible world-model evaluation.Source: §1 · p. 4
- Hy-Nomas is reported to outperform existing world models across interactive tasks, including text interaction, camera navigation, interleaved control, and spatial memory.The authors also identify remaining challenges in world modeling and release benchmarks for reproducible evaluation.Source: §1 · pp. 3–4
2 Data
The data pipeline turns heterogeneous videos into training samples through preprocessing, filtering, labeling, trajectory annotation, and structured captioning. Adaptive sampling shifts the training distribution to address weaknesses found in earlier stages.
2 Data
- The end-to-end pipeline processes videos through preprocessing, filtering, semantic deduplication, camera-trajectory annotation, and structured captioning.Sampling is adjusted across training stages to address previously identified weaknesses.Source: §2 · p. 4
2.1 Data Filtering
- Videos are divided into five-minute segments, split at detected shot boundaries, transcoded, and filtered for basic properties, visual quality, derivative content, and abrupt transitions.The shot-boundary detector is based on OmniShotCut and fine-tuned on high-quality annotations.Source: §2.1 · p. 4
2.2 Data Distribution
- A hierarchical taxonomy labels semantic content and visual attributes, while intermediate-level cluster annotations make unsupervised visual clusters usable for data selection.Subsequent training stages adjust sampling weights for selected subsets that need improvement.Source: §2.2 · p. 5
2.3 Camera Trajectories
- Camera trajectories are estimated for a subset of suitable videos, then filtered to exclude static cameras, cuts, distortion, and unstable or degenerate estimates.The filtering pipeline includes optical-flow checks, tracking, calibration, trajectory estimation, and a final audit.Source: §2.3 · p. 5
2.4 Video Captioning
- Structured captions maintain subject identities and timestamp actions separately from chronological scene and camera descriptions; later segments describe changes rather than repeat unchanged context.Action intervals remain within segment boundaries and form a non-overlapping chronological sequence.Source: §2.4 · p. 6, Figure 4 · p. 7
- Camera-motion sentences are removed when estimated trajectories are available, but retained as textual supervision for segments without trajectory annotations.This supports camera control through trajectory-mapped keyboard inputs or natural-language motion instructions.Source: §2.4 · p. 6, Figure 3 · p. 6
- A multi-turn VLM annotation process establishes initial scenes and subject profiles, then uses accumulated context to describe later changes and refine identities.A final text-only review corrects inconsistencies and validates the English caption and aligned Chinese translation.Source: §2.4 · pp. 6–7
3 Model Architecture
Hy-Nomas adapts a diffusion transformer for segment-wise autoregressive world generation, combining temporally structured captions, cached history, and camera conditioning. Action-aware masking keeps environment descriptions available across segments while localizing action instructions; camera trajectories provide continuous control.
3.1 Segment-wise Causal Attention
- Segment-wise attention allows access to current and preceding segments, while action-aware masking restricts each action window to its associated action text and the segment description.Descriptive heads remain accessible to later segments, but action tails are restricted to their originating segment.Source: §3.1 · p. 10
- Later-stage action-aware causal attention substantially reduces action-order errors and cross-segment action repetition.Source: §3.1 · p. 10
3.2 Camera-Trajectory Conditioning
- Camera conditioning is injected through cross-attention to temporally aligned camera tokens, and randomly disabling it per segment during training supports interleaved camera- and text-controlled segments.Camera poses are computed from relative motion independently within each segment.Source: §3.2 · pp. 10–11
- Relative-pose camera trajectories keep both translation and rotation continuous, enabling finer-grained control over direction, speed, and scale than the cited discrete-action interfaces.Source: §3.2 · p. 11
4 Training Infrastructure
The infrastructure supports long, variable-length video training by combining distributed parallelism, memory management, and workload-aware throughput optimizations.
4.1 Model and Data Parallelism
- Sequences of up to 360K tokens are supported by combining sequence and data parallelism with FSDP2 sharding across eight-GPU groups.The layout localizes parameter-sharding and sequence-parallel communication; FLOPs-balanced packing helps reduce replica workload variation.Source: §4.1 · pp. 11–12
4 Training Infrastructure
- The infrastructure combines hybrid sharding, asynchronous checkpointing, and specialized optimization to support long-running training jobs.Checkpoint resumption stores dataloader and random-number-generator states to preserve data ordering and stochastic operations.Source: §4.1 · pp. 11–12, §4.3 · p. 13
4.2 GPU Memory Optimization
- During segment forcing, streaming backward bounds peak memory, so attention-output retention and activation offload are disabled.Retaining attention outputs increased memory by approximately 1.6 GB per rank without meaningful speed improvement.Source: §4.2 · p. 12
4.3 Training Throughput Optimization
- FLOPs-based packing balances estimated per-sample compute across replicas, while Flex-Flash-Attention handles structured masks without materializing dense masks.The packing plan is constructed offline and assigned with a longest-processing-time-first greedy algorithm.Source: §4.3 · pp. 12–13
5 Model Training
Hy-Nomas is trained through a progressive sequence that develops causal generation, text and camera interaction, longer context, and efficient autoregressive generation, with a separate lightweight adaptation workflow.
5 Model Training
- The curriculum progressively adapts a bidirectional DiT to causal generation, multi-turn text and camera interaction, and longer histories before downstream adaptation.The training plan has five stages, with continual pre-training divided into three regimes.Source: §5 · p. 13, §5.1 · p. 13, Table 2 · p. 14
5.1 Training Curriculum
- Training progresses from segment-causal to action-aware attention, then segment forcing and distillation; text interaction precedes continuous camera control, while context grows to 360K tokens.The curriculum sets context to 180K tokens before action-aware CPT and 360K tokens from that stage onward.Source: §5.1 · pp. 13–14, Table 2 · p. 14
5.2 Continual Pre-Training
- Continual pre-training establishes segment causality, then restricts action instructions to their annotated temporal windows and adds camera control through frame-wise cross-attention.Camera trajectories and camera-related text are dropped with 50% probability per segment to support both camera-control and text-prompt interaction.Source: §5.2 · pp. 14–15
5.3 High-Quality Annealing
- High-quality annealing uses detailed captions and routes videos with reliable camera trajectories to camera-conditioned training, while retaining other videos as text-conditioned samples.The model increases maximum text length from 768 to 1,280 tokens during annealing.Source: §5.3 · p. 15
5.4 Segment Forcing
- Segment Forcing trains on its own cached predictions while applying flow-matching supervision to each current segment and releasing its computation graph after backpropagation.One-step low-noise estimates supply context for later segments; streaming backward accumulates gradients for one optimizer step after the sequence.Source: §5.4 · p. 15, Figure 7 · p. 16
5.5 Diffusion Distillation
- Two-stage causal distillation reduces sampling from 100 to 4 NFEs per segment, first establishing few-step generation and then refining fidelity and motion stability.The refinement stage uses hybrid forcing, combining a ground-truth prefix with autoregressively generated history on longer sequences.Source: §5.5 · p. 16
5.6 Downstream Adaptation via LoRA
- The recommended LoRA workflow trains adapters on the base autoregressive model before merging them into the distilled model, enabling customization without full-model fine-tuning or repeated distillation.The recipe uses on the order of 100 task-specific samples.Source: §5.6 · pp. 16–17, Figure 8 · p. 17
6 Model Inference
Inference pairs a history-aware Prompt Enhancer with autoregressive video generation and optimizations for deployment on a single GPU.
6 Model Inference
- A vision-language Prompt Enhancer translates user intent into structured instructions, while inference optimizations target efficient interactive deployment.The PE conditions the next segment on generated video, the current instruction, and optional camera-control signals.Source: §6 · p. 17
6.1.1 Prompt Enhancer Workflow
- At each turn, the Prompt Enhancer creates a dense caption for the next segment from the current input and complete dialogue and video history.Text mode supplies scene changes and actions; camera-control mode supplies scene changes while the camera trajectory is passed separately to the DiT.Source: §6.1.1 · pp. 17–18, Figure 9 · p. 18
6.1.2 Prompt Enhancer Training
- The lightweight Prompt Enhancer is supervised on multi-turn conversations built from synthesized prompts and augmented real-user prompts paired with interaction histories and captions.The training examples span subjects, environments, viewpoints, control modes, and action orderings.Source: §6.1.2 · pp. 18–19
6.2 Inference Optimization
- Single-GPU inference combines INT8 storage, sliding-window attention, CPU offloading, and sparse attention; video decoding uses a one-step PixelUMM-based diffusion decoder.A 100-second, seven-segment video at 1280×704 and 24 FPS requires 20.53 GiB of peak GPU memory; CPU offloading trades memory reduction for transfer latency.Source: §6.2 · p. 19
7 Evaluation
Hy-Nomas-Bench evaluates interaction and spatial memory across four benchmarks, using pairwise human judgments tailored to different models’ interfaces. Hy-Nomas performs strongly on text and interleaved control, while spatial memory remains weaker than Genie 3; LoRA adaptation and Prompt Enhancer results add further capability-specific findings.
7.1 Hy-Nomas-Bench and Metrics
- Hy-Nomas-Bench contains 280 cases across multi-turn text interaction, camera navigation, interleaved control, and spatial memory, with tests spanning varied scenes and viewpoints.The benchmarks include interactions requiring videos longer than 60 seconds and are evaluated on prompt following, physical plausibility, camera control, and memory consistency.Source: §7 · p. 19, §7.1 · p. 20, Figure 10 · p. 20
- Bench-3 tests text and camera inputs interleaved within a continuous rollout, while related WBench evaluates those control capabilities separately.Bench-4 tests memory by taking an anchor object out of view and revisiting matched camera positions and orientations.Source: §7.1 · pp. 20–21
7.2 Evaluation Protocol
- Models are evaluated through their native interaction interfaces, and three trained annotators assess pairwise wins, ties, and losses for each applicable metric.Genie 3 is not evaluated on text interaction, and HappyOyster does not support interleaved text and camera control.Source: §7.2 · p. 22
7.3 Comparison with Other World Models
- Across Bench-1 to Bench-3, Hy-Nomas achieves the best performance; it also has stronger prompt following and physical-plausibility results than the compared models on Bench-1.Bench-2 camera-control advantages are smaller, and Bench-3 camera-control performance is more comparable to LingBot-World v2.Source: §7.3 · pp. 22–23
- Hy-Nomas trails Genie 3 on spatial memory, performs comparably to HappyOyster, and substantially outperforms LingBot-World v2.Against Genie 3, Hy-Nomas has a 13.3% win rate and 30.0% loss rate; against LingBot-World v2, its win rate is 58.3% and loss rate 1.7%.Source: §7.3 · p. 24, Figure 17 · p. 26
7.4 LoRA Adaption Results
- Cinematic LoRA-SFT improves prompt following and physical plausibility, but camera control regresses, with 19% wins versus 37% losses.The authors hypothesize that limited camera-trajectory diversity in the SFT data may weaken underrepresented camera motions.Source: §7.4 · p. 25
- The robotics-specific adapted checkpoint performs out-of-distribution dual-arm tasks, whereas the baseline confuses dual-arm and human-hand generation.The authors attribute the baseline issue to the training set’s heavy distribution of human egocentric data.Source: §7.4 · p. 25
7.5 PE Results
- Hy-Nomas-PE improves prompt following and physical plausibility over Qwen3.8-27B while maintaining camera control, and nearly always follows the required response format.The reported format adherence is nearly 100%, versus approximately 50% for Qwen3.8-27B and 70% for Kimi-K3.Source: §7.5 · p. 26
8 Related Work
Related work spans predictive visual representations, persistent 3D scene models, reconstruction, and video-based generation. Hy-Nomas focuses on video-based interactive generation, combining free-form text and camera control; open questions include robust physical dynamics, spatial memory, and long-horizon generation.
World models
- World-model approaches range from predictive visual representations and persistent 3D models to geometry reconstruction and video-based modeling of action-conditioned transitions.The paper places Hy-Nomas in the video-based direction, motivated by the scalability and diversity of available video data.Source: §8 · pp. 26, 28
- Whether scaling data and model capacity with reinforcement learning can overcome limits in physical generalization remains an open question.The paper notes that controlled studies have found limits in physical generalization despite growth in model capacity and training data.Source: §8 · p. 27
Video-based world models
- Video-based world models must respond to actions introduced during a rollout and preserve their consequences in later observations.Related systems include Genie’s latent-action approach, LingBot-World v2’s long-horizon generation, and HappyOyster’s separate Directing and Wandering modes.Source: §8 · pp. 28–29
- Hy-Nomas targets native alternation between free-form text and continuous camera control in one 8B model, using temporally structured supervision, causal context, and a VLM-based director.The evaluation examines text interaction, camera control, interleaved control, and spatial memory separately.Source: §8 · p. 29
9 Discussion and Future Directions
The discussion identifies data and captioning, memory management, infrastructure, and robotics adaptation as important directions for interactive world models. It emphasizes that visual quality alone cannot establish action following, physical consistency, or long-horizon scene memory.
Gaming and robotics
- Robotic control requires predictions tied to executable robot actions and evaluation of task success; the paper presents LoRA fine-tuning as an initial adaptation step.The authors note that available datasets have limited coverage across tasks, environments, and robot embodiments.Source: §9 · p. 29
Agentic world models
- Long-horizon rollouts make full GPU KV-cache retention costly, motivating retrieval of relevant older context from CPU memory or persistent storage.The discussion proposes coordinating geometry, reference images, text summaries, and cached features as complementary memory representations.Source: §9 · p. 29
- Retrieval alone cannot update off-screen state changes, so a future agentic system would need to update explicit world state alongside retrieved visual references.The paper illustrates this with a clock that should retain its appearance while displaying a later time after the user returns.Source: §9 · p. 30
Data and captioning
- Detailed captions may reduce the diffusion transformer’s burden of inferring unspecified content, but caption accuracy and Prompt Enhancer specificity become central challenges.The authors propose curating data for accurate descriptions of scene changes, camera motion, blur, and editing effects.Source: §9 · p. 30
Infrastructure and scaling
- Higher resolution and longer context increase training cost, inference latency, and KV-cache memory, making efficient attention and cache management important for longer rollouts.The discussion says world-model capabilities require direct evaluation and cannot be inferred from visual-quality gains alone.Source: §9 · p. 30