Source-linked AI summary
HelloWorld: Enabling Socially Interactive Characters in Video World Models
Liangyang Ouyang, Ruicong Liu, Xuangeng Chu, Kaipeng Zhang, Yoichi Sato
TL;DR
Video world models have not supported social interaction between users and in-world characters. HelloWorld introduces an interactive model trained through self-distillation and controlled at inference with temporal attention masking, and reports superior interaction performance while maintaining strong video quality and camera following.
Problem
Existing video world models lack support for social interaction between users and in-world characters, while benchmarks overlook characters within generated worlds.
Method
HelloWorld uses self-distillation on self-generated interaction-and-camera videos and a training-free temporal cross-attention mask to control interaction timing.
Results
HelloWorld surpasses existing world models on interaction metrics while maintaining state-of-the-art video quality.
Takeaways & Limitations
The model enables users to create social interactions with in-world characters through an interaction button and provides a benchmark for evaluating them.
Takeaways & Limitations
HelloWorld does not yet support real-time user interaction and relies on pre-specified camera trajectories and interaction scripts.
Abstract
from arXiv · showhide
Despite the remarkable recent progress of video world models, social interaction between users and the characters within these worlds remains unsupported. To fill this gap, we present HelloWorld, a video world model that enables social interaction with in-world characters. With a single button press, users can prompt the on-screen character to respond toward the camera, e.g., turning to the viewer, waving, nodding, or speaking a short greeting. To make these interactions natural, we propose a self-distillation pipeline that finetunes the video generation model on data synthesized by itself. Each synthesized clip contains both social interactions and camera motion, allowing the model to learn camera-pose conditioning without degrading interaction quality. At inference, we further introduce a training-free module that determines when the interaction occurs. Upon a button press, it modulates the cross-attention masks of the DiT so that the interaction-related text prompt attends only to the frames within the press window, temporally localizing the character's response. We further build HelloWorldBench, a 400-sample benchmark with three social interaction metrics alongside three conventional metrics, for evaluation. Experiments demonstrate that HelloWorld surpasses a variety of baselines in interaction quality, while maintaining state-of-the-art picture aesthetics and camera-pose following. Project page: https://github.com/AlayaLab/HelloWorld
1 Introduction
HelloWorld addresses the lack of user-directed social interaction in video world models by combining interactive character control, self-distillation, and temporal attention control. HelloWorldBench evaluates these capabilities, and experiments report strong interaction, video-quality, and camera-following performance.
- Motivation: Existing video world models support camera control or event triggering, but not social interaction between users and in-world characters.Characters may remain static, or exhibit only ambient rather than user-directed motion.
- HelloWorld: HelloWorld adds an interaction button alongside camera trajectory and text prompt, enabling users to trigger character responses toward the camera.The character follows the interaction described in the text prompt when the button is pressed.
- Training: Its self-distillation pipeline converts a pretrained video generation model into a controllable world model using synthesized videos containing social interactions and camera motion.The pipeline curates prompts, annotates camera trajectories, and uses point-cloud warping during training.
- Evaluation: HelloWorldBench combines 120 images with designed prompts and camera trajectories to create 400 evaluation samples and introduces three interaction-specific metrics.The benchmark evaluates interaction behavior in addition to conventional video quality and camera accuracy.
- Results: HelloWorld substantially outperforms existing methods on all three interaction metrics while remaining competitive with or surpassing recent world models in video quality and camera following.The authors report state-of-the-art performance overall.
2 Related Work
Prior interactive video world models focus on exploring environments, camera and control inputs, or selected object and NPC interactions. HelloWorld instead targets multimodal, viewer-directed social interaction with diverse human and non-human characters.
- Interactive Video World Models: Interactive world models provide controls for camera, keyboard, skills, and events, enabling exploration of generated environments.These controls primarily steer the world or trigger events rather than establish social interaction with its subjects.
- Interactive Video World Models: Existing work has studied object interactions and NPC-to-NPC control, but the latter is limited to a single game.Interaction with subjects in the generated world remains comparatively underexplored.
- Interactive Video World Models: HelloWorld is presented as the first video world model focused on social interaction between in-world characters and the user across diverse stylized characters and social behaviors.Its scope includes viewer-directed interaction rather than only environmental exploration or object control.
- Multimodal Social Interactions: Multimodal social interaction includes communication through speech, facial expressions, gaze, gestures, and body movements.Prior generative work covers human social interaction, while social world models are limited to text modality.
- Multimodal Social Interactions: HelloWorld studies multimodal social interaction in video world models, covering actions, gestures, facial expressions, speech, and non-human characters.The paper positions this as broader than prior text-only social world models.
3 Proposed Method
HelloWorld converts a pretrained video generator into a camera-controllable social world model through self-distillation, then temporally localizes interactions with a training-free attention mask.
- Task formulation: HelloWorld formulates generation around a first frame, text prompt, camera trajectory, and interaction window, producing camera-following videos with viewer-directed character responses.The interaction window specifies when the social interaction occurs, while the camera trajectory specifies movement across video frames.
- Warp video condition: The warp video re-renders the first frame along the target camera trajectory, providing frame-aligned geometric guidance while leaving missing regions for the model to complete.It is constructed by lifting the first frame into a point cloud and reprojecting it onto each target camera pose.
- Self-distillation training: Self-distillation uses the base model’s interaction-rich videos with camera motion, then finetunes a lightweight LoRA under warp-video conditioning.The pipeline recovers point clouds and camera trajectories, renders warp videos, selects visible tokens, and reconstructs the generated clips.
- Self-distillation training: The camera prompt is removed during training so camera following is controlled solely by the warp video, without external data collection or human annotation.The training objective uses noisy latent interpolation and predicts the velocity field under flow matching loss.
- Inference: At inference, keyboard inputs determine the camera trajectory and warp video, while a button press opens an interaction window for temporal control.The forward pass follows the same conditioning procedure used during training.
- Inference: The temporal cross-attention mask blocks frames outside the press window from attending to interaction text, localizing the response without training or meaningful computational overhead.Frames outside the window remain ambient rather than responding to the interaction prompt.
4 HelloWorldBench
HelloWorldBench evaluates viewer-directed social interaction alongside conventional world-model quality and controllability, using diverse first-frame scenes and interaction-specific metrics.
- Benchmark construction: HelloWorldBench combines 120 high-quality images with designed interactions, four camera trajectories, and randomized early, middle, or late interaction timing.The images cover humans, animals, toys, and robots across diverse scenes and visual styles.
- Evaluation metrics: The benchmark’s three social metrics separate what interaction occurs, when it occurs, and whether it is directed toward the viewer.These metrics complement standard measures of background consistency, aesthetics, and camera controllability.
- Evaluation metrics: ActAcc measures whether the generated character performs the prompted interaction using an eight-way VLM multiple-choice judgment.It is the fraction of videos for which the VLM selects the correct prompted action.
- Evaluation metrics: TimeAcc measures whether the interaction occurs in the user-specified temporal segment, excluding videos judged to contain no interaction.The judge selects among three uniformly divided temporal segments and an additional no-interaction option.
- Evaluation metrics: GazeDev measures viewer direction by computing mean angular deviation between character gaze and the camera’s optical axis during the interaction window.For human samples, a 90° deviation is assigned when no face is detected.
5 Experiments
Experiments show that HelloWorld provides stronger social interaction control than competing world models while preserving video quality and camera following. Ablations, qualitative comparisons, computational analysis, and a user study examine the sources and costs of these results.
- Main Results: HelloWorld substantially outperforms existing world models on all three social interaction metrics while remaining competitive with or surpassing them in video quality and camera following.The model achieves the best camera-following score, and self-distillation preserves the base model’s video quality.
- Main Results: 81.7% TimeAcc contrasts with baselines near 30%, showing substantially stronger control over when interactions occur than the 33.3% random-guess level.Baseline models can produce character actions but generally do not control their timing.
- Qualitative Comparison: Qualitatively, HelloWorld produces faithful viewer-directed actions, whereas baselines often miss the prompted interaction, fail to face the camera, or hallucinate extra content.LingBot-World and SANA-WM generate some actions but remain prone to hallucination.
- Training-Data Ablation: Self-generated interaction data improves action accuracy and gaze deviation relative to real-video training, while other metrics are barely affected and timing remains unchanged.Real videos without interactions teach inpainting but provide no signal for engaging with the camera; self-generated clips contain gazing and acting toward it.
- Temporal-Mask Ablation: Applying a temporal mask makes interactions fall within the designated window, whereas removing it leaves interactions occurring at arbitrary times.The ablation evaluates no mask, video-only masking, and video-plus-audio masking; masking both streams gives the best overall performance.
- Computational Cost: Warp-video tokens increase inference time by roughly 20% and FLOPs by 36% over LTX-2.3, while HelloWorld’s 0.26 s per-frame time remains competitive with baselines.The reported changes are 50.3s →60.2s and 6.9 →9.4×10^15 FLOPs; per-frame time is comparable to SANA-WM and below WorldPlay and LingBot-World.
- User Study: In a 30-rater study on 41 samples, HelloWorld is preferred over all four competing methods on action naturalness, viewer interaction, and scene quality.Every confidence interval lies above 66%, with the largest interaction margin against the LoRA trained on real videos.
6 Conclusion
HelloWorld enables social interaction with in-world characters through a self-distillation pipeline and training-free temporal control, without manually collected or annotated data. It surpasses existing world models on interaction metrics while maintaining state-of-the-art video quality, but does not yet support real-time interaction.
- HelloWorld turns a base video generation model into an interactive world model using self-distillation without manually collected or annotated data.
- A temporal cross-attention mask provides training-free control over when interactions occur.
- HelloWorldBench is introduced as the first benchmark for socially interactive world models.
- HelloWorld surpasses existing world models on interaction metrics while maintaining state-of-the-art video quality.
- HelloWorld does not yet support real-time interaction, and generation depends on pre-specified camera trajectories and interaction scripts.