EVA: An Embodied World Model for Future Video Anticipation

• Existing robotics video-generation models focus on conditional simulation, limiting them to fixed-frame, single-round predictions and leaving flawed outputs undetected without human intervention.These limitations degrade performance in OOD scenarios.Source: §1 · p. 1
• RoG inserts intermediate reasoning between understanding and generation, enabling iterative self-correction, deeper understanding, modular interpretation, and longer video sequences.It can combine any visual-language model (VLM) with a video-generation model (VDM) as a unified world model.Source: §1 · p. 2
• EVA-Bench evaluates embodied video anticipation through four meta-tasks—Action-Description, How-To, Finish-Thinking, and Next-Step—using in-domain and OOD videos and both VQA and pixel-level prediction metrics.It draws on egocentric and robotics datasets and covers understanding and generation.Source: §1 · p. 2, Figure 2 · p. 2
• EVA combines understanding and generation in a chunk-wise autoregressive model that uses reflection conditions to adaptively extend video length and VLM supervision for consistent, self-correcting generation.The paper presents EVA as a model tailored to RoG for embodied scenarios.Source: §1 · p. 2
• Experiments on EVA-Bench show RoG performs strongly on in-domain and OOD tasks, while robot-simulator evaluations show EVA and RoG support real-world task execution.Source: §1 · p. 3

Video Generation

• Diffusion-based models, including VideoCrafter (Chen et al., 2023; 2024) and VideoPoet (Kondratyuk et al., 2023), have advanced high-quality video generation.Image-conditioned models such as Dynamicrafter (Xing et al., 2023), Stable Video Diffusion (Blattmann et al., 2023), and Animatediff (Guo et al., 2023) also achieve impressive generation quality and are used in many areas.Source: §2 · p. 3
• Video-generation models still struggle with reasoning, consistency, and understanding, prompting work that combines generation models with LLMs (He et al., 2024).The passage notes that this weakness also occurs in some long-video-generation methods (Wang et al., 2023a; Yin et al., 2023).Source: §2 · p. 3

Embodied World Model

• Embodied world models support gaming, driving, video prediction, and robot simulation, but a task-level video predictor is still needed.Examples include Genie (Bruce et al., 2024) for gaming, Vista (Gao et al., 2024; Wang et al., 2023b) for autonomous driving, and Seer (Gu et al., 2024) and AID (Xing et al., 2024) for future-frame motion prediction; VLP (Du et al., 2023b) integrated language and video generation for robot planning but remained at the concept level.Source: §2 · p. 3

3.1. Hierarchy of Video Prediction

• Prediction is organized as frame-level next-frame synthesis, task-level coherent execution, and serial task-level subtask sequencing; RoG centers task-level world modeling that combines frame generation with task reasoning.Source: §3.1 · p. 3

3.2. Reflection of Generation World Model

• RoG encodes the initial observation, instruction, and current predictions, then reflects to extend, regenerate, or output predictions until they converge or meet stopping criteria.Figure 3 depicts chunk-wise frame extension with completion checks.Source: §3.2 · pp. 3–4, Figure 3 · p. 4

3.3. RoG World Model Task Decomposition

• RoG evaluation decomposes prediction into Action-Description, Finish-Thinking, How-To, and Next-Step, assessed with language, video-quality, completion, and task-level prediction metrics.Action-Description uses text similarity, GPT-4 (OpenAI, 2024) keyword matching, and CLIP (Radford et al., 2021); Finish-Thinking uses yes/no accuracy, GCE, Fréchet Video Distance (Unterthiner et al., 2018), and VBench (Huang et al., 2024). How-To combines language and video scores as EVA-Score, while Next-Step uses How-To metrics plus task-level QA accuracy and frame-ground-truth intermodality correlation.Source: §3.3 · p. 4

4. Embodied Video Anticipator

• EVA combines pretrained multimodal models, multistage training, and domain-specific Ensemble-LoRA for embodied video prediction.The paper identifies these as the model’s key elements.Source: §4 · p. 4

4.1. Model Structure

• EVA pairs a 7B VLM with a 1.5B VDM, using clustered video tokens, interaction tokens, and cross-attention alignment for generation.The VLM combines a CLIP (ViT-L/14) visual encoder with Vicuna-v1.5; the VDM generates 16-frame videos in 2 seconds, and task-token gating combines domain-specific LoRAs.Source: §4.1 · p. 5

4.2. Multi-stage Training Strategy

• Training progresses from visual-adapter warm-up on COCO and curated CC3M-595K, to embodied instruction tuning, then adaptation of the generation adapter and task-specific VDM Ensemble-LoRA.The instruction-tuning stage uses open-domain multimodal instruction data and EVA’s instruction-tuning dataset.Source: §4.2 · p. 5

4.3. Autoregressive Frame Extention

• After initial VDM generation, RoG prompts check task completion; if unfinished, EVA feeds selected final frames into another round, concatenating chunks until the VLM judges completion.The process iteratively extends the video based on task-completion verification.Source: §4.3 · p. 5

5. Embodied Video Anticipation Benchmark

• Under RoG, the benchmark section divides complex interleaved multimodal generation into four subtasks and introduces the dataset and benchmark.Fine-grained metric descriptions and benchmark composition are provided in Appendix C.Source: §5 · p. 5

5.1. Dataset Construction

• The 500K-pair EVA instruction-tuning dataset combines four sources: GPT-4o-extended How-To and Action-Description prompts, RoboVQA-based unfinished-task questions, and ordered steps from Ego-Exo4D and RoboVQA.Annotations are standardized into subject-verb-object format.Source: §5.1 · p. 5

5.2. Benchmark Construction

• EVA-Bench comprises 125 curated samples across real and simulated robots and egocentric activities, with annotated completion frames and expert-refined OOD robotics prompts.Scenarios include pick-and-place, cooking, bike repair, COVID testing, and indoor organization.Source: §5.2 · p. 5

6.5. Qualitative analysis

• Figure 4 shows EVA generating robot and human-ego motion from instructions, extending video over long horizons, and generating video from reasoning results.An example has EVA answer a Next-Step question with “Wipe hand” and generate the corresponding video.Source: Figure 4 · p. 6, §6.5 · p. 8

6.1. Video Question Answering

• 62.63: EVA achieves the best score under GPT-4o evaluation, while zero-shot Qwen2 is the best-performing open-source model at 29.58.Fine-tuned comparison models use a 50K subset of EVA-Instruct with LoRA or full-parameter training.Source: §6.1 · p. 7, Table 1 · p. 7

6.2. Longer Video Generation

• 97.11 SC, 96.01 BC, and 99.31 MS: EVA excels on these metrics and achieves the lowest FVD at 184.81.EVA-Gen records GCE 86.83 and FVD 177.28, which the authors attribute to its LoRA design.Source: §6.2 · p. 7, Table 2 · p. 7

6.3. Interleaved Generation

• 77.83 EVAS-V and 81.64 EVA-Score: EVA outperforms other approaches under RoG in two-stage generation evaluation.For Next-Step, EVA performs best; inaccurate next-step descriptions from LLAVA-OneVision and Qwen2-VL-7B are associated with lower EVAS-V scores of 51.75 and 57.23, respectively.Source: Table 3 · p. 7, §6.3 · p. 8

6.4. Evaluation on Robot Planning

• 28% higher overall success than AVDC: EVA reaches 100% on Move Object, has five more OOD successes than AVDC, and exceeds EVA without RoG by seven cases.Open/Close is challenging for examples involving transparent glass doors, such as opening a refrigerator door.Source: §6.4 · p. 8, Table 4 · p. 8
• EVA significantly outperforms the no-RoG baseline in CALVIN video and simulation evaluations, and EVA with CMLP2 performs best in simulation.CMLP and CMLP2 condition on predicted video and first-action status to output 16 action frames; CMLP2 adds techniques including simple action chunks.Source: §6.4 · p. 8, Table 5 · p. 8

7. Conclusion

• EVA combines video generation with VQA-style reasoning and chunk-wise autoregression to produce high-fidelity predictions and robust generalization for longer sequences.The conclusion presents these capabilities as key advances in video prediction for embodied scenarios.Source: §7 · p. 8
• EVA-Bench enables comprehensive, fine-grained evaluation and validates RoG across diverse downstream tasks.The conclusion describes the benchmark as supporting evaluation of embodied video prediction.Source: §7 · p. 8
• These contributions pave the way for large-scale pre-trained models in real-world video prediction and embodied AI applications.Source: §7 · p. 9

Impact Statement

• EVA and EVA-Bench provide a unified benchmark for assessing world-model capabilities and advancing embodied intelligence; the authors identify no societal consequence requiring special emphasis.Source: Impact Statement · p. 9

A. Appendix: Model Architecture and Training

• EVA combines a pre-trained diffusion generator and visual language model for autoregressive world prediction, with a visual projector, image-conditional resampler, generation adapter, and domain-specific Ensemble LoRA.The components are trained in three stages.Source: Figure 5 · p. 13, A. Appendix: Model Architecture and… · p. 13

A.1. Vision Language Model

• To fit limited compute, EVA uses ChatUniVi’s parameter-free adaptive token clustering to represent more frames within 2048 tokens and fully fine-tunes VLMs for prediction and planning.The VLM is independently trained on the Embodied-Video-Description dataset; existing models struggled with prediction and planning due to limited relevant training data.Source: §A.1 · p. 13

A.2. Latent Diffusion Model

• For video, EVA maps framewise <image> tokens into visual features and uses prefix <IMG P> query embeddings as the VDM condition instead of text prompts.Source: §A.2 · p. 14

A.3. Ensemble of LoRA

• Ensemble-LoRA adapts video diffusion across domains with task-token-controlled gating over domain-specific low-rank modules, enabling efficient adaptation without discarding earlier adaptations.The described domains include human-egocentric, real-robot, and simulation-robot settings.Source: §A.3 · p. 14

A.4. Cross-Attention Generation Adapter

• The generation adapter aligns VLM hidden features with VDM text-embedding features through cross-attention and is trained using diffusion denoising loss.Source: §A.4 · p. 15

A.5. EVA Model Architecture

• EVA uses a Vicuna-based VLM following ChatUniVi and a 3D U-Net VDM with spatial downsampling, upsampling, skip connections, 3D convolutions, and attention layers.Source: §A.5 · p. 15

A.6. VLM Training Details

• VLM training proceeds from caption alignment on COCO and CC3M, through multimodal QA instruction tuning, to domain-specific adapter and Ensemble-LoRA tuning; the adapter uses all data while LoRA updates per domain.Stage 1 uses one epoch; Stage 2 uses two epochs, with 336 × 336 image and frame inputs.Source: §A.6 · pp. 15–16

A.7. Training Dataset

• EVA-Instruct contains 500K QA pairs across four tasks, drawing on Open-X-Embodiment, Ego4d, Ego-Exo4d, and CALVIN.ChatGPT-4v generated question-answer pairs, and the source annotations can be restructured into subject, verb, object, location, and destination components.Source: §A.7 · p. 16
• Table 8 identifies EVA-Instruct’s dataset sources, training-example counts, and mixture weights.The supplied passage gives no individual table values.Source: Table 8 · p. 16
• Action-description prompts use natural-language variants, with GPT-4V-generated scene-understanding guidelines included at 50% probability.The instructions for How-to, Finish-Think, and Next-Step are constructed separately for their respective formats and dataset differences.Source: §A.7 · p. 16
• How-To-200K and Action-Description-200K standardize annotations into subject, verb, object, location, and destination; Finish-Think-50K and Next-Step-50K use task-specific examples.Finish-Think draws on the first 25% of videos and convertible RoboVQA completion questions; Next-Step uses Ego-Exo4D key steps and ordered steps from long-horizon RoboVQA annotations.Source: §A.7 · pp. 16–17

B. Appendix: Extensive experiments

• On simulation robot data, EVA follows text instructions with different actions from the same input video, and the reported following is quite accurate with proper action.Source: B. Appendix: Extensive experiments · p. 17, Figure 6 · p. 17
• On real robot videos, Figures 7 and 8 demonstrate EVA’s action-following generation, while Figure 7 shows it answering a question by generating video.Source: B. Appendix: Extensive experiments · p. 17, Figure 7 · p. 18
• With proper training, EVA shows some planning ability in an egocentric example, generating motion that first turns right and then gets the sugar.Source: B. Appendix: Extensive experiments · p. 17

B.1. Comparison in RT1

• On seen RT-1 tasks, EVA’s total success rate was 28% higher than AVDC’s, whose success rate was 58%.The evaluation randomly selected 50 validation tasks spanning six task types.Source: Table 9 · p. 18, §B.1 · p. 18
• On seen Move Object tasks, EVA achieved a 100% success rate, while Open/close remained difficult in cases such as opening a transparent fridge door.Source: §B.1 · p. 18
• On unseen prompts, AVDC performed better than EVA without finish-thinking on Place, knock, and Open/Close tasks because their trajectories exceeded what EVA could generate simultaneously; extending the video improved EVA’s success rate.Unseen prompts were created by manually changing the subject or object in existing cases.Source: §B.1 · p. 18

C. Appendix: EVA-Benchmark

• EVA-Bench curates 125 high-quality samples spanning real-world robots, simulated robots, and egocentric daily activities across diverse scenarios.Examples include pick-and-place, cooking, bike repair, COVID testing, and indoor organization.Source: C. Appendix: EVA-Benchmark · p. 18

C.1. Benchmark Examples

• Benchmark examples show cooking, COVID testing, and bike repair alongside robotic-arm object manipulation and simulated-robot scenes.The examples span egocentric human videos, real-world robots, and simulated robots.Source: §C.1 · p. 19, C. Appendix: EVA-Benchmark · p. 19

C.2. EVA Score Language Metrics

• Language evaluation combines BLEU, METEOR, ROUGE-L, SPICE, and reference-free CLIPScore with GPT-4o judging based on object, action, location, and attribute.GPT-4o is used because conventional text metrics may miss important action distinctions or reward responses that omit key information.Source: §C.2 · pp. 19–20

C.3. EVA Score Video Metrics

• Video metrics assess text-video consistency with ViCLIP, motion smoothness using interpolation-model motion priors, and subject and background consistency using DINO features.The subject and background metrics use order-statistics schemes.Source: §C.3 · pp. 20–21

C.4. Last Frame Comparison Feature Selection

• For goal completion estimation, EVA-Bench selects a DINOv2-finetuned DreamSim variant for perceptual alignment and normalizes its scores so higher values indicate closer target alignment.DreamSim emphasizes foreground objects and semantic content while remaining sensitive to color and layout; CLIP and DINOv2 visualizations have limitations including irrelevant attention and manually adjusted PCA parameters.Source: §C.4 · p. 21

C.5. Model Inference Prompts

• For each EVA-Bench meta-task, models other than EVA use the same task-specific inference prompts listed in Table 17.The prompts guide VLMs toward concise, non-redundant answers focused on annotated key actions.Source: §C.5 · p. 21, Table 17 · p. 27

C.6. Evaluation Prompts

• GPT-4o scores step descriptions on object, action type, location, and attribute, assigning each criterion 1, 0.5, or 0 and averaging the four scores.The approach follows the use of GPT-4o for VLM assessment because BLEU, METEOR, and ROUGE-L poorly capture action, object, and location similarities.Source: §C.6 · pp. 21–22