Source-linked AI summary
Active Intelligence in Video Avatars via Closed-loop World Modeling
Xuanhua He, Tianyu Yang, Ke Cao, Ruiqi Wu, Cheng Meng, Yong Zhang, Zhuoliang Kang, Xiaoming Wei, Qifeng Chen
TL;DR
Video avatars can preserve identity and follow driving signals but lack agency for long-term, adaptive goals. The paper introduces L-IVA and ORCA, which combines closed-loop outcome verification with hierarchical planning and execution. On L-IVA, ORCA outperforms open-loop and non-reflective baselines in task success and behavioral coherence.
Problem
Current video avatars follow driving signals but cannot autonomously pursue long-term goals through multi-step planning and adaptive environmental interaction.
Source: §1 · p. 1Method
The paper introduces L-IVA for evaluating goal-directed planning in stochastic generative environments and ORCA, which uses an OTAR loop and dual-system planning and action grounding.
Source: §1 · p. 2Results
ORCA outperforms open-loop and non-reflective baselines in task success rate and behavioral coherence on L-IVA.
Source: §1 · p. 3Takeaways & Limitations
The authors conclude that reflection supports coherent multi-step task completion by preventing belief corruption in generative environments.
Source: §6 · p. 9ORCA’s capabilities are bounded by its underlying VLM and I2V foundation models.
Source: §11 · p. 13
Abstract
from arXiv · showhide
Current video avatar generation methods excel at identity preservation and motion alignment but lack genuine agency, they cannot autonomously pursue long-term goals through adaptive environmental interaction. We address this by introducing L-IVA (Long-horizon Interactive Visual Avatar), a task and benchmark for evaluating goal-directed planning in stochastic generative environments, and ORCA (Online Reasoning and Cognitive Architecture), the first framework enabling active intelligence in video avatars. ORCA embodies Internal World Model (IWM) capabilities through two key innovations: (1) a closed-loop OTAR cycle (Observe-Think-Act-Reflect) that maintains robust state tracking under generative uncertainty by continuously verifying predicted outcomes against actual generations, and (2) a hierarchical dual-system architecture where System 2 performs strategic reasoning with state prediction while System 1 translates abstract plans into precise, model-specific action captions. By formulating avatar control as a POMDP and implementing continuous belief updating with outcome verification, ORCA enables autonomous multi-step task completion in open-domain scenarios. Extensive experiments demonstrate that ORCA significantly outperforms open-loop and non-reflective baselines in task success rate and behavioral coherence, validating our IWM-inspired design for advancing video avatar intelligence from passive animation to active, goal-oriented behavior.
1. Introduction
Current video avatars preserve identity and follow driving signals but lack agency for long-term, adaptive goals. The section introduces L-IVA and ORCA to address this gap through closed-loop state tracking and hierarchical planning and execution.
1. Introduction
- Video avatars lack genuine agency for long-term goals, despite progress in identity preservation and alignment with speech, text, or pose signals.The section links this gap to limited application in broader, more dynamic scenarios.Source: §1 · p. 1
- The task requires inferring hidden state from observation, predicting future states, and planning toward goals; the section characterizes this setting as a POMDP and motivates Internal World Models.The agent sees generated clips rather than the complete world state.Source: §1 · p. 1
- Stochastic generations and partial visual observations complicate state tracking, while open-domain semantic actions require detailed, model-specific control signals.The section motivates closed-loop verification and hierarchical planning to address these distinct challenges.Source: §1 · pp. 1–2
- L-IVA evaluates goal-directed planning in stochastic generative environments through sequential clips that depict task completion via meaningful object interactions.Unlike single-clip aesthetics benchmarks, it starts from an initial scene and high-level goal.Source: §1 · p. 2
- ORCA’s OTAR loop estimates and updates the internal state, plans sub-goals, then verifies outcomes and triggers corrections against generative uncertainty.The section describes reflection as verifying outcomes to support robust state tracking and execution.Source: §1 · p. 2
- Its dual-system architecture assigns strategic progress evaluation, sub-goal planning, and state prediction to System 2, while System 1 grounds plans in model-specific action captions.The two specialized VLM modules address open-domain planning and precise execution.Source: §1 · p. 2
- ORCA and L-IVA target autonomous, multi-step goal completion, and ORCA outperforms open-loop and non-reflective baselines in task success and behavioral coherence.The authors present ORCA as an Internal World Model-inspired framework for active intelligence in generative video avatars.Source: §1 · pp. 2–3
2. Related Work
Prior video-avatar and video-agent systems focus on reactive generation or refining individual clips, while this paper frames avatar control as long-horizon interaction under partial observability and generative stochasticity. It introduces L-IVA and ORCA, combining strategic planning, model-specific action grounding, and outcome verification.
2.1. Video Avatar Model
- Earlier avatar methods preserve identity or generate motion from signals, but frame generation as reactive processing rather than planning actions toward long-term goals [13] [14] [17] [18] [19].The paper positions cognitive reasoning before synthesis as its alternative to reactive generation.Source: §2.1 · p. 3
2.2. Agent for Video Generation
2.3. World Models for Agentic Planning
- Prior internal-world-model agents address embodied planning, games, or navigation, often assuming low-variance environments unlike stochastic video generation, [1], [2] [7] [24].The paper identifies that environmental difference as a distinction from its setting.Source: §2.3 · p. 3
3.1. L-IVA Task as POMDP
- L-IVA formulates avatar control as a POMDP: the agent infers hidden states from partial video observations and maintains a belief state to choose actions.The I2V model serves as a stochastic world simulator, and reward is sparse and terminal.Source: §3.1 · p. 3
3. Method
- L-IVA requires autonomous achievement of high-level intentions through sequential interaction with a generative environment, rather than animation driven by predefined signals.The agent acts through natural-language captions and observes generated video frames.Source: §3.1 · p. 3
3.2.1. Overview
- ORCA combines closed-loop outcome verification with a dual-system architecture to address generative uncertainty and the distinct demands of planning and execution.Its designs use pre-trained VLMs through structured prompting and require no task-specific training.Source: §3.2.1 · p. 4
3.2.2. Hierarchical Dual-System Architecture
- System 2 performs strategic reasoning, while System 1 grounds its plans in detailed action captions tailored to the I2V model.System 2 uses broad VLM knowledge for open-domain compositional reasoning without handling model-specific formatting.Source: §3.2.2 · p. 4
3.2.3. Closed-loop OTAR Cycle
- ORCA’s OTAR cycle verifies generated outcomes against predictions, revising actions or replanning to prevent belief corruption from stochastic generations.The cycle continues until the remaining subgoals are complete.Source: §3.2.3 · p. 5
4. L-IVA Benchmark
L-IVA evaluates long-horizon, goal-directed task completion, addressing the gap between per-clip aesthetics benchmarks and interactive planning. Its benchmark uses multi-step tasks and combines VLM-based judgments with human annotations.
4.1. Overview
- L-IVA contains 100 tasks across five real-world scenario categories, each requiring 3–8 interaction steps and involving more than three objects.Tasks use fixed-viewpoint, single-room settings to avoid spatial inconsistencies in current I2V models.Source: §4.1 · p. 6
4.2. Dataset Construction
- Each task pairs a scene with an object inventory, a high-level intention, and a reference action sequence, while accepting alternative valid action orderings.Evaluation prioritizes goal achievement over exact trajectory imitation.Source: §4.2 · p. 6
4.3. Evaluation Protocol and Metrics
- Evaluation combines VLM judgments and human annotations, using Task Success Rate for subgoal progress and Best-Worst Scaling for human preference rankings.Subgoal completion is manually verified; diagnostic measures assess physical plausibility and action fidelity.Source: §4.3 · p. 6
5. Experiments
The experiments evaluate L-IVA across multiple scenarios and metrics, comparing ORCA with open-loop, reactive, and world-model-based alternatives. ORCA leads on average task success and physical plausibility, while qualitative comparisons and ablations examine reflection and hierarchical control.
5.1. Benchmark and Dataset
- L-IVA covers five balanced scene categories, uses 92 synthetic and 8 real images, and averages 5.0 subgoals per task; comparisons include open-loop, reactive, and deterministic-world-model paradigms.The evaluation spans five scenarios and measures task success, execution quality, video quality, and human preference.Source: Figure 3 · p. 6, §5.1 · p. 6
5.2. Implement Details
- All methods use the same vision-language model and video-generation settings, with prompts tailored to each method’s reasoning structure.ORCA is training-free and uses Gemini-2.5-Flash and Wanx2.2 with distilled LoRA.Source: §5.2 · p. 6
Evaluation on Task Success Rate and Execution Quality
- ORCA leads average Task Success Rate (71.0%) and Physical Plausibility (3.72), with its advantage clearest in Garden and Workshop while open-loop remains competitive in Kitchen and Livestream.ORCA also achieves the highest Subject Consistency and Best-Worst Scaling preference; open-loop has the lowest Subject Consistency and can spend its fixed steps attempting all subgoals in simpler tasks, whereas ORCA retries can consume its budget.Source: Table 1 · p. 7, Evaluation on Task Success Rate and… · p. 8
5.3.1. Quantitative Comparison
- Table 1 reports results across five scenarios and multiple dimensions, including task completion, video quality, and human preference.Its results expose trade-offs between task completion, execution quality, and video coherence.Source: §5.3.1 · p. 6, Table 1 · p. 7
5.3.2. Qualitative Comparison
- On the Transfer Plant task, ORCA completes the subgoals, while open-loop errors accumulate, reactive behavior repeats actions, and unreflected VAGEN errors corrupt the final state.The comparison describes reflection as detecting errors early and preventing these three failure modes.Source: Figure 4 · p. 7, §5.3.2 · p. 8
5.4. Ablation Study
- Ablations associate removing belief tracking with severe TSR degradation, removing reflection with lower Subject Consistency and BWS, and removing System 1 grounding with lower TSR and BWS.The reported explanation is that abstract System 2 commands yield imprecise I2V generation; the ablation re-evaluates the Workshop scene.Source: §5.4 · pp. 8–9, Table 2 · p. 8
6. Conclusion
The conclusion presents L-IVA and ORCA as contributions toward active, goal-oriented video avatars. It attributes robust long-horizon behavior to closed-loop state tracking and hierarchical planning and execution.
6. Conclusion
- L-IVA is presented as a benchmark for autonomous goal completion in interactive scenarios, and ORCA as a framework grounded in Internal World Model theory.The paper characterizes these contributions as a shift from passive animation toward active intelligence in video avatars.Source: §6 · p. 9
- ORCA’s design combines a closed-loop OTAR cycle for tracking state under generative uncertainty with a dual-system architecture for hierarchical planning and execution.Source: §6 · p. 9
- The conclusion reports that reflection prevents belief corruption and enables coherent multi-step task completion.Source: §6 · p. 9
7. Supplementary Material
The supplementary material describes its contents as supporting details for the main paper, covering benchmark construction, evaluation, comparisons, prompts, and limitations.
7. Supplementary Material
- Section A details construction of the L-IVA benchmark.Source: §7 · p. 11
- Section B covers evaluation protocols and metrics, while Section C provides additional qualitative visualizations comparing methods.Source: §7 · p. 11
- Section D lists ORCA prompts, and Section E discusses failure cases and limitations.Source: §7 · p. 11
8. Benchmark Construction Details
The benchmark construction combines real-world photography and AI-synthesized imagery to support diverse scenes and task interactions. The two sources use different procedures for aligning scenes with goals.
8. Benchmark Construction Details
- L-IVA uses a hybrid construction pipeline combining real-world photographs with AI-synthesized imagery.Source: §8 · p. 11
- For real-world samples, selected Pexels images contain interactive objects; each is paired with a manually defined intention and model-generated hierarchical annotations.Gemini-2.5-Pro generates sequential subgoals, object descriptions, and reference action prompts.Source: §8 · p. 11
- For synthetic samples, a goal-first design specifies solvable object interactions before prompts generate a scene aligned to those task requirements.Source: §8 · p. 11
9. Evaluation Protocols
The evaluation combines VLM-based automated metrics with human judgments, which assess task completion, physical plausibility, and holistic preference. The human study uses multiple evaluators and comparative sets to judge videos from ORCA and three baselines.
Human study
- Eight evaluators assessed 520 videos across 130 comparative sets, each containing outputs from ORCA and three baselines.The study measured physical score and Task Success Rate (TSR).Source: §9 · p. 11
Task Success Rate
- TSR measures completed subgoals as a ratio of the total subgoals for each test case.Evaluators count successfully completed subgoals based on visual evidence.Source: §9 · p. 11
Action Fidelity Score
- Action Fidelity Score (AFS) is a binary classification of whether the clip matches the command's core action, key objects, and direction.The metric tolerates minor visual artifacts but treats semantic errors such as cutting instead of peeling as mismatches.Source: §9 · p. 11
Physical Plausibility Score
- Physical Plausibility Score (PPS) rates each video sequence on a 1–5 Likert scale for physical consistency.The rubric ranges from perfect interactions to complete breakdowns in adherence to basic physical laws.Source: §9 · pp. 11–12
Human preference ranking
- Best-Worst Scaling ranks holistic quality by prioritizing task completion, then logical coherence, then visual aesthetics.Scores range from -100% to +100%; a positive score means a method was selected Best more often than Worst.Source: §9 · pp. 12–13
10. Prompt Details for ORCA
ORCA's prompts divide initialization, observation, strategic reasoning, action grounding, and reflection across its modules. They are described as model-agnostic and optimized for Gemini-2.5-Flash.
Initialization
- The Initialization Prompt parses the initial scene and decomposes the high-level intention into a structured plan.The prompts are model-agnostic but optimized for Gemini-2.5-Flash.Source: §10 · p. 13
Observation and thinking
- The Observation Prompt updates the belief state from generated clips, while the Thinking Prompt supports strategic reasoning, discrepancy detection, and next-step planning.Source: §10 · p. 13
Action grounding and reflection
- The Action Grounding Prompt translates abstract plans into high-fidelity I2V-compatible captions, and the Reflection Prompt verifies execution and triggers error correction.Source: §10 · p. 13
11. Discussion on Failure Cases and Limitations
ORCA's performance is bounded by limitations in its underlying VLM and I2V models, rather than primarily by its algorithmic design. The reported failure cases include VLM perceptual errors and I2V control and consistency failures.
Overall limitation
- ORCA's capabilities remain bounded by its foundation models, with observed failures attributed primarily to pretrained-model limitations rather than the framework's algorithmic design.Source: §11 · p. 13
VLM-centric failures
- VLM sampling can miss fleeting glitches, while limited depth perception can make background objects appear reachable and prompt geometrically impossible actions.Figure 6 illustrates temporal information loss and mistaken reach instructions.Source: §11 · p. 13, Figure 6 · p. 14
I2V-centric failures
- I2V models may disregard fine-grained instructions despite retries, and object disappearance or hallucination can corrupt the belief state when the VLM misses it.Figure 6 includes examples of failed interaction execution and an object disappearing mid-clip.Source: §11 · p. 13, Figure 6 · p. 14
Conclusion
- The authors posit that improved VLM spatial reasoning and I2V controllability would let ORCA's performance scale without architectural changes.Source: §11 · p. 13