InteractiveWorld: A Verifiable Human-Centric Environment for Building Better Interactive MLLM Assistants

1 Introduction

• Static video benchmarks cannot test closed-loop adaptation or the combination of skills needed when assistant guidance changes human actions and subsequent observations.Real-world evaluations are costly and difficult to scale with people.Source: §1 · pp. 1–2
• InteractiveWorld models an assistant giving natural-language guidance to a human agent, whose actions update the environment and the next first-person view.The human agent interprets guidance and executes high-level commands; the environment executes them and renders the resulting view.Source: §1 · p. 2, Figure 1 · p. 2
• InteractiveBench contains 120 human-reviewed, multi-turn tasks across six categories, with success automatically verified from final simulator object states.Tasks require perception, memory, reasoning, and communication.Source: §1 · p. 2
• Claude Fable 5.1 succeeds on only 41.7% of tasks; all other models stay at or below 33.3%, and the best Guidance and Planning results are 20.0% and 25.0%.Experiments evaluate 12 frontier MLLMs.Source: §1 · p. 2
• The contributions frame the environment as a human-centric collaboration simulator and the benchmark as a way to evaluate whether assistant outputs shape task outcomes.Success is measured with verifiable, state-based rewards.Source: §1 · p. 2
• The analyses identify perception, memory, and communication as key bottlenecks in current MLLMs’ dynamic interaction.These findings are presented as a basis for future interactive MLLM research.Source: §1 · pp. 2–3

2 InteractiveWorld

• InteractiveWorld uses three layers—Physical, Human Action, and Semantic—to make the human an independent agent that observes, understands instructions, and acts.This design targets the shift from direct robot control to human guidance through natural language.Source: §2 · p. 3

2.1 Physical Layer

• The Physical Layer supplies realistic humans, assets, and physics states as a foundation for high-level planning.The human model uses a head-mounted first-person camera and kinematic execution for valid pose sequences; paths and reachability are checked before execution, and unsafe actions are rejected.Source: §2.1 · p. 3

2.2 Human Action Layer

• The Human Action Layer executes high-level actions through collision-aware navigation and posture- and target-adaptive manipulation.It finds a reachable standing position, plans a safe path, adapts posture and arm trajectory, and replans when no feasible trajectory exists.Source: §2.2 · pp. 3–4, Figure 2 · p. 4

2.3 Semantic Layer

• The Semantic Layer maps natural-language guidance to 36 executable action primitives, while the human interprets guidance and the environment updates state and first-person views after actions.The unified interface specifies actions and targets; success can be checked against object positions and relationships. Accessibility tasks intentionally withhold visual inputs from human agents.Source: §2.3 · p. 4, Table 1 · p. 5

3 InteractiveBench

• InteractiveBench contains 120 human-reviewed tasks across six everyday assistance categories, designed to require multi-turn use of first-person information and verifiable outcomes.Tasks address realistic assistance needs rather than actions the human can complete unaided.Source: §3 · p. 5
• Each evaluation pairs the target assistant with an auxiliary human agent instructed to follow guidance; after warmup, both can respond as actions update the scene and visual input.The interaction can end when the assistant outputs done or when the interaction limit is reached.Source: §3 · p. 5
• A task succeeds only when every predefined core state is met at termination, making the outcome depend on actual simulator state.The benchmark emphasizes long-horizon interaction, combined perception, memory, reasoning, and communication, and verifiable outcomes.Source: §3 · pp. 5–6
• Performance is assessed across six categories using Pass and mean completion rate, while action efficiency compares actual actions with oracle-required actions on individually passed tasks.The same Human agent has a 100% pass rate with an Oracle Assistant and 0% without assistance under the evaluation protocol.Source: Table 2 · p. 6, Figure 3 · p. 6

4 Evaluation

• The evaluation measures task success and action efficiency, assesses nine capabilities, classifies initial failures into six types, and examines task-length effects.The main experiments use Gemini 3.1 Pro as the Human agent; visual inputs are capped at 32 frames per turn.Source: §4.1 · p. 7, §4 · p. 7, Table 3 · p. 7

4.1 Benchmark Results and Analysis

• Claude Fable 5.1 achieves the highest average success rate at 41.7%, while every other model reaches at most 33.3%.An Oracle Assistant achieves 100% success with the same Human agent.Source: §4.1 · p. 7
• 85.0% is the best-model success rate on Decision tasks, versus 20.0% on Guidance and 25.0% on Planning.The authors attribute the gap to continuous tracking, anomaly detection, and replanning demands, and report that performance drops as tasks get longer, especially for Guidance and Planning.Source: §4.1 · pp. 7–8
• Visual perception and memory errors account for 70.4%–85.7% of first errors across models.A misperception or forgotten state can cascade into subsequent instructions; Gemini 3.1 Pro has 69.8% visual-perception first errors, while Kimi K3 has more memory than perception errors (39.2% vs. 33.8%).Source: §4.1 · p. 8
• Similar average capability scores do not ensure similar success: GLM-5.3-Flash scores 58.2% versus Claude Fable 5.1’s 59.2%, but has lower communication and task-completion scores.Their communication scores are 69.8% versus 79.6%, and GLM-5.3-Flash completes 30.0% of tasks versus 41.7% for Claude Fable 5.1.Source: §4.1 · pp. 8–9
• Models use 2.35×–3.66× the Oracle’s action cost in Accessibility and Guidance, and efficiency varies by task.Claude Fable 5.1 reaches 6.75× Oracle cost on Accessibility but 1.20× on Prevention; models also proactively seek views, labels, and spatial information.Source: §4.1 · p. 9

4.2 Ablation Study

• Inference-time memory augmentation improves Prevention and Finding success rates by 5–15% across three models, but effects vary by model on other categories.Planning success drops by 5% for two models, and the authors conclude this simple augmentation alone is insufficient for reliable long-horizon assistance.Source: §4.2 · p. 9
• Across 16 Assistant–Human pairings, Assistant performance varies with its Human partner, but model rankings remain relatively stable.Fable 5.1 ranks first and GPT-5.6 last; all four Assistants achieve their highest observed MCR when Gemini 3.1 acts as Human, and behavioral calibration toward real users is left for future work.Source: §4.2 · pp. 9–10

5 Conclusion

• InteractiveWorld and InteractiveBench provide a verifiable environment for closed-loop human-assistant interaction and a dynamic benchmark of 120 multi-turn tasks.InteractiveWorld measures interactive abilities with verifiable rewards, while InteractiveBench evaluates how models assist people in the real world.Source: §5 · p. 10

Appendix Contents

• The appendices cover environment implementation, benchmark construction and evaluation, additional experiments, real-human validation, and future work.The contents also list the task catalog and evaluation metrics.Source: Appendix Contents · p. 15

A Background & Related Work

• Existing egocentric-assistant benchmarks are largely open-loop, while user-in-the-loop studies involve few participants and are costly to scale and hard to reproduce.InteractiveBench closes the loop in simulation: instructions change human actions and the world, with success checked from final object states.Source: A Background & Related Work · p. 15
• In robot-centric simulators, the evaluated model directly commands the robot; InteractiveWorld instead lets an autonomous human interpret the assistant’s natural-language guidance and act.HumanCLAW uses a simulated human body, but its VLM still decides every move.Source: A Background & Related Work · p. 16
• Other interaction-oriented platforms place AI in the physical-actor role, whereas InteractiveWorld evaluates a disembodied assistant guiding an autonomous MLLM-driven human.This reversal supports closed-loop, verifiable evaluation of assistants that help people rather than act for them.Source: A Background & Related Work · p. 17

B.1.1 Stable Human Representation

• The human uses a fixed-shape SMPL+H kinematic model and reuses locomotion clips, with distance-based gait phase and layered manipulation poses for reproducible motion.Collision checks screen desired poses before commitment, while a 17-capsule proxy rig and head sphere support collision queries.Source: §B.1.1 · p. 17, §B.1.2 · p. 17

B.1.2 Collision-Free Interaction

• The collision-free guarantee screens planned routes and manipulations against shared geometry, but does not promise exact separation between continuous triangle surfaces.Small body-only contacts may remain as telemetry if the camera/eye volume is clear; object penetration above 5 cm fails the final placement gate.Source: §B.1.2 · p. 18

B.1.3 Rich and Scalable Asset Support

• BEHAVIOR-1K v3.9.0 contributes 1,829 categories and 51 layouts; generic task admission accepts 44 layouts, and automated baking and metadata validation enable asset additions without simulator-code changes.The importer retains 11,428 interactive instances across 3,189 distinct models; materials and particle systems are catalogued separately from manipulable rigid objects.Source: §B.1.3 · p. 18

B.2.1 Collision-Aware Autonomous Navigation

• Navigation selects a reachable, collision-checked stance near the task contact point, plans on a 0.10 m occupancy grid, and validates the resulting 3-D gait trajectory.For closed doors it routes to a validated handle stance, opens the door, updates occupancy, and replans; unsafe routes return structured failures rather than teleporting the human.Source: §B.2.1 · p. 19

B.2.2 Target-Driven Adaptive Manipulation

• Manipulation combines reusable pose prototypes, target-relative trajectories, and online inverse kinematics, adapting posture, grasp, and hand pose to object geometry and reachability.Trajectories are checked at sampled points; grasps require valid contact from at least four of five fingers, and successful objects follow the wrist rigidly until release.Source: §B.2.2 · pp. 19–20

B.3.1 Unified Action Interface

• The public interface uses normalized entity names and deterministic command parsing to compile requests into typed goals, inserting and deduplicating navigation when needed.The audited runtime registry contains 36 implemented canonical primitives, while task-facing prompts may expose only a relevant subset.Source: §B.3.1 · pp. 20–21, Table 5 · p. 21

B.3.2 Verifiable Multi-Turn Interaction

• Commands run against a persistent physical state and are verified against live outcomes, while scoring reflects the world’s state rather than an assistant’s completion claim.The interface includes gaze, optional waiting, and a confirm halt; scoring excludes predicates already true before handoff. Authored policies can additionally require assistant semantics to match independently queried world predicates.Source: §B.3.1 · p. 22, §B.3.2 · p. 23

C Implementation Details of InteractiveBench

• InteractiveBench evaluates assistants through a separate Human executor across 120 frozen episodes in six equally weighted families.Each versioned TaskPack fixes the scene, entities, opening history, utterances, executable actions, scoring contract, and provenance hashes.Source: C Implementation Details of… · p. 24

C.1 Benchmark Construction

• Benchmark tasks are built from fixed assessment contracts, then checked, replayed, and frozen only after their executable outcomes and rendered views are validated.Construction includes independent proposal reviews, deterministic syntax, affordance, geometry, and goal checks, plus oracle, no-help, incorrect-help, and early-help controls before release.Source: §C.1 · p. 24

C.1.1 Design objective and candidate generation

• Each episode is designed so visible evidence, timely help, and intervention can change a verifiable physical outcome.Derived interaction TaskPacks add opening actions, handoff speech, frame selection, role-specific contexts, intervention windows, and completion requirements; pre-sweep controls test wiring, executor limits, and whether failures are observable.Source: §C.1.1 · p. 24, §C.1.2 · p. 25

C.2.1 Two-agent closed loop

• The closed loop pairs each evaluated Assistant with a fixed Human executor and supplies the Assistant chronological first-person observations without private symbolic state.Frame selection preserves command intervals and true result frames; the Human sees only information needed to ground instructions, while the Assistant is denied simulator identifiers, answer keys, and score.Source: §C.2.1 · pp. 25–26, §C.2.2 · p. 26

C.2.3 Assistant prompts

• Common prompts require the Assistant to rely on received speech and video, while the Human executor must ground actions in available evidence and clarify ambiguous instructions.The Assistant uses one response mode per turn; the Human prompt requires exact command output and forbids substituting objects, destinations, or actions.Source: §C.2.3 · p. 27, §C.2.4 · p. 27

C.2.5 Validity and causal attribution

• Invalid infrastructure or evidence failures void a cell, whereas incorrect or incomplete assistance in a sound run counts as model failure; family MCR equally weights task-normalized completion.Human- or simulator-caused outcomes are not credited to the Assistant, and tasks with more objects or verification points do not receive extra weight.Source: §C.2.5 · p. 28, §C.3.1 · p. 28

C.3.2 Capability-level metrics

• Capability scores count only clean_pass evaluation points, retain missed opportunities as unobserved, and pool points equally rather than averaging task pass rates.Capability checks remain independent of whole-task outcomes: a failed task can contain passed capability points, and task completion does not prove every capability.Source: §C.3.2 · pp. 28–29

C.3.3 Efficiency and reproducibility

• Efficiency is measured only on strict-success episodes against a task-matched shortest legal oracle, with ratios paired per task before aggregation.Each Human macro action, including refused actions, counts as one step; efficiency is reported separately from MCR and Pass Rate.Source: §C.3.3 · p. 29
• Released manifests preserve task hashes, rubric and protocol versions, evaluation status, and evidence paths, allowing aggregate tables to be recomputed from per-task and per-atom ledgers.The manifest distinguishes source and assigned families, and observed hashes from frozen-rubric fallback hashes.Source: §C.3.3 · p. 29

C.4 Task Catalog

• Tables 7–12 list each released task’s goal, isolated capability, and source-family verification; “Baseline” is the state at Assistant handoff after the deterministic opening.Source: §C.4 · p. 29

• Guidance evaluates converting visible documents, domain knowledge, earlier observations, and numerical constraints into grounded, incremental instructions.Source: §C.4.1 · p. 30
• Table 7 lists the 20 Guidance tasks.Source: Table 7 · p. 30

• Planning evaluates strategic decomposition, resource and route optimization, constraint satisfaction, and replanning after interruptions or state changes.Source: §C.4.2 · p. 30
• Route tasks require moving objects to specified endpoints across maps, with shortest-route quality scored separately from task completion.Examples include moving three objects across a house map and four objects across a five-room map.Source: §C.4.2 · p. 31
• Cooking tasks combine sequencing, equipment and time limits, and final-state requirements such as correctly plated food and safely reset appliances.Tasks include scheduling multiple foods across appliances and selecting a feasible subset under time and equipment constraints.Source: §C.4.2 · p. 31
• Some tasks require recovering from a mistaken storage state, interruption, or newly appearing spill while completing a reset.Source: §C.4.2 · p. 31

• The Human executor receives no camera images, so the Assistant must supply timely, body-anchored, goal-relevant visual information.Accessibility tasks use CFSS-StateScore; STG denotes the critical transition gates reported as STG-CFSS.Source: §C.4.3 · p. 31
• Table 9 lists the 20 Accessibility tasks.Source: Table 9 · p. 31
• Tasks include identifying a shoe hazard before storing milk and recalling a phone’s location after intervening activity.Source-memory grounding is evaluated separately because the phone’s endpoint state alone cannot prove recall.Source: §C.4.3 · p. 32

C.4.4 Decision (20 tasks)

• Decision tasks link situated questions to physical consequences and test evidence integration, arithmetic, state reconstruction, and explicit action.The listed task outcomes include placing a phone on an office desk and a badge on a lobby desk while preserving completed endpoints.Source: §C.4.4 · p. 32

C.4.5 Prevention (20 tasks)

• Prevention tasks require staying quiet during correct work and intervening before the deadline of each consequential error in a mostly correct private plan.Intervention correctness and timing are measured separately by C7/C8 under WATCH-binary.Source: §C.4.5 · p. 32

C.4.6 Finding (20 tasks)

• Finding tests episodic object memory and evidence-grounded retrieval, including distinguishing targets from similar distractors and recovering objects after occlusion.Examples include retrieving specific bottles, nested squeegees, and a folder hidden beneath a cover.Source: §C.4.6 · pp. 33–34

D.1 Experiment setting detail of error type analysis.

• The error taxonomy was derived from model results; each valid failed episode received one primary label across six categories, with ambiguous cases rechecked.The categories cover visual misreading, lost state, rule errors, planning errors, unexecutable instructions, and mistimed or missing assistance; simulator and backend failures were excluded from model errors.Source: §D.1 · p. 34

D.2 Experiment setting detail of Memory Augmentation.

• Memory augmentation appends detailed new factual entries to a chronological ledger and includes the accumulated memory in the next-step input.Visual input remained unchanged, capped at 32 images per step; the prompt emphasizes object facts and actions, while completion checks require evidence for each pending physical change.Source: §D.2 · pp. 34–35

D.3 Additional Experiments about Comparative of Dynamic vs. Static

• The static benchmark reuses the dynamic benchmark’s 120 tasks, six families, scenes, and recorded first-person episodes.Each family contains 20 tasks.Source: §D.3 · p. 35
• Static items present fixed image sequences with one question, stopping before a decision or intervention and withholding future frames, action results, and feedback.Models cannot act, check outcomes, or recover from errors.Source: §D.3 · p. 36
• Eight models were evaluated once per static item by a frozen semantic judge; Table 13 reports accuracy by family with 20 tasks per entry, counting insufficient evidence as failure.The judge accepts matching key facts or instructions without requiring exact wording, provided there is no harmful contradiction.Source: §D.3 · p. 36, Table 13 · p. 36
• Static adaptations inflate eight models’ scores, and the large Static Only and Dynamic Only regions indicate a severe evaluation gap.The figure compares static-only and dynamic-only scores.Source: Figure 8 · p. 36

D.4 Additional Experiment of Task Horizon and Model Performance

• Task length is measured from the best available reference action count, not the model’s executed actions, avoiding bias from repeated or truncated failed runs.The study uses 120 tasks from six families and evaluates eight models on each task.Source: §D.4 · p. 37
• Reference action counts are normalized separately within each family, so 0 and 1 mark that family’s shortest and longest tasks; the axis compares tasks, not episode progress.The visualization fits separate Firth logistic regressions for each model and family; the main statistical analysis averages the eight binary outcomes per task and uses a family-fixed-effects task-level linear probability model.Source: §D.4 · p. 37
• A −0.431 family-adjusted slope corresponds to a 43.1-percentage-point decrease in complete-task success from the shortest to longest normalized horizon (95% bootstrap CI: −58.1 to −28.9; one-sided permutation p < 0.0001).All six family-level point estimates and 45 of 48 model–family curves were negative, although not every family-level test was individually significant; the authors treat the result as provisional because some labels were reconstructed from saved traces.Source: §D.4 · p. 37

D.5 Additional Human-Agent Validation with Real Human Participants

• Across seven executors, Assistant pass-count ranges were at most three tasks out of 36 (8.3 percentage points), and Fable 5.1 ranked best while Gemini 3.1 was lowest or tied for lowest.Muse 1.1 and GPT-5.6 Sol changed relative ordering across executors, so the full ranking was not invariant.Source: §D.5 · p. 38, Table 14 · p. 39
• Human-participant average pass rates differed from Gemini 3.1-executor rates by +6.5, +2.8, −1.9, and +1.9 percentage points for the four Assistants, respectively.The comparisons use the same 36 tasks and are relatively close, but executor-related differences are not zero.Source: §D.5 · p. 38, Table 14 · p. 39
• For Fable 5.1, Gemini 3.1 agreed with the three human participants on 30/36, 33/36, and 28/36 task outcomes; mean agreement was 84.3%, versus 85.2% for human–human pairs.Figure 10 shows task-level outcomes across the same 36 tasks.Source: §D.5 · p. 38, Figure 10 · p. 39
• These comparisons are descriptive, reuse the same tasks, and are not independent samples or a statistical equivalence test; agreement counts shared successes and failures, not identical actions or dialogue.The evidence supports preliminary use of prompted Human agents as a practical proxy for human instruction following in InteractiveWorld, not unrestricted agents or physical-world deployment.Source: §D.5 · pp. 38–39

E Future Work

• Future work will develop an InteractiveWorld reinforcement-learning platform, make Human agents more like people with varied personalities and skill levels, and add scanned real-world scenes.Source: E Future Work · p. 39