PaperlayerView exact revision on arXiv

Source-linked AI summary

VibeWorlding: Can Multimodal Agents Construct 3D Open Worlds End-to-End?

Yansong Ning, Jingwen Ye, Zhongkai Wu, Yang Sun, Yiqin Zhu, Xingyi Li, Weidong Zhang, Hao Liu

arXiv:2608.15265v1cs.AI

TL;DR

VibeWorlding addresses the lack of unified evaluation and training for agents constructing interactive 3D worlds from user queries. It introduces a multimodal benchmark and RL framework, finding that VibeWorlder-30B-A3B achieves the best overall Pass@1 at 59.3%.

  • Problem

    Existing 3D asset tools are fragmented across incompatible interfaces, hindering unified end-to-end construction and systematic verification of 3D worlds.

  • Method

    VIBEWORLDING combines a multi-turn multimodal tool-use task, VWE-BENCH, and a dual-constraint verifier with joint multimodal agentic RL training.

  • Results

    59.3% overall Pass@1 makes VibeWorlder-30B-A3B the best-performing evaluated model, surpassing GPT-5.5 at 57.3% and Qwen3.8-Max at 56.9%.

  • Takeaways & Limitations

    The results identify precise, collision-free 3D editing as the dominant bottleneck while showing that reliable-reward RL can substantially improve this capability.

  • Takeaways & Limitations

    Precise 3D distance editing remains unresolved because agents can select the correct asset and target but emit incorrect coordinates.

Abstract

from arXiv · show

Constructing an interactive 3D open world from a user query is important. However, existing methods are primarily evaluated on idealized, simple queries, making it difficult to systematically analyze and compare how multimodal agents understand user intent, use 3D tools, and reason over textual and visual 3D world information. To this end, we propose VibeWorlding, a unified framework for benchmarking and training vibe worlding agents: a multimodal agent that can autonomously infer user intent, plan scene layout, invoke 3D tools, and reflect on the multimodal feedback in a multi-turn agent-environment interaction process. To achieve this, we first build VWE-BENCH, a benchmark of 2,616 high-quality 3D assets, 323 human-annotated seed 3D worlds, and 6,828 reverse-synthesized multimodal user queries, split into verified queries with ground-truth and unverified queries with carefully designed rubrics. Moreover, we develop VibeWorlding-Gym, a joint multimodal RL post-training framework that integrates (1) a sandbox environment unifying asset retrieval, editing, and image rendering as MCP tools, and (2) a rubric-based verifier that combines physical feasibility and intent fulfillment verification, supporting both fair model evaluation and scalable multimodal RL reward service. Our experiments show that current frontier MLLMs are far from solving the vibe worlding agent task, with even GPT-5.5 and Qwen3.8-Max reaching below 60% success rate, and trace the bottleneck to precise 3D world editing. We further find that RL training can ease this weakness and enable open-source MLLMs to even surpass closed-source frontiers: our VibeWorlder-8B is comparable to frontier MLLMs, while our flagship VibeWorlder-30B-A3B attains the best overall Pass@1 among all evaluated models.

1 INTRODUCTION

VibeWorlding introduces a unified benchmark and training framework for multimodal agents that construct or edit interactive 3D open worlds through multi-turn, tool-integrated reasoning. It combines large-scale assets, annotated worlds, multimodal queries, a dual-constraint verifier, and multimodal RL, while showing that precise 3D world editing remains the central bottleneck.

  • Framework: VibeWorlding formulates 3D world construction as multi-turn multimodal reasoning in which agents infer intent, plan layouts, invoke 3D tools, and respond to feedback.The task supports both building worlds from plain-text queries and editing existing worlds from textual instructions.
  • Benchmark: VWE-BENCH contains 2,616 high-quality 3D assets, 323 human-annotated seed worlds, and 6,828 reverse-synthesized user queries.Queries are split into verified instances with ground-truth and unverified instances evaluated with designed rubrics.
  • Training and evaluation: VibeWorlding-Gym unifies asset retrieval, editing, and image rendering as MCP tools with a rubric-based verifier for joint multimodal RL training.The verifier checks physical feasibility, including height and collision, alongside intent fulfillment such as ecological plausibility, 3D understanding, reasoning, and asset retrieval.
  • Results: Below 60% success on VWE-BENCH is achieved by both GPT-5.5 and Qwen3.8-Max, showing that current frontier multimodal agents remain far from solving vibe worlding.The six-capability analysis attributes the gap to inaccurate editing through 3D tools despite understanding the world and user intention.
  • Contributions: The framework provides an end-to-end stack spanning annotated assets, seed worlds, user queries, an embedding model, an interactive sandbox, a dual-constraint verifier, and SFT plus agentic RL post-training.The contribution also includes a comprehensive six-capability analysis of multimodal agents on end-to-end 3D world construction.

2 PRELIMINARY

The paper defines 3D assets as structured atomic building blocks of 3D worlds and equips agents with tools for manipulating them. It formulates world construction as a multi-turn process in which an agent uses query- and history-conditioned actions and multimodal sandbox observations to produce a final interactive world.

  • Core definitions: A 3D asset is an atomic world component represented by the 5-tuple (id, name, category, face count, bbox).The face count specifies mesh complexity, while bbox specifies native physical size.
  • Core definitions: The agent uses five 3D tools: asset retrieve, asset add, asset rotate, asset translate, and asset delete.These tools respectively retrieve, add, rotate, reposition, and remove assets in the scene.
  • Problem formulation: Given a multimodal query, the agent interacts with a sandbox for T turns and constructs an interactive 3D world represented by a final 3D map.The formulation treats the completed world as the outcome of agent-environment interaction.
  • Problem formulation: At turn i, the policy generates a thought and one or more tool-call actions conditioned on the query and prior interaction history.The action may consist of one or multiple calls selected from the available 3D tools.
  • Problem formulation: Each sandbox observation combines tool responses with the current 3D map’s text and rendered multi-view images, and the interaction ends when no further tool call is issued.The resulting trajectory contains thoughts, actions, and observations across T turns.

3 VWE-BENCH CONSTRUCTION

VWE-BENCH is constructed through an art-annotator- and MLLM-assisted pipeline that produces a physically consistent asset library, seed worlds, and reverse-synthesized multimodal queries. The benchmark combines verified precise editing queries with unverified open-ended construction and refinement queries evaluated by rubrics.

  • Asset synthesis: The asset pipeline uses artist-annotators to define inventories and control quality, while generative models produce reference images and image-to-3D meshes followed by quality filtering and size annotation.The process uses text-to-image generation and Hunyuan3D 3.1 for mesh creation, filtering assets whose geometry or appearance differs substantially from the concept image.
  • Asset synthesis: 2,616 high-quality assets span 20 semantic categories, from small props to buildings and terrain, and include physical-size annotations and semantic descriptions.The assets are used to construct seed worlds and support retrieval across varied open-world scenes.
  • Seed worlds: 323 professional-annotated seed 3D worlds jointly cover the synthesized asset library and serve as coarse, functional starting points for later world-refinement query synthesis.Annotators are instructed to create functional rather than highly polished environments.
  • Query synthesis: Construction queries contain theme-only, theme-plus-elements, or full-blueprint specifications, testing autonomous asset retrieval, constrained layout generation, or execution from near-complete world descriptions.Because these queries are open-ended and lack a single ground-truth world, they are evaluated by an MLLM judge using a rubric.
  • Query synthesis: The benchmark yields six query types: one precise asset-level editing type with ground-truth verification, plus five open-ended types evaluated with carefully designed rubrics.The unverified types include fuzzy asset edits and holistic scene critique, guidance, restatement, and complex-description requests.
  • Benchmark statistics: 6,828 reverse-synthesized queries comprise 1,364 3D world construction queries and 5,464 3D world refinement queries after annotator review and filtering.The final benchmark combines 2,616 assets, 323 seed worlds, and these queries.

4 VIBEWORLDING-GYM

VIBEWORLDING-GYM is a unified framework for scalable agentic RL training, combining an interactive 3D sandbox, end-to-end verifier, and multimodal post-training. Its environment provides tools and rendering, while verification checks physical feasibility and user-intent fulfillment.

  • Framework Overview: VIBEWORLDING-GYM combines a stable interactive 3D sandbox, reliable end-to-end verifier, and unified post-training framework for scalable agentic RL.The framework is designed to investigate how vibe worlding agents improve through agent post-training.
  • Unified 3D Tools: Five unified tools retrieve, add, delete, rotate, and translate assets, supporting natural-language asset retrieval and parameterized 3D-world editing.Asset retrieval maps natural-language intent to placeable candidates; the other tools manipulate assets in the world.
  • Unified 3D Rendering: Blender renders each agent-modified world after every turn and returns images from five fixed camera viewpoints.The rendered multimodal feedback is provided to the agent at every interaction turn.
  • Dual-Constraint Verification: The dual-constraint verifier requires physical feasibility and user-intent fulfillment, combining Python geometric checks with a rubric-based MLLM judge.Physical checks cover collision and support constraints, while intent evaluation considers ecological plausibility, 3D understanding, and 3D reasoning.
  • Multi-Dimension Reward Calculating: For Unverified queries, worlds must pass physical verification before rubric judging and pass all dimensions; Verified queries are scored by the proportion of correctly modified assets.This produces dimension-sensitive rewards for both open-ended intent fulfillment and ground-truth editing accuracy.
  • Joint Multimodal RL: Joint multimodal RL trains from both pure-text construction and multimodal refinement queries using GRPO and the dual-constraint verifier.Training starts from a cold-started agent policy and uses an outcome-based reward.

5 EXPERIMENTAL SETUP

The experiments use disjoint seed-world training and testing splits with preserved query-type proportions, and evaluate agents through automatic and blind human assessments. All methods share the same sandbox, asset library, and verifier, while comparisons cover frontier MLLMs, agent scaffolds, and open models with supervised and RL training settings.

  • Dataset partition: VWE-BENCH uses completely disjoint seed 3D worlds for training and testing while preserving query-type proportions across splits.Cold-start data synthesis uses the SFT query set, whereas the RL query set is used directly for joint multimodal RL.
  • Auto Evaluation: Automatic evaluation uses Gemini 3.5-flash as the verifier backbone and reports Pass@1, the fraction of queries solved by a single rollout.Unverified queries pass only when all verifier dimensions pass; Verified queries use a rule-based score based on correctly modified assets.
  • Human Evaluation: Blind human evaluation presents anonymous annotators with the query, initial and final multi-view screenshots, and the agent’s final response.Annotators evaluate both the constructed 3D world and the agent’s final response without access to model identity.
  • Evaluation Quality Analysis: The verifier agrees with holistic human Pass@1 on 83.4% of cases and preserves system ordering with Spearman’s ρ = 0.88 across evaluated systems.Aspect-level agreement reaches Cohen’s κ = 0.54 for ecological plausibility and κ = 0.53 for intent fulfillment.
  • Compared methods: All methods use the same sandbox tools, asset library, and verifier, spanning zero-shot frontier MLLMs, training-free agent-scaffold baselines, and open base models.The frontier models range from 8B to 2.8T parameters and include GPT-5.5, Gemini 3.5-flash, Gemini 3.1-pro, Claude-Opus-4.8, Kimi-K3, Qwen3.8-Max, Qwen3-VL-8B, and Qwen3-VL-30B-A3B.
  • Training setup: Cold-start uses full-parameter supervised fine-tuning for 2 epochs, while RL uses GRPO for one epoch with groups of 8 rollouts per prompt.Cold-start uses learning rate 2 × 10−5 and maximum sequence length 122,880 tokens; RL uses learning rate 5 × 10−7 and KL coefficient 0.05.

6 RESULT ANALYSIS

Current multimodal LLMs remain far from reliably constructing 3D open worlds end-to-end, with precise 3D editing the central bottleneck. Multimodal RL substantially improves open-source models, enabling VibeWorlder-30B-A3B to achieve the best overall Pass@1 among evaluated models.

  • Overall performance: Less than ∼60% overall Pass@1 is achieved by GPT-5.5 and Qwen3.8-Max, while untrained Qwen3-VL-8B and Qwen3-VL-30B-A3B reach 5.3% and 13.6%.These results show that end-to-end 3D world construction remains far from solved.
  • RL gains: 59.3% overall Pass@1 makes VibeWorlder-30B-A3B the best-performing evaluated model, surpassing GPT-5.5 at 57.3% and Qwen3.8-Max at 56.9%.VibeWorlder-8B reaches 41.4% overall and matches Gemini 3.1-pro at 42.7% approximately, while exceeding its Verified score, 59.3 vs. 44.4%.
  • Query difficulty: 3D world construction from scratch challenges all MLLMs because it jointly requires asset retrieval, spatial reasoning, plausible placement, and collision avoidance.Complex Description and Scene Restatement are the most difficult refinement sub-types because they require interpreting longer, more complex descriptions.
  • Capability bottlenecks: Collision remains the weakest capability at 59–68% after RL, making collision-free 3D spatial editing the key unresolved challenge.By contrast, RL raises 3D reasoning from 6–20% for base backbones to 56–85%, with VibeWorlder-30B-A3B reaching 85%.
  • Failure modes: The dominant editing failure is executing the requested distance along the wrong direction, reflecting coordinate-frame misunderstanding rather than incorrect asset or target selection.In the example, the agent moves an object 7 m in the opposite direction, producing a 14 m positional error under a 0.5 m tolerance.
  • Failure modes: Overediting is a second dominant failure mode: during refinement, agents modify assets the user did not request, such as deleting a large tree while adding another beside a chest.The plan captured both requested intents and placed the new tree correctly, but the extra deletion violated scope.

7 RELATED WORK

Prior MLLM-based 3D world construction has progressed from fixed, modular workflows toward autonomous agents that iteratively plan, use 3D tools, and revise worlds. In parallel, RLVR has expanded from long-horizon reasoning and tool use to multimodal agent training.

  • 3D World Construction: MLLM-based 3D world construction is evolving from fixed workflows to autonomous agentic systems with iterative planning, tool invocation, and world revision.Autonomous systems enable an MLLM to plan scenes, invoke external 3D tools, and revise constructed worlds.
  • 3D World Construction: Fixed-workflow methods decompose construction into predefined stages handled by specialized modules or sub-agents.SceneCraft generates executable Blender programs through critique-and-refine, while 3D-GPT translates natural language into procedural scene-generation parameters using multiple sub-agents.
  • Multimodal RLVR: RLVR improves long-horizon reasoning and tool use by providing rewards through executable programs or structured verifiers, and recent work extends it to multimodal agents.Examples include Kimi K2.5, which jointly optimizes vision and language through RL, and OpenSearch-VL, which trains a multimodal agent.

8 DISCUSSION

VIBEWORLDING introduces a unified benchmark and training framework for end-to-end interactive 3D open-world construction. The discussion identifies open needs for richer tools, larger-scale construction, more efficient multimodal RL, and stronger 3D spatial reasoning.

  • Contributions: VIBEWORLDING unifies benchmarking and training for agents that construct interactive 3D open worlds end-to-end, supported by VWE-BENCH and VIBEWORLDING-GYM.VWE-BENCH covers high-quality assets, human-annotated seed worlds, and reverse-synthesized multimodal queries; VIBEWORLDING-GYM unifies asset retrieval, editing, and rendering.
  • Directions for Improvement: Future agents need richer, composable 3D tools and reusable construction skills instead of assembling scenes solely through low-level atomic edits.Examples include directly instantiating a football field or scattering assets across a rectangular region to populate a forest.
  • Directions for Improvement: 258 assets comprise the most complex VWE-BENCH world, motivating larger-scale construction approaches such as decomposing tasks among collaborating multimodal agents.The passage characterizes this benchmark scale as relatively small compared with truly open-ended vibe worlding requirements.
  • Directions for Improvement: More efficient multimodal RL methods remain an open direction for 3D world construction alongside the framework’s broader research and engineering limitations.The current recipe scores the entire trajectory with an outcome-based reward, as stated in the discussion passage.
  • Challenges for Vibe Worlding Agents: Agents can combine 3D maps and rendered images but still struggle to reason precisely about spatial quantities such as distance and angular relationships.The discussion links this limitation to capability and failed-case analyses and suggests that stronger base MLLMs may be needed.

A APPENDIX · A.1 ILLUSTRATIVE CASE

The appendix makes the abstract interaction protocol concrete by replaying two complete episodes, covering 3D world construction and refinement. Each replay follows the agent’s actual alternating interaction with the environment, including reasoning, tool calls, and multimodal feedback.

  • A APPENDIX: Two complete episodes concretize the interaction protocol specified abstractly in Sections 2.1 and 4.The episodes are replayed turn by turn.
  • A.1 ILLUSTRATIVE CASE: The first episode constructs a 3D world from a query.It is one of the two query types represented in the appendix.
  • A.1 ILLUSTRATIVE CASE: The second episode refines a 3D world against a ground-truth world.This episode represents the other query type in the replay.
  • A.1 ILLUSTRATIVE CASE: The construction episode is scored by an MLLM judge.Its evaluation differs from the ground-truth comparison used for refinement.
  • A.1 ILLUSTRATIVE CASE: The refinement episode is scored against a ground-truth world.The appendix explicitly contrasts this scoring setup with the construction episode.
  • A.1 ILLUSTRATIVE CASE: Each turn alternates the agent’s reasoning and tool calls with the environment’s reply.The reply contains the updated 3D map plus five renders.

A.1.1 CASE A: 3D WORLD CONSTRUCTION

In this full-blueprint case, the agent plans a modern commercial street from text, retrieves assets before placement, and iteratively constructs the scene through visual feedback. The episode terminates after six turns with a world containing 34 assets across 11 types.

  • Case setup: The text-only query requires the agent to retrieve every asset before placing anything, then complete five acting turns with four rounds of visual feedback.The agent terminates autonomously by emitting no tool call.
  • Initial planning: Before tool use, the agent separates focal subjects from set dressing and partitions the ground plane into four coordinate-defined zones.The plan centers the asphalt street, places shops on the left, high-rises on the right, and advertising columns in the distance while avoiding direct AABB collisions.
  • Candidate selection and revision: In the second turn, the agent rejects stylistically incompatible or oversized candidates, repeats selected assets for structure, and reformulates retrieval when no suitable electronic signage is found.It temporarily uses a dark pillar as an advertising-column placeholder while querying for an electronic billboard or neon screen.
  • Final construction: 34 assets over 11 distinct types remain in the completed world after extra turns add street lamps and a second lantern type.Turns 4–5 follow the same retrieve-then-place pattern, and turn 6 ends the episode with no tool call.

A.1.2 CASE B: 3D WORLD REFINEMENT

This refinement episode tests whether an agent can resolve a spatial referent from multimodal world state and execute the requested edit in one turn. The agent correctly deletes the statue nearest the altar and stops after completing the single-intent instruction.

  • Task setup: The episode provides an existing snow world as a map and five rendered views, restricts additions to a closed component whitelist, and evaluates structural criteria in one acting turn.Available tools are asset_rotate/asset_translate, asset_delete, and asset_add.
  • Spatial reasoning: The two candidate statues are identical Statue 01 assets, so identifying the requested target requires computing distance to the altar rather than matching names or appearance.The nearer statue is 3.08 m away versus 3.37 m for the other, a margin of 0.29 m.
  • Action: The agent calls asset_delete on the statue at (18.78, 20.8, 2.0), explicitly reasoning that it is closest to Altar 01.The tool call directly implements the user’s request to delete the statue nearest the altar.
  • Result: After deletion, the returned world retains the statue at (14.13, 18.08, 2.0) and removes the statue at (18.78, 20.8, 2.0), leaving other assets untouched.The world is re-rendered after the edit; scatter differences arise from sandbox re-sampling rather than agent movement.
  • Completion: The agent emits no second tool call, correctly treating the single-intent instruction as complete after the one edit.This demonstrates termination without unnecessary embellishment.

A.2 ASSET RETRIEVAL SERVICE CONSTRUCTION

The asset retrieval service models relevance across entity and attribute matches, synthesizes graded supervision from metadata, and deploys a trained encoder for multi-positive evaluation. On held-out queries, it finds an acceptable asset first for 89.5% of queries and in the top 10 for 99.8%.

  • Graded relevance: Relevance is graded by entity and attribute matches: G0 matches both, G1 matches the entity only, G2 matches attributes only, and G3 matches neither.G0 is the gold target, G1 a tolerable substitute, and G2 a deceptive distractor.
  • Interaction-free data pipeline: The metadata pipeline generates K=10 asset-grounded names per card, applies a head-noun constraint, creates G2 negatives through attribute flips, and grades pairs on two dimensions.Entity and attribute judgments are combined through an auditable decision tree.
  • Interaction-free data pipeline: 28,835 graded records contain 241,548 hard negatives, with G2 comprising approximately 88.6% and appearing for roughly 89% of queries.The records span 26,213 entity-only queries and 2,622 attribute-bound queries.
  • Backbone and serving: The retriever fine-tunes Qwen3-Embedding-4B with InfoNCE plus an Error-Cost-Ordered hinge chain, then serves normalized asset embeddings through exhaustive inner-product search.At VWE-BENCH scale, exhaustive search is exact and adds negligible latency.
  • Retrieval Performance Analysis: 89.5% of held-out queries rank an acceptable asset first, while 99.8% return one in the top 10 under the multi-positive protocol.The evaluation uses 1,311 held-out queries and reports six complementary ranking metrics across multi-positive and anchor-only protocols.
  • Retrieval Performance Analysis: 44.0% precision@10 under the multi-positive protocol is already close to the |GT|/k ceiling when |GT| < k.This ceiling applies because multiple G0 assets may be correct for a query.

A.3 AUTO EVALUATION PROTOCOL

The auto-evaluation protocol defines structural criteria for verified queries and specifies a dual-constraint verifier for unverified queries, serving as both an offline metric and an online RL reward.

  • A.3 AUTO EVALUATION PROTOCOL: The protocol first establishes structural criteria for verified queries, then fully specifies the dual-constraint verifier for unverified queries.The verifier is from Section 4.2 and supports both evaluation and training.
  • A.3 AUTO EVALUATION PROTOCOL: The dual-constraint verifier functions as both the offline metric and the online RL reward.This links automatic evaluation with reinforcement-learning reward provision.

A.3.1 STRUCTURAL CRITERIA FOR VERIFIED QUERIES

Verified queries use machine-checkable criteria for each requested atomic edit, with rewards equal to the fraction of satisfied criteria. A hard verification mechanism rejects unauthorized world changes to prevent agents from exploiting partial credit.

  • Structural criteria: Each verified query assigns one machine-checkable criterion per requested atomic edit, and reward equals the fraction of criteria satisfied.The reward is defined as r = |{criteria passed}|/|{criteria}| ∈[0, 1].
  • Structural criteria: The criterion types include proximity for additions, exact absence matching for deletions, position deltas for translations, and yaw matching for rotations.Deletion matching uses a 0.1 m tolerance, while translation uses a default 0.5 m tolerance; the supplied passage does not state the rotation tolerance.
  • Reward verification: Hard reward verification rejects unauthorized edits by diffing initial and final worlds and deriving licensed edits from the requested changes.This prevents agents from satisfying individual criteria while making sweeping unrequested changes, such as deleting surrounding assets to simplify placement.

A.3.2 PHYSICAL FEASIBILITY VERIFICATION FOR UNVERIFIED QUERY … A.5 VIBE WORLDING AGENT SYSTEM PROMPT

The paper combines geometric physical-feasibility checks, diagnostic MLLM-based intent verification, strict verifier prompts, human agreement analysis, and a structured agent system prompt for planning 3D worlds.

  • A.3.2 PHYSICAL FEASIBILITY VERIFICATION FOR UNVERIFIED QUERY: Each asset is represented by an axis-aligned bounding box derived from its position and half-extent, anchored at its ground-contact height.The box spans [x ± ex], [y ± ey], and [z, z + ez].
  • A.3.2 PHYSICAL FEASIBILITY VERIFICATION FOR UNVERIFIED QUERY: Collision fails when one asset pair has positive residual overlap on all axes after 0.5 m slack and intersection volume of at least 10 m3.A single qualifying pair fails the collision aspect, while slack prevents penalizing intended contact or interlocking.
  • A.3.3 INTENT FULFILLMENT VERIFICATION FOR UNVERIFIED QUERY: Intent fulfillment is split into 3D understanding and 3D reasoning so the verifier distinguishes scene misreading from incorrect tool execution.Retrieval plausibility evaluates how the agent uses returned candidates, including whether it rejects an unsuitable list rather than force-fitting an asset.
  • A.3.3 INTENT FULFILLMENT VERIFICATION FOR UNVERIFIED QUERY: A trajectory is correct only when every applicable check passes: five checks for from-scratch generation and four for unverified refinement.The checks cover collision, height, ecological plausibility, intent fulfillment, and retrieval plausibility, with retrieval plausibility omitted for refinement.
  • A.3.4 VERIFIER SYSTEM PROMPTS: For from-scratch construction, the verifier first derives requirements from the query, then checks fulfillment, tool execution, and distractor recognition.The distractor check rewards explicitly identifying infeasible or missing-asset requirements and offering a reasonable substitute or clarification.
  • A.4 HUMAN EVALUATION PROTOCOL: Human intent-fulfillment labels are compared with verifier rewards using Spearman’s ρ, while binary ecological and physical-feasibility judgements use Cohen’s κ.The comparisons include human ecological plausibility versus verifier ecological assessment and human physical feasibility versus geometric collision checking.
  • A.4 HUMAN EVALUATION PROTOCOL: Claude-Opus-4.8 receives the highest response-intelligence score, 4.54/5, but also the highest hallucination rate, 69.2%.The reported pattern indicates that articulate responses are not necessarily faithful to the edits performed.
Loading 2608.15265v1…