PaperlayerView exact revision on arXiv

Source-linked AI summary

Search-to-World: Evaluation of 3D World Delivery from User Request through Web Search

Zixiao Gu, Yabo Chen, Xunzhi Xiang, Yu He, Haibin Huang, Chi Zhang, Yunbo Wang, Xuelong Li

arXiv:2609.07605v2cs.CV

TL;DR

Agentic systems can search the live web, but their ability to turn retrieved content into a usable 3D world lacks an end-to-end evaluation. Search-to-World and the WorldSearcher harness evaluate retrieval through delivery, and results show model-dependent delivery, a gap between observing relevant content and delivering a world, and improved delivery and action efficiency with jointly trained recovery agents.

  • Problem

    The ability of agentic systems to transform web-retrieved content into a usable 3D world has not been systematically evaluated end to end.

    Source: §1 · pp. 1, 3

  • Method

    Search-to-World evaluates request-to-world delivery with ORR and WDR, while WorldSearcher combines reusable-world retrieval, video-based reconstruction, and structured recovery.

    Source: §1 · p. 3

  • Results

    The agent model affects overall performance, relevant-content observation does not ensure delivery, and jointly training recovery agents improves delivery and action efficiency.

    Source: §1 · p. 4

  • Takeaways & Limitations

    Search-to-World makes agentic 3D-world delivery measurable, and WorldSearcher provides an evaluation harness with recovery capabilities.

    Source: §6 · p. 14

Abstract

from arXiv · show

Agentic systems can interpret user requests, search the live web, and use external tools, but their ability to transform retrieved web content into a usable 3D world has not been systematically evaluated. No established end-to-end pipeline or benchmark exists for this capability. We introduce Search-to-World, an end-to-end evaluation task covering request understanding, web visual-content retrieval, and 3D-world delivery. We define Observed Retrieval Rate (ORR) and World Delivery Rate (WDR) to distinguish observing relevant content from successfully delivering a request-aligned, perceptually acceptable world. We also present WorldSearcher, a reuse-then-reconstruction harness that connects existing search agents to world delivery: it first retrieves reusable 3D worlds and, when none are available, reconstructs a world from video. A structured recovery controller revises temporal grounding, replaces source videos, or reformulates queries after failure. Using WorldSearcher, we benchmark representative models on Search-to-World and study supervised fine-tuning (SFT) for recovery subagents. Results show that delivery depends on the underlying agentic model, and that relevant-content observation does not ensure world delivery. Jointly training recovery agents improves delivery success and action efficiency. Search-to-World makes agentic 3D-world delivery measurable, while WorldSearcher provides a practical evaluation harness with recovery capabilities.

1 Introduction

Search-to-World frames usable 3D-world delivery as an end-to-end task from a user request through live-web visual-content acquisition. It distinguishes retrieval opportunity from accepted delivery and introduces WorldSearcher to reuse or reconstruct worlds with structured recovery.

1 Introduction

  • Prior evaluation families cover parts of the process, but the paper identifies no benchmark evaluating the complete path from user request through live-web search to usable 3D-world delivery.The motivating questions include resolving visual mismatch, grounding failure, unsuitable clips, and poor reconstruction quality.Source: §1 · pp. 2–3
  • Search-to-World evaluates whether a user request becomes a usable 3D world through web-sourced visual content, rather than treating retrieval alone as success.The delivered world must preserve request alignment and perceptual quality across the complete execution pipeline.Source: §1 · pp. 1, 3, 4
  • Observed Retrieval Rate (ORR) measures relevant 3D-world or video-resource retrieval, while World Delivery Rate (WDR) measures end-to-end world-delivery fulfillment.Together, the metrics distinguish observed retrieval opportunity from successful delivery.Source: §1 · pp. 3–4
  • WorldSearcher first seeks an acceptable reusable 3D world, then uses video-based reconstruction and structured recovery when reuse or reconstruction fails.Recovery can revise temporal grounding, substitute the source video, or reformulate search and grounding queries.Source: §1 · pp. 3–4
  • The agent model affects overall Search-to-World performance, relevant-content observation does not ensure delivery, and jointly training recovery agents improves delivery and action efficiency.The evaluation uses 300 scored requests across 20 fine-grained semantic domains, indoor and outdoor environments, and five spatial scales.Source: §1 · p. 4

2 Related Work

Prior evaluation families address web acquisition, 3D reconstruction or generation, and behavior within instantiated worlds. Search-to-World connects these stages by evaluating a request-level path from web-sourced visual content to usable 3D-world delivery.

2 Related Work

  • The evaluation boundary distinguishes web-sourced visual content, delivered 3D worlds, and interactive rollouts; Search-to-World spans a request through content acquisition to world delivery.Interactive-world rollouts instead assess behavior after a world has already been instantiated.Source: §2 · p. 4, Table 1 · p. 5

2.1 Web Content Retrieval and Acquisition Evaluation

  • Web-agent and multimodal-search benchmarks end at web states, answers, reports, grounded content, videos, or video moments, while Search-to-World additionally assesses usable 3D-world fulfillment.These benchmarks characterize web-sourced visual-content acquisition but do not assess whether that content becomes a request-fulfilling world.Source: §2.1 · p. 5

2.2 3D Reconstruction and Generation Evaluation

  • 3D reconstruction and generation benchmarks typically start with supplied visual content, conditioning signals, or candidate outputs, whereas Search-to-World starts from a user request and includes web acquisition.The cited benchmarks assess reconstruction or generated-world qualities such as text alignment, perceptual quality, consistency, and plausibility.Source: §2.2 · pp. 5–6

2.3 Interactive World Model Evaluation

  • Interactive-world benchmarks begin with an instantiated world or rollout and assess behavior, interaction, consistency, or stability, complementing Search-to-World’s web-to-delivery boundary.Their endpoint is an interactive rollout rather than delivery of a world discovered from a user request.Source: §2.3 · p. 6

3 Search-to-World

Search-to-World evaluates the full path from a user request to a usable 3D world, distinguishing relevant-content retrieval from successful delivery that meets alignment and perceptual-quality criteria.

3 Search-to-World

  • Search-to-World evaluates a complete request-to-world pipeline, separating relevant-content observation from delivery that meets alignment and perceptual-quality criteria.Source: §3 · p. 6

3.1 Task Definition

  • The task requires live-web discovery of request-relevant visual content before delivering a world matching the requested content and spatial extent.Unlike world generation from supplied conditions, the system must discover corresponding visual content on the live web.Source: §3.1 · p. 6

3.2 Execution Pipeline

  • Execution acquires web visuals, then either reuses a discovered 3D world or reconstructs one from a corresponding video after temporal grounding.Intermediate bottlenecks can include incorrect 3D retrieval, temporal grounding, clip suitability, or reconstruction quality.Source: §3.2 · p. 7

3.3 Automatic World Delivery Assessment

  • Two VLM-based evaluators assess multiview renderings for request alignment and visual quality; a positive label requires an aligned final world above the quality threshold.Source: §3.3 · pp. 7–8

3.4 Evaluation Metrics

  • ORR counts runs retrieving at least one relevant 3D-world or grounded-video item, while WDR counts end-to-end delivery fulfillment.The metrics distinguish observing relevant visual content from fulfilling the world request.Source: §3.4 · p. 8

4 WorldSearcher: Reuse-Then-Reconstruction

WorldSearcher implements Search-to-World as a reuse-then-reconstruction harness with structured recovery during video-based reconstruction. It first seeks an accepted reusable world, then uses video retrieval, temporal grounding, and reconstruction as a fallback.

4 WorldSearcher: Reuse-Then-Reconstruction

  • WorldSearcher combines reusable-world retrieval, video-based reconstruction, and structured recovery control to address bottlenecks in end-to-end delivery.The harness updates a compressed recovery state after runtime failure and selects a state-admissible action.Source: §4 · p. 8

4.1 Reuse-Then-Reconstruction

  • WorldSearcher accepts an existing 3D world only after it passes automatic delivery assessment, and reconstructs from video if no candidate is accepted.The reconstruction branch retrieves a video, locates a request-relevant interval, and reconstructs a world.Source: §4.1 · pp. 8–9

4.2 Structured Recovery Control

  • Recovery offers temporal revision within the same video, source substitution under the same query, or query reformulation while preserving requested world semantics.The three action types retain different portions of the current trajectory and have no fixed selection order.Source: §4.2 · p. 9
  • The harness constrains the recovery policy to actions compatible with the current state, separating feasibility control from action selection.The compressor updates the state before the policy selects from the feasible action set.Source: §4.2 · p. 10
  • The compressed recovery state records the failure, concepts to avoid, visual targets to preserve, and guidance for the next query.It provides compact context without requiring replay of the full trajectory history.Source: §4.2 · p. 10

5 Experiments

Experiments compare models on a shared Search-to-World evaluation and test supervised fine-tuning of recovery agents. Outcomes vary by model, and jointly fine-tuning both recovery agents produces the strongest SFT configuration.

5.1 Evaluation Settings

  • The evaluation compares eight open-weight and four closed-source models on the same 300-request set, reporting ORR, WDR, and mean actions.The four primary agents use the same evaluation model within each full-model configuration, while other harness components and the action cap are fixed within comparisons.Source: §5.1 · p. 10

5.2 Data Engine

  • The data engine turns audited requests paired with scene templates into controlled natural-language requests through drafting, polishing, and verification.Accepted requests form an 8,900-request training-source pool and a 300-request evaluation set.Source: Figure 6 · p. 11, §5.2 · p. 11
  • Both request sets span 20 fine-grained domains, indoor and outdoor environments, and five spatial scales.The evaluation set contains 141 indoor and 159 outdoor requests; the training-source pool is evenly split between indoor and outdoor requests.Source: §5.2 · p. 11, Figure 7 · p. 12

5.3 Full Model Comparison

  • 88.3% ORR and 78.7% WDR: DeepSeek-V4-Flash leads the evaluated configurations, with 17.3 mean video-path actions.GLM-5.2 uses the fewest actions overall (13.9) and reaches 52.7% WDR; within closed-source models, GPT-5.6 Sol has the highest WDR (53.0%).Source: §5.3 · p. 11, Table 2 · p. 12

5.4 Recovery Ablation

  • Recovery-agent adapters are trained on teacher-generated trajectories, with role-specific filtering aligned to each agent’s inference-time output contract.Policy examples require positive world-delivery labels and admissible actions; compressor examples must follow the recovery-state schema and encode the observed failure and relevant context.Source: §5.4 · pp. 12, 14
  • 64.0% ORR, 49.7% WDR, and 24.5 mean actions: the Both SFT recovery configuration is the strongest SFT variant.Relative to the base model, Policy-SFT Only raises ORR from 49.0% to 57.0% and WDR from 40.3% to 44.3%; Compressor-SFT Only raises ORR to 52.3% and WDR to 42.0%.Source: §5.4 · p. 14, Table 3 · p. 14

6 Conclusion

Search-to-World makes end-to-end 3D-world delivery from web-sourced visual content evaluable, while WorldSearcher combines direct reuse, video reconstruction, and structured recovery. Evaluation shows that model choice affects retrieval, delivery, and action cost, and joint recovery-agent adaptation is the strongest SFT variant.

6 Conclusion

  • Search-to-World defines an end-to-end evaluation task for turning a user request into a usable 3D world through web-sourced visual content.ORR and WDR distinguish relevant-content retrieval from accepted world delivery.Source: §6 · p. 14

A Visualization and metrics of Delivered 3D World

The appendix presents representative delivered worlds from both WorldSearcher acquisition routes, pairing each request with its source resource and rendered views.

A.1 Reuse 3D World Delivery

  • Reuse-branch examples pair each user request with the retrieved 3D resource and several multi-view renderings.Source: §A.1 · p. 15

A.2 Reconstruction 3D World Delivery

  • Reconstruction-branch examples pair each user request with the retrieved video resource and several multi-view renderings.Source: §A.2 · p. 15

A.3 Metrics of Delivered 3D World

The appendix reports additional delivered-world metrics and specifies harness settings, including the recovery agents’ LoRA configuration.

A.3 Metrics of Delivered 3D World

  • Delivered-world quality and alignment are additionally assessed with PSNR, LPIPS, SSIM, CLIP Score, and Q-ReAlign.Table 4 reports these metrics for worlds delivered using GPT-5.6 Sol.Source: §A.3 · p. 15, Table 4 · p. 15

B Harness Settings

  • The harness coordinates query generation, query refinement, recovery compression, and recovery policy, while holding auxiliary retrieval, grounding, reconstruction, and quality-gate components fixed within each comparison.Source: B Harness Settings · p. 15, Table 5 · p. 15

B.1 Recovery SFT/LoRA Configuration

  • Recovery components use Qwen3.5-9B with role-specific LoRA adapters targeting all linear layers at rank 8, alpha 32, and dropout 0.05; the vision encoder and aligner are frozen.Source: §B.1 · p. 15

B.2 Recovery SFT/LoRA Training Data

This section specifies recovery-adapter training data and procedures, including the supporting state, prompts, and agent actions. It also defines recovery constraints and clip-acceptance criteria that guide search, grounding, and reconstruction while preserving the user’s visual target.

B.2 Recovery SFT/LoRA Training Data

  • Each recovery adapter receives a subset of training data as toy examples, with separate toy compressor and toy policy SFT files.The passage names “toy compressor sft.jsonl” and “toy policy sft.jsonl,” respectively.Source: §B.2 · p. 15

B.3 Recovery State and Action Schema

  • The recovery state retains the request target, visual-content assessment, failure, compressed context, admissible and selected actions, and remaining budget; its admissible actions depend on state.The compressor passes failure information, retained visual targets, and next-query guidance to the subsequent policy decision; the schema lists a video-action cap of 36.Source: §B.3 · p. 18

C Prompt Registry

  • The prompt registry records each interface’s role, model assignment, input/output schema, parsing, stopping, and retry rules, with complete prompts shown separately.The supplied registry passage describes these recorded fields; the table caption identifies the prompt-registry table.Source: C Prompt Registry · p. 18, Table 9 · p. 19

query generation

  • Query generation produces separate 3D-resource, video-source, and temporal-grounding queries, using the original request as the target authority without adding unsupported scene requirements.Search prioritizes 3D resources before video fallback; video-query wording may aid retrieval of continuous multi-view coverage, while grounding describes only what must appear in the retrieved segment.Source: query generation · p. 20

query refinement

  • Query refinement repairs video-search and grounding queries using failure advice only to change retrieval strategy, preserving request-supported target semantics and avoiding previously attempted video queries.The repaired queries must be compact, use complete words, and exclude avoid terms.Source: query refinement · p. 21

recovery compressor

  • The recovery compressor carries forward compact, transferable failure advice for later action selection and query repair, rather than noisy candidate-specific trace details.It updates a summary, avoid terms, retained visual targets, and next-query guidance, removing stale or one-off information.Source: recovery compressor · pp. 21–22

recovery policy

  • WorldSearcher’s video-recovery policy selects one currently allowed action for a video-to-3D path that requires an aligned, clear, continuous clip and a render passing quality gates.The harness dynamically supplies the legal action menu; the policy uses the immediate failure and cumulative failure advice, while next-video or next-clip attempts may still require budget for reconstruction.Source: recovery policy · pp. 22–23
  • The allowed_actions list governs legal recovery choices, and the policy prefers trying available higher-quality candidates before query refinement, which starts a new downstream chain.This is a preference rather than a rigid rule; a medium-quality candidate may be worth trying when budget is limited.Source: recovery policy · pp. 23–24
  • Q-ReAlign clips pass only when quality_score exceeds the configured clip_vqa_threshold, which defaults to 0.5; reranker matching scores are reference signals, not acceptance thresholds.Video acceptance remains the responsibility of a separate semantic-match gate.Source: recovery policy · pp. 23–24

semantic matcher

  • The semantic matcher accepts a VQA-passing clip only when sampled frames visibly support the original request, rejecting absent, incidental, occluded, or conflicting targets.For image-based requests, it allows broad matches in scene type, spatial category, and primary objects; it need not reject differences in colors or materials unless those are core targets.Source: semantic matcher · p. 25

D Data Engine and Verification Standard

The data-engine appendix maps implementation names to manuscript roles and documents prompt-based drafting, revision, and judging alongside deterministic validation checks.

D Data Engine and Verification Standard

D.1 Data-Engine Prompts

  • The appendix reproduces drafting, revision, and inspection prompts, while the Validator is specified through its interface and checks rather than an LLM prompt.Source: §D.1 · p. 26

Generator/Drafter and Naturalizer/Reviser

  • Drafted descriptions follow catalog blueprints, use 6–24 words, preserve an anchor term and required feature phrases, and serve as semantic input for a later naturalizer.Source: Generator/Drafter and… · p. 26

Judge/Inspector

  • The Judge/Inspector accepts concise descriptions only when they faithfully identify the scene, remain spatially useful, and avoid unsupported constraints or prohibited prompt-like language.Source: Judge/Inspector · pp. 26–27

Validator interface

D.2 Judge Output and Deterministic Checks

  • The Judge returns an identifier and Boolean acceptance decision without a score; deterministic audit categories cover five specified checks, as summarized in Table 11.Source: §D.2 · pp. 27–28, Table 11 · p. 28

E Evaluation Set Inventory

The sealed evaluation inventory contains 300 held-out requests, distributed across 20 semantic domains.

E Evaluation Set Inventory

E.1 Distribution Check

  • The 20 semantic domains each contain 13–23 requests in the evaluation set.Source: Figure 11 · p. 29

E.2 Row-Level Inventory

The appendix provides a complete 300-row evaluation inventory, pairing request IDs with scene descriptions and environment labels.

E.2 Row-Level Inventory

  • The inventory is identified as the complete set of 300 evaluation requests.The listed entries include request IDs, scene descriptions, and environment labels.Source: Table 13 · p. 30, §E.2 · p. 30
  • The supplied entries describe varied indoor and outdoor scenes, spanning hospitality, residential, transit, industrial, and natural settings, with examples including buildings, streets, and landscapes.Each entry pairs a scene description with domain and setting labels.Source: §E.2 · p. 31

Related papers

Loading 2609.07605v2…