PaperlayerView exact revision on arXiv

Source-linked AI summary

Think-Capture: Active 3D Geometry Acquisition for Spatial Reasoning in Vision-Language Models

Junjie Wang, Zhihao Yuan, Yunhan Tang, Sheng Xu, Jiaqi Mao, Xiaoyu Ye, Yueru Luo, Xu Yan, Chun-Mei Feng, Zhen Li

arXiv:2610.29232v1cs.CV

TL;DR

VLMs can struggle with consistent 3D relations across sparse viewpoints, while dense geometry fusion adds computation and context overhead. Think-Capture selects task-critical objects, acquires and compresses their geometry, and trains the VLM to use or discount tool evidence. It reports gains across three benchmarks and identifies limits tied to tool observability and fine-manipulation detail.

  • Problem

    VLMs struggle with cross-view consistency, occlusion, and relative 3D position, while dense geometry fusion can add computation and context overhead.

    Source: §1 · p. 1

  • Method

    Think-Capture selects query-relevant objects, converts their geometry into compact symbolic state, and uses hierarchical GRPO to train the VLM on helpful and misleading tool evidence.

    Source: §1 · pp. 1–2

  • Results

    Think-Capture-4B reaches 54.4% on MindCube-Tiny (+21.5 points over Qwen3-VL-4B), 39.1% on the MMSI positional subset (+7.9), and 42.8% on ViewSpatial (+5.2).

    Source: §4.3 · p. 5

  • Takeaways & Limitations

    Consistent benchmark, perturbation, scaling, and real-robot results support active acquisition as a practical route to spatially grounded VLMs.

    Source: §5 · p. 8

    The framework cannot recover geometry absent from every view, and fine manipulation requires higher-resolution local shape and contact information.

    Source: §4.10 · p. 8

Abstract

from arXiv · show

Vision-Language Models (VLMs) excel at 2D recognition but often fail to maintain consistent 3D relations across sparse viewpoints, leading to spatial hallucinations under occlusion and perspective changes. Existing methods commonly fuse all frames with dense geometry, incurring substantial computation and context overhead while encouraging blind reliance on imperfect estimators. We present Think-Capture, a query-conditioned framework that first identifies task-critical objects and then acquires their geometry on demand. A 3D toolkit lifts zero-shot masks into a common coordinate frame, and a Symbolic 3D State Abstraction compresses object and camera states into structured text. Hierarchical Group Relative Policy Optimization (GRPO) trains the VLM on cases in which geometry either helps or misleads, teaching it when to use tool evidence and when to fall back on visual cues. Think-Capture-4B reaches 54.4% on MindCube-Tiny (+21.5 points over its backbone), 39.1% on the MMSI-Bench positional subset (+7.9), and 42.8% on ViewSpatial (+5.2). It also remains robust under coordinate and mask perturbations. The full stack fits one 24 GB GPU and serves as an asynchronous high-level reasoner on an Agibot G1. Our code is available at https://github.com/819346675/Think-Capture.

1. Introduction

VLMs struggle to maintain spatial consistency across views, while dense fusion can waste computation and context on irrelevant geometry. Think-Capture instead acquires query-relevant geometry and trains the VLM to weigh it alongside visual evidence.

1. Introduction

  • Cross-view consistency, occlusion, and relative 3D position remain difficult for VLMs, which may substitute image-plane heuristics for a persistent spatial model [7] [20] [21] [40] [44] [50] [53] [55] [57].These capabilities matter for embodied systems that navigate and manipulate objects [2] [13] [31] [32] [49] [51] [64].Source: §1 · p. 1
  • Dense fusion processes geometry before seeing the query, increasing computation and context length while potentially adding distracting evidence [8] [23] [40].External 3D tools make geometry explicit, but errors in tool output or interpretation can still propagate to answers.Source: §1 · p. 1
  • Think-Capture selects query-needed objects, estimates their masks, coordinates, and cross-view relations, then compresses them into a symbolic object-and-camera state.The abstraction filters irrelevant geometry while preserving query-relevant relations [3] [42].Source: §1 · p. 1
  • Hierarchical GRPO trains a 4B VLM on cases where geometry helps or misleads, rewarding useful tool integration while retaining correct answers when augmentation is misleading.The evaluation covers MindCube, MMSI-Bench, and ViewSpatial-Bench, along with scaling, transfer, tool corruption, a matched 2D-box baseline, and real-robot deployment.Source: §1 · pp. 1–2

2. Related Work

Related work spans implicit spatial learning, multi-view evaluation, and explicit geometry or tool use. Think-Capture combines query-conditioned target selection, a shared symbolic state, and training to accept or reject imperfect geometric evidence.

2.1. Implicit Spatial Learning in VLMs

2.2. Spatial Reasoning From Multi-views

  • Multi-view benchmarks test cross-view understanding [54], mental rotation [40], relations between entities that never co-occur [53], and perspective transformations [20].Mental models and evaluations of VLM spatial understanding [6] [34] inform this line of work.Source: §2.2 · p. 2

2.3. Explicit Spatial Reasoning

  • Reconstruct-then-reason systems represent the whole scene, while Think-Capture combines query-conditioned target selection, a shared object-and-camera state, and hierarchical GRPO to accept or reject imperfect tool evidence.Tool-using alternatives include systems that manipulate point clouds or estimate camera anchors [42] [60]; controlled corruption and matched 2D-box experiments evaluate robustness and representation value.Source: §2.3 · p. 2

3. Method

Think-Capture plans which geometry to acquire, gathers target-specific 3D state, formats it as compact text, and reasons over that state with the original images. Its training rewards useful geometry use while guarding against misleading tool evidence.

3.1. Overview: The Think-Capture Paradigm

  • Given multi-view images and a spatial query, the two-stage Reason-then-Act loop plans needed geometry, acquires it, and combines the symbolic state with the images to produce an answer.Stage 1 emits semantic targets; Stage 2 reasons over the acquired state and original images.Source: §3.1 · p. 2

3.2. 3D Perception via Tool-Calling

  • For each target, SAM3 produces a zero-shot mask in each relevant frame; in parallel, π3 predicts local point maps and camera transforms, mapping points into a world frame anchored at the first frame.Source: §3.2 · p. 2

3.3. 3D Templated Information Generation

  • Object streams represent targets with world positions, bounding-box scales, and pairwise relations, while camera motion is decomposed into translation and Euler angles; the bottleneck averages 100 tokens per query versus 205 for unstructured tool output (51% reduction).Source: §3.3 · pp. 3–4

3.4. 2D-3D Joint Reasoning

  • The clipped GRPO objective uses sampled trajectories and a KL penalty, with normalized advantages based on within-group reward mean and standard deviation.Constant-reward groups provide no within-group preference signal.Source: §3.4 · p. 4
  • GRPO training uses 3,032 disagreement-mined examples: positive cases reward correct answers when tool use helps, while negative cases reward correct answers despite misleading augmentation.The sample-conditioned subsets contain 1,927 positive and 1,105 negative examples; screening agreement cases are excluded, and the labels indicate screening-time disagreement rather than uniform question difficulty. The reward combines accuracy and format compliance, with the required output format earning 0.5.Source: §3.4 · p. 4

4. Experiment

Think-Capture is evaluated on three spatial-reasoning benchmarks, with results spanning matched-backbone gains, robustness controls, qualitative cases, and robot deployment. Its experiments also identify scope limits: geometry depends on perception observability and calibration, and the current waypoint-oriented representation does not provide fine-manipulation detail.

4.1. Datasets

  • The evaluation covers MindCube-Tiny mental rotation, MMSI cross-frame relations, and ViewSpatial perspective taking, using Qwen3-VL-4B trained on 3,032 disagreement-mined examples.Training used four A100 GPUs for approximately 38 hours; the vision encoder was frozen.Source: §4.1 · p. 4, §4.2 · p. 4

4.3. Main Results

  • 54.4% on MindCube-Tiny (+21.5 points over Qwen3-VL-4B), 39.1% on the MMSI positional subset (+7.9), and 42.8% on ViewSpatial (+5.2).Table 1 compares accuracy across the three benchmarks; Think-Capture also improves unseen MMSI and ViewSpatial results without training on either benchmark.Source: Table 1 · p. 5, §4.3 · p. 5
  • With 0.4 m coordinate noise, accuracy is 46.2% versus 26.7% for zero-shot tool use; with 30% mask dropout, scores are 52.1% versus 36.2%.The reported controls also compare symbolic 3D state against matched 2D-box inputs; the full model retains higher accuracy across tested corruption levels.Source: §4.3 · p. 5, Table 2 · p. 6

4.4. Generalization and Matched Controls

  • Across 2B, 4B, and 8B backbones, MindCube gains remain within 19.9–21.5 points and full-MMSI gains within 8.8–10.0 points.The matched 2D-box control trails symbolic 3D state by 17.5 points on MindCube, 7.1 on full MMSI, and 4.9 on ViewSpatial.Source: §4.4 · p. 5, Table 3 · p. 6, Table 4 · p. 6

4.5. Ablation Study

  • On MindCube-Tiny, combining tools with GRPO reaches 54.4%, versus 39.9% for zero-shot tool use and 33.3% for GRPO alone.Replacing the template with an unstructured coordinate dump reduces accuracy from 54.4% to 49.8%; the template averages 100 tokens per query versus 202 for unified format and 205 for raw output.Source: §4.5 · p. 5, Table 5 · p. 7

4.6. Qualitative Analysis

  • The qualitative cases show the model inferring an unseen bench behind the observer and converting a mirror’s world-frame location to photographer-relative front-left.The first case uses the blue table as a facing anchor; the second captures the reference camera pose and mirror geometry.Source: Figure 3 · p. 6, §4.6 · p. 7, Figure 4 · p. 8

4.7. Joint Reasoning Dynamics

  • During GRPO training, positive-sample accuracy stabilizes near 85%, while negative-sample accuracy rises from about 30% to above 60%.The subsets respectively represent cases where tool augmentation repairs a vision-only error and cases where it overturns a correct visual answer.Source: §4.7 · p. 7

4.8. Real-World Deployment

  • On an Agibot G1, the high-level reasoner supplies a global waypoint asynchronously while StarVLA performs high-frequency arm control; the full stack peaks at 21.5 GB on one RTX 4090.The 21.8-second pipeline spends 3.1 seconds on geometric capture; direct-answer decoding cuts latency by over 40% while retaining 53.2% MindCube-Tiny accuracy.Source: §4.8 · p. 7, Figure 5 · p. 8

5. Conclusion

Think-Capture combines query-conditioned 3D acquisition, a compact symbolic interface, and GRPO training to integrate visual and geometric evidence. Benchmark, perturbation, scaling, and robot results support active acquisition as a practical route to spatially grounded VLMs.

5. Conclusion

  • The conclusion presents active 3D acquisition and compact symbolic state as a practical route to spatially grounded VLMs, supported by benchmark, perturbation, scaling, and robot results.Future work will extend the interface to uncertainty-calibrated, temporally persistent geometry for state refresh in dynamic environments.Source: §5 · p. 8

Supplementary Material

The supplement documents the perception pipeline, symbolic abstraction, prompts, training, and controlled experiments. Its overview maps these details to supplementary Sections B–F.

B.1.1. Core Algorithm Setup

  • The AgentToolWrapper combines text-prompted SAM3 segmentation with π3 geometry estimation, then formats selected target points into structured spatial text.The pipeline proposes targets, estimates geometry, segments targets, and fuses observations into object state.Source: §B.1 · p. 13, §B.1.1 · p. 13, §B.1.2 · p. 13

B.1.2. Implementation Specifics

  • The implementation downsamples sequences to four frames, segments only requested targets, and merges cross-view observations by centroid distance.The reported fusion threshold is 0.5 m; absolute metric accuracy depends on reconstruction quality, and robot execution also requires mapping to the robot frame.Source: §B.1.2 · p. 13

B.2. Symbolic 3D Abstraction: From Geometry to Semantics

  • Symbolic 3D abstraction exposes on-demand object positions, extents, and relations through a compact interface rather than dense scene descriptions.The authors note that richer object-level context could extend the interface to shape- and contact-related questions.Source: §B.2 · pp. 13–14

B.2.1. 3D Back-Projection & Attribute Extraction

  • Masked point maps are transformed into world coordinates, invalid points are discarded, and axis-aligned bounding boxes provide object centers and scales.The extraction algorithm returns no attributes when the masked point set is empty.Source: §B.2.1 · p. 14, §B.2 · p. 14

B.2.2. Spatial Reasoning & Formatting

  • The formatter filters details by query type and projects relative directions onto the first frame’s basis to align terms with the initial viewpoint.For object queries, the formatted state includes selected objects and their pairwise relations.Source: §B.2.2 · p. 14, §B.2 · p. 14

B.3.1. Stage 1: Intent Analysis

  • Stage 1 prompts the VLM to list the objects requiring 3D measurement to answer the question.The prompt requests one target per line in an objects-to-segment block.Source: §B.3.1 · p. 14

B.3.2. Stage 2: 3D-Augmented Reasoning

  • Stage 2 is instructed to combine the original images with formatted 3D spatial information and return a multiple-choice answer.Its prompt specifies separate reasoning and answer tags.Source: §B.3.2 · p. 14

B.4. Training Hyperparameters & Details

  • GRPO fine-tuning uses four A100 GPUs and stops at 1,600 steps to avoid overfitting the 3,032-example training set.The supplementary material also provides a table of detailed GRPO hyperparameters.Source: §B.4 · p. 14, Table S1 · p. 15

B.4.1. Training Dynamics

Training curves show improved accuracy on both positive and negative screening subsets. The supplement also examines tool robustness, backbone transfer, and explicit 3D state.

B.4.1. Training Dynamics

  • Positive-sample accuracy stabilizes near 85%, while negative-sample accuracy rises from roughly 30% to above 60% during GRPO training.The accompanying figures show training accuracy and mean reward, alongside optimization loss.Source: §B.4.1 · p. 15, Figure S1 · p. 16, Figure S2 · p. 16

C. Additional Controlled Experiments

  • The additional experiments assess robustness to imperfect tools, transfer across model scales and datasets, and explicit 3D state beyond visual highlighting.The controlled comparisons include tool perturbations, backbone scaling, and a matched 2D-box baseline.Source: C. Additional Controlled Experiments · p. 16

C.1. Robustness to Tool Perturbations

Controlled experiments test performance under tool corruption, across backbone sizes, and against matched 2D-box visual prompting. The reported comparisons favor hierarchical training and explicit symbolic 3D state in the tested settings.

C.1. Robustness to Tool Perturbations

  • At strongest coordinate perturbation, hierarchical GRPO improves accuracy from 40.3 to 46.2%; at 30% mask dropout, it improves 49.0 to 52.1%.Tool returns no valid target in fewer than 1.6% of MindCube-Tiny cases; a deterministic fallback omits the empty symbolic state and answers from images alone.Source: Table S2 · p. 16, §C.1 · p. 16

C.2. Backbone Generalization

  • Gains are consistent across 2B, 4B, and 8B Qwen3-VL backbones on MindCube-Tiny and full MMSI-Bench.The supplementary scaling table reports the full MMSI-Bench, whereas the main table reports its positional-relation subset.Source: §C.2 · p. 16, Table S3 · p. 16

C.3. Explicit 3D State versus Visual Prompting

  • Matched 2D-box prompting remains substantially below explicit 3D state on all three benchmarks, especially for egocentric and cross-view questions requiring coordinate transformations.Both representation variants use the same selected targets and masks.Source: §C.3 · pp. 16–17, Table S4 · p. 17

D.1. Efficiency & Latency Analysis

Latency is dominated by VLM text generation, while the complete stack fits within 24 GB of GPU memory. On the Agibot G1, Think-Capture provides asynchronous waypoint updates, with geometry captured at initialization.

D.1.1. Latency Breakdown

  • More than 40% lower end-to-end latency is possible without intermediate chain-of-thought tokens, with MindCube-Tiny accuracy at 53.2% versus 54.4% with full reasoning.Across 1,050 MindCube-Tiny samples on one NVIDIA A100 (80 GB), end-to-end time is 21.8 s; Stage 1 takes 6.8 s and Stage 2 takes 11.9 s, while tool execution takes 3.1 s.Source: §D.1.1 · p. 17, §D.1 · p. 17

D.1.2. Resource Consumption

  • 21.5 GB peak GPU memory allows the complete VLM-4B, SAM3, and π3 stack to fit on a 24 GB RTX 4090.The paper contrasts this with systems requiring 40 GB or 80 GB accelerators.Source: §D.1.2 · p. 17

D.2. Real-World Deployment Details

  • Think-Capture sends low-frequency global spatial waypoints to a separate StarVLA motor policy, keeping VLM generation outside the time-critical control loop.At initialization, it identifies instruction-relevant objects, acquires their geometry, and formats target coordinates and relations for reasoning.Source: §D.2 · p. 17
  • Geometry is acquired at initialization, so moving targets or substantial scene changes require re-acquisition; the 21.8 s reasoning cycle yields asynchronous waypoint updates.Faster decoding and state refresh would extend the design to more reactive tasks.Source: §D.2 · p. 17

E. Limitations

The paper identifies limits from foundation-model errors, symbolic information loss, and runtime and scene dynamics. It points to broader training, uncertainty modeling, efficient state updates, and temporal evaluation as directions for addressing these boundaries.

E. Limitations

  • Severe π3 reconstruction or SAM3 segmentation failures can still propagate to answers despite hierarchical GRPO and the empty-return fallback.The authors also identify omitted fine shape, object orientation, and contact geometry as limitations for dexterous manipulation.Source: E. Limitations · p. 17
  • VLM generation dominates runtime, and initialization-only geometry capture constrains reactive deployment; broader training, uncertainty modeling, and efficient geometric updates could improve robustness in dynamic environments.The paper says reactive deployment will require faster decoding and state refresh.Source: E. Limitations · p. 17
  • Evaluating temporal extensions remains future work: the authors call for controlled sequences and verified, time-dependent spatial labels to test when geometry should be refreshed under motion or appearance changes.The proposed tests separate motion from appearance variation, including lighting changes and visual edits.Source: E. Limitations · p. 17

F. Additional Qualitative Analysis

The qualitative examples contrast image-plane guesses with query-conditioned target selection and geometry-grounded reasoning. They illustrate egocentric relations, cross-view references, and the progression from selecting targets to a spatial answer.

F.1. Ambiguous Egocentric Relations

  • In Figure S3, the baseline answers “back-right” from image-plane cues, while Think-Capture selects the sofa and bag, reconstructs their geometry, and infers “front” in the requested frame.The example shows the system interpreting “facing shelves” in the common coordinate frame.Source: §F.1 · p. 18, Figure S3 · p. 18

F.2. Egocentric and Cross-View Reference Frames

  • Figure S4 shows the baseline confusing image-plane background with the observer’s rear; Think-Capture uses the Blue Table’s facing direction and the Seating Bench relation to identify the unseen rear region.Figure S5 illustrates that explicit camera pose and object geometry resolve the requested photographer-relative direction on MMSI-Bench.Source: Figure S4 · p. 19, Figure S5 · p. 19, §F.2 · p. 19

F.3. Additional Target-Selection Case

  • Figure S6 illustrates the sequence from target selection through geometry acquisition to the final spatial answer.The supplement identifies it as an additional target-selection case.Source: §F.3 · p. 20, Figure S6 · p. 20
Loading 2610.29232v1…