PaperlayerView exact revision on arXiv

Source-linked AI summary

GeneralVLA-2: Geometry-Aware Reconstruction and Governed Memory for Robot Planning

Haoyu Wang, Guoqing Ma, Zeyu Zhang, Yandong Guo, Boxin Shi, Hao Tang

arXiv:2606.17480v1cs.CVcs.RO

TL;DR

GeneralVLA-2 addresses unreliable monocular 3D reconstruction and uncontrolled experience reuse in robot planning with geometry-aware multi-view fusion and governed long-term memory. It improves reconstruction on GSO-30 and memory performance across Terminal-Bench 2.0 and SWE-Bench Verified.

  • Problem

    Generalist vision-language-action planning lacks stable object-centric 3D evidence from monocular reconstruction and governed, geometrically relevant reuse of manipulation experience.

  • Method

    GeneralVLA-2 combines GeoFuse-MV3D’s mask-verified, conservatively fused multi-view geometry with KnowledgeBank records governed by quality, confidence, lifecycle, verifier, and conflict metadata.

  • Results

    Across reconstruction and long-horizon agent benchmarks, GeoFuse-MV3D improves all reported GSO-30 metrics and KnowledgeBank improves over ReasoningBank by 4.53% on Terminal-Bench SR.

  • Takeaways & Limitations

    The results support explicitly governed memory and more faithful object geometry as planner-facing evidence improvements for hierarchical vision-language-action systems.

  • Takeaways & Limitations

    The method depends on reliable calibrated observations, masks, poses, and verifier quality, and evaluation does not cover several challenging real-world manipulation settings.

Abstract

from arXiv · show

Generalist vision-language-action systems need object-centric 3D evidence and reusable manipulation experience to plan reliable robot trajectories. GeneralVLA provides a hierarchical interface for converting language and RGB-D observations into 3D end-effector paths, but two bottlenecks remain. First, monocular SAM3D-style object reconstruction can hallucinate pose and unseen geometry, while manipulation benefits from stable object shape when calibrated multi-view observations are available. Second, the original KnowledgeBank mainly retrieves semantically similar snippets and appends new knowledge, which makes it difficult to control memory quality, conflicts, confidence, and geometric relevance. To address the first challenge, we introduce GeoFuse-MV3D, a geometry-prior-guided MV-SAM3D reconstruction branch that verifies external geometry cues with input-view masks, applies soft visual-hull support, performs axis-wise refinement, and fuses only geometry while preserving appearance. To address the second challenge, we upgrade KnowledgeBank into a governed long-term memory system with explicit quality, confidence, lifecycle, verifier, and conflict metadata, together with precision-oriented retrieval. Finally, we evaluate the reconstruction branch on GSO-30 and the memory module on Terminal-Bench 2.0 and SWE-Bench Verified; GeoFuse-MV3D improves over the MV-SAM3D baseline by reducing CD and LPIPS by 2.20% and 2.02% while increasing PSNR and SSIM by 2.36% and 1.03%, and KnowledgeBank improves over ReasoningBank by 4.53% on Terminal-Bench SR and 3.73% on SWE-Bench resolve rate, while reducing AS by 4.95% and 5.65%, respectively. Code: https://github.com/AIGeeksGroup/GeneralVLA-2. Website: https://aigeeksgroup.github.io/GeneralVLA-2.

1 Introduction

GeneralVLA-2 strengthens GeneralVLA by addressing unreliable monocular object reconstruction and uncontrolled experience reuse. It introduces geometry-aware multi-view fusion and governed memory while retaining the hierarchical interface for 3D-conditioned robot planning.

  • Challenge: Monocular SAM3D-style reconstruction is vulnerable to pose ambiguity and hallucinated backside geometry, which can alter grasps, clearance, and collision relations.Reliable manipulation requires faithful object shape and pose from robot observations.
  • Challenge: Semantic similarity alone cannot ensure that retrieved KnowledgeBank memories are safe, current, non-conflicting, or geometrically applicable to the present scene.The original memory system stores natural-language memories from past trajectories, motivating governed experience reuse.
  • Motivation: Calibrated short multi-view RGB-D observations can reduce monocular hallucination, but reconstruction must conservatively avoid missing parts, over-shrunk geometry, and appearance drift.Downstream planning is sensitive to reconstruction errors in geometry, color, and opacity.
  • Contribution: GeoFuse-MV3D validates external geometry priors against input masks, applies appearance calibration, performs soft visual-hull and axis-wise refinement, and fuses geometry conservatively while preserving fixed input views.The method treats external 3D estimation as a geometry-prior provider rather than as the reconstruction method itself.
  • Summary: GeoFuse-MV3D supplies more stable object geometry under multi-view reconstruction, while governed memory provides more controlled experience reuse within GeneralVLA-2’s hierarchical interface.On GSO-30, GeoFuse-MV3D improves over the MV-SAM3D baseline across CD, PSNR, SSIM, and LPIPS.

2 Related Work

Related work spans object-centric 3D reconstruction, generalist vision-language-action planning, and governed memory for long-horizon agents. GeneralVLA-2 builds on these directions by retaining hierarchical planning while strengthening geometry and long-term experience inputs.

  • Object-centric 3D reconstruction: Single-image 3D methods face pose ambiguity and hallucinated unseen structure, while MV-SAM3D uses complementary multi-view observations and adaptive fusion without additional training.SAM3D reconstructs geometry, texture, and layout from one image; Fast-SAM3D targets the speed-quality tradeoff.
  • VLA and 3D robot planning: Generalist VLA systems connect visual observations, language instructions, and robot actions, with RT-2, Octo, OpenVLA, π0, and TinyVLA demonstrating the value of large-scale data.A complementary line uses foundation models to produce intermediate plans or 3D representations.
  • VLA and 3D robot planning: GeneralVLA separates SAM-based affordance segmentation, iterative localization, 3DAgent planning, and low-level execution into an interpretable intermediate-representation hierarchy.GeneralVLA-2 retains this hierarchy while refining object-centric geometry and retrieved long-term experience conditioning 3DAgent.
  • Memory, verification, and governance: Long-horizon agents reuse past experience through systems such as Synapse, Agent Workflow Memory, and ReasoningBank, but semantic relevance alone cannot ensure memory safety, currency, consistency, or geometric applicability.Verifier-oriented methods and LLM-as-a-judge provide scalable signals for evaluating open-ended trajectories when criteria are decomposed.

3 Preliminaries

GeneralVLA formulates tabletop manipulation as generating executable trajectories from language and calibrated RGB-D observations through affordance perception, 3D planning, and low-level execution. Its original KnowledgeBank supports test-time learning through semantic retrieval but lacks geometric applicability, failure constraints, and lifecycle control.

  • Task setting: At task t, the robot receives language instruction q_t and RGB-D observation o_t from one or more calibrated cameras, then generates an executable trajectory.The trajectory must satisfy the instruction under the current object arrangement, obstacle layout, and grasping constraints.
  • Affordance segmentation and 3D point construction: SAM-based segmentation and iterative localization identify task-relevant affordances, which calibrated depth and camera intrinsics project into a 3D scene representation.The representation associates object or region labels with 3D points.
  • Affordance segmentation and 3D point construction: Multiple 3D points per object let the planner infer extent, pose, opening direction, obstacle height, and other spatial relations unavailable from a single point.These geometric cues support planning in the current scene.
  • Original KnowledgeBank: The original KnowledgeBank stores natural-language items from prior trajectories and retrieves semantically similar snippets, but semantic similarity alone does not ensure geometric applicability.Failed trajectories should become constraints rather than action recipes, while growing memory requires lifecycle control to suppress duplicates and conflicts.

4 Method

GeneralVLA-2 strengthens the planner-facing interfaces with GeoFuse-MV3D for conservative multi-view object reconstruction and a governed KnowledgeBank for reliable experience reuse. The resulting 3DAgent conditions planning on current 3D scene information, refined object evidence when available, and structured memory context.

  • GeoFuse-MV3D: GeoFuse-MV3D uses calibrated multi-view RGB-D observations, masks, and camera poses to regularize MV-SAM3D geometry and reduce reliance on monocular reconstruction.Its two geometry sources combine an external geometry-prior branch, instantiated with VGGT, and a provider-free input-view axis-compensation branch.
  • GeoFuse-MV3D: Both geometry sources are checked against input-view masks, with low-support points converted into bounded inward geometry corrections rather than deleted or made transparent.The branch also applies axis-wise center refinement while preserving compatibility with the original reconstruction.
  • GeoFuse-MV3D: Final fusion blends only geometry coordinates, leaving color, opacity, scale, rotation, and spherical-harmonic appearance unchanged.When topology differs or mask support is weak, the method falls back to the trusted source-A result.
  • Governed KnowledgeBank: Governed KnowledgeBank stores reusable manipulation lessons as structured records containing content type, lifecycle state, confidence, verifier quality, conflict links, and verifier metadata.Memory types distinguish procedural hints, failure-avoidance constraints, and tool-usage guidance, while lifecycle states distinguish provisional, active, summary, and archived records.
  • Governed KnowledgeBank: A verifier scores candidate memories using task completion, spatial consistency, collision safety, execution validity, and generalizability, promoting only sufficiently supported knowledge.Retrieval is precision-oriented, and consolidation supports add, merge, replace, discard, summarization, and archival operations under a fixed active-memory budget.
  • Planner integration: At inference, 3DAgent receives the instruction, current 3D scene representation, refined object evidence when available, and structured KnowledgeBank context, then outputs the existing waypoint-and-gripper trajectory format.The affordance segmentation, depth-based 3D point, and low-level execution interfaces remain unchanged.

5 Experiments

Experiments evaluate GeoFuse-MV3D reconstruction, governed KnowledgeBank memory, and GeneralVLA-2 planning across controlled benchmarks, RLBench simulation, and real-world manipulation. Results show improved memory reuse, broad task coverage, and effective geometry-aware reconstruction and execution.

  • KnowledgeBank evaluation: The controlled memory evaluation isolates admission, verification, retrieval, and lifecycle management on Terminal-Bench 2.0 and SWE-Bench Verified before integration with 3DAgent.Terminal-Bench uses SR and AS, while SWE-Bench Verified uses resolve rate and AS.
  • KnowledgeBank evaluation: KnowledgeBank improves over ReasoningBank by 4.53% on Terminal-Bench SR and 3.73% on SWE-Bench resolve rate across four model backbones.It also reduces AS by 4.95% and 5.65%, respectively.
  • Simulation evaluation: GeneralVLA-2 generates successful trajectories for all 14 RLBench tasks, while Hamster, VoxPoser, and CAP cover 10, 9, and 7 tasks, respectively.It outperforms baselines in 10 tasks, and removing KnowledgeBank consistently lowers success.
  • GeoFuse-MV3D ablation: Conservative geometry fusion achieves the best PSNR, SSIM, and LPIPS while retaining CD clearly better than the MV-SAM3D baseline.The geometry-prior branch is strongest for CD, while the provider-free axis branch gives the strongest single-source appearance metrics.
  • Real-world evaluation: GeneralVLA-2 succeeds on all four real-world tasks and outperforms CAP and RoboPoint.The tasks are move spray bottle, open drawer, open jar, and sort object, evaluated over 10 episodes with varying object poses across three trials.

6 Limitations

GeneralVLA-2 remains sensitive to the reliability of calibrated observations, masks, poses, and verifier quality, with failures propagating to reconstruction or memory retrieval. Its conservative design yields consistent but modest gains, while real-world evaluation does not yet cover long-horizon mobile manipulation, heavy occlusion, or deformable objects.

  • Input and verifier dependence: GeneralVLA-2 depends on reliable calibrated observations, masks, poses, and verifier quality.Failures in these inputs can propagate to reconstruction or memory retrieval.
  • Input and verifier dependence: Calibration or mask failures can directly affect GeoFuse-MV3D in multi-view reconstruction.
  • Evaluation scope: The conservative method achieves consistent but modest gains over MV-SAM3D.
  • Evaluation scope: Real-world evaluation does not yet cover long-horizon mobile manipulation, heavy occlusion, or deformable objects.

7 Conclusion … A.3 Additional GeoFuse-MV3D Details

GeneralVLA-2 extends GeneralVLA with geometry-aware reconstruction and governed experience reuse, while the appendix details GeoFuse-MV3D’s fixed evaluation protocol, qualitative gains, and reconstruction mechanisms.

  • 7 Conclusion: GeneralVLA-2 extends GeneralVLA with GeoFuse-MV3D for calibrated multi-view 3D evidence and a governed KnowledgeBank for reusable manipulation experience.The reconstruction branch uses mask-verified geometry support, provider-agnostic priors, axis-wise refinement, and appearance-preserving fusion; memory uses verifier-aware quality, confidence, lifecycle, conflict, and usage metadata.
  • Appendix: The appendix consolidates GeoFuse-MV3D’s quantitative results, qualitative comparisons, implementation equations, and component ablations under a reproduced MV-SAM3D protocol.Experiments use the same GSO-30 objects, five input views, masks, camera poses, and held-out target views as the baseline.
  • A.1 GSO-30 Reconstruction Results: GeoFuse-MV3D’s GSO-30 reconstruction evaluation uses the same five-view, mask, camera-pose, and held-out-view setup as the reproduced MV-SAM3D baseline.The appendix identifies this as the main quantitative reconstruction evaluation protocol.
  • A.2 Qualitative Reconstruction Results: Under a fixed evaluation protocol, GeoFuse-MV3D produces more complete and geometrically consistent reconstructions in several challenging GSO-30 cases.The comparison covers CD, PSNR, SSIM, and LPIPS against the MV-SAM3D baseline.
  • A.2 Qualitative Reconstruction Results: For qualitative comparisons, MV-SAM3D and GeoFuse-MV3D use identical five input views, object masks, and camera poses for every GSO-30 object.Figures 5 and 6 provide comparisons across two object sets.
  • A.3 Additional GeoFuse-MV3D Details: GeoFuse-MV3D projects 3D points into input views and computes average mask reprojection disagreement as a conservative geometry diagnostic.Low support is converted into bounded shrink strength rather than triggering hard deletion.
  • A.3 Additional GeoFuse-MV3D Details: Appearance affine calibration estimates per-channel gains and biases on masked pixels, with identity-oriented regularizers preserving the input object’s original tone.Axis compensation separately chooses scale and shift parameters, while geometry fusion can use confidence-weighted residual fusion when support scores are available.

A.4 GeoFuse-MV3D Component Ablation · B Additional KnowledgeBank Evaluation · B.1 Additional KnowledgeBank Details

The ablation isolates GeoFuse-MV3D components and shows complementary strengths from geometry-prior and provider-free axis branches. KnowledgeBank adds typed lifecycle governance, verifier-based quality assessment, and consolidation rules for maintaining reusable memories.

  • A.4 GeoFuse-MV3D Component Ablation: Ablations use the same GSO-30 objects, five input views, and held-out target views to isolate geometry-prior guidance, soft visual-hull support, axis compensation, and multi-source fusion.
  • A.4 GeoFuse-MV3D Component Ablation: The geometry-prior main-provider branch is strongest for CD, whereas the provider-free axis branch yields the strongest single-source PSNR, SSIM, and LPIPS gains.
  • A.4 GeoFuse-MV3D Component Ablation: Adding softVH improves CD and SSIM over source A but slightly lowers PSNR and LPIPS by prioritizing geometric support over appearance fidelity.
  • A.4 GeoFuse-MV3D Component Ablation: Combining the two sources achieves the best PSNR, SSIM, and LPIPS among ablated rows while keeping CD cl.
  • B.1 Additional KnowledgeBank Details: The full KnowledgeBank record includes source status, usage statistics, deduplication keys, verifier metadata, lifecycle state, and memory type.Lifecycle states are provisional, active, summary, or archive; memory types are procedural hint, failure avoidance, or tool usage.
  • B.1 Additional KnowledgeBank Details: The verifier evaluates robot memories using completion, spatial consistency, collision safety, execution validity, and generalizability, while software-agent evaluation uses root-cause analysis, code quality, and empirical verification.The aggregate outcome is verified success, verified fail, or uncertain, with midpoint fallback only when no valid score token is found.
  • B.1 Additional KnowledgeBank Details: Consolidation compares candidates against active records using relevance, similarity, deduplication, and conflict checks, then applies ADD, MERGE, REPLACE, or DISCARD.Governance retires stale failures, resolves conflicts, summarizes repeated successful clusters, and archives overflow beyond the active budget.

B.2 KnowledgeBank Benchmark Results · B.3 KnowledgeBank Component Ablation

KnowledgeBank improves benchmark performance over no-memory and prior-memory baselines while trading verifier and governance overhead for shorter executions. Component ablations show that governance and retrieval quality are central to its gains.

  • B.2 KnowledgeBank Benchmark Results: KnowledgeBank outperforms no-memory and prior-memory baselines across Terminal-Bench 2.0, SWE-Bench Verified, and all evaluated backbones.All metrics are reported as mean ± standard deviation over five seeds under matched backbone models and benchmark protocols.
  • B.2 KnowledgeBank Benchmark Results: 4.0 points: ReasoningBank improves Terminal-Bench SR over No Memory on average.ReasoningBank also improves SWE-Bench resolve rate by 3.8 points and reduces AS.
  • B.2 KnowledgeBank Benchmark Results: Verifier and governance calls increase extra tokens, but shorter executions reduce total tokens and latency relative to AWM and ReasoningBank for every backbone.No Memory remains the lowest-cost reference because it removes retrieval and verification, but it does not provide the reported accuracy gains.
  • B.2 KnowledgeBank Benchmark Results: AS is the unweighted macro-average for cost-accounting settings, while extra tokens cover verifier, re-verification, induction, and governance calls outside AS.Storage denotes the final serialized memory-bank footprint.
  • B.2 KnowledgeBank Benchmark Results: Persistent verifier metadata can improve future retrieval and reduce agent steps, making storage of verifier outputs central to the cost tradeoff.The tradeoff is therefore not simply the amount of verifier usage.
  • B.3 KnowledgeBank Component Ablation: The ablation removes admission, replay, governance, or failure-handling modules and compares them with a semantic-retrieval-only variant.The semantic variant retrieves memory by textual similarity without the full governance pipeline.
  • B.3 KnowledgeBank Component Ablation: 4.2 points: removing governance reduces Terminal-Bench SR on average and increases AS on both benchmarks.The same removal decreases SWE-Bench resolve rate by 4.4 points.
  • B.3 KnowledgeBank Component Ablation: 4.4 Terminal-Bench SR points: Sem. Ret. falls relative to the full system, while SWE-Bench resolve rate drops by 5.0 points.The results support governance and high-value memory processing as central design choices.

C Additional Robot Evaluation Details

The additional evaluations describe simulation and real-world protocols, representative manipulation rollouts, and task characteristics that highlight the roles of 3D pose estimation, dynamic correction, and retrieved experience. Real-world gains are attributed to calibrated 3D perception, retrieval, and planning-time memory use rather than policy fine-tuning.

  • Simulation evaluation: Simulation uses a Franka Panda with a parallel gripper, CoppeliaSim, PyRep, four RGB-D cameras, and waypoint actions executed by a motion planner.The 14 RLBench tasks vary object categories, object poses, and task horizons; CAP generates programs invoking hand-crafted primitives, while VoxPoser predicts waypoints through 3D voxel value maps.
  • Simulation evaluation: The remaining difficult simulation tasks mainly involve non-prehensile or fine-grained manipulation requiring precise 3D pose estimation or dynamic execution correction.VoxPoser is limited when arm movement exceeds 4-DoF, whereas CAP is constrained by its hand-written primitive set.
  • Simulation evaluation: The gap between GeneralVLA-2 and GeneralVLA-2 without KnowledgeBank indicates that retrieved manipulation experience supplies useful task-level constraints beyond current observations.
  • Simulation evaluation: Figure 7 visualizes button pressing, cup manipulation, and lamp operation through temporally ordered frames from the same simulation episodes.
  • Real-world evaluation: In real-world experiments, the agent explores each task ten times before testing and stores reusable experience in KnowledgeBank.Model parameters remain unchanged, so improvement comes from calibrated 3D perception, retrieval, and planning-time memory use rather than policy fine-tuning.

D Additional Limitations and Failure Modes

GeoFuse-MV3D is constrained by consistent multi-view masks and accurate camera poses, while governed KnowledgeBank performance depends on verifier reliability and task coverage. Current real-world evaluation is limited to tabletop manipulation with static objects and short horizons.

  • GeoFuse-MV3D limitations: GeoFuse-MV3D can become unreliable when masks leak, object regions are missing, or camera calibration drifts.Its projection-based soft visual-hull support checks assume consistent target-object masks and sufficiently accurate camera poses.
  • KnowledgeBank limitations: KnowledgeBank may promote weak memories or suppress useful ones when verifier judgments are incorrect.Rare failure modes can also remain underrepresented until the robot observes them.
  • Evaluation scope: The real-world evaluation covers tabletop manipulation with static objects and short horizons, leaving broader settings for future testing.Suggested extensions include longer mobile-manipulation tasks, stronger occlusions, deformable objects, and interactive recovery.
Loading 2606.17480v1…