PaperlayerView exact revision on arXiv

Source-linked AI summary

RefGuard: Identity-Aware Language-Guided Robot Manipulation via Joint Target-Anchor-Frame Grounding

Lan Wei, Kangyi Lu, Yongchen Wang, Chenmeng Bi, Qi Chen, Hanlin Niu, Yip Fun Yeung, Dandan Zhang

arXiv:2609.06221v1cs.RO

TL;DR

In clutter, instructions may leave multiple plausible targets, anchors, or reference frames, making the intended physical object uncertain. RefGuard preserves joint uncertainty over those variables and delays commitment until action; its evaluations report fewer wrong-identity outcomes while retaining correct executions.

Abstract

from arXiv · show

Vision-language-action (VLA) models have substantially advanced language-guided robot manipulation, yet reliable execution still hinges on identifying which physical object an instruction refers to. In cluttered scenes containing repeated objects, ambiguous anchors, or frame-dependent spatial terms, a robot can execute a geometrically valid action on a semantically compatible but unintended instance; we call this failure an identity switch. The referent is jointly determined by three coupled latent variables: the target, the anchor, and the reference frame, so committing to any one of them before execution turns residual ambiguity into a silent and irreversible error. We propose RefGuard, an identity-aware grounding framework that delays commitment by maintaining a joint posterior over all three variables. RefGuard builds a frame-conditioned object-centric scene graph from RGB-D observations, separating frame-independent geometry from directional relations, and routes the posterior through a decision policy that executes, clarifies, reobserves, or aborts. On a real UF850 arm, RefGuard records no identity switch on any ambiguity-stress trial and executes correctly on 90.0% of them, whereas fine-tuned VLA and LLM (Large Language Model)-based baselines switch identity in 33-46% of the same trials, while retaining 93.3% success on unambiguous scenes and recovering from post-grounding scene changes in 86.7% of trials. On a 3200-episode procedural suite, it raises correct execution on solvable instructions from 56.6% to 80.5% over the ablation that commits to the anchor and frame before the target, while deferring less often (19.5% vs. 43.4%).

I. INTRODUCTION

Language-guided manipulation can produce a geometrically valid action on the wrong physical object when ambiguity over the target, anchor, and reference frame is resolved prematurely. RefGuard preserves their joint uncertainty and delays execution commitment until the grounding is reliable.

I. INTRODUCTION

  • An identity switch is an irreversible action on the wrong physical referent after a premature grounding commitment, despite a geometrically valid action.The referent depends jointly on target, anchor, and reference-frame assignments.Source: §I · p. 1
  • Existing VLA, LLM-driven, and vision-language systems often commit to a single target, anchor, and frame before planning, hiding residual ambiguity.Tracking can preserve assigned identities but does not determine which identity the instruction intended.Source: §I · p. 1

A. System Overview and Problem Definition

  • RefGuard uses a frame-conditioned scene graph and joint target–anchor–frame inference, then executes, clarifies, reobserves, or aborts according to the grounding evidence.The task contract includes a decision and reason code; when execution is not allowed, the target remains uninstantiated.Source: Figure 2 · p. 2, §I · p. 2, A. System Overview and Problem… · pp. 2–3

I. INTRODUCTION

  • 81 of 90 ambiguity-stress trials were executed correctly without a single identity switch, and correct execution rose from 56.6% to 80.5% on solvable procedural-suite episodes.The procedural comparison is against a matched ablation that commits to the anchor and frame before the target.Source: §I · p. 2

A. System Overview and Problem Definition

  • A grounding is unsafe when plausible assignments imply different executable goals; if they share the same physical goal, they remain action-equivalent.When no unique safe goal is determined, RefGuard does not execute an arbitrary top-ranked interpretation.Source: A. System Overview and Problem… · p. 3

B. Language Parsing into Primitive Queries

The language parser converts instructions into symbolic primitives without binding phrases to object instances. RefGuard then evaluates candidate instances using a scene graph that distinguishes frame-independent geometry from frame-dependent directions and combines object, relation, and action evidence.

B. Language Parsing into Primitive Queries

  • GPT-4o decomposes instructions into ordered symbolic primitive queries but does not bind phrases to detected instances or choose a target, anchor, or frame.Instance-level assignments are left to the grounding module.Source: B. Language Parsing into Primitive… · p. 3
  • Each primitive can represent an action, target phrase, multiple anchor–relation pairs, frame cues, and action constraints.A frame cue records the suggested frame without resolving it; an underspecified “left” leaves candidate frames active for grounding.Source: B. Language Parsing into Primitive… · p. 3

C. Object-Centric Scene Graph and Candidate Evidence

  • Scene-graph nodes combine instance geometry, semantics, affordances, and uncertainty, while the graph is rebuilt and identity re-inferred after each observation.The local node identities allow recovery when objects move or are swapped before execution.Source: C. Object-Centric Scene Graph and… · p. 3
  • The graph separates frame-independent geometry for relations such as near or inside from frame-conditioned directional offsets such as left-of or in-front-of.Directional offsets are computed only for frames relevant to the current query, without committing to one frame in advance.Source: C. Object-Centric Scene Graph and… · p. 3
  • Candidate interpretations combine soft object and relation evidence with action feasibility, while missing attributes are omitted rather than penalized.Weak or near-tie evidence remains uncertain for later clarification or reobservation, and hard feasibility violations are pruned before normalization.Source: C. Object-Centric Scene Graph and… · p. 3

D. Joint Hypothesis Table for Target–Anchor–Frame Grounding

RefGuard represents each candidate grounding as a complete target–anchor–frame interpretation and scores it using language, spatial, frame-prior, and action-feasibility evidence. It aggregates row weights into target beliefs and diagnoses which grounding variable remains uncertain.

D. Joint Hypothesis Table for Target–Anchor–Frame Grounding

  • Each hypothesis-table row represents a complete target, anchor assignments, and reference frame, with a score and normalized weight.An explicit frame cue concentrates the prior on the requested frame; an unspecified cue spreads it across available frames.Source: D. Joint Hypothesis Table for… · p. 3
  • Rows combine target, anchor, frame-prior, spatial-relation, and action-feasibility evidence, and invalid assignments are rejected before normalization.This avoids committing to an anchor or frame before searching for the target.Source: D. Joint Hypothesis Table for… · p. 4
  • RefGuard sums row weights by target to form target beliefs, then uses disagreement among top hypotheses to identify target, anchor, or frame ambiguity.Weak visibility, detector confidence, or stale poses can instead indicate weak perception or a stale scene, guiding the execute, clarify, reobserve, or abort decision.Source: D. Joint Hypothesis Table for… · p. 4

E. Posterior-to-Decision Policy

RefGuard uses posterior concentration and competing grounding hypotheses to choose whether to execute, clarify, reobserve, or abort. Posterior-aware planning, pre-execution checks, and scene updates preserve the grounding through action.

E. Posterior-to-Decision Policy

  • The policy uses target-marginal concentration and decision thresholds to route grounding to EXECUTE, CLARIFY, REOBSERVE, or ABORT.Its concentration score supports decisions but is not treated as a formal calibration metric.Source: E. Posterior-to-Decision Policy · p. 4
  • After clarification, RefGuard adds the user's reply as a constraint, removes inconsistent hypotheses, and re-grounds the same primitive.Execution resumes if the updated decision is EXECUTE.Source: E. Posterior-to-Decision Policy · p. 4

F. Posterior-Aware Planning and Verification

  • During planning, RefGuard retains high-weight anchor and frame hypotheses, accepting poses only when feasible and consistent with those hypotheses.Conflicting goals detected during pose checks route the query back to CLARIFY or REOBSERVE.Source: F. Posterior-Aware Planning and… · p. 4
  • Before motion, a contract check verifies that targets and anchors remain visible, poses and spatial constraints still match, and the action remains feasible.A failed check routes to REOBSERVE, CLARIFY, or ABORT without moving the robot.Source: F. Posterior-Aware Planning and… · p. 4
  • After each primitive, RefGuard rebuilds the scene graph and re-grounds the next primitive rather than permanently locking the initial target–anchor–frame binding.Memory-backed nodes support temporarily occluded objects with reduced confidence and are overridden when objects are re-detected.Source: F. Posterior-Aware Planning and… · pp. 4–5

III. EXPERIMENTS

The experiments evaluate identity-aware grounding on a real UF850 tabletop robot across six settings, using adapted VLA policies and an LLM-based baseline. RefGuard's outcome decomposition on solvable ambiguity settings reports more correct executions and no identity switches than the baselines.

III. EXPERIMENTS

  • The evaluation separates ordinary manipulation from ambiguity through six settings, progressing from clean scenes to duplicate targets, joint target–anchor ambiguity, frame cues, perturbations, and hard negatives.The suite includes irreducibly ambiguous instructions for which correct behavior is to defer.Source: §III · p. 5, B. Metrics and Identity-Switch Stress… · pp. 5–6

A. Real-Robot Setup and Baselines

  • The real-robot platform is a UF850 tabletop kitchen scene observed by two calibrated Intel RealSense D435i RGB-D cameras, with each episode starting from a reset layout and instruction.The cameras are an external third-person view and a wrist-mounted view.Source: A. Real-Robot Setup and Baselines · p. 5
  • The baselines are platform-adapted GR00T and π0.5 VLA policies fine-tuned on the same demonstration dataset, plus an LLM-based pipeline that makes one hard target assignment and executes without re-observation.The LLM-based pipeline uses RefGuard's perception and motion planner.Source: A. Real-Robot Setup and Baselines · p. 5

B. Metrics and Identity-Switch Stress Suite

  • The six settings use task-level metrics: clean-scene task success, execution-conditional identity-switch rate for S2–S4, post-perturbation recovery for S5, and correct defer rate for S6.The identity-switch denominator is executed trials, C + I, while deferral is counted separately.Source: B. Metrics and Identity-Switch Stress… · pp. 5–6, Table I · p. 6
  • On S2–S4 (N=90), RefGuard records 81/0/3/6 correct/identity-switch/low-level-failure/defer outcomes, versus GR00T 45/30/15/0, π0.5 42/36/12/0, and LLM-based 45/41/4/0.The figure reports episode-level outcome fractions without macro-averaging.Source: Figure 4 · p. 6

C. Main Results

RefGuard preserves joint target–anchor–frame uncertainty to reduce wrong-instance execution, with matched controls indicating that the benefit is not explained by abstention or scene updates alone.

C. Main Results

  • Across duplicate-target, duplicate-anchor, and frame-conditioned settings, RefGuard had no observed identity switches, while baselines often executed on the wrong physical instance.The duplicate-anchor setting requires resolving the anchor jointly with the target; the separate duplicate-container variant likewise uses a secondary spatial relation to select the intended receptacle.Source: C. Main Results · p. 6
  • RefGuard retains more correct executions while eliminating observed wrong-ID outcomes, rather than replacing executions with deferrals.This is the overall comparison across solvable ambiguity settings S2–S4.Source: C. Main Results · p. 6

D. Ablation and Shared-Update Controls

  • In the sequential episode, the selected pepper is retained as an in-gripper memory node while the scene graph is rebuilt before grounding PLACE.Only one joint target–anchor assignment satisfies “near” in the depicted scene.Source: Figure 5 · p. 7
  • Matched controls show that joint target–anchor–frame grounding yields higher correct execution and removes early-commitment identity switches, with the largest gap under duplicate-anchor ambiguity.Hard-A/F shares RefGuard’s perception, decision interface, planner, and controller, but commits to the anchor and frame before inferring the target.Source: D. Ablation and Shared-Update Controls · pp. 6–7
  • Giving controls the same post-change observation and graph update improves recovery, but identity switches remain; RefGuard has the strongest recovery without observed switches.The comparison indicates that scene updating alone is insufficient when re-binding after a change.Source: D. Ablation and Shared-Update Controls · p. 7

E. Sequential Composition and Generalisation

RefGuard re-grounds each primitive as scenes change, maintaining identity through sequential tasks and scaling to a procedural clutter evaluation. The paper concludes that preserving referential uncertainty through execution is central to reducing wrong-instance actions within the tested settings.

E. Sequential Composition and Generalisation

  • After each primitive, RefGuard reobserves the scene, updates held and placed object states, and re-grounds the next primitive on the updated graph.In the illustrated episode, the grasped pepper becomes an in-gripper memory node, and PLACE uses the rebuilt graph rather than the initial binding.Source: E. Sequential Composition and… · p. 7
  • RefGuard maintains high primitive completion as sequence length increases while producing no identity-switch executions, including on two held-out object categories.Free-form paraphrase evaluation also shows generalisation beyond instruction templates; post-action checkpoints favor dynamic re-grounding over locked binding.Source: E. Sequential Composition and… · p. 7, Table IV · p. 8

F. Procedural LIBERO-Style Evaluation at Scale

  • Across procedural clutter levels, RefGuard achieves higher correct execution and reduces unnecessary deferral on solvable episodes; it safely defers on all hard negatives.The evaluation uses 2,400 solvable and 800 hard-negative episodes; it tests controlled ambiguity and clutter rather than unrestricted open-world manipulation.Source: F. Procedural LIBERO-Style Evaluation… · p. 7, Table IV · p. 8

IV. CONCLUSION

  • The conclusion attributes reduced wrong-ID execution to preserving joint uncertainty through grounding, verification, and execution, with commitment only when the intended instance is sufficiently determined.The controlled comparisons distinguish joint delayed commitment from abstention or scene-update access alone.Source: §IV · p. 8

APPENDIX

The UF850 tabletop set includes duplicate and singleton produce categories alongside common tableware, arranged to test referent ambiguity.

APPENDIX

  • The UF850 set has 15 produce instances across 10 categories, including five duplicate pairs and five singleton categories, plus common tableware.Objects can serve as targets or anchors, and layouts use duplicates or visually similar instances to test relational and frame-dependent cues.Source: A. UF850 Setup · p. 9

B. Baseline Implementation Details

The baseline implementations comprise fine-tuned VLA policies, a one-shot LLM pipeline, and matched modular controls that vary grounding commitment or post-change updating.

B. Baseline Implementation Details

  • The π0.5 baseline is fine-tuned on 300 teleoperated UF850 demonstrations for 20,000 steps, using LoRA modules in the PaliGemma backbone and action expert.Training uses four A100 80GB GPUs, a global batch size of 64, and a 5 × 10^-5 learning rate after 1,000 warm-up steps.Source: B. Baseline Implementation Details · p. 9
  • At inference, π0.5 receives two RGB views, the seven-dimensional robot state, and the instruction, then executes actions in a repeated receding-horizon loop.Commands use UF850 servo control, with the gripper command clipped to [0, 1] before mapping to the physical range.Source: B. Baseline Implementation Details · p. 9
  • GR00T-N1.7-3B is fine-tuned on the same UF850 dataset for 20,000 steps, using four A100 GPUs and a global batch size of 128.Its learning rate is 1 × 10^-4, and deployment averages valid predicted action chunks using temporal action ensembling.Source: B. Baseline Implementation Details · p. 9
  • The LLM pipeline commits once to a hard target and any required anchor, then executes without re-observation or an explicit defer/abort option.Hard-A/F instead shares RefGuard’s perception and action stack but fixes the anchor and frame before target inference; update controls receive the same post-change observation and graph update.Source: B. Baseline Implementation Details · p. 9

C. Sequential and Generalisation Evaluation

The evaluation tests sequential re-grounding, held-out object categories, paraphrase parsing, and dynamic re-grounding against a locked-binding control.

Sequential composition

  • At lengths L = 2, 4, and 6, sequential composition uses 10 episodes each and totals 120 primitives, with fresh observation and re-grounding at every primitive boundary.Each boundary also updates object state and the scene graph.Source: C. Sequential and Generalisation… · p. 9

Object generalisation

Paraphrases

Post-action checkpoints

D. Instruction Templates and Scene Layouts

The six evaluation settings range from unambiguous sanity checks to duplicate-object, frame-conditioned, perturbed-scene, and hard-negative cases.

S1: Unambiguous sanity check

S2: Duplicate-target grounding

S3: Joint target-anchor grounding

  • S3 includes duplicate targets and anchors that cannot be identified in isolation; one joint assignment satisfies the RGB-D next-to relation, while near-tied assignments trigger CLARIFY.A duplicate-container PLACE variant uses a secondary relation to uniquely identify the receptacle.Source: D. Instruction Templates and Scene… · p. 10

S4: Frame-conditioned grounding

  • S4 tests directional relations with explicit frame cues and duplicate targets, including cases where robot and external-camera frames imply different target instances.Ignoring the frame cue or fixing the anchor before evaluating the frame-conditioned relation leads to the wrong instance.Source: D. Instruction Templates and Scene… · p. 10

S5: Post-grounding perturbation

S6: Hard negatives

  • S6 uses hard negatives such as duplicate identical targets, equally valid targets, unspecified frames, or missing targets or anchors, for which no unique safe physical goal is specified.The expected response is to clarify, reobserve, or abort rather than execute an arbitrary valid action.Source: D. Instruction Templates and Scene… · p. 10
Loading 2609.06221v1…