PaperlayerView exact revision on arXiv

Source-linked AI summary

RoboIRGBench: Benchmarking Implicit Referential Grounding in Vision-Language-Action Models

Aernaer Akelijiang, Jiannan Li, Zhineng Chen, Jingjing Chen, Bin Zhu

arXiv:2609.34384v1cs.ROcs.CV

TL;DR

Existing robotic manipulation benchmarks typically state task-relevant information explicitly, although real instructions often leave objects, quantities, and relations implicit. RoboIRG-Bench evaluates this grounding across four challenge types, and the results show a robustness gap between explicit and implicit instructions that persists in real-world manipulation.

  • Problem

    Existing benchmarks typically specify task-relevant information explicitly, leaving implicit references from linguistic or perceptual context under-evaluated.

    Source: §I · p. 2, A. Robotic Manipulation Benchmarks · p. 3

  • Method

    RoboIRG-Bench reformulates tasks from RoboMME into 40 variants across direct, reasoning-mediated, spatial, and contextual grounding challenges.

    Source: §I · p. 2

  • Results

    Strong explicit-instruction performance does not reliably transfer to implicit references, and the robustness gap persists in physical manipulation.

    Source: §I · p. 2, §V · p. 8

  • Takeaways & Limitations

    IRG is a distinct, underexplored capability involving interpretation and grounding of contextual information beyond retaining or retrieving memory.

    Source: §V · p. 8

Abstract

from arXiv · show

Vision-Language-Action (VLA) models have shown strong capabilities in robotic manipulation, yet existing benchmarks typically assume that task-relevant information is explicitly specified in the instruction. In practice, however, humans frequently refer to objects, quantities, and relations implicitly, requiring robots to recover the intended target from linguistic and perceptual context. We study this capability as Implicit Referential Grounding (IRG) and introduce RoboIRG-Bench, a manipulation benchmark designed to systematically evaluate it. Built upon RoboMME, RoboIRG-Bench contains 40 variants derived from 11 tasks and covers four challenges, including direct, reasoning-mediated, spatial, and contextual referential grounding. As IRG often requires retaining and retrieving previously established context, we evaluate representative VLAs spanning different memory mechanisms. Our evaluation reveals a noticeable referential robustness gap. Models that perform well under explicit instructions can degrade sharply when the same task-relevant information must be recovered from context. Reasoning-mediated and spatial references are particularly challenging, while models using external VLMs show greater robustness but still exhibit significant failures. Moreover, replacing the external VLM with a stronger model does not eliminate these gaps. We further validate these findings on a Franka Research 3 robot arm, where the gap persists under real-world manipulation and manifests as both incorrect referent grounding and downstream execution failures. These results establish IRG as a distinct and underexplored capability for reliable robotic instruction following and highlight the need for VLAs that can robustly integrate language, perception, reasoning, and action.

I. INTRODUCTION

The paper frames Implicit Referential Grounding (IRG) as resolving task-relevant information from linguistic or perceptual context, a capability that existing benchmarks often leave untested. RoboIRG-Bench evaluates this gap across four challenges, and the reported results show that explicit-instruction performance does not reliably transfer to implicit references.

I. INTRODUCTION

  • Existing benchmarks often specify task-relevant information explicitly, while natural interaction relies on linguistic and perceptual context to resolve objects, quantities, or relations.The paper distinguishes IRG from memory: memory makes context available, but IRG interprets and grounds it for action.Source: §I · p. 2
  • Models strong under explicit instructions can degrade sharply on implicit formulations, and stronger external VLMs do not eliminate the remaining failures.The reported gap also persists in Franka robot experiments, with failures involving incorrect grounding and downstream execution.Source: §I · p. 2
  • The benchmark frames IRG as an underexplored VLA evaluation dimension: recovering task-relevant information from context rather than explicit target specification.Table I compares IRG coverage across existing robotic manipulation benchmarks.Source: §I · p. 2, Table I · p. 3
  • RoboIRG-Bench evaluates implicit referential grounding through 40 task variants spanning direct, reasoning-mediated, spatial, and contextual challenges.The benchmark is built upon RoboMME and derives its variants from 11 manipulation tasks.Source: §I · p. 2

A. Robotic Manipulation Benchmarks

Existing manipulation benchmarks generally specify task-relevant entities explicitly, and prior implicit-reference evaluation does not capture diverse scenarios in direct manipulation. Human dialogue frequently uses reduced references, motivating evaluation beyond sequential action pronouns.

A. Robotic Manipulation Benchmarks

  • Existing manipulation benchmarks generally name task-relevant entities rather than testing how context is interpreted and grounded for control.Pronouns in some benchmarks mainly occur in sequential action chains, not in references that must be resolved from linguistic or perceptual context.Source: A. Robotic Manipulation Benchmarks · p. 3

B. Implicit Referential Grounding

A. Benchmark Overview

RoboIRG-Bench reformulates task-defining information as implicit references while preserving the manipulation objective. Its four dimensions probe how robots recover entities, quantities, and relations from context or the observed scene.

A. Benchmark Overview

  • RoboIRG-Bench evaluates implicit referential grounding in closed-loop manipulation through 40 variants derived from 11 RoboMME tasks.The selected tasks come from RoboMME's Counting, Permanence, and Reference suites.Source: A. Benchmark Overview · p. 3
  • The benchmark excludes five RoboMME tasks whose objectives are determined primarily by the visual scene and lack task-defining variables suitable for implicit reformulation.Included variants are evaluated at Easy, Medium, and Hard levels using rule-based instruction templates.Source: A. Benchmark Overview · p. 3
  • Its design preserves the manipulation objective while changing how task-relevant entities, quantities, or relations must be recovered from preceding context or the scene.The four dimensions are direct, reasoning-mediated, spatial, and contextual grounding; memory availability and correct interpretation for action are treated as distinct.Source: A. Benchmark Overview · p. 3

B. Implicit Referential Grounding Challenges

The benchmark separates implicit grounding into direct, reasoning-mediated, spatial, and contextual challenges. The settings vary whether the referent is recovered from prior context, additional reasoning, spatial cues, or intervening dialogue.

B. Implicit Referential Grounding Challenges

  • Direct grounding tests whether the model binds a final reduced reference to a target introduced earlier in the dialogue.Expressions include pronouns and nominal references such as “the former,” “the latter,” and “the first/second/third.”Source: B. Implicit Referential Grounding… · p. 3
  • Reasoning-mediated grounding derives the final target through color exclusion, number deduction, sequential relations, or contextual logic.These patterns require combining a referenced entity, quantity, or event with an additional relation.Source: B. Implicit Referential Grounding… · p. 4
  • Spatial grounding tests whether references such as “the cube there” can be linked to landmarks defined from object positions in the robot coordinate frame.It is applied to seven selected tasks; four tasks are excluded because their targets depend primarily on counts or temporal information.Source: B. Implicit Referential Grounding… · p. 4
  • Contextual grounding tests recovery of a referent across a five-stage sequence that inserts reinforcement and distractor information before the final implicit instruction.The sequence also includes referent introduction and an action cue.Source: B. Implicit Referential Grounding… · p. 4

IV. EXPERIMENT

The evaluation finds that strong explicit-instruction performance does not reliably transfer to implicit referential grounding, even for models with memory or external VLMs. It compares representative memory configurations and reports recurring grounding failures.

A. Experimental Setup

  • The evaluation covers symbolic, perceptual, and recurrent memory configurations, retrieval-based MemER, and vanilla π0.5; each configuration is measured over 150 episodes.The 150 episodes comprise 50 episodes under each of three random seeds, with success rate averaged across seeds.Source: A. Experimental Setup · p. 4

B. Main Results and Analysis

  • Reasoning-Mediated is consistently the hardest setting; memory-augmented models still degrade, while external-VLM subgoal systems generally retain more performance.Source: B. Main Results and Analysis · p. 4
  • FrameS-Modul leads Explicit at 45.82% but falls to 29.76% Direct and 16.85% Reasoning-Mediated; GroundSG-QwenVL leads three IRG challenges, while MemER leads Spatial at 26.29% versus 25.33%.Source: B. Main Results and Analysis · p. 4, Table II · p. 5
  • Figure 3 compares suite-averaged success for four implicit referential strategies with their explicit counterparts, showing overlap and annotated gains or drops.Source: Figure 3 · p. 5
  • FrameS-Modul fails on implicit repetition counts and target identities, while GroundSG has spatial-grounding and long-context tracking failures that produce erroneous subgoals.Source: B. Main Results and Analysis · pp. 4, 6, Figure 4 · p. 6

C. Isolating the Effect of Implicit Referential Grounding

Paired explicit counterparts help distinguish referential-grounding difficulty from manipulation difficulty. Making references explicit improves performance in many settings, but does not consistently resolve reasoning-mediated failures.

D. Does a Stronger External VLM Solve IRG?

Replacing GroundSG-QwenVL’s external VLM with Gemini 3.1 Pro improves average success on three representative IRG challenges. The explicit-to-implicit robustness gap nevertheless persists, especially for spatial grounding.

E. How Does Longer Context Affect IRG?

The contextual-grounding evaluation progressively inserts linguistic stages between referent introduction and the final instruction. Success shows an overall downward trend as more stages are added.

  • Success shows an overall downward trend as more contextual stages are introduced in Contextual Referential Grounding.Source: E. How Does Longer Context Affect IRG? · p. 7
  • The manipulation task remains unchanged while Referent Reinforcement, Distractor Context, and an Action Cue are inserted between the initial referent and final instruction.Source: E. How Does Longer Context Affect IRG? · p. 7
  • The accompanying real-world success-rate table evaluates Explicit instructions and four IRG challenges over 10 trials per setting.Source: Table V · p. 7

F. Real-World Experiments

Real-world trials show that strong explicit-instruction performance does not reliably transfer to implicit referential grounding. Failures include both choosing the wrong referent and failing during execution.

Tasks and IRG Construction

  • The study tested PutInBox and PickByCount across explicit and four implicit conditions, with policies trained only on explicit instructions.The tasks used color exclusion and left/right cues for grounding; spatial relations were defined in the robot base frame.Source: F. Real-World Experiments · p. 7

Experimental Protocol

  • Two policies were evaluated for 10 trials per instruction variant after fine-tuning on 50 demonstrations per task.π0.5 and FrameS-Modul were fine-tuned for 5k steps; the physical evaluation used a 7-DoF Franka Research 3 arm.Source: F. Real-World Experiments · p. 7

Result and Analysis

  • Both policies failed completely on PutInBox Reasoning-Mediated trials and achieved 10% and 20% success on PickByCount, respectively.Direct grounding retained 90% success on PutInBox for both policies; contextual and spatial grounding also degraded performance.Source: F. Real-World Experiments · p. 8
  • Spatial PutInBox trials had 69% wrong-target selections, unlike PickByCount, and some failures occurred after the intended referent was correctly grounded.The paper distinguishes grounding errors from downstream execution errors.Source: F. Real-World Experiments · p. 8, Figure 5 · p. 8

V. CONCLUSIONS

The paper introduces RoboIRG-Bench to evaluate implicit referential grounding across four challenge types. Its findings show that explicit-instruction performance does not reliably transfer to implicit references, including in physical manipulation.

V. CONCLUSIONS

  • RoboIRG-Bench evaluates direct, reasoning-mediated, spatial, and contextual grounding by converting explicit manipulation instructions into implicit settings while preserving task objectives.Source: §V · p. 8
  • Strong explicit-instruction performance does not reliably transfer to implicit references, and stronger external VLMs still leave significant failures.Retaining or retrieving relevant context also does not guarantee correct interpretation, grounding, and action.Source: §V · p. 8
  • Real-world results confirm that the gap persists in physical manipulation, with failures arising from incorrect grounding and downstream execution.The paper concludes that VLAs need tighter integration of contextual language understanding, visual grounding, reasoning, and control.Source: §V · p. 8
Loading 2609.34384v1…