PaperlayerView exact revision on arXiv

Source-linked AI summary

Does Continual Imitation Learning Remain Grounded? A Language-Perturbed Benchmark for Robotic Task Retention

Siddeshwar Raghavan, Ziqin Yuan, Fengqing Zhu, Byung-Cheol Min

arXiv:2610.00542v1cs.RO

TL;DR

Continual-learning task performance does not establish whether robotic behavior remains guided by language, since policies may rely on scene cues or learned associations. This paper evaluates instruction variants across continual-learning checkpoints with goal-based and behavioral diagnostics. Its results show that task retention alone does not guarantee semantic invariance, reliable goal switching, or appropriate responses to scene-incompatible instructions.

  • Problem

    Task-retention metrics do not establish whether a policy’s retained behavior remains appropriately guided by language, as it may rely on scene layout or object associations.

    Source: §I · p. 1

  • Method

    The protocol evaluates instruction variants across successive learning stages, using success objectives for valid goals and behavioral statistics for incompatible requests.

    Source: §I · p. 2, A. Problem Setting · p. 2

  • Results

    Across LIBERO, task retention did not guarantee semantic invariance, reliable goal switching, or appropriate behavior under scene-incompatible instructions.

    Source: §VII · p. 8

  • Takeaways & Limitations

    The diagnostics expose complementary language-grounding limitations that task success alone cannot identify.

    Source: §VII · p. 8

    The study uses a common behavior-cloning architecture, does not evaluate large-scale VLAs, and tests minimal contrasts and collisions only at the final checkpoint.

    Source: §VII · p. 8

Abstract

from arXiv · show

Continual imitation learning evaluates whether a robot can learn new knowledge without forgetting previously learned skills. However, retaining task performance does not ensure the behavior remains grounded in language because policies may rely on scene cues, object associations, or memorized task structure. We introduce a benchmark protocol to study how language-guided behavior changes as robotic policies learn successive tasks. We construct meaning-preserving and meaning-changing instruction variants for the Goal, Spatial, Object, and Long suites of LIBERO. Policy experiments focus on LIBERO-Goal, evaluating Original and Paraphrase instructions after each continual-learning stage. We compare representative continual imitation learning methods under their original assumptions while separating task competence from language sensitivity. The proposed diagnostics complement standard learning and forgetting metrics by measuring semantic robustness, goal adaptation, and language sensitivity. Results show that strong continual-learning performance does not always translate to reliable language grounding, and our diagnostics help determine whether retained skills remain correctly guided by their instructions. Additional materials are available at https://sites.google.com/view/stillgrounded

I. INTRODUCTION

Conventional continual-learning metrics measure task acquisition and retention, but do not establish whether retained behavior remains guided by language. The paper introduces a temporal protocol that tests language sensitivity across learning checkpoints using instruction variants and complementary goal-based or behavioral diagnostics.

I. INTRODUCTION

  • Task success and retention metrics do not establish whether a policy’s retained behavior remains appropriately guided by language.Policies may instead rely on scene layout, object associations, or memorized task routing.Source: §I · p. 1
  • A successful paraphrase does not by itself show language use, and a changed behavior does not show that a modified goal was completed.The changed goal may be infeasible in the current scene, so incompatible requests have no valid success target.Source: §I · p. 1
  • The protocol tests original instructions, meaning-preserving paraphrases, valid minimal goal changes, and scene-incompatible requests across continual-learning checkpoints.Goal-based metrics apply when an instruction defines a valid goal; incompatible requests are assessed through behavioral responses.Source: §I · pp. 1–2
  • The study evaluates representative continual imitation-learning methods on LIBERO-Goal without changing their algorithmic designs.The protocol’s instruction construction and evaluation are designed to extend beyond LIBERO’s scene and goal definitions.Source: §I · p. 2

II. RELATED WORK

Prior work supplies continual-learning methods and language-robustness benchmarks, while this paper examines whether retained robotic behaviors remain dependent on language. Its problem setting evaluates checkpoints across observed tasks and instruction conditions, using behavioral outcomes when an instruction has no valid goal.

A. Continual robotic imitation learning

  • Continual robotic imitation work spans replay, regularization, parameter isolation, and architectural expansion; newer approaches also use pretrained models, adapters, retrievable skills, or routed expert libraries.The paper evaluates implementations of TAIL, L2M, IsCiL, and DMPEL in a common robotic imitation framework rather than proposing a new retention mechanism.Source: A. Continual robotic imitation learning · p. 2

B. Language robustness and diagnostic evaluation

  • Language-robustness benchmarks test variations in objects, initial states, instructions, environments, and targeted language changes that standard task success may miss.LIBERO-PRO, LIBERO-Plus, and Contrast Sets are cited as examples of this diagnostic work.Source: B. Language robustness and diagnostic… · p. 2

A. Problem Setting

  • At each continual-learning stage, the protocol evaluates the checkpoint on observed tasks and instruction conditions; incompatible instructions are represented by behavioral outcomes rather than goal success.The benchmark does not constrain the retention mechanism, which may include replay, stored parameters, task-specific masks, adapters, skills, or routed experts.Source: A. Problem Setting · p. 2

B. Dataset-Agnostic Task Interface

The protocol represents each task through a dataset adapter and constructs controlled instruction variants using language generation followed by dataset-side validation. Its checks constrain which goal changes are retained, but scene compatibility does not guarantee physical feasibility.

B. Dataset-Agnostic Task Interface

  • A dataset adapter exposes each task’s canonical instruction, scene entities, supported predicates, original goal, and rollout-success evaluator; its metadata is not passed to the policy.For LIBERO, these fields come from BDDL task definitions; other datasets can implement the interface using task metadata and success annotations.Source: B. Dataset-Agnostic Task Interface · p. 3

C. Controlled Instruction Construction

  • LLM-proposed instruction variants undergo deterministic dataset-side validation, and minimal contrasts are retained only when their entities, predicates, and goal registration are supported.The semantic slots are action, manipulated object, relation, target, and attribute.Source: C. Controlled Instruction Construction · p. 3
  • Paraphrases preserve every semantic slot, are evaluated against the original goal, and undergo redundancy checks and an independent semantic audit.The generation process proposes five paraphrases per task.Source: C. Controlled Instruction Construction · p. 3
  • Scene-compatible modified goals may still be physically infeasible, so failure can reflect either insufficient language grounding or limited task execution capability.Incompatible collisions are sensitivity probes because they have no correct alternate rollout.Source: C. Controlled Instruction Construction · pp. 3–4
  • LIBERO-Goal retains 34 minimal contrasts—10 action, 8 object, 8 target, and 8 relation variants—but no attribute variant meets the construction and validation criteria.Task-slot pairs without an honest one-slot alternative are reported as unavailable rather than replaced artificially.Source: C. Controlled Instruction Construction · pp. 3–4

D. Continual Grounding Metrics

The grounding metrics separate original-task competence and paraphrase invariance from goal adaptation, collision-conditioned behavior, and action-level language sensitivity. They pair task outcomes with diagnostics that distinguish changing actions from achieving the requested goal.

D. Continual Grounding Metrics

  • Original-condition reporting includes NBT, AUC, and final success, while task-balanced aggregation averages evaluated variants equally within each task and condition.Metrics are computed independently per seed and reported as the mean and standard deviation across seeds.Source: D. Continual Grounding Metrics · p. 4
  • The signed paraphrase gap compares original-instruction and paraphrase success; values near zero indicate similar success, while negative values indicate higher paraphrase success.The signed gap should be read alongside absolute success rates.Source: D. Continual Grounding Metrics · p. 4
  • Goal-Switch Accuracy measures whether executable minimal contrasts reach the modified goal, not merely whether the policy changes its actions.This separates goal adaptation from action change alone.Source: D. Continual Grounding Metrics · p. 4
  • For incompatible collisions, Original-Goal Persistence measures continued achievement of the original goal, while achieved-goal statistics distinguish original, other audited, multiple, or unrecognized goals.The achieved-goal distribution is reported separately from action initiation, which can overlap with goal outcomes.Source: D. Continual Grounding Metrics · p. 4
  • Action Divergence compares mean actions for the same observation under original and perturbed instructions, while the expert-action language margin tests whether the expert action is more likely under the correct instruction.Action-level metrics are reported alongside task outcomes because action changes alone do not establish correct grounding.Source: D. Continual Grounding Metrics · p. 4

A. Experimental Setup

The experiments use LIBERO’s sequential task suites and compare representative continual imitation learning methods under shared policy and optimization settings. Frozen checkpoints are evaluated with a fixed rollout budget, and the LIBERO-Goal table reports semantic-invariance metrics.

A. Experimental Setup

  • LIBERO contains Goal, Spatial, Object, and Long suites with ten sequential tasks each; main evaluations use LIBERO-Goal for its varied goals involving actions, objects, targets, and spatial relations.At task t, policies are evaluated on all previously observed tasks j ≤ t.Source: A. Experimental Setup · p. 4
  • Eight representative continual imitation learning methods are compared, preserving each method’s native routing mechanism and reporting any use of task identity.The methods are SeqFT, ER, EWC, PackNet, DMPEL, IsCiL, L2M, and TAIL.Source: A. Experimental Setup · pp. 4–5
  • The LIBERO-Goal semantic-invariance table lists Original AUC, Paraphrase AUC, and signed paraphrase gap as its metrics.The caption identifies the table’s scope but provides no result values in the supplied passage.Source: Table I · p. 5
  • All methods share a pretrained policy initialization, behavior-cloning architecture, and optimization configuration, with method-specific hyperparameters following public implementations.Training runs for 10 epochs per task; experiments use three random seeds.Source: A. Experimental Setup · p. 5
  • Frozen checkpoints are evaluated separately from training, and the full instruction profile uses 20 simulator initial states per evaluated variant.Paraphrases are evaluated at each continual-learning task; minimal contrasts and collisions are evaluated after the final task.Source: A. Experimental Setup · p. 5

A. Semantic Invariance Across Learning Stages

DMPEL leads on both original and paraphrase performance, but every method loses performance under paraphrasing. Continual-learning retention and language invariance are related but distinct, so paraphrase gaps must be read alongside absolute success.

  • DMPEL achieves the highest Original and Paraphrase AUC, at 78.51 and 71.24, respectively.Source: A. Semantic Invariance Across Learning… · p. 5
  • All methods degrade under paraphrasing, with AUC drops ranging from 1.61 points for L2M to 7.27 points for DMPEL.L2M’s smaller gap accompanies low overall performance, while DMPEL still has the best Paraphrase AUC; interpret gaps alongside absolute performance.Source: A. Semantic Invariance Across Learning… · p. 5
  • DMPEL has the strongest Original Final SR and zero NBT yet retains a nonzero paraphrase gap, showing that task retention does not ensure semantic invariance.Source: A. Semantic Invariance Across Learning… · p. 5
  • Replay methods retain broader language-task associations, while sequential fine-tuning is more susceptible to representation drift; isolation and regularization do not explicitly align equivalent instructions.High Paraphrase AUC together with a small signed gap is treated as stronger evidence of semantic invariance than the gap alone.Source: A. Semantic Invariance Across Learning… · p. 5

B. Response to Minimal Semantic Changes

Controlled semantic changes expose limits to goal switching: even strong methods can continue the original task, and sensitivity varies by edited instruction slot. Under scene-incompatible requests, policies commonly act or fall back to learned goals, so action-level sensitivity must be distinguished from behavioral switching.

B. Response to Minimal Semantic Changes

  • DMPEL leads on final-stage Target SR (30.91) and GSA (21.68), but its Original Persistence remains 44.72.Target SR can count rollouts that reach the modified goal without abandoning the original; GSA additionally requires that the original goal is not achieved.Source: B. Response to Minimal Semantic Changes · p. 6
  • Target edits yield the highest average GSA (∼16.78%), while object, relation, and action edits remain around 8 to 9%.No attribute results are available because the benchmark contains no retained attribute minimal contrasts.Source: B. Response to Minimal Semantic Changes · p. 6

C. Behavior Under Incompatible Instructions

  • With scene-incompatible instructions, Action-Initiation Rate remains 83 to 92% across methods, and DMPEL reaches the original goal in 50.80% of rollouts.DMPEL has the highest CGAR at 64.90%; action-level sensitivity does not always produce a behavioral switch, and these outcomes are not collision success rates because no valid target exists.Source: Figure 4 · p. 6, C. Behavior Under Incompatible… · p. 6

D. Instruction-Manifest Coverage

Final-checkpoint and across-stage evaluations can give different accounts of paraphrase sensitivity. The instruction manifest retains only variants that pass validation, producing unequal coverage across conditions.

D. Instruction-Manifest Coverage

  • Across four LIBERO suites, 460 instruction variants are retained, including all 40 originals and 200 paraphrases; 180 generated candidates are excluded.The retained set includes 130 of 200 minimal contrasts and 90 of 200 scene-incompatible collisions.Source: D. Instruction-Manifest Coverage · p. 7
  • Minimal contrasts and collisions are excluded when they introduce unsupported objects, relations, or goal objectives, leaving different eligible subsets across tasks and semantic slots.Source: D. Instruction-Manifest Coverage · p. 7

A. Effect of Temporal Evaluation

B. Complementarity of Collision Diagnostics

Action-level measures do not reliably track achieved behavior, while collision action initiation tracks canonical-goal activation without establishing correct instruction following. The conclusion presents these diagnostics as complementary to task success, with explicit scope limitations.

B. Complementarity of Collision Diagnostics

  • AIR and CGAR are strongly associated (ρ = 0.80), but this does not establish correct instruction following because collision instructions have no valid scene goal.The reported association is across method-task pairs.Source: B. Complementarity of Collision… · p. 7
  • AD and LM correlate only weakly with OGP (ρ = 0.26 and 0.21), so action-distribution changes do not reliably predict changes in achieved behavior.The paper recommends reporting both action- and outcome-level diagnostics because they capture complementary policy responses.Source: Table III · p. 7, B. Complementarity of Collision… · p. 7

VII. LIMITATIONS AND CONCLUSION

  • The paper introduces a temporal diagnostic protocol for assessing whether language grounding persists as continual policies acquire new tasks.The conclusion presents the protocol as a way to examine behavior beyond task success alone.Source: §VII · pp. 7–8
  • Across LIBERO, task retention did not guarantee semantic invariance, reliable goal switching, or appropriate behavior under scene-incompatible instructions.Paraphrase, minimal-contrast, and collision diagnostics expose complementary limitations that task success alone cannot identify.Source: §VII · p. 8
  • The study uses a common behavior-cloning architecture, excludes large-scale VLAs, and evaluates minimal-contrast and collision diagnostics only at the final checkpoint.Failures on modified goals may reflect limited task-execution capability rather than language interpretation; the diagnostics do not establish a causal mechanism for grounding failure.Source: §VII · p. 8
  • The protocol provides a more detailed view of language grounding than task success alone.The authors hope it supports more rigorous evaluation in continual robot learning.Source: §VII · p. 8

Related papers

Loading 2610.00542v1…