PaperlayerView exact revision on arXiv

Source-linked AI summary

Flexible Language-Guided Representation Editing for Adaptive Robot Behavior

Audrey Lee, Nathaniel Dennler, Ilia Sucholutsky, Andreea Bobu

arXiv:2610.28893v1cs.RO

TL;DR

Demonstrations show robots how to behave but may not reveal which task features matter, while language can convey feature meaning. FLEX uses language models to refine feature representations through adding, generalizing, and masking; it learns desired behaviors across evaluated settings and transfers to realistic human feedback.

  • Problem

    Demonstrations can leave robots to guess which task features matter, and existing feature-editing methods support only a single edit type.

    Source: §1 · p. 2, §2 · p. 3

  • Method

    FLEX uses language models to refine task-feature representations from natural-language corrections through ADD, GENERALIZE, and MASK operations.

    Source: §1 · p. 2, §4 · pp. 4–5

  • Results

    FLEX learns desired robot behaviors across evaluated feature-misalignment scenarios and transfers effectively from simulated domains to realistic human feedback.

    Source: §6 · p. 9

  • Takeaways & Limitations

    FLEX’s three operations let language corrections refine feature sets by adding missing features, generalizing specific ones, and masking irrelevant ones.

    Source: §1 · p. 2, §4 · p. 4

    FLEX focuses on quasi-static rigid-object manipulation, relies on language-model common-sense reasoning, ignores correction content about reward weights, and was not evaluated in a closed-loop physical-robot user study.

    Source: §6 · p. 9

Abstract

from arXiv · show

Trajectory demonstrations show robots how to behave but not why, leaving them to guess which features of the task actually matter. Incorrect guesses produce misaligned internal representations and lead to incorrect robot behaviors. Because task features are often semantically defined, we argue that natural language is better suited for specifying features rather than relying on demonstrations. Although existing representation editing methods have proposed using language to specify new features, only allowing additive edits produces feature sets that are often over-specified or too narrowly scoped, especially in sequential editing tasks. We propose the Flexible Feature Editing Pipeline (FLEX), which uses language models to iteratively refine a robot's internal representation of task features from natural language corrections via three complementary operations: Add missing features, Generalize existing specific features, and Mask irrelevant features. We evaluate FLEX on 2D navigation scenarios, 7DoF Franka tabletop manipulation tasks, and a real-world study using naturalistic human language corrections and real-robot demonstrations. Comparisons against single-operation and unscaffolded baselines show that all three operations are necessary in interactive scenarios, and that FLEX achieves consistently higher normalized reward compared to other feature editing approaches while maintaining compact, transferable representations. Project available at https://mit-clear-lab.github.io/FLEX/

1 Introduction

Robots can misread which task features matter because instructions often leave them implicit, and demonstrations may not reveal their semantic meaning. FLEX uses language corrections to add, generalize, or mask features as needed.

1 Introduction

  • Human instructions often leave task-relevant features implicit, so robots may form misaligned representations and fail to behave as desired.Examples include avoiding sensitive electronics and staying close to a table to prevent spills.Source: §1 · p. 2
  • Demonstrations can be consistent with multiple feature interpretations, while language can express both what matters and why.The paper argues that language is better suited to feature specification and demonstrations to reward optimization.Source: §1 · p. 2
  • Feature representations can be incomplete, over-specified, or abstracted at the wrong level, while existing methods may fail to generalize or accumulate redundant features.FLEX responds with ADD, GENERALIZE, and MASK, illustrated by adding a laptop, generalizing to electronics, and masking a bowl feature.Source: §1 · p. 2

2 Related Work

Prior work addresses reward learning, feature specification, and language feedback, but feature-editing methods have generally supported only one kind of edit. FLEX unifies three edits to address different forms of representation misalignment.

2 Related Work

  • Reward-learning approaches use demonstrations, corrections, rankings, or other behaviors, but underspecified feedback can require impractical amounts of data.Some approaches combine feedback types but collect data and train rewards offline; FLEX focuses on iterative feedback at deployment time.Source: §2 · p. 2
  • Prior feature-editing methods support only adding, removing, or generalizing features individually, whereas FLEX unifies all three operations.The paper states that the three edit types are necessary to address the full range of representational misalignment types.Source: §2 · p. 3
  • Language-based robot-learning work often uses language to clarify goals or redirect behavior, rather than to specify which environmental features matter.The paper identifies language’s semantic content and language models’ shared-preference priors as useful for sparse feature-editing corrections.Source: §2 · p. 3

3 Problem Definition

The paper studies learning user-preferred robot behavior from demonstrations and language by jointly refining task features and reward weights. It argues that language corrections communicate feature meaning more directly than demonstrations do.

3 Problem Definition

  • The task is to learn reward functions reflecting user preferences and generate robot behavior from limited feedback, which includes demonstrations and language.The reward is modeled as a linear combination of state features, with preference weights expressing tradeoffs among features.Source: §3 · p. 3
  • Reward optimization estimates preference weights from demonstrations, using a maximum-entropy model that assigns higher probability to higher-reward trajectories while allowing suboptimal behavior.After inferring the weights, the robot selects the trajectory with the highest weighted feature reward.Source: §3 · p. 3
  • Jointly learning features and reward weights matters because assuming the relevant features are known before deployment can be unrealistic in evolving, unstructured environments.Earlier approaches separate feature specification under fixed weights from reward optimization under a fixed feature set.Source: §3 · pp. 3–4
  • A single demonstrated trajectory may fit many possible representations, making additional demonstrations labor-intensive and an inefficient way to convey representational information.Language corrections can communicate representational content directly; the paper assigns language to feature specification and demonstrations to reward optimization.Source: §3 · p. 4

4 Flexible Feature Editing Pipeline

FLEX uses language models to refine task-feature representations from natural-language corrections through three complementary edits. It assumes corrections identify relevant features rather than preference weights and applies the edits in a fixed order.

4 Flexible Feature Editing Pipeline

  • The pipeline assumes a correction arises after the user compares the robot’s current trajectory with an imagined desired trajectory, and uses it to infer features rather than preference weights.The model treats the correction as conditionally independent of the current and desired reward weights given the representations.Source: §4 · p. 4
  • FLEX decomposes language-based feature refinement into ADD, GENERALIZE, and MASK operations, which can be motivated simultaneously by one correction.Each operation maps the current feature set to a refined set using a task and environment description together with the correction.Source: §4 · pp. 4–5
  • ADD checks whether the current feature set captures the correction and proposes a new feature when it does not.The proposed feature is appended to the existing set.Source: §4 · p. 5
  • GENERALIZE replaces existing features with more general ones by extracting a higher-level intent and checking which features can be broadened to match it.For example, a laptop-distance feature may be generalized to cover other electronic devices; the implementation uses text-based object properties and LM-generated code.Source: §4 · p. 5
  • MASK removes features judged irrelevant to the correction, reducing the feature set to a subset of its current contents.The language model estimates a binary relevancy score for each feature.Source: §4 · p. 5
  • FLEX applies ADD, then GENERALIZE, then MASK so new features are processed and masking decisions apply to features in their final forms.Other operation orders require re-iteration to reach a consistent representation.Source: §4 · p. 5

5 Experiments

The experiments test FLEX across 2D navigation and 7DoF Franka manipulation, comparing its three editing operations with single-operation and unscaffolded baselines. They assess scenario-level alignment, sequential performance, and adaptation to real human corrections and demonstrations.

5 Experiments

  • The evaluation compared FLEX’s ADD, GENERALIZE, and MASK operations with ALGAE, MFE, and an unscaffolded LM baseline across randomized navigation and manipulation scenarios.Normalized reward ranges from 0 for worst-case behavior to 1 for optimal behavior.Source: §5 · p. 6

5.1 Isolation of FLEX’s Components (RQ1)

  • FLEX achieved the highest normalized reward in all seven navigation misalignment scenarios, while single-operation baselines degraded when scenarios required other operations, particularly GENERALIZE.USF was inconsistent, scoring 0.64 in ADD ONLY and 0.99 when combining ADD+MASK.Source: §5.1 · p. 7, Table 1 · p. 7

5.2 Evaluation in Simulated Continuous Control Domain (RQ2)

  • Across five-stage Franka correction sequences, FLEX learned the ground-truth reward after one or two corrections and maintained high performance, while baseline performance degraded as corrections accumulated.The baselines gathered spurious features, failed to re-abstract earlier features, and overfit to their training stage.Source: §5.2 · p. 7, Figure 4 · p. 8

5.3 Evaluation with Real Corrections and Demonstrations (RQ3)

  • The Franka user study involved 12 participants in a counterbalanced, within-subjects comparison of FLEX, ALGAE, and MFE, with five sequential correction stages per method.Participants provided typed corrections toward predefined task preferences; their robot interaction was simulated for safety.Source: §5.3 · pp. 7–8
  • With real human corrections and Franka demonstrations, FLEX learned users’ rewards by the sequence’s end; participants also rated it highest on all five questionnaire dimensions.Participants interacted with the methods in simulation for safety; corrections were separately tested offline with expert demonstrations collected on the physical Franka.Source: §5.3 · pp. 8–9, Figure 5 · p. 9, Figure 6 · p. 9

6 Conclusion

FLEX uses language corrections to edit task-feature representations with ADD, GENERALIZE, and MASK, and is evaluated across simulated and real-world feedback settings. The authors report accurate behavior learning across misspecified feature sets and effective transfer to realistic human feedback.

6 Conclusion

  • FLEX edits task-feature representations through ADD, GENERALIZE, and MASK, and is evaluated in 2D navigation and 7DoF manipulation using simulated and real-world user feedback.The reported results include accurate learning under several types of misspecified feature sets.Source: §6 · p. 9
  • FLEX transfers effectively from simulated domains to realistic human feedback.Source: §6 · p. 9
  • FLEX focuses on quasi-static rigid-object manipulation; dynamic tasks and deformable objects may require modalities beyond language to specify relevant features.The authors also identify reliance on language-model common-sense reasoning, limited handling of corrections that jointly encode features and reward weights, and a simulated closed-loop user study with hardware feasibility checks rather than a full physical-robot study.Source: §6 · p. 9

7 Appendix

The appendix documents prompts for FLEX’s feature-editing operations and executable feature generation, alongside baseline prompts and study procedures. It also presents scenario analyses of the three operations as task configurations change.

7.1 LM Prompts

  • Add prompts use the instruction, correction history, addendum, and current feature list to propose only one missing, computable feature relevant to the task.They reject additions when existing features suffice and average distances when multiple objects of the same type are relevant.Source: §7.1 · p. 15
  • Generalization prompts extract the addendum’s intent, then rename only semantically relevant features using explicitly listed environment properties and avoiding spurious correlations.Features may remain unchanged when already sufficiently general or intent-aligned, and semantically equivalent features receive the same name.Source: §7.1 · p. 16
  • Masking prompts judge each feature separately for inclusion in the minimal reward-relevant set, a per-feature approach reported as more consistent than requesting the full mask at once.The downstream reward model has access to feature values but not the instruction or corrections.Source: §7.1 · p. 17
  • Code-generation prompts require executable, environment-generalizable feature functions that compute cumulative values over navigation trajectories or batches of Franka trajectories.The prompts specify trajectory inputs and output shapes, and allow object lookup through an environment utility.Source: §7.1 · pp. 18, 20
  • The LM prompt examples include code for book-proximity rewards and coffee-orientation features computed from the end-effector rotation matrix.The book example averages summed per-timestep distance rewards over matched objects; the coffee example uses the end-effector x-axis z-component to score downward orientation.Source: §7.1 · p. 21

7.2 Baseline Implementations

  • MFE prompts request distinct, relevant computable features, while USF prompts incorporate the task, correction history, and addendum to generate feature values with confidence scores.The prompts ask for features computable from agent position and scene objects, averaging distances across objects of the same type where applicable.Source: §7.2 · p. 22

7.3 RQ1 Additional Details

  • ADD had an 8.6% error rate (9/105), MASK 3.8% (4/105), and GENERALIZE 5.7% (6/105) when tested individually across seven 2D Navigation scenarios.Each operation was tested on all seven scenarios with 15 additional seeds, for 105 trials; errors included unwarranted or omitted edits and incorrect features. The scenarios cover combinations of operations needed to respond to “Don’t burn yourself on the stove.”Source: §7.3 · pp. 22–23, Table 2 · p. 23

7.4 RQ2 Additional Details

  • Five CoffeeRobot configurations show that aligned trajectories can conceal representational misalignment, while changing objects expose the need to ADD, MASK, and GENERALIZE features.The sequence begins with bowl proximity and laptop avoidance, then masks an irrelevant bowl feature; apparent laptop avoidance also avoids an overlapping phone, but fails once the laptop is removed, requiring generalization to electronics.Source: §7.4 · pp. 23–24

7.5 RQ3 Additional Details

  • Participants entered natural-language corrections through a simulated interface with replay controls and three preset camera views; drawn-out trajectories were not visible to them.Figure 7 depicts the interface, including controls for replaying ground-truth and current trajectories and instructions for rotating, panning, and zooming the scene.Source: §7.5 · p. 24, Figure 7 · p. 24

Strongly disagree (1) to Strongly agree (7)

The listed study questions cover collaboration, ease of use, task performance, understanding of user intent, and robustness to unintended consequences.

Strongly disagree (1) to Strongly agree (7)

Related papers

Loading 2610.28893v1…