PaperlayerView exact revision on arXiv

Source-linked AI summary

SIRL: Similarity-based Implicit Representation Learning

Andreea Bobu, Yi Liu, Rohin Shah, Daniel S. Brown, Anca D. Dragan

arXiv:2301.00810v3cs.ROcs.AIcs.HCcs.LG

TL;DR

Robots must learn both task-relevant features and how to combine them into rewards, but reward-oriented input can yield representations containing spurious correlations. SIRL learns representations from users’ trajectory-similarity judgments; simulations and a user study report improved alignment and generalization over self-supervised and task-input alternatives, with limitations in alignment and query difficulty.

  • Problem

    Learning rewards from raw states requires both task-relevant representations and reward functions, while reward-oriented input can produce representations with spurious correlations that fail to generalize.

    Source: §1 · p. 2

  • Method

    SIRL trains a trajectory representation from users’ choices of the two most similar behaviors in triplets, then uses that representation for downstream reward learning.

    Source: §3.2 · p. 3

  • Results

    Simulations show better causal-feature alignment and more generalizable rewards than alternatives, while a user study finds a significant effect in recovering individual preferences that is smaller than in simulation.

    Source: §1 · p. 2, §4.4 · p. 6

  • Takeaways & Limitations

    Similarity queries are a step toward recovering human-aligned representations, and the paper identifies interface design and query selection as areas for further exploration.

    Source: §6 · p. 8

    SIRL representations are not perfectly aligned with causal features, and judgments become harder when multiple important features change.

    Source: §6 · p. 8

Abstract

from arXiv · show

When robots learn reward functions using high capacity models that take raw state directly as input, they need to both learn a representation for what matters in the task -- the task ``features" -- as well as how to combine these features into a single objective. If they try to do both at once from input designed to teach the full reward function, it is easy to end up with a representation that contains spurious correlations in the data, which fails to generalize to new settings. Instead, our ultimate goal is to enable robots to identify and isolate the causal features that people actually care about and use when they represent states and behavior. Our idea is that we can tune into this representation by asking users what behaviors they consider similar: behaviors will be similar if the features that matter are similar, even if low-level behavior is different; conversely, behaviors will be different if even one of the features that matter differs. This, in turn, is what enables the robot to disambiguate between what needs to go into the representation versus what is spurious, as well as what aspects of behavior can be compressed together versus not. The notion of learning representations based on similarity has a nice parallel in contrastive learning, a self-supervised representation learning technique that maps visually similar data points to similar embeddings, where similarity is defined by a designer through data augmentation heuristics. By contrast, in order to learn the representations that people use, so we can learn their preferences and objectives, we use their definition of similarity. In simulation as well as in a user study, we show that learning through such similarity queries leads to representations that, while far from perfect, are indeed more generalizable than self-supervised and task-input alternatives.

1 INTRODUCTION

Reward functions can change across tasks, users, and contexts even when the features underlying the representation remain shared. SIRL elicits judgments of trajectory similarity to learn those features directly, and its evaluations report better-aligned representations and more generalizable rewards than alternatives.

1 INTRODUCTION

  • Reward functions vary across tasks, users, and contexts, while the features needed for a reusable representation can remain shared.Learning that representation can support reward learning when the task, user, or context changes.Source: §1 · p. 2
  • Trajectory similarity queries ask users to select the two most similar of three trajectories, using their judgments to train a feature representation.The proposed intuition is that similar behaviors share important features even when their low-level behavior differs.Source: §1 · p. 2
  • Unlike visually defined similarity in self-supervised contrastive learning, SIRL uses users’ judgments to target the representation they use.The paper argues that visual heuristics can retain distractor features that users consider irrelevant.Source: §1 · p. 2
  • Simulation results show better alignment with causal features and more generalizable downstream rewards, while the user study reports a significant but smaller effect.The authors attribute part of the smaller user-study effect to interface difficulty analyzing trajectories; some users also struggled to trade off features.Source: §1 · p. 2

2 RELATED WORK

Prior work learns representations implicitly from reward-oriented human input or from signals tied to particular goals, while this paper targets task-agnostic features for multiple preferences. It uses similarity judgments to learn a shared latent space without requiring a pre-defined feature set or a curated distribution of reward tasks.

Learning from Human Input

  • Human feedback methods infer policies or rewards from demonstrations, corrections, comparisons, or rankings, whereas this work explicitly seeks a preference-agnostic latent space for downstream tasks.Imitation learning copies demonstrations [47], while inverse reinforcement learning infers a reward; demonstrations may generalize poorly beyond training [40].Source: §2 · p. 2

Representation and Similarity Learning

  • Representation methods using unsupervised or self-supervised signals need not capture features people care about, while task labels and rankings can tie learned spaces to specific goals or preferences.This work instead targets task-agnostic feature similarity for learning multiple preferences.Source: §2 · pp. 2–3
  • Unlike approaches requiring selected features, physical demonstrations, or a known test-task distribution, this work learns a lower-dimensional representation from trajectory triplets.The authors describe their approach as the first use of triplet loss for a general, task-agnostic model of human trajectory similarity.Source: §2 · p. 3

3 METHOD

SIRL learns a preference-agnostic trajectory representation from users’ relative similarity judgments, then reuses it for downstream reward learning across preferences. Its training objective brings embeddings of similar trajectories together and separates dissimilar ones.

3 METHOD

  • SIRL learns a preference-agnostic representation from trajectory triplets in which users choose the two most similar behaviors.The resulting shared latent space is intended for downstream preference learning with multiple people.Source: §3 · p. 3

3.1 Preliminaries

  • The robot first learns a representation from human input, rather than relying on reward-specific input to learn the representation and reward simultaneously.The downstream reward is a parameterized neural-network approximation learned from preference labels.Source: §3.1 · p. 3

3.2 Training the Feature Representation via Trajectory Similarity Queries

  • Users compare three trajectories because relative similarity judgments avoid requiring a binary or quantitative similarity threshold.A triplet loss trains the embedding to bring similar trajectories closer and dissimilar trajectories farther apart; the loss accounts for queries without an explicit anchor.Source: §3.2 · p. 3

3.3 Using SIRL for Reward Learning

  • For reward learning, the robot collects pairwise trajectory preferences and learns a reward function from those labels using a cross-entropy loss.The learned embedding can also support downstream tasks using other types of human feedback.Source: §3.3 · pp. 3–4

3.4 Adapting to Different User Preferences

  • For new user preferences, the robot reuses the shared similarity-trained representation to learn a reward function rather than training each preference independently.The authors hypothesize that this preference-agnostic, human-trained feature space is more transferable than a space pretrained on specific task objectives.Source: §3.4 · p. 4

4 EXPERIMENTS IN SIMULATION

The simulation compares SIRL with alternative representation-learning methods in GridRobot and JacoRobot. With enough representation data, SIRL learns representations more predictive of true features and supports more generalizable reward learning, though causal alignment is imperfect.

4.1 Environments

  • GridRobot is a 5-by-5 grid with 19-dimensional inputs, while JacoRobot has a 7-DoF arm and 2,037-dimensional inputs; both use four ground-truth features.Similarity and preference labels are generated from sampled trajectories using feature similarity and ground-truth rewards, respectively.Source: §4.1 · p. 4

4.2 Qualitative Examples

  • In the GridRobot examples, SIRL groups trajectories sharing relevant features despite different states, and separates trajectories with similar states but different features.The examples contrast laptop clearance and cup orientation with low-level path similarity.Source: §4.2 · pp. 4–5, Figure 3 · p. 5

4.3 Experimental Setup

  • The study compares SIRL with VAE, MultiPref, SinglePref, and Random representations, matching human-data budgets across the preference-based and similarity-query methods.MultiPref is tested with 10 or 50 simulated humans; embeddings use the same network size within each environment.Source: §4.3 · p. 5
  • Representation quality is measured by linear-probe feature prediction error, while reward generalization is measured by preference accuracy on held-out queries across 20 test rewards.FPE is test-set MSE; TPA is averaged across test human preferences.Source: §4.3 · p. 5

4.4 Results

  • VAE pre-training helps SIRL in GridRobot but can hurt in JacoRobot; preference-query baselines also face a trade-off between reward diversity and data per reward.With 1,000 representation queries, both SIRL variants outperform baselines on test preference accuracy in both environments.Source: §4.4 · pp. 5–6
  • With enough representation data, SIRL better predicts true features and supports more generalizable rewards than baselines in both environments, though causal-feature alignment is imperfect.For JacoRobot, SIRL without VAE pre-training performs better than the VAE-pretrained version.Source: §4.4 · pp. 5–6, Figure 4 · p. 6

5 USER STUDY

The JacoRobot user study tested whether novice similarity judgments could yield useful shared representations. SIRL improved feature prediction and reward generalization over Random, with interface and study-scope caveats.

5.1 Experiment Design

  • The JacoRobot study focused on table and laptop features and used trajectory views showing each path's Euclidean trace to aid comparison.Participants could change the view and replay trajectories in the interface.Source: §5.1 · p. 6
  • Ten campus participants, mostly with technical backgrounds, each answered 100 recorded similarity queries and 100 preference queries after practice.Preference scenarios varied across participants.Source: §5.1 · p. 6
  • The study compared SIRL without VAE against Random, training a shared representation from 100 similarity queries per participant; evaluation included real, held-out-user, and simulated preference settings.Real preference data was split for reward training and evaluation; simulated preferences covered 10 different rewards.Source: §5.1 · pp. 6–7

5.2 Analysis

  • With simulated preference queries, performance improved slightly for both methods; SIRL was also slightly better for users assigned preferences involving both features, hinting at feature entanglement.The feature-subset result is tentative; the authors describe it as a hint that the representation may have entangled the features.Source: §5.2 · p. 7
  • SIRL recovered representations twice as predictive of true features as Random (p < .0001) and improved average real and held-out-user TPA over Random.Figure 6 reports the feature-prediction result and TPA results for real, held-out, and simulated preferences.Source: §5.2 · p. 7, Figure 6 · p. 8
  • Some users found viewpoint-dependent judgments in the 2D interface deceptive and struggled to trade off the two features; most participants' technical backgrounds limit the study's usability scope.The authors suggest that less structured, in-the-wild studies could test whether users draw on their unaltered representations.Source: §5.1 · p. 6, §5.2 · p. 8

6 DISCUSSION AND LIMITATIONS

SIRL uses trajectory similarity queries to learn representations that better capture relevant features and support reward learning across tasks. The authors describe the approach as promising but imperfect, with open challenges in query difficulty, study setting, and scale.

6 DISCUSSION AND LIMITATIONS

  • Trajectory similarity queries target representations that capture and disentangle relevant features while excluding spurious ones, and the paper reports better test-time reward ranking than self-supervision or multi-task learning.The authors frame similarity queries as a way to tap into the features people use across tasks.Source: §6 · p. 8
  • The learned representations are not perfectly aligned with causal features, and similarity judgments become harder when multiple important features change together.The authors suggest iteratively building the representation and using current feature estimates to guide later queries.Source: §6 · p. 8
  • The study was not in the wild: researchers explained contexts and what might matter, potentially engineering participants' internal representations.The authors propose a future study with less structure and users drawing on familiar everyday contexts.Source: §6 · p. 8
  • Learning a good representation can require hundreds of queries, making sample complexity important as robotic tasks become more complex.The authors propose crowd-sourcing and active querying as possible ways to reduce user burden or query volume.Source: §6 · p. 8
  • The authors view similarity queries as complementary to self-supervision, while applying SIRL directly to policy learning or exploration and combining the approaches remain open directions.Self-supervised heuristics might remove spurious features before SIRL fine-tunes the representation.Source: §6 · p. 8

A.1 Trajectory generation

The JacoRobot trajectory space is sampled using 10,000 start-goal pairs and shortest paths, then expanded through localized random-torque deformations.

A.1 Trajectory generation

  • 10,000 randomly sampled start-goal pairs are used to compute shortest paths in JacoRobot’s configuration space.Each path has horizon H and n-dimensional states.Source: §A.1 · p. 11
  • Up to three states on each trajectory are randomly selected and deformed with different random torques.The deformation is localized to the selected state index; its magnitude and index are randomized.Source: §A.1 · p. 11
  • The deformation adds a scaled, localized torque vector transformed by A, which defines a trajectory-space norm and deformation shape.For smooth deformations, A is based on acceleration; other norm choices are possible.Source: §A.1 · p. 11

A.2 Training details

The implementation details describe the embedding-network sizes used for GridRobot and JacoRobot and their shared activation choice.

A.2 Training details

  • The appendix presents architecture and optimization details to support reproduction of the training setup.Source: §A.2 · p. 11
  • GridRobot embeddings use two-hidden-layer MLPs with 128 units per layer and six outputs; JacoRobot uses 1,024 units for its larger input space.Both environments use ReLU after every linear layer.Source: §A.2.1 · p. 11
  • All embedding networks use ReLU nonlinearities after every linear layer.Source: §A.2.1 · p. 11

A.2.1 Feature networks.

The feature-network details specify VAE pre-training, preference-model optimization, SIRL training, TPA evaluation, and a fixed-length trajectory assumption.

A.2.1 Feature networks.

  • VAE pre-training uses a variational reconstruction loss with KL regularization weighted at 0.01, trained for 2,000 epochs with Adam.The learning rate starts at 0.01 and decays exponentially; batch size is 32.Source: §A.2.1 · p. 11
  • SinglePref and MultiPref use preference loss with L2 reward regularization, weighted 10 for GridRobot and 1 for JacoRobot.GridRobot training uses 5,000 Adam epochs at learning rate 0.01; JacoRobot uses 0.001 for more stable training.Source: §A.2.1 · p. 11
  • SIRL’s similarity objective is trained separately after optional VAE pre-training, for 3,000 Adam epochs with batch size 64.The learning rate starts at 0.004 and decays exponentially.Source: §A.2.1 · p. 11
  • The current architectures assume fixed-length trajectories, though an LSTM-based architecture could handle varying lengths.Source: §A.2.1 · p. 11
  • TPA evaluation places preference networks on each method’s embeddings, using two-layer MLPs and the same environment-specific L2 weights.Training uses 500 epochs for GridRobot and 1,000 for JacoRobot.Source: §A.2.2 · p. 11

A.3 Ablations

The ablations compare frozen and unfrozen representations and the effect of VAE pre-training across GridRobot and JacoRobot.

A.3 Ablations

  • SIRL performs better with its learned representation frozen, whereas the other methods perform better with representations unfrozen.Figure 7 reports the ablation for GridRobot and JacoRobot.Source: §A.3 · p. 11, Figure 7 · p. 12
  • SinglePref and MultiPref perform better without VAE pre-training, while SIRL sometimes benefits from pre-training in simple environments such as GridRobot.Source: §A.3 · p. 11, Figure 7 · p. 12
  • Figure 5 uses frozen SIRL and unfrozen baselines without VAE pre-training, the best configurations found for each method.Source: §A.3 · p. 11
Loading 2301.00810v3…