PaperlayerView exact revision on arXiv

Source-linked AI summary

LATTE: LAnguage Trajectory TransformEr

Arthur Bucker, Luis Figueredo, Sami Haddadin, Ashish Kapoor, Shuang Ma, Sai Vemprala, Rogerio Bonatti

arXiv:2208.02918v3cs.ROcs.AIcs.CLcs.CVcs.LG

TL;DR

Robots need to translate human language into motion while balancing semantic intent with geometric and dynamics constraints across varied platforms. LATTE uses pretrained language and image encoders with geometric features to reshape trajectories, and the authors report successful intent-following in simulated and real-world experiments across multiple robot types.

  • Problem

    Robots need to combine high-level human instructions with geometric and kinodynamic constraints, while task-specific solutions offer limited generalizability across hardware platforms.

    Source: §I · p. 1

  • Method

    LATTE uses BERT and CLIP to encode free-form commands and scene images, fuses these with geometric features, and decodes a modified robotic trajectory.

    Source: §I · p. 1, B. Proposed Network Architecture · p. 3, §V · p. 6

  • Results

    The authors report that the model follows human intent by modifying trajectory shape and speed in simulated and real-world experiments across manipulation, aerial, and legged platforms.

    Source: B. Simulation Experiments · p. 5, §V · p. 6, D. User study experiments · p. 6

  • Takeaways & Limitations

    The output trajectory can be post-processed for use across platforms including manipulators, aerial vehicles, and legged robots.

    Source: §V · p. 6

Abstract

from arXiv · show

Natural language is one of the most intuitive ways to express human intent. However, translating instructions and commands towards robotic motion generation and deployment in the real world is far from being an easy task. The challenge of combining a robot's inherent low-level geometric and kinodynamic constraints with a human's high-level semantic instructions traditionally is solved using task-specific solutions with little generalizability between hardware platforms, often with the use of static sets of target actions and commands. This work instead proposes a flexible language-based framework that allows a user to modify generic robotic trajectories. Our method leverages pre-trained language models (BERT and CLIP) to encode the user's intent and target objects directly from a free-form text input and scene images, fuses geometrical features generated by a transformer encoder network, and finally outputs trajectories using a transformer decoder, without the need of priors related to the task or robot information. We significantly extend our own previous work presented in Bucker et al. by expanding the trajectory parametrization space to 3D and velocity as opposed to just XY movements. In addition, we now train the model to use actual images of the objects in the scene for context (as opposed to textual descriptions), and we evaluate the system in a diverse set of scenarios beyond manipulation, such as aerial and legged robots. Our simulated and real-life experiments demonstrate that our transformer model can successfully follow human intent, modifying the shape and speed of trajectories within multiple environments. Codebase available at: https://github.com/arthurfenderbucker/LaTTe-Language-Trajectory-TransformEr.git

I. INTRODUCTION

Robots in human-centered settings must adapt to commands and context while respecting safety and dynamics, but traditional task-specific systems offer limited flexibility. LATTE reshapes planner-generated trajectories using language, scene images, and geometry, extending the approach to 3D and velocity and evaluating it across robot forms.

I. INTRODUCTION

  • Robots need to adapt trajectories to human intent and context while respecting safety and dynamics, yet many remain pre-programmed for specific tasks and unstructured environments.This requires combining language and vision with geometric information to shape trajectories.Source: §I · p. 1
  • LATTE adds semantic objectives to an initial trajectory from a geometric planner, which handles obstacle avoidance and dynamics constraints.The framework is intended to let users reshape trajectories with language instructions.Source: §I · p. 1, Figure 1 · p. 1
  • The method extends waypoint representation from planar XY to 3D coordinates and velocity.Source: §I · p. 1
  • Instead of textual object labels, the system uses environment images to infer command targets and evaluates outputs on aerial and legged robots beyond manipulators.The model's outputs are described as amenable to different robot dynamics and motion controllers.Source: §I · pp. 1–2

II. RELATED WORK

Prior language-based human-robot interfaces have used rigid instructions or complex action-and-object models, while deep approaches often require extensive training data. LATTE builds on transformer and multimodal work, distinguishing itself from a close planner-based approach by aligning semantic and geometric cues to reshape trajectories.

II. RELATED WORK

  • Language interfaces have relied on rigid instruction sets or mathematically complex action-and-object models, while deep models often require vast training datasets.Source: §II · p. 2
  • Navigation research combines language, perception, and planning through cost functions or reinforcement learning, while manipulation and multimodal work use CLIP or pretrained language-vision representations.Source: §II · p. 2
  • Robotics transformers support trajectory forecasting, reinforcement learning, and multimodal feature extraction; unlike [9]’s semantic-cost-map-guided optimization planner, this method uses an encoder-decoder to align semantic and geometric cues and recast trajectories.The authors’ first paper validated the approach in 2D; this paper extends it to higher dimensions and more realistic experimental settings.Source: §II · p. 2

III. APPROACH

LATTE modifies a robot's trajectory over time in response to a natural-language command, using scene geometry and images as context. It starts from a trajectory assumed to satisfy system constraints and aims to produce one that meets the user's semantic objective while remaining feasible.

III. APPROACH

  • The platform-agnostic system predicts waypoint sequences from scene geometry, environment images, and language using a transformer encoder-decoder pair.The robot body or end-effector behavior is represented as a 3D trajectory over time.Source: §III · p. 2

A. Problem Definition

  • The input is a normalized trajectory of N waypoints, each containing x, y, z, and velocity, assumed to satisfy system constraints but not necessarily the full task specification.The user supplies a natural-language correction, such as asking the robot to move more slowly near fragile glasses.Source: A. Problem Definition · pp. 2–3
  • The learned mapping takes the original trajectory, command, and environment objects—with positions and images—and targets a modified trajectory that satisfies semantic objectives within the feasible domain.Source: A. Problem Definition · p. 3

B. Proposed Network Architecture

LATTE encodes language, object imagery, trajectory waypoints, and object poses, then decodes a modified trajectory sequentially. Its data-driven mapping addresses multimodal inputs and ambiguity in the set of trajectories that can satisfy a command.

B. Proposed Network Architecture

  • The data-driven model approximates the mapping from inputs to modified trajectories, a task complicated by multiple modalities and multiple valid trajectories for a semantic objective.Source: B. Proposed Network Architecture · p. 3
  • Pretrained BERT encodes the command, while CLIP encodes command text and object images; cosine similarities identify possible target objects and combine with BERT features as the semantic embedding.The text-image embedding approach gives equivalent target-identification results to the earlier use of textual object descriptions.Source: B. Proposed Network Architecture · p. 3
  • A geometry encoder projects trajectory waypoints and object poses into a shared feature space, then uses a transformer to extract geometric relations across the sequences.Source: B. Proposed Network Architecture · p. 3
  • A multimodal transformer decoder combines language and geometry features to predict the reshaped trajectory one waypoint at a time, trained by imitation learning with Huber loss.Source: B. Proposed Network Architecture · p. 3

C. Post-processing and execution

The method post-processes modified trajectories to respect Cartesian constraints under simplifying assumptions, then uses inverse kinematics to produce trajectories for higher-degree-of-freedom robots.

C. Post-processing and execution

  • The post-processing modules are intended to keep the method agnostic to specific robotics platforms.Source: C. Post-processing and execution · p. 3
  • The method assumes the planner supplies hard Cartesian constraints and an initially admissible trajectory, then stops updating any waypoint that reaches an inadmissible region.It moves each waypoint incrementally from its original position toward the modified one; the authors note more complex constraint algorithms are possible.Source: C. Post-processing and execution · p. 3
  • Inverse kinematics can convert the final trajectory for higher-degree-of-freedom robots, and the evaluation includes manipulators, aerial robots, and legged robots.Source: C. Post-processing and execution · p. 3

D. Synthetic Data Generation

The paper uses procedural generation to address the cost of collecting robotics data aligned across language, vision, and geometry. Its synthetic examples pair trajectories, language instructions, and object poses and images to train trajectory modifications.

D. Synthetic Data Generation

  • Aligned language, vision, and geometry data can be challenging and expensive to collect in robotics.Source: D. Synthetic Data Generation · p. 3
  • The procedural dataset examples show original trajectories in red, ground-truth edits, and model predictions in blue, including object images and speed-profile changes.Source: Figure 3 · p. 4
  • Procedural generation supplies trajectory-language pairs, based on the hypothesis that pretrained BERT and CLIP encoders reduce vocabulary-diversity data requirements.The authors state that experiments validate this hypothesis.Source: D. Synthetic Data Generation · p. 4
  • Each sample combines a base trajectory, language input, modified trajectory, and objects represented by poses and web-crawled images.Base trajectories are splines through random-walk points; object names are sampled from ImageNet.Source: D. Synthetic Data Generation · p. 4
  • The generated instructions cover absolute position, speed, and object-relative changes, with a handcrafted force field guiding trajectory optimization.Intensity words vary field strength, and BART paraphrases are also explored.Source: D. Synthetic Data Generation · p. 4
  • A locality factor lets users set the desired range of a trajectory modification after training, from localized changes to long-range effects.Source: D. Synthetic Data Generation · p. 4

IV. EXPERIMENTS

The experiments are designed to assess 3D and velocity-space trajectory modification, architectural components, and applicability across robot platforms. Training uses procedurally generated data and frozen BERT and CLIP encoders.

IV. EXPERIMENTS

  • The experiments target effectiveness in 3D and velocity space, architectural contributions, and applicability across multiple robotic platforms.Source: §IV · p. 4

A. Model training details

  • The model is trained on 70,000 of 100,000 procedurally generated examples, with 10,000 for validation and 20,000 for testing.BERT and CLIP encoder weights are frozen.Source: A. Model training details · p. 4
  • Training uses a one-block transformer encoder and five-block decoder, with AdamW, batch size 16, and 500 epochs taking approximately two hours on a Tesla V100 GPU.The model uses 512-neuron fully connected layers and eight attention heads per transformer.Source: A. Model training details · p. 4

B. Simulation Experiments

In simulated scenes, the model modifies trajectories and speed profiles in response to language commands, with demonstrations across different robot platforms and evaluations of dataset size, architecture, and locality factor.

Qualitative and multi-platform results

  • For simulated scenes with web-sourced object images, the model predicts spatial and speed-profile changes for commands to move closer or farther, or faster or slower near objects.The multi-platform experiments also show trajectory modifications for aerial and legged robots.Source: B. Simulation Experiments · pp. 4–5, Figure 4 · p. 5

Baseline architectures

  • The architecture comparison reports no improvement from adding encoder blocks, while three decoder blocks perform slightly better than the five-block baseline.Source: Table I · p. 5

Locality factor

Dataset size and augmentations

  • Increasing training data from 1k to 10k significantly improves validation metrics; geometric shifting and scaling yield a modest performance increase.Procedurally generated data keeps the stated challenge minimal aside from longer training time.Source: B. Simulation Experiments · pp. 5–6, Table III · p. 6

C. Real Robot Experiments with Manipulation

The authors deploy the model on a 7-DOF PANDA arm, using a workbench camera to capture obstacle scenes and an off-the-shelf CPU/GPU setup for computation. A sample interaction depicts online language-directed speed modification.

Real-world setup

Sample interaction

  • The sample interaction depicts online speed modification, with the original trajectory in red and the modified trajectory in blue.Source: Figure 6 · p. 6

D. User study experiments

In the user study, participants rated the trained model and procedural ground truth highly, while intentionally incorrect, language-free, and projected-2D baselines received poor ratings. In free-form interactions, 48% of inputs contained words unseen during training, and the model failed on 24% of cases.

  • In a user study of 10 participants providing 300 data-points on a 1–5 Likert scale, Ground Truth and Ours received high ratings, while intentionally incorrect Ground Fake received the lowest score.Source: D. User study experiments · p. 6
  • The baseline evaluation collected 300 data points from 10 participants rating trajectory changes from five approaches on a 1–5 Likert scale.Source: D. User study experiments · p. 6
  • 48% of free-form user inputs included words unseen during training, and the model failed on 24% of cases.Table IV compares in- and out-of-distribution settings; its caption identifies the evaluation but provides no cell values here.Source: D. User study experiments · p. 6, Table IV · p. 6

V. CONCLUSION AND DISCUSSION

The conclusion presents a language-based interface that uses text, scene images, and trajectory geometry to generate modified trajectories. The work reports simulated and real-world experiments and identifies additional modalities and longer interaction horizons as future directions.

  • The framework encodes intent and target objects from free-form text and scene images, fuses geometric features, and outputs modified trajectories through a transformer decoder.Source: §V · p. 6
  • The model operates in 3D and velocity spaces, with post-processed trajectories applicable to manipulation, aerial vehicles, and legged robots.The authors report simulated and real-world experiments demonstrating effectiveness.Source: §V · p. 6
  • Future work will explore force inputs and interaction over longer horizons with multiple instructions.Source: §V · p. 6
Loading 2208.02918v3…