Source-linked AI summary
Reshaping Robot Trajectories Using Natural Language Commands: A Study of Multi-Modal Data Alignment Using Transformers
Arthur Bucker, Luis Figueredo, Sami Haddadin, Ashish Kapoor, Shuang Ma, Rogerio Bonatti
TL;DR
Rigid language interfaces and data-intensive interaction make it difficult for users to express trajectory changes to robots. The paper combines pretrained language encoders with trajectory geometry in a multimodal transformer that predicts reshaped paths. The authors report better trajectory results than baselines and significant user preference for the natural-language interface.
Problem
Natural-language robot interfaces may require rigid instruction sets or large amounts of data, while users need to specify semantic constraints on existing trajectories.
Source: §I · p. 1Method
The system encodes commands and object labels with pretrained BERT and CLIP, combines language and trajectory geometry in a multimodal transformer, and predicts the reshaped trajectory sequentially.
Source: B. Proposed Network Architecture · p. 3Results
The model outperforms baseline trajectory-reshaping approaches in loss values and result quality, while users significantly prefer its interface, which is faster to use and has a higher success rate.
Source: §V · p. 6Takeaways & Limitations
The authors report that pretrained foundational models let them train the robotics system with a smaller dataset and find similarities when users employ novel vocabulary.
Source: §V · p. 6The study does not address visual input and uses semantic object labels; future work considers object images and causal relations among objects.
Source: §V · p. 6
Abstract
from arXiv · showhide
Natural language is the most intuitive medium for us to interact with other people when expressing commands and instructions. However, using language is seldom an easy task when humans need to express their intent towards robots, since most of the current language interfaces require rigid templates with a static set of action targets and commands. In this work, we provide a flexible language-based interface for human-robot collaboration, which allows a user to reshape existing trajectories for an autonomous agent. We take advantage of recent advancements in the field of large language models (BERT and CLIP) to encode the user command, and then combine these features with trajectory information using multi-modal attention transformers. We train the model using imitation learning over a dataset containing robot trajectories modified by language commands, and treat the trajectory generation process as a sequence prediction problem, analogously to how language generation architectures operate. We evaluate the system in multiple simulated trajectory scenarios, and show a significant performance increase of our model over baseline approaches. In addition, our real-world experiments with a robot arm show that users significantly prefer our natural language interface over traditional methods such as kinesthetic teaching or cost-function programming. Our study shows how the field of robotics can take advantage of large pre-trained language models towards creating more intuitive interfaces between robots and machines. Project webpage: https://arthurfenderbucker.github.io/NL_trajectory_reshaper/
I. INTRODUCTION
Robotics interfaces often depend on rigid instructions, while robotics models are frequently task- and hardware-specific. This work uses pretrained language models and multimodal attention to let users reshape existing robot trajectories with natural-language commands.
I. INTRODUCTION
- Pretrained foundational models are appealing because training across broad datasets and tasks can provide skills usable directly or with minimal fine-tuning.The cited examples include models that combine modalities such as text, images, video, and depth.Source: §I · p. 1
- Robotics models are often task- and hardware-specific, requiring retraining or redesign when robot dynamics, environments, or objectives change.Source: §I · p. 1
- Natural-language robot interaction can require large amounts of data or constrain users to rigid instruction sets.Source: §I · p. 1
- Users can reshape an existing trajectory relative to environmental objects to meet additional semantic or safety constraints beyond collision avoidance.Examples include moving farther from fragile objects or closer to a cup for pouring.Source: §I · p. 1, Figure 1 · p. 1
- The system combines pretrained language representations with geometrical trajectory cues through multimodal attention for semantic trajectory generation.The authors also report that users preferred the interface over baseline methods in ease of use and performance.Source: §I · pp. 1–2
II. RELATED WORK
Related work spans language-based human–robot interaction, multimodal representation learning, and transformer applications in robotics. The paper positions its contribution as a multimodal transformer for aligning visual-language understanding with robot actions during trajectory reshaping.
II. RELATED WORK
- Prior language-based human–robot interaction work covers instruction understanding, motion planning [9] [25], cooperation, semantic belief propagation [19], and visual-language navigation [11].Source: §II · p. 2
- Visual-language representations support tasks such as question answering, grounding, retrieval, and captioning –[31], while robotics representations must account for perception conditioned on motion policies and dynamics [4].Source: §II · p. 2
- Prior robotics transformer applications include trajectory forecasting and reinforcement learning; the authors describe their work as the first multimodal transformer to align visual-language understanding with robot actions for trajectory reshaping.Source: §II · p. 2
A. Problem Definition
The problem is to modify a reasonable, collision-avoiding robot path so it also satisfies a user’s semantic intent. The output should stay close to the original path and preserve its start and goal while accounting for the command and objects in the environment.
A. Problem Definition
- The modified trajectory should remain close to the original, preserve its start and goal, and obey the user’s semantic intent.Source: A. Problem Definition · p. 2
- The original trajectory is assumed to be a reasonable, collision-avoiding path from start to goal that can be pre-calculated but may fall short of the full task specifications.Source: A. Problem Definition · p. 2
- The target mapping takes the original trajectory, natural-language command, and environmental objects as input and returns a trajectory satisfying the semantic objective.Objects are represented by positions and semantic labels, such as a glass.Source: A. Problem Definition · p. 2
B. Proposed Network Architecture
The proposed architecture encodes language and geometry separately, then combines their features in a multimodal transformer decoder that predicts a reshaped trajectory waypoint by waypoint. It uses pretrained BERT and CLIP language encoders and imitation learning for training.
B. Proposed Network Architecture
- The architecture processes initial trajectory, object position and semantic labels, and a language command to predict a new trajectory using pretrained CLIP and BERT with trainable transformer encoder and decoder components.Source: Figure 3 · p. 3, B. Proposed Network Architecture · p. 3
- BERT encodes command semantics, while CLIP text embeddings represent the command and object labels to estimate their similarity and help identify the target object.The authors associate pretrained language encoders with synonym flexibility and lower data requirements.Source: B. Proposed Network Architecture · p. 3
- A learnable linear transform projects trajectory waypoints and object positions into higher-dimensional features, which a transformer encoder processes as a sequence.Source: B. Proposed Network Architecture · p. 3
- The decoder combines language and geometry embeddings to generate the modified trajectory sequentially; imitation learning uses Huber loss between predicted and ground-truth waypoint locations.Source: B. Proposed Network Architecture · p. 3
C. Synthetic Data Generation
The authors create 10,000 language-labeled trajectory modifications in randomized environments, using CHOMP to reshape A*-planned paths. Validation examples compare original paths, CHOMP targets, and model outputs.
C. Synthetic Data Generation
- 10,000 trajectory labels pair A*-planned initial paths with CHOMP modifications in randomized environments.The modifications use pre-determined semantic combinations of direction relative to an object and intensity.Source: C. Synthetic Data Generation · p. 3
- The vocabulary spans directions and intensity changes relative to objects, plus 1,000 object labels sampled from ImageNet.Examples include closer or farther, left/right/front/back, and degrees such as “a bit” or “very.”Source: C. Synthetic Data Generation · p. 3
- Fig. 4 shows randomly selected validation problems, coloring original A* paths, CHOMP ground-truth reshaped paths, and model outputs.The figure provides examples of the generated original and reshaped trajectories.Source: C. Synthetic Data Generation · p. 3, Figure 4 · p. 4
D. Implementation Details
The implementation combines frozen pretrained BERT and CLIP language encoders with transformer-based geometry and trajectory processing. Training uses data augmentation followed by fine-tuning on fixed start and goal locations.
D. Implementation Details
- Frozen pretrained BERT and CLIP encoders produce sentence and object-similarity features that are scaled to 256 dimensions.The 768-dimensional sentence embedding is concatenated with a CLIP similarity vector of size 1 × M.Source: D. Implementation Details · p. 4
- Geometry features combine object poses with 100 original-trajectory waypoints, while the model uses a 2-block encoder and 4-block decoder.Each transformer has three hidden layers with 512 fully connected neurons; normalization steps were removed to better retain geometry information.Source: D. Implementation Details · p. 4
- Training first augments planning problems through random rotation and rescaling, then fine-tunes with start and goal locations fixed to opposite map corners.The model is trained for 500 epochs using AdamW, batch size 64, and a linear 15-epoch warm-up.Source: D. Implementation Details · p. 4
IV. EXPERIMENTS
The experiments assess the model’s efficiency and generalizability as a human-robot interface, compare it with other interface classes from users’ perspectives, and test applicability to real robots.
IV. EXPERIMENTS
- The experiments test whether pretrained large language models and multimodal transformers can create efficient, generalizable human-robot interfaces.This is the first stated experimental goal.Source: §IV · p. 4
- The study quantitatively and qualitatively compares the approach with other classes of human-robot interfaces from the user’s perspective.This is the second stated experimental goal.Source: §IV · p. 4
- The experiments also assess whether the natural-language method applies to real robots.This is the third stated experimental goal.Source: §IV · p. 4
A. Simulation Experiments
Simulation experiments examine language-conditioned trajectory changes across commands and planning problems, including commands with unseen vocabulary. The reported architecture comparison favors the multimodal transformer without normalization and with a sequential predictive decoder.
A. Simulation Experiments
- For a fixed environment, direction and intensity gradients in language commands correctly modify the decoded path, while changes toward an object are constrained as unsafe.Fig. 5 illustrates the influence of different language inputs on the same environment configuration.Source: A. Simulation Experiments · p. 4, Figure 5 · p. 4
- Across randomly sampled validation problems, reshaped trajectories usually model user intent and fall close to the CHOMP ground truth.Fig. 4 displays original A* paths, CHOMP ground-truth paths, and model outputs.Source: A. Simulation Experiments · p. 4
- The model executes expected behavior for novel commands containing vocabulary absent from the training labels.The paper attributes this result to the large capacity of the BERT and CLIP text encoders.Source: A. Simulation Experiments · p. 5, Figure 6 · p. 5
- The multimodal transformer without layer and batch normalization and with a sequential predictive decoder is the best-performing architecture in Table I.The comparison includes a fully connected regression baseline and a naive predictor that copies the original trajectory.Source: A. Simulation Experiments · p. 5, Table I · p. 5
- The authors compare transformer variations with a five-layer fully connected regressor and a naive predictor that copies the original path.The fully connected model takes concatenated trajectory, object-position, and embedded-language features and predicts waypoint coordinates.Source: A. Simulation Experiments · p. 5
B. User Evaluation in Real Robot Experiments
The real-robot study compared natural-language trajectory reshaping with three other interfaces on a bartender task. Users found natural language fastest, failure-free, easiest, and most natural, though less predictable than direct-control methods.
B. User Evaluation in Real Robot Experiments
- Ten interviews tested four interfaces for redirecting a bartender robot’s wine-pouring path away from crystal glasses, using a 7-DOF PANDA arm on a 1 × 1m tabletop.The study compared natural language, kinesthetic teaching, trajectory drawing, and keyboard-and-mouse programming of obstacle-avoidance weights.Source: B. User Evaluation in Real Robot… · p. 5, Figure 8 · p. 6
- Natural language was fastest and had no failures; programming took longest and required many iterations, while kinesthetic teaching and drawing often failed on kinematic infeasibility.The authors describe natural language as the most robust interface in the study.Source: B. User Evaluation in Real Robot… · p. 5, Table II · p. 6
- Natural language was rated easiest and most natural, but less predictable than kinesthetic teaching and drawing, whose users directly controlled the final trajectory.Most methods received similar satisfaction ratings, except programming, which was rated lower, likely because interaction was difficult.Source: Table III · p. 6, B. User Evaluation in Real Robot… · p. 6
- The authors conclude that natural language is a strong alternative; kinesthetic teaching depends on robot size, form factor, and actuators, while drawing may violate acceleration and kinematic constraints.Source: B. User Evaluation in Real Robot… · p. 6
V. CONCLUSION AND DISCUSSION
The paper presents multimodal attention for reshaping trajectories from language and geometric cues, using pretrained language models to support flexible commands and smaller training datasets. In simulation and real-world evaluations, the system outperformed baseline approaches, and users preferred its natural-language interface; visual inputs remain outside the study.
V. CONCLUSION AND DISCUSSION
- A multimodal attention system reshapes trajectories from flexible language commands and an initial path by jointly aligning language features with geometric cues.Source: §V · p. 6
- BERT and CLIP pretrained models support a flexible interface and smaller robotics training datasets, while language-model sentence similarities can accommodate novel vocabulary.Source: §V · p. 6
- The model outperformed baseline trajectory-reshaping approaches in simulation and real-world scenarios, while users preferred its faster, higher-success natural-language interface.The reported baseline comparison concerns loss values and result quality; the user study compared natural language with kinesthetic teaching and programming interfaces.Source: §V · p. 6
- The study does not address visual inputs; future work proposes using object images instead of semantic labels and modeling causal relations for more complex tasks.Source: §V · p. 6