PaperlayerView exact revision on arXiv

Source-linked AI summary

FRAM: Trajectory-Guided Visual Feature Selection for Compact Language-Conditioned Robot Manipulation

Hiroshi Ito, Hyogo Hiruma, Yoshiki Kanai, Takahiro Yoshida, Akira Kanazawa, Hiroyuki Yamada

arXiv:2609.30965v2cs.RO

TL;DR

Large Vision-Language-Action models can demand substantial resources, motivating compact policies that still perform well. FRAM uses predicted future trajectories to select local visual features from current images, and achieves strong results across LIBERO suites and perturbed environments with a small policy.

  • Problem

    Large Vision-Language-Action policies often require substantial computing resources for fine-tuning and inference, motivating small models that still perform well.

    Source: §I · p. 1

  • Method

    FRAM predicts future end-effector trajectories and uses their image coordinates as spatial pointers to read action-relevant local features from the current image.

    Source: §I · p. 1, A. Design Concept · p. 2

  • Results

    With 138.7M parameters, FRAM reaches 92.2% average success across the four standard LIBERO suites and 67.3% on LIBERO-Plus without extra training on perturbed environments.

    Source: §I · p. 2, §VI · p. 8

  • Takeaways & Limitations

    FRAM also stacks cups on a dual-arm UR5e using wrist cameras, including choosing and switching between the left and right arms.

    Source: §I · p. 2

    LIBERO-Long performance is lower than on the other suites, and the real-robot evaluation is limited to cup stacking.

    Source: §VI · p. 8

Abstract

from arXiv · show

Vision-Language-Action models achieve strong performance in robot manipulation, but often require large numbers of parameters. In this work, we propose the Future Representation Action Model (FRAM), a small policy that explicitly links the future end-effector trajectory to the current visual input. FRAM uses the image coordinates of the predicted trajectory as spatial pointers and reads local visual features related to the motion from the current image. This organizes the information for action generation into the reference position (Where), the visual state (What), and the future motion (Future). Trajectory labels are generated automatically from demonstrations and camera geometry, so no manual annotation is needed. With 138.7M parameters, including a frozen language encoder, FRAM reaches an average success rate of 92.2% over the four standard LIBERO suites, close to the 94.2% of $\pi_0$ with 3.3B parameters. Without extra training, it also reaches an average of 67.3% on LIBERO-Plus. Ablations confirm that both the future trajectory and the local visual features improve performance and robustness. On a real dual-arm UR5e, FRAM stacks cups using only wrist cameras, including choosing and switching between the left and right arms. These results show that selecting visual information based on future motion is an effective way to obtain both high performance and robustness in a small robot policy.

I. INTRODUCTION

Compact language-conditioned robot policies must select action-relevant visual information despite limited model capacity. FRAM links future end-effector motion to local visual features, achieving strong LIBERO performance with a small policy and demonstrating dual-arm cup stacking with wrist cameras.

I. INTRODUCTION

  • Image coordinates alone do not preserve enough information about object appearance and surroundings or explicitly represent the future motion path.Existing trajectory methods use trajectories as motion predictions, action representations, or instructions, but do not explicitly read current-image features along the predicted path.Source: §I · p. 1
  • FRAM predicts future end-effector trajectories and uses their image coordinates as pointers to read local visual features for action generation.The representation links reference position (Where), current visual state (What), and future motion (Future).Source: §I · p. 1
  • 92.2% average success across the four standard LIBERO suites with 138.7M parameters, close to π0’s 94.2% with 3.3B parameters.Without extra training on perturbed environments, FRAM reaches 67.3% average success on LIBERO-Plus.Source: §I · pp. 1–2
  • On a dual-arm UR5e cup-stacking task, FRAM uses wrist cameras and selects and switches between the left and right arms.The predicted trajectory is updated step by step according to the target object and placement location.Source: §I · p. 2

A. Efficient and Compact VLA

  • For compact policies, selecting and structuring action-relevant visual information matters alongside model structure and capacity allocation.The motivation is to support strong performance where the computing resources needed for fine-tuning and inference of large policies are limited.Source: §I · p. 1, A. Efficient and Compact VLA · p. 2

B. Action Generation via Trajectory and Future Representations

Prior work uses trajectories or future representations to inform action generation, but does not fully explore directly reading current-image visual state along a predicted path. FRAM ties trajectory coordinates to local image features to represent where to look, what is there, and where to move.

B. Action Generation via Trajectory and Future Representations

  • Trajectory-based methods use future paths as predictions, action representations, or instructions, while future-state methods condition action generation on subgoals or latent plans.The cited examples include ATM, MT-π, RT-Trajectory [6], CoT-VLA [15], and ThinkAct [16].Source: B. Action Generation via Trajectory… · p. 2
  • FRAM uses predicted trajectory image coordinates as pointers into the current image’s feature map, explicitly linking future motion to current visual information.This addresses an underexplored step in prior methods: directly reading visual state along the path and at the target location.Source: B. Action Generation via Trajectory… · p. 2

A. Design Concept

  • FRAM’s trajectory represents where to look and move, while features read at those positions represent the object and its surroundings.The image positions are geometrically tied to future motion, rather than relying only on a global feature compressing the whole image.Source: A. Design Concept · p. 2

B. Language and Vision Encoders

FRAM encodes language and images into language-conditioned visual features, alongside a global observation feature used for action generation.

B. Language and Vision Encoders

  • A frozen SigLIP text encoder produces language tokens and a sentence embedding, while shared ImageNet-pretrained ResNet-18 weights encode both views.The image encoder uses the intermediate stage through the third residual stage, followed by a 1×1 convolution projection.Source: B. Language and Vision Encoders · pp. 2–3
  • FiLM conditioning applies language-derived scale and shift at each spatial position to produce visual feature maps whose emphasized regions vary with the instruction.The two-view global visual features, end-effector state embedding, and language embedding are concatenated into a global observation feature for action generation.Source: B. Language and Vision Encoders · p. 3

C. Future Trajectory Prediction and Label Generation

FRAM predicts a future end-effector path and automatically generates trajectory labels from demonstrations using camera geometry. The labels encode a path sampled at equal arclength intervals and projected into each view.

C. Future Trajectory Prediction and Label Generation

  • K learnable queries in a separate Transformer decoder for each view attend to image tokens, language tokens, and an end-effector state token to regress the future trajectory.Each predicted point includes image coordinates, camera-frame depth, and a projected orientation offset.Source: C. Future Trajectory Prediction and… · p. 3
  • Trajectory labels are generated by splitting demonstrations at gripper-command switches, fitting a piecewise cubic B-spline, and sampling K points at equal arclength intervals.The spline passes through the current position, interval end, and gripper-switch positions; point orientations are interpolated from the corresponding times.Source: C. Future Trajectory Prediction and… · p. 3
  • Future end-effector poses are projected into each view using camera geometry at the current time, so wrist-camera labels require no future camera poses.For wrist views, the camera transform is computed from the current end-effector pose and the hand-eye transform; pixel coordinates are normalized to [−1, 1].Source: C. Future Trajectory Prediction and… · p. 3
  • Points behind the camera and points far from the optical axis are excluded from the loss, while out-of-image projected coordinates remain labels.Keeping out-of-image coordinates preserves information that the motion heads out of view.Source: C. Future Trajectory Prediction and… · p. 4

D. Trajectory-Guided Image Feature Extraction

FRAM uses predicted trajectory image coordinates to select current-image features along the path and around the target. These features represent the visual state associated with the predicted future motion, not a predicted future image.

D. Trajectory-Guided Image Feature Extraction

  • Predicted trajectory image coordinates point to locations on the current feature map, where FRAM samples a neighborhood by bilinear interpolation and averages its features.Coordinates outside the image are clipped to the feature-map edge.Source: D. Trajectory-Guided Image Feature… · p. 4
  • The average across trajectory points represents visual state along the path, while the endpoint feature represents the target location.The target can be the next object to grasp or the placement location.Source: D. Trajectory-Guided Image Feature… · p. 4
  • An attention layer uses the target feature as query and current-image tokens as keys and values to provide context when the predicted endpoint is slightly off target.The resulting context, path, and endpoint features are combined into the local visual feature.Source: D. Trajectory-Guided Image Feature… · p. 4
  • FRAM’s What is the current image’s visual state selected by predicted future motion, not a prediction of a future image.Source: D. Trajectory-Guided Image Feature… · p. 4

E. Training and Generation with the Action Expert

The action expert conditions action generation on global observations, trajectory information, local visual features, and language. It generates action chunks through flow matching, using trajectory tokens as cross-attention memory.

E. Training and Generation with the Action Expert

  • The action condition concatenates global observation features, an embedding of each view’s predicted trajectory, and local visual features.Source: E. Training and Generation with the… · p. 4
  • Each predicted trajectory point becomes a token with a position embedding, and action-generation steps cross-attend to these tokens.Each token retains the point’s position, depth, and orientation information.Source: E. Training and Generation with the… · p. 4
  • During training, flow matching interpolates Gaussian noise and the demonstrated action chunk, while the velocity field predicts chunk-step updates using conditioned MLP and Transformer blocks.The gripper command is generated as a continuous value whose sign determines opening or closing at execution.Source: E. Training and Generation with the… · p. 4
  • The predicted trajectory is used in training and inference, but a stop-gradient prevents action loss from backpropagating into the trajectory predictor.Auxiliary losses regress the action chunk and the trajectory’s position, shape, and projected orientation.Source: E. Training and Generation with the… · p. 4
  • At inference, the velocity field is numerically integrated from sampled Gaussian noise, and the policy repeats prediction and generation after each executed chunk.Source: E. Training and Generation with the… · p. 4

A. Setup

The evaluation uses LIBERO’s simulated manipulation tasks and tests standard-suite performance as well as transfer to perturbed environments. The setup also compares ablations that remove trajectory reference positions or the local visual features read from them.

A. Setup

  • LIBERO uses a simulated 7-DoF Franka Panda with external and wrist cameras, and its four suites test spatial relations, object identification, goal understanding, and long-horizon manipulation.Each suite contains 10 tasks.Source: A. Setup · pp. 4–5
  • Each task provides 50 successful demonstrations, split into 45 training and 5 validation episodes, with two RGB views, end-effector state, and action as inputs and outputs.The future trajectory has 16 points per view and the action chunk has 16 steps; image translations also shift trajectory labels to preserve geometric consistency.Source: A. Setup · p. 5
  • LIBERO-Plus evaluates the standard-LIBERO policy without extra training across camera, robot-state, language, lighting, background, sensor-noise, and object-layout perturbations.The evaluation covers 10,030 perturbed tasks.Source: A. Setup · p. 5
  • The ablations remove either local visual features while retaining the trajectory or the trajectory and its associated features and tokens.These comparisons isolate local-feature contribution and explicit image reference positions, including under changes to object layout, camera viewpoint, and robot initial pose.Source: A. Setup · p. 5

B. Success Rates on Standard Tasks

FRAM performs strongly across the standard LIBERO suites and is competitive with much larger models. On LIBERO-Plus, its average success rate is competitive, with performance varying by perturbation; ablations show contributions from both trajectory reference positions and local visual features.

B. Success Rates on Standard Tasks

  • 92.2% average success across standard LIBERO suites, with 92.2% Spatial, 96.2% Object, 93.8% Goal, and 86.4% Long; FRAM is on par with π0 at 94.2%.It exceeds the small models compared and is the smallest model in the table, while OpenVLA-OFT and FLOWER score 97.1% and 96.9%.Source: B. Success Rates on Standard Tasks · pp. 5–6

C. Robustness to Environment Changes

  • 67.3% average success on LIBERO-Plus, close to OpenVLA-OFT at 69.6%; FRAM reaches 68.6% for camera viewpoint and 40.2% for robot initial state, but 68.5% on background versus OpenVLA-OFT’s 93.3%.The examples show predicted trajectories following targets under background, layout, viewpoint, and robot-initial-state changes.Source: C. Robustness to Environment Changes · p. 6, Table II · p. 6

D. Ablation

  • Removing local features or the trajectory lowers standard-suite average success from 92.2% to 88.4% or 87.7%, and LIBERO-Plus success from 67.3% to 57.8% or 50.8%.Both ablations score below FRAM on all seven perturbations; examples show FRAM updating its trajectory from grasp target to placement while both ablations fail.Source: D. Ablation · p. 6

V. REAL-ROBOT EXPERIMENTS

The real-robot experiment evaluates FRAM on dual-arm UR5e cup stacking using only wrist cameras. The policy completes runs with either arm and switches between arms according to cup layout and task progress.

A. Setup

  • The real setup uses a dual-arm UR5e with two wrist cameras and no external camera.Each arm has 6 DoF and a parallel gripper; four differently sized and colored cups are stacked in a fixed order.Source: A. Setup · p. 6
  • Demonstrations comprise 62 teleoperated episodes, and the policy must select both the next cup and which arm to move.The episodes total about 44 minutes and 79,669 frames; 44 use both arms, 8 only the left, and 10 only the right.Source: A. Setup · p. 6

B. Results

  • In left-only and right-only runs, trajectories shift between the next cup to grasp and the base cup for placement as stacking progresses.The same trajectory prediction and local-feature reading occur when the policy uses either arm.Source: B. Results · p. 6
  • The figure distinguishes left-arm, right-arm, and both-arm runs; its white line marks the predicted trajectory, star the endpoint, and yellow frame the moving arm.In the both-arm panel, c1 is the left wrist view and c2 the right wrist view.Source: Figure 4 · p. 7
  • In mixed-arm runs, each arm can predict a later target while waiting, then move after the other arm finishes its preceding operation.For example, the waiting left arm points toward the blue cup while the right arm handles orange; the waiting right arm later points toward yellow.Source: B. Results · p. 7
  • A single policy stacks cups by choosing, waiting, and switching between the left and right arms while updating trajectories toward current and subsequent targets.With cups spread across both sides, it uses the sequence right arm, left arm, right arm.Source: B. Results · pp. 6–8

VI. CONCLUSION

FRAM uses predicted future trajectories as spatial pointers to connect current visual features with action generation. It achieves strong LIBERO results with a compact policy, while long-horizon performance and broader real-robot validation remain future work.

VI. CONCLUSION

  • FRAM uses future end-effector trajectories as image-coordinate pointers to read local visual features, linking reference position, current visual state, and future motion.Trajectory labels come automatically from demonstrations and camera geometry, requiring no manual annotation.Source: §VI · p. 8
  • FRAM reached 92.2% average success across the four standard LIBERO suites with 138.7M parameters, matching models several to tens of times larger.The policy uses a frozen language encoder; without extra training on perturbed environments, it reached 67.3% on LIBERO-Plus.Source: §VI · p. 8
  • The real-robot evaluation is limited to cup stacking, and the authors identify long-horizon manipulation and broader real-task validation as future work.They plan to test different objects, tasks, environment conditions, and robot configurations.Source: §VI · p. 8

Related papers

Loading 2609.30965v2…