PaperlayerView exact revision on arXiv

Source-linked AI summary

Bridging Perception and Action: Spatially-Grounded Mid-Level Representations for Robot Generalization

Jonathan Yang, Chuyuan Kelly Fu, Dhruv Shah, Dorsa Sadigh, Fei Xia, Tingnan Zhang

arXiv:2506.06196v1cs.RO

TL;DR

Dexterous robot policies may not generalize when grounding remains high-level or fails to match task-specific spatial demands. The paper conditions a diffusion policy on specialized mid-level representations and reports higher success than standard diffusion and language-representation baselines. It also finds that weighting imitation by representation alignment improves success over standard training.

  • Problem

    Existing forms of grounding may not transfer from simpler tasks to dexterous tasks, while useful representations vary with task-specific spatial and interaction demands.

    Source: §I · p. 2

  • Method

    The approach conditions a diffusion policy on outputs from multiple specialists, each providing a distinct mid-level representation, and uses representation alignment to weight imitation-learning data.

    Source: §I · p. 2, §III · p. 3

  • Results

    Mid-Level MoE achieves 24% higher success than a standard diffusion policy baseline and 11% higher success than a language-representation baseline across bimanual dexterous tasks.

    Source: §I · p. 2

  • Takeaways & Limitations

    Different mid-level representations excel at different tasks, and combining their task-specific benefits achieves higher performance across a wide range of environments.

    Source: §VIII · p. 10

    Policies that rely heavily on structured representations can be more susceptible to overfitting and reduced robustness in noisy environments.

    Source: §I · p. 2

Abstract

from arXiv · show

In this work, we investigate how spatially grounded auxiliary representations can provide both broad, high-level grounding as well as direct, actionable information to improve policy learning performance and generalization for dexterous tasks. We study these mid-level representations across three critical dimensions: object-centricity, pose-awareness, and depth-awareness. We use these interpretable mid-level representations to train specialist encoders via supervised learning, then feed them as inputs to a diffusion policy to solve dexterous bimanual manipulation tasks in the real world. We propose a novel mixture-of-experts policy architecture that combines multiple specialized expert models, each trained on a distinct mid-level representation, to improve policy generalization. This method achieves an average success rate that is 11% higher than a language-grounded baseline and 24 percent higher than a standard diffusion policy baseline on our evaluation tasks. Furthermore, we find that leveraging mid-level representations as supervision signals for policy actions within a weighted imitation learning algorithm improves the precision with which the policy follows these representations, yielding an additional performance increase of 10%. Our findings highlight the importance of grounding robot policies not only with broad perceptual tasks but also with more granular, actionable representations. For further information and videos, please visit https://mid-level-moe.github.io.

I. INTRODUCTION

The paper argues that grounding for dexterous manipulation must combine spatially informative, actionable representations with broad generalization. It studies task-specific mid-level representations and combines specialists to improve policy performance.

I. INTRODUCTION

  • Existing grounding can support generalization, but its benefits for simpler tasks may not transfer to dexterous tasks requiring more object interaction.The paper motivates representations that balance high-level abstraction with actionable detail across task-specific requirements.Source: §I · pp. 1–2

III. SPATIALLY-GROUNDED MID-LEVEL REPRESENTATIONS

  • The study examines object-centric, motion-centric, depth-aware, and pose-aware representations, whose usefulness depends on task requirements.These representations address object properties and interaction points, dynamics, three-dimensional structure, and object or robot pose.Source: §I · p. 2, §III · p. 3

I. INTRODUCTION

  • Mid-Level MoE achieves 24% higher success than a standard diffusion policy baseline and 11% higher success than a language-representation baseline across bimanual dexterous tasks.The comparisons are reported across a series of tasks.Source: §I · p. 2
  • Weighted imitation learning yields a 10% higher success rate than standard training while improving reliance on mid-level representations and preserving robustness to perturbations.It weights demonstrations according to their alignment with the mid-level representations.Source: §I · p. 2

II. RELATED WORKS

  • Prior grounding approaches include auxiliary tasks and explicit representations, but many do not capture the spatial and contextual detail needed for dexterous manipulation.The paper positions spatial mid-level representations between high-level inputs and robot actions.Source: §II · pp. 2–3

III. SPATIALLY-GROUNDED MID-LEVEL REPRESENTATIONS

  • The policy combines state input with outputs from multiple experts, each providing a distinct type of mid-level grounding.The policy is trained to leverage these representations across diverse scenarios.Source: §I · p. 2, §III · p. 3

IV. THE SENSITIVITY-ROBUSTNESS TRADEOFF

The section frames mid-level representations as a bridge between perception and action, with policy adherence and resilience to representation errors as distinct concerns. It defines metrics for these properties and describes the diffusion-policy implementation.

IV. THE SENSITIVITY-ROBUSTNESS TRADEOFF

  • Policies face a tradeoff between following mid-level representations and remaining robust when those representations contain errors.The representations are described as a bridge between sensory inputs and lower-level joint actions.Source: §IV · pp. 3–4
  • The Sensitivity Score measures adherence to supplied representations; a lower score indicates closer following, while a higher score indicates less reliance.Adherence measures alignment between a policy trajectory and a particular representation.Source: §IV · p. 4
  • The Robustness Index averages performance ratios under Gaussian perturbations relative to performance with unperturbed representations; higher values indicate greater resilience.The perturbations are applied to the mid-level representations.Source: §IV · p. 4

V. ARCHITECTURE

  • The diffusion policy encodes four viewpoints with separate ResNet50 encoders, then uses a transformer to predict ten action chunks simultaneously.The four images comprise two third-person views and two wrist views; actions include joint positions and gripper values.Source: §V · p. 4

A. Mid-level Experts

The architecture combines four specialized mid-level experts with image features to balance task-relevant grounding against robustness to noisy signals. Experts are trained separately, then frozen while the policy is trained; self-consistency weights emphasize reliable expert guidance.

A. Mid-level Experts

  • The policy is designed to balance responsiveness to relevant representations with robustness when those representations are noisy.The architecture uses multiple experts and attention-based integration to manage this sensitivity-robustness tradeoff.Source: A. Mid-level Experts · pp. 4–5
  • Four experts encode object-centric, motion-centric, pose-aware, and depth-aware information, with attention and cross-attention integrating their outputs with image features.The design dynamically weights expert contributions using the current state and visual context.Source: A. Mid-level Experts · pp. 4–5, Figure 4 · p. 5

B. Training

  • Experts are trained separately on representation-specific data relabeled from demonstrations and proprioception, then frozen while the policy is trained end-to-end.Motion-centric labels use future arm trajectories projected into the observation frame; object and pose representations use OWL-ViT and retrained ResNet34 models, respectively.Source: B. Training · p. 5

C. Self-Consistency

  • Standard behavioral cloning can produce inconsistent actions when mid-level predictions are noisy or partly incorrect, degrading use of expert signals.Self-consistency weights emphasize reliable guidance, reflecting how well expert outputs align with ground truth or improve task success.Source: C. Self-Consistency · p. 6
  • The self-consistency objective weights behavioral-cloning loss to promote alignment between policy actions and mid-level expert outputs.The method iteratively refines training data and adjusts weights for consistent samples.Source: C. Self-Consistency · p. 6

VI. EXPERIMENTS

The experiments assess whether spatial grounding improves performance across varied tasks and environments, including the use of multiple representations and robustness to noise. The simulation evaluation uses a bimanual workcell observed from four camera views.

VI. EXPERIMENTS

  • The experiments ask whether spatial representations improve generalization, which tasks benefit from each representation, and how multiple signals and architectures affect performance and robustness.They also assess whether these representations can serve as training supervision for precision and generalization.Source: §VI · p. 6

A. Simulation Environment

  • Simulation uses the Aloha Unleashed bimanual parallel-jaw workcell, with overhead, robot-facing, and two wrist-camera views.The evaluation includes tasks such as peg insertion, block assembly, fruit arrangement, and pen handover.Source: A. Simulation Environment · pp. 6–7

B. Real-World Environment

The evaluation spans real-world bimanual tasks and compares representations, baselines, and policy architectures. Mid-Level MoE performs strongly across tasks, while individual representation benefits vary with task requirements.

B. Real-World Environment

  • The real-world evaluation uses two 6-DoF ViperX arms with parallel-jaw grippers and includes kitchen stacking, cup stacking, shirt hanging, and shoelace tying.The observation perspectives match those used in simulation.Source: B. Real-World Environment · p. 7

C. Experiment Setup

  • The ablations compare single experts, MOKA-style keypoints, and an RT-H-style language baseline under varied object locations, distractors, and lighting.The keypoint baseline queries a vision-language model for image points of interest; RT-H demonstrations are relabeled with directional arm commands.Source: C. Experiment Setup · p. 7

A. Mid-level Representations have Task-Specific Benefits

  • Representation ablations outperform the no-representation baseline, but the best representation varies by task: motion for Single Insertion and Cup Stack, object-centric for Fruit Bowl and Kitchen Stack, pose-aware for FMB and Shirt Hang, and depth-aware for Pen Handover and Shoelace Tying.For ground-truth representations, trajectory traces reach 94% versus 85% for object-centric representations in Single Insertion; Grasp Plans reach 67% in FMB.Source: A. Mid-level Representations have… · pp. 7–8

B. Mid-Level MoE can effectively utilize multiple different

  • 69.6% average success across tasks for Mid-Level MoE compares with 45.1% for no representations, 51.5% for language, and 58% for keypoints.The method also achieves 24% higher success than the standard diffusion policy baseline.Source: B. Mid-Level MoE can effectively… · p. 8, Figure 7 · p. 8
  • Mid-Level MoE remains consistently strong across tasks, although it does not always outperform the best single representation on an individual task.Bounding boxes outperform it on Kitchen Stack, while language and keypoint baselines each do well on some tasks without consistent gains across tasks.Source: B. Mid-Level MoE can effectively… · p. 8

C. Different Architectures offer Different Tradeoffs between Sensitivity and Robustness

Mid-Level MoE outperforms alternative fusion architectures while balancing sensitivity to mid-level representations and robustness to noise. Weighted imitation learning improves representation adherence but introduces a robustness trade-off, and its scaling depends on task-specific metrics.

C. Different Architectures offer Different Tradeoffs between Sensitivity and Robustness

  • Mid-Level MoE outperforms early fusion by an average of 11% and concatenation by 7.5%, with lower sensitivity and higher robustness than concatenation.Its average sensitivity score is 0.2175, versus 0.2575 for concatenation and 0.315 for early fusion; its robustness index is 0.86, versus 0.75 for concatenation.Source: C. Different Architectures offer… · p. 9

D. Self-Consistency Leads to Higher Sensitivity Scores

  • Weighted imitation learning yields an average sensitivity score 25.5% lower than unweighted approaches across three architectures.The authors interpret this as more precise focus on pertinent features, potentially reducing noisy or irrelevant information.Source: D. Self-Consistency Leads to Higher… · pp. 9–10
  • Weighted imitation learning also leads to a 9% lower average robustness index, and self-consistency metrics require extensive manual tuning or automated discovery to scale beyond evaluated tasks.The robustness decrease may compromise performance in dynamic or unpredictable environments.Source: D. Self-Consistency Leads to Higher… · p. 10
  • Weighted Mid-Level MoE averages 10% higher performance than Mid-Level MoE across four real-world tasks, while weighted concatenation averages 26% lower performance than its unweighted version.The passage attributes the concatenated architecture’s difficulty to noise and spurious correlations alongside a robustness index of 0.51.Source: D. Self-Consistency Leads to Higher… · p. 10

VIII. CONCLUSION

  • The conclusion frames representation effectiveness as task-dependent and identifies sensitivity versus robustness as a central trade-off in grounding dexterous, bimanual policies.It reports that weighted imitation learning reduces sensitivity while preserving a high level of robustness.Source: §VIII · p. 10

APPENDIX

The appendix describes spatial and motion representations through normalized coordinates, object points, and trajectories. The supplied passages also identify language commands and distinguish 2D from depth-aware trajectories.

A. Representations

  • Bounding Box 2D uses normalized image-space corner coordinates, while Bounding Box 3D uses normalized coordinates for box corners.The 2D coordinates are randomized before concatenation into a tensor.Source: A. Representations · p. 13
  • Keypoints denote points of interest on an object, and Trajectory Trace 2D represents an object's path over time as image-space points.The 2D trajectory trace is concatenated into a single tensor.Source: A. Representations · p. 13
  • A depth-aware trajectory trace uses normalized image coordinates with depth, while the language representation specifies future movement with directional commands.The listed commands are “left,” “right,” “up,” and “down” relative to image space.Source: A. Representations · p. 13

B. Sensitivity Metrics

Sensitivity metrics evaluate how closely achieved robot trajectories follow mid-level representations. The supplied descriptions define adherence and distinguish grasp-plan, bounding-box, and trajectory-based measures.

B. Sensitivity Metrics

  • Sensitivity metrics evaluate how closely an achieved robot trajectory follows its representation, with lower adherence indicating closer following.The metric set includes representation-specific measures for grasp plans, bounding boxes, and trajectory traces.Source: B. Sensitivity Metrics · p. 13
  • Bounding-box adherence measures the minimum Euclidean distance between achieved trajectory points and the representation's box corners.The supplied description defines distance using the minimum distance to any of the four corners.Source: B. Sensitivity Metrics · p. 13
  • Grasp-plan adherence combines the minimum pixel-space distance to the planned grasp location with the minimum geodesic distance to its planned orientation.The planned grasp location is expressed relative to the bounding-box centroid, then converted to an absolute pixel location.Source: B. Sensitivity Metrics · p. 13
  • Trajectory Trace 2D adherence averages distances between corresponding points, while the depth-aware metric combines spatial distance and absolute depth difference.A weighting factor λ balances depth and spatial adherence, and smaller values indicate closer alignment in both.Source: B. Sensitivity Metrics · pp. 13–14

C. Self-Consistency Weights

The weighting procedure converts representation adherence into weights for imitation learning, using adherence-based pseudo-rewards in a Reward-Weighted Regression interpretation. The passages also report that scale-based demonstration weighting improves success rate over setting λ to 1.

C. Self-Consistency Weights

  • Adherence-based weights approach −1 when a policy closely follows a representation and approach 0 when adherence is large.The total weight averages adherence across experts before applying the exponential.Source: C. Self-Consistency Weights · p. 14
  • In the Reward-Weighted Regression interpretation, the pseudo-reward is the negative adherence error averaged across mid-level experts.Weighting each behavior-cloning loss term is equivalent, up to a constant sign, to weighting by exp(R(s, a)).Source: C. Self-Consistency Weights · p. 14
  • Weighting demonstrations by scale leads to a higher success-rate improvement than setting λ equal to 1.Source: Figure 10 · p. 14
  • The coefficients λE determine how much each expert's adherence contributes to the combined pseudo-reward, and are set inversely proportional to maximum adherence.The passage says this choice avoids imposing a hand-crafted prior that one distance metric matters more than another.Source: C. Self-Consistency Weights · p. 14

D. Architecture Details

The section identifies a table of model architecture details.

D. Architecture Details

E. Dataset

The dataset uses teleoperated demonstrations collected in simulation and the real world.

E. Dataset

  • The dataset was collected through teleoperation with an ALOHA system in simulation and the real world.ViperX robots transfer the operator’s joint information to the robot.Source: E. Dataset · p. 14
  • Simulation datasets contain around 1,000 trajectories per task, compared with around 500 trajectories per task in the real world.Source: E. Dataset · p. 14
  • Figure 11 presents comprehensive simulation results.Source: Figure 11 · p. 15

F. Ablation: Image-Space versus Lower-Level Embedding

The section reports that lower-level embeddings perform slightly better than image-space embeddings, presents comprehensive representation results, and evaluates zero-shot performance on novel cups and a marker.

F. Ablation: Image-Space versus Lower-Level Embedding

  • Across tasks, lower-level embeddings performed slightly better than image-space embeddings; the proposed explanation is that fine geometric and kinematic information may be lost in image-space embeddings.Table VI compares representations across tasks, but the supplied passages do not report its cell values.Source: F. Ablation: Image-Space versus… · p. 15, Table VI · p. 15

G. Comprehensive Results

  • Figures 11 and 12 show comprehensive simulation and real-world results, with representations color-coded by object-, motion-, pose-, and depth-centric axes.Averaging success rates by color yielded consistent results for the representation types that benefit each task.Source: G. Comprehensive Results · p. 15, Figure 12 · p. 16

Zero-Shot Generalization Results

  • 85% success on Kitchen Stack and 90% on Pen Handover were achieved across 20 real-world trials per task without additional demonstrations or parameter tuning.The trials varied object geometry, color, and surrounding clutter; Figure 13 depicts the two tasks.Source: Zero-Shot Generalization Results · p. 15, Figure 13 · p. 16
  • The depth-trace expert maintained a stacking trajectory for cups with rim diameters varying by up to 5 mm, while the bounding-box and orientation experts handled a marker 30% thicker than training pens.The reported expert outputs supported adaptation without retraining.Source: Zero-Shot Generalization Results · pp. 15–16

Related papers

Loading 2506.06196v1…