PaperlayerView exact revision on arXiv

Source-linked AI summary

Bi-MoDe: Bilateral Control-based Imitation Learning via Modifier-Conditioned Decoding for Modulation of Execution Speed and Contact Intensity

Takumi Kobayashi, Masato Kobayashi, Yuki Uranishi

arXiv:2609.16040v1cs.RO

TL;DR

Bilateral imitation learning captures position and force, but operators have limited ways to specify how a learned task should be executed at inference time. Bi-MoDe conditions every Transformer action-decoder layer on a constrained latent; on real-world wiping, it improves physical directive following while maintaining comparable temporal control.

  • Problem

    Existing approaches provide limited means for operators to specify execution speed and contact intensity at inference time.

    Source: §I · p. 1

  • Method

    Bi-MoDe injects a constrained latent aligned with modifier directives into every Transformer action-decoder layer via adaLN-Zero.

    Source: §I · p. 2

  • Results

    Bi-MoDe reached a 0.968 slope ratio versus Baseline's 0.896 and a 0.036 MDE versus 0.123, while temporal directive following remained comparable.

    Source: 2) Physical directive following · p. 6, 3) Temporal directive following · p. 8, §V · p. 8

  • Takeaways & Limitations

    A 2 × 2 ablation showed that decoder conditioning and latent-space composition jointly contribute to physical directive following.

    Source: §I · p. 2, §V · p. 8

    The evaluation covers one contact-rich task and only directive levels represented in the training data.

    Source: §V · p. 8

Abstract

from arXiv · show

Bilateral control-based imitation learning captures both position and force information, making it well suited to contact-rich manipulation. However, existing approaches provide limited means for an operator to specify how a learned task should be executed at inference time, such as slowly or quickly, gently or firmly. We propose Bi-MoDe, a modifier-conditioned decoding framework that injects a constrained latent into every layer of the Transformer action decoder via adaLN-Zero, allowing behavioral directives to directly influence action-chunk generation. We evaluate the method on a real-world whiteboard wiping task with combinations of temporal and physical modifiers. Bi-MoDe improves physical directive following over the action-chunking baseline while maintaining comparable temporal control. An ablation further shows that decoder conditioning and latent-space composition interact, and that their combination is important for accurate physical directive following. Additional material is available at the https://mertcookimg.github.io/bi-mode/

I. INTRODUCTION

Bi-MoDe addresses inference-time control over how a learned manipulation task is executed, conditioning action-chunk generation on operator-specified behavioral modifiers. It is evaluated on real-world wiping, with an ablation indicating that decoder conditioning and latent-space composition jointly improve physical directive following.

I. INTRODUCTION

  • The paper targets inference-time control over execution style, since the same wiping task may call for different speeds and contact forces.The stated challenge is making behavioral variability something an operator can specify after policy learning.Source: §I · p. 1
  • Prior modifier conditioning acted only at the encoder, leaving the action-chunk decoder to recover directives from a weakly constrained latent.The paper identifies this as a conditioning-design limitation, especially for behavior that must persist across an action chunk.Source: §I · pp. 1–2
  • Bi-MoDe injects the constrained latent into every Transformer action-decoder layer via adaLN-Zero, directly conditioning action-chunk generation.This lets temporal and physical directives influence execution at inference time.Source: §I · p. 2
  • Bi-MoDe demonstrates inference-time modulation of execution speed and contact intensity on a real-world contact-rich wiping task.Source: §I · p. 2
  • A 2 × 2 ablation shows that decoder conditioning and latent-space composition jointly improve physical directive following.Source: §I · p. 2

II. RELATED WORK

Related work spans bilateral-control imitation learning and behavior-level conditioning. Bi-MoDe distinguishes itself by exposing both temporal and physical characteristics as inference-time commands and propagating modifier information through the action decoder.

A. Bilateral Control-based Imitation Learning

  • Bilateral control-based imitation learning captures position and force information for contact-rich manipulation, while prior speed approaches encoded behavior in demonstrations or trajectories before training.Bi-MoDe instead enables temporal and physical characteristics to be specified at inference time through action-decoder conditioning.Source: A. Bilateral Control-based Imitation… · p. 2

B. Behavior-Level Conditioning of Manipulation Policies

  • Natural-language conditioning can modulate force but is ambiguous for continuous quantities, while scalar approaches separately address speed or autonomous physical interaction.The cited approaches do not directly provide operator-specified control over both temporal and physical characteristics.Source: B. Behavior-Level Conditioning of… · p. 2
  • Modifier directives align constrained latent dimensions with temporal and physical labels, but the action-chunking variant followed directives less reliably than its LSTM counterpart.Bi-MoDe retains modifier directives while propagating the constrained latent directly through the action decoder.Source: B. Behavior-Level Conditioning of… · pp. 2–3

III. METHOD

Bi-MoDe extends a Transformer-driven CVAE with modifier-conditioned decoding, propagating the constrained latent through each decoder layer. Its data collection uses bilateral teleoperation and paired scalar labels for intended speed and contact force.

A. Overview

  • Bi-MoDe extends the modifier-based action-chunking Transformer with adaLN-Zero, propagating constrained latent zc into every decoder layer as well as the shared encoder token.This lets modifier directives directly influence trajectory generation.Source: A. Overview · p. 3

B. Data Collection

  • Data collection uses four-channel bilateral control to synchronize positional and force information between the operator and robot.Joint angles are measured by encoders, while torque responses are estimated with a disturbance observer and a reaction force observer rather than dedicated force/torque sensors.Source: B. Data Collection · p. 3
  • Before each demonstration, the operator assigns temporal and physical labels from {0.0, 0.5, 1.0} for intended speed and contact force.The labels are recorded with joint-state trajectories and assigned before the trial.Source: B. Data Collection · p. 3

C. Learning Model

Bi-MoDe builds on a Transformer CVAE with constrained and unconstrained latent components, then gives the constrained latent direct access to every decoder layer. Zero-initialized modulation preserves encoder-side conditioning at the start of training, while the model is trained with reconstruction, KL, and modifier-prediction losses.

C. Learning Model

  • The CVAE encoder infers unconstrained latent zu during training, but Bi-MoDe excludes it from the latent token supplied to the Transformer.A weakly supervised head aligns zc with modifier labels; zu remains KL-regularized during training.Source: C. Learning Model · p. 3
  • The model predicts leader-robot action chunks from follower-robot joint angles, velocities, and torques.The encoder, modifier prediction head, and decoder are optimized under the combined objective.Source: C. Learning Model · pp. 3–4
  • Bi-MoDe appends constrained latent zc to the encoder input and uses it to modulate every decoder layer through adaLN-Zero.The modulation covers self-attention, cross-attention, and feedforward sub-layers.Source: C. Learning Model · p. 3
  • Zero initialization makes direct adaLN modulation vanish initially while preserving encoder-memory conditioning and unit residual gates.The (1 + αi) gates allow observation-dependent information to pass through decoder branches from initialization.Source: C. Learning Model · p. 4
  • Training uses ℓ1 action-chunk reconstruction, KL regularization, and binary-cross-entropy modifier prediction across S = 2 constrained latent dimensions.Modifier labels take values 0.0, 0.5, or 1.0.Source: C. Learning Model · p. 4

D. Inference

At inference, the operator selects directive levels that map to fixed constrained-latent commands, and the model uses follower-robot state to predict the next leader-robot action chunk.

D. Inference

  • The operator specifies behavior by selecting a directive level on each axis, mapped to a fixed zc value derived from training demonstrations.For each label level, the command is the median constrained latent from demonstrations assigned that level.Source: D. Inference · p. 4
  • Each model has its own latent commands, whose signs are arbitrary, but each directive level uses the same command on every rollout..Source: D. Inference · p. 4
  • Using the follower robot’s most recent joint data, Bi-MoDe predicts the leader robot’s subsequent action chunk with directives propagated through every decoder layer..Source: D. Inference · p. 4

A. Hardware

The study uses bilateral OpenManipulator-X robots for repeated whiteboard wiping, with joint torques and motion serving as measurable indicators of the physical and temporal directives.

A. Hardware

  • The OpenManipulator-X setup uses leader and follower arms, with Joint2 and Joint3 bearing the load when the cleaner is pressed against the board.Their torque is the physical feature used in evaluation.Source: A. Hardware · p. 4

B. Task Setting

  • The follower holds a cleaner against the whiteboard and sweeps end-to-end and back for three repeated strokes..Source: B. Task Setting · p. 4
  • At equal elapsed times, fast, moderate, and slow demonstrations complete three, one and a half, and one stroke over six seconds.The arm follows nearly the same path across physical levels, while Joint2 and Joint3 torques separate by commanded force.Source: Figure 5 · p. 5, B. Task Setting · p. 5
  • The evaluation covers nine temporal-by-physical modifier conditions, with five demonstrations and five rollouts per condition.This yields 45 demonstrations and 45 rollouts per configuration; success requires completing all three strokes.Source: B. Task Setting · p. 5

C. Training Setup

Training uses proprioceptive joint data and evaluates four configurations in a 2×2 ablation of decoder conditioning and unconstrained-latent inclusion.

C. Training Setup

  • Both robots’ joint angles, angular velocities, and torques are recorded at 1000 Hz, while models use proprioceptive state alone as input.Each robot’s joint data are 15-dimensional, and each model’s inference commands come from its own latent space.Source: C. Training Setup · p. 5

D. Ablation Design

  • The 2×2 ablation varies whether zc modulates every decoder sub-layer and whether zu is included in the encoder’s latent token.All four configurations share an encoder modifier token, latent structure, and training objective.Source: D. Ablation Design · pp. 5–6
  • The unconstrained style latent zu is sampled from the posterior during training but fixed to the prior mean at inference, creating a phase mismatch through the encoder output.This latent component follows the conditional variational formulation of ACT.Source: D. Ablation Design · p. 6
  • The action-chunking baseline disables adaLN-Zero and includes zu, whereas Bi-MoDe enables adaLN-Zero and removes zu.The other two configurations apply one of these changes in isolation.Source: D. Ablation Design · p. 6

E. Evaluation Metrics

The evaluation measures directive fidelity by comparing fitted generated-behavior lines with demonstration reference lines. It also reports how strongly the generated behavior's slope reflects the commanded modifier.

E. Evaluation Metrics

  • MDE is the coefficient-space distance between the generated line y = cx + d and reference line y = ax+b, with both terms normalized by R.Lower MDE indicates closer agreement between the intended directive and generated motion.Source: E. Evaluation Metrics · p. 6
  • Trials are grouped at modifier levels x ∈ {0.0, 0.5, 1.0}, with 15 trials per group, and unweighted least squares fits lines to the group means and demonstrations.Source: E. Evaluation Metrics · p. 6
  • R is the range of individual reference features pooled across all three levels, measured as 4.643 s temporally and 0.830 Nm physically.It is computed from individual trials rather than group means to reflect the full spread of demonstrated behavior.Source: E. Evaluation Metrics · p. 6
  • Using R avoids the physical-axis sensitivity of the original coefficient-wise normalization, where a near-zero reference intercept dominates the metric.For Bi-MoDe, the original intercept term contributes 0.875 versus 0.032 from the slope term.Source: E. Evaluation Metrics · p. 6
  • The slope ratio c/a isolates the slope component: 1.0 matches the demonstrations' dynamic range, while a value near 0 indicates the directive is effectively ignored.Source: E. Evaluation Metrics · p. 6

F. Results

All configurations completed the wiping task, while Bi-MoDe's profiles show temporal control comparable to the alternatives and physical directive following close to the demonstrations. The ablation indicates that decoder conditioning and latent-space composition work together for stronger physical following.

1) Overall:

  • Bi-MoDe's physical slope ratio is 0.968, and its force-axis separation follows the demonstrations; all configurations completed the task, with TSR 45/45.Fig. 7 plots realized features against commanded levels; Fig. 8 shows duration-separated angle traces and contact-phase torque separated by magnitude.Source: 1) Overall · p. 6, Figure 7 · p. 7, Figure 8 · p. 7

2) Physical directive following:

  • On the physical axis, slope ratio was 0.896 for Baseline, 0.754 without z_u, 0.854 with adaLN-Zero retaining z_u, and 0.968 for Bi-MoDe.Source: 2) Physical directive following · p. 6
  • Physical-axis MDE was 0.123 for Baseline, 0.300 without z_u, 0.139 with adaLN-Zero retaining z_u, and 0.036 for Bi-MoDe.Source: 2) Physical directive following · p. 6
  • The two metrics agree that the combined design improves physical following, and the pattern is not an artifact of error normalization.Source: 2) Physical directive following · pp. 6–7
  • The authors attribute the interaction to z_u's training–inference mismatch: once the decoder receives the directive directly, removing z_u improves following, while neither change alone helps.During training z_u is sampled from the posterior; at inference it is fixed to the prior mean.Source: 2) Physical directive following · p. 7

3) Temporal directive following:

Bi-MoDe improved physical directive following while maintaining comparable temporal modulation, and all four configurations tracked temporal directives closely. The evaluation was limited to one contact-rich task and directive levels represented in training.

3) Temporal directive following:

V. CONCLUSIONS

  • Bi-MoDe improved physical directive following while maintaining comparable temporal modulation; decoder conditioning and latent-space composition jointly contribute to physical following.Source: §V · p. 8
  • Evaluation is limited to a single contact-rich task and directive levels represented in the training data.Source: §V · p. 8
Loading 2609.16040v1…