PaperlayerView exact revision on arXiv

Source-linked AI summary

MorphAct: Robot Policies Morphing with the Worlds They Act In

Ziyan Liu, Jiting Liu, Weihong Zhong, Wenda Yu, Yeqiu Chen, Jason Li, Xiaojiang Peng, Shuyuan Hu, Kai Wang, Bo Zhao

arXiv:2610.20019v1cs.ROcs.AI

TL;DR

Conventional robot policies keep effective parameters fixed across interaction contexts, while different operations can favor competing updates. MorphAct uses coupled, context-dependent parameter generators trained through the native action objective. It improves manipulation performance across three VLA backbones, with Meta-World average-success gains of 7.57–16.30%.

  • Problem

    Conventional VLA policies use shared, static effective parameters across diverse interactions, even when different operations favor competing updates.

    Source: §1 · pp. 1–2

  • Method

    MorphAct uses coupled generators to produce context-dependent low-rank updates for vision–language and action modules, trained jointly through the native action objective.

    Source: §1 · p. 2

  • Results

    MorphAct improves Meta-World average success by 7.57–16.30% across three VLA backbones and also improves results on LIBERO and four real-world tasks.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Evaluations and ablations support contextual generation, adaptation of both modules, and coupling through adapted vision–language features for closed-loop manipulation.

    Source: §7 · p. 10

Abstract

from arXiv · show

Progress toward general intelligence in the physical world requires scaling robot learning across diverse environments and interactions. However, existing robot policies use shared parameters that remain fixed across interaction contexts. When different operations favor competing parameter adjustments, fitting their mixture can leave individual operations insufficiently adapted. Our atomic-task diagnostics reveal competing update directions in both vision-language and action parameters, with source-improving updates increasing loss on other tasks. This motivates MorphAct, a contextual parameterization framework that dynamically adjusts robot policy parameters as interaction contexts change. Two coupled generators produce context-dependent low-rank updates for the vision-language and action modules. Their coupling follows the policy's computational dependency: vision-language updates shape the representations that condition action-side parameter generation. Both generators are trained jointly through the native action objective, without operation labels or expert-adapter targets. MorphAct improves performance across three VLA backbones on LIBERO and Meta-World, with gains of up to 16.30%. In real-world evaluation, it achieves gains of up to 14% over $\pi_{0.5}$. Ablations support the benefits of contextual parameter generation and representation coupling.

1 INTRODUCTION

Robot policies must accommodate differing perceptual and control demands across environments and even within one interaction, but conventional policies keep their effective weights fixed across contexts. MorphAct generates context-dependent updates for vision–language and action modules, and evaluations report gains across simulation benchmarks and real-world tasks.

1 INTRODUCTION

  • Diverse operations within a single interaction can require different visual cues and control behaviors even under an unchanged instruction.The introduction frames accommodating these heterogeneous demands as a challenge for shared models.Source: §1 · p. 1
  • Operation-level diagnostics find competing update directions in both modules, with every measured source-improving update increasing loss on batches from other tasks.Opposing parameter-group contributions can also cancel within a module.Source: §1 · p. 2
  • MorphAct generates context-dependent low-rank updates for vision–language and action modules, coupling action-side generation to adapted representations and training both generators through the native action objective.Training uses no operation labels or expert-adapter targets.Source: §1 · p. 2
  • Meta-World average success improves by 7.57–16.30% across three VLA backbones, with gains also reported on LIBERO and four real-world tasks.On LIBERO, X-VLA, π0, and π0.5 gain 4.15%, 2.50%, and 1.75%, respectively; real-world gains with π0.5 are 8.00–14.00%.Source: §1 · p. 2

2 PARAMETER CONFLICTS ACROSS OPERATIONS

At a common policy checkpoint, operation-level diagnostics examine gradient alignment, cross-task loss changes, and parameter-group contributions in the vision–language and action modules. They find competing update demands both between tasks and among parameter groups within modules.

2 PARAMETER CONFLICTS ACROSS OPERATIONS

  • The diagnostics use 65 RoboCasa365 atomic tasks and a fixed RoboCasa-pretrained π0.5 checkpoint, measuring gradient alignment, one-step cross-task loss changes, and parameter-group contributions.Measurements use disjoint episodes for source and target batches.Source: §2 · pp. 2–3

2.1 CONFLICTING DIRECTIONS IN BOTH MODULES

  • Negative gradient alignment occurs in 47.80% of cross-task VLM comparisons and 48.28% of action comparisons, versus 38.94% and 42.69% within tasks.The within-task reference shows update directions also vary among episodes of the same task.Source: §2.1 · p. 3, Figure 2 · p. 3

2.2 CROSS-TASK INTERFERENCE FROM PARAMETER UPDATES

  • Each of 260 source updates reduces source-batch loss but increases loss on batches from 22–46 of the other 64 tasks, with a median of 32.The mean relative source-loss reduction is 2.38%.Source: §2.2 · p. 3, Figure 2 · p. 3

2.3 CONFLICTING CONTRIBUTIONS ACROSS PARAMETER GROUPS

  • Cross-task parameter-group contributions cancel by 36.13% in the VLM and 30.90% in the action module, compared with 23.51% and 22.70% within tasks.A small module-level cosine can reflect opposing group contributions rather than uniformly weak interactions.Source: Figure 2 · p. 3, §2.3 · p. 4

3 MORPHACT: MORPH WITH THE ENVIRONMENT AND THEN ACT

MorphAct adjusts policy parameters for the current interaction context before predicting an action chunk. Its coupled generators produce updates for vision–language and action modules, learn through the native action objective, and reuse generated outputs within each action chunk.

3.1 REPRESENTATION-COUPLED GENERATION

  • The two generators share weights across contexts but produce context-dependent effective parameters; action-side generation uses features from the adapted vision–language representation.The adapted VLM pass also provides the native action-conditioning path, while the action generator receives robot state.Source: §3.1 · pp. 4–5
  • Structured low-rank synthesis generates factors for selected blocks, avoiding materialization of dense parameter updates and accommodating different module widths and depths.The rank is defined per generated block, not for the assembled multi-head projection.Source: §3.1 · p. 5

3.2 LEARNING THROUGH THE ADAPTED REPRESENTATION

  • Both generators are trained jointly through the base policy’s native action objective, with generated factors treated as differentiable intermediate outputs and backbone parameters frozen.For π0 and π0.5, training predicts velocity along a linear action–noise path rather than the clean inference chunk.Source: §3.2 · p. 5

3.3 CONTEXT-LEVEL ADAPTATION, CHUNK-LEVEL REUSE

  • At each replanning observation, MorphAct generates context-specific updates that are reused throughout action-chunk prediction and refreshed at the next replanning step, without test-time optimization.The condition depends on the current observation context, not noisy actions or internal solver time.Source: §3.3 · pp. 5–6

3.4 IMPLEMENTATION DETAILS

  • Implementation uses small, nonzero readout weights and zero biases; generated factors and residual accumulation are computed in FP32 before conversion to the backbone output dtype.Variable-length conditions are grouped by sequence length, with original batch order restored after generation.Source: §3.4 · p. 6

4 EXPERIMENTS

MorphAct is evaluated through closed-loop task success on LIBERO, Meta-World, and four physical manipulation tasks, with inference cost also measured. It improves reported success across the simulation backbones and real-world tasks, while adding action-chunk latency.

4.1 EVALUATION SETTING

  • Closed-loop task success is the primary metric; simulation compares three VLA backbones with their corresponding baselines under the same task settings.LIBERO training covers four ten-task suites, while Meta-World training jointly covers all 50 MT50 tasks.Source: §4.1 · p. 6, §4 · p. 6
  • The simulation tables report MorphAct means across three random seeds, using 50 rollouts per LIBERO task and 20 per Meta-World task.Baseline variances are not reported.Source: §4.1 · p. 6

4.2 LIBERO

  • MorphAct raises LIBERO mean success by 4.15, 2.50, and 1.75 percentage points on X-VLA, π0, and π0.5, respectively.The table reports suite-level closed-loop success and means across the four LIBERO suites.Source: §4.2 · p. 6, Table 1 · p. 7

4.3 META-WORLD

  • MorphAct improves Meta-World average success by 7.57, 16.30, and 15.72 percentage points on X-VLA, π0, and π0.5, respectively.The results cover joint training on MT50 and average success across its four difficulty groups.Source: Table 2 · p. 7, §4.3 · p. 7

4.4 REAL-WORLD MANIPULATION

  • MorphAct improves π0.5 success by 8.00–14.00 percentage points across four physical manipulation tasks, with 50 rollouts per method per task.Figure 4 pairs task examples with baseline and MorphAct success rates; reported gains are absolute percentage-point differences.Source: §4.4 · pp. 7–8, Figure 4 · p. 8

4.5 INFERENCE COST

  • MorphAct generates updates once per replanning context and reuses them during action-chunk prediction, outside the solver loop.End-to-end chunk latency increases by approximately 18.00–21.00% relative to shared LoRA on π0 and π0.5.Source: §4.5 · p. 8

5 ANALYSIS

The ablations compare fixed and contextual generation conditions, adaptation of individual or both modules, and which VLM features condition action-side generation. Their results support context-sensitive generation and coupling through adapted representations.

5 ANALYSIS

  • The ablations vary generation conditions, adapted modules, and the features passed between generators using X-VLA on LIBERO.Table 3 reports success rates for these design comparisons.Source: §5 · p. 8

5.1 CONTEXT BEYOND TASK IDENTITY

  • Current-context generation reaches 97.20% mean success, exceeding Constant and Task-ID by 9.70 and 8.95 points, respectively.MorphAct improves all four suites, whereas Task-ID raises mean success from 87.50% to 88.25% but reduces Spatial by 3.40 points.Source: §5.1 · pp. 8–9

5.2 ADAPTING BOTH MODULES

  • Action-only reaches 81.00% mean success versus 43.50% for VLM-only, while Unadapted raises Action-only performance to 86.00%.Unadapted adapts the representation supplied to the action predictor, even though its action generator still uses pre-adaptation features.Source: §5.2 · p. 9

5.3 COUPLING THROUGH ADAPTED FEATURES

  • Using adapted VLM features to condition action-side generation raises mean success from 86.00% to 97.20%, a gain of 11.20 percentage points.The comparison is between Unadapted, which uses pre-adaptation features for action-update generation, and MorphAct, which uses adapted features.Source: §5.3 · p. 9

6 RELATED WORK

The related work situates MorphAct among VLA policies, parameter-efficient adaptation, multi-task learning, and parameter generation. It distinguishes MorphAct by coupling context-dependent low-rank updates across perception and action.

VLA models

  • MorphAct studies adaptation of representation and action modules to changing manipulation contexts within a shared policy.The section situates this focus alongside generalist robot policies, VIMA, RT-2, OpenVLA, and the π family.Source: §6 · p. 9

Parameter-efficient fine-tuning

  • MorphAct generates context-dependent low-rank updates while retaining the host policy’s action interface and objective.The discussion places it alongside adapters, LoRA, DoRA, and VLA-specific adaptation methods.Source: §6 · p. 9

Parameter sharing and generation

  • Prior multi-task learning work addresses structural sharing, joint optimization, task transfer, and layerwise conflicts; parameter generation synthesizes weights or updates from conditioning inputs.The cited examples include hypernetworks and diffusion models.Source: §6 · p. 9
  • Unlike task- or prompt-conditioned parameter generation, MorphAct couples adapted vision–language representations to action-side update generation and trains both generators through the native action objective.The discussion contrasts this design with Hyper-LoRA, HyperVLA, and PhaseLoRA.Source: §6 · p. 10

7 CONCLUSION

The conclusion presents MorphAct as a shared adaptation mechanism for context-dependent updates across vision–language and action modules. It reports improved closed-loop manipulation across simulation and real-world evaluations, with ablations supporting its design choices.

7 CONCLUSION

  • MorphAct generates context-dependent low-rank updates for both vision–language and action modules through a shared adaptation mechanism.The generators are coupled through adapted representations and jointly trained with the native action objective.Source: §7 · p. 10
  • Atomic-task diagnostics reveal conflicting adaptation demands in both vision–language and action modules.Source: §7 · p. 10
  • Evaluations on LIBERO, Meta-World, and four real-world tasks demonstrate improved closed-loop manipulation.Ablations support contextual generation, adaptation of both modules, and coupling through adapted VLM features.Source: §7 · p. 10

A ATOMIC-TASK DIAGNOSTIC PROTOCOL AND COMPLETE RESULTS

The appendix specifies the atomic-task diagnostic protocol and reports cross-task transfer harms and parameter-group contributions, with visualizations that distinguish local conflict from module-level contribution.

A ATOMIC-TASK DIAGNOSTIC PROTOCOL AND COMPLETE RESULTS

A.1 PARAMETER SCOPE AND SAMPLING

  • Alignment and contribution measurements use 16 rounds of disjoint-episode batch sampling, while one-step transfer uses four complete source-to-target rounds at the same checkpoint.The reported rounds are repeated batch samples at one checkpoint, not independent training seeds.Source: Table A1 · p. 14, §A.1 · p. 14
  • The diagnostic quantities use separate numerical realizations for alignment and contribution accounting, and confidence intervals and zero-norm exclusion counts are not reported.Negative-cosine frequencies count within-round observations, unlike the sign of a pair’s 16-round mean cosine.Source: §A.1 · p. 14

A.2 GRADIENT ALIGNMENT AND DIRECTIONAL EFFECTS

  • All 260 source updates reduced source-batch loss by 2.38% on average, yet each harmed 22–46 other target batches, with a median of 32/64.Each target also received antagonistic updates from at least 20 other sources within its round; the maximum cross-task relative loss increase was 0.80%.Source: Table A3 · p. 15, §A.2 · p. 15

A.3 PARAMETER GROUPS AND CONTRIBUTION ACCOUNTING

  • The 47 VLM groups and 19 action groups separate Transformer blocks by depth from parameters outside Transformer blocks.The outside-block groups include distinct mappings, projections, and normalization parameters.Source: §A.3 · p. 15
  • The three largest action groups—outside Transformer blocks, block 00, and block 11—account for 78.88% of negative cross-task contribution before rounding.The largest VLM groups are outside Transformer blocks (20.40%), language block 00 (17.44%), and vision block 00 (7.83%).Source: §A.3 · p. 16

A.4 PARAMETER-GROUP VISUALIZATIONS

  • Group-local cosine profiles and full-module-normalized contribution shares measure different aspects of interference and should not be conflated.Figure 5 shows the five largest negative-contribution groups per module and aggregates the rest; Figure 6 gives complete Transformer depth profiles.Source: §A.4 · p. 20, Figure 5 · p. 20, Figure 6 · p. 21

B PARAMETER GEOMETRY AND ARCHITECTURE INTERFACES

MorphAct’s generated low-rank factors accommodate different projection layouts and preserve architecture-specific policy interfaces.

B PARAMETER GEOMETRY AND ARCHITECTURE INTERFACES

  • MorphAct assembles updates from output-head blocks for Q/K/V projections and sums input-head contributions for attention-output projections.Key and value projections use their actual key–value head counts; feed-forward projections use matrix-level factors.Source: B PARAMETER GEOMETRY AND ARCHITECTURE… · p. 22
  • Each generated factor product has rank at most r, while an assembled multi-head projection can reach the sum of block ranks, bounded by its dimensions.Rank is defined per generated block, not for the assembled projection; equal block rank does not imply equal generated or trainable capacity.Source: B PARAMETER GEOMETRY AND ARCHITECTURE… · p. 22
  • The architecture-specific integrations preserve each base policy’s action-conditioning and loss definitions, while X-VLA adds separate generated Q/K/V residuals to its fused QKV projections.X-VLA also uses auxiliary-view visual features for its action generator; this optional extension does not apply to the π integrations.Source: Table A7 · p. 22, B PARAMETER GEOMETRY AND ARCHITECTURE… · p. 22

C JOINT TRAINING AND POLICY EXECUTION

MorphAct generates vision–language updates before action-side updates, couples the generators through adapted representations, and trains them jointly using the policy’s native action objective.

C JOINT TRAINING AND POLICY EXECUTION

  • Only the generator parameters, action-state projection, and configuration-retained projector parameters are optimized; the remaining backbone parameters stay frozen.Generated factors are intermediate outputs, not independent optimization variables.Source: C JOINT TRAINING AND POLICY EXECUTION · p. 23
  • For each context, MorphAct generates a VLM update, computes adapted representations, then conditions action-update generation on those representations and robot state.The adapted features remain differentiable through the action generator, allowing the native action loss to train the VLM generator through both conditioning paths.Source: C JOINT TRAINING AND POLICY EXECUTION · p. 23
  • Training applies the native prediction target and action objective for each backbone, without operation labels or operation-specific adapter targets.π0 and π0.5 predict flow velocity, while X-VLA predicts clean actions.Source: C JOINT TRAINING AND POLICY EXECUTION · pp. 23–24
  • At inference, MorphAct caches generated updates and adapted conditioning for the action-sampling procedure, then recomputes them when a new observation triggers replanning.The control loop executes the first Hexec actions of the predicted chunk before acquiring a new observation.Source: C JOINT TRAINING AND POLICY EXECUTION · pp. 23–24
  • Small nonzero readout initialization avoids blocking generator-trunk gradients, while FP32 residual calculations and activation checkpointing support numerical handling and memory reduction.A sufficiently small residual may still be quantized when converted to the base output dtype.Source: C JOINT TRAINING AND POLICY EXECUTION · p. 24

D.1 ABLATION CONFIGURATIONS

The ablations vary generation conditions, module adaptation, and the representation used to generate action-side updates.

D.1 ABLATION CONFIGURATIONS

  • Task-ID assigns a fixed generation condition per task, whereas Constant uses conditions fixed across tasks and observations.Both retain the policy’s normal image, language, and robot-state inputs.Source: §D.1 · p. 25
  • VLM-only disables generated action-side updates, while Action-only disables generated VLM updates but retains visual inputs.Source: §D.1 · p. 25
  • Unadapted feeds the action generator the pre-update VLM representation, while the action predictor still receives the adapted representation.This changes the condition for action-side parameter generation specifically.Source: §D.1 · p. 25

D.2 ADAPTATION BUDGET AND RUNTIME

MorphAct’s adaptation budget and runtime measurements characterize the execution cost of contextual updates. Parameter generation is paid per context and reused during action-chunk refinement, but computing and applying updates still adds cost.

D.2 ADAPTATION BUDGET AND RUNTIME

  • Generator counts report learned weights separately from active-context scalar A/B entries; generated factors are outputs, unlike shared LoRA’s reused learned factors.Explicit factor counts distinguish head-aware from matrix-level layouts, which nominal rank alone does not capture.Source: §D.2 · p. 25, Table A8 · p. 25
  • Latency covers preprocessing and full action-chunk prediction, with generation and residual computation included where applicable, under matched hardware and execution settings.Settings include solver steps, predicted horizon, and executed-prefix length; control rate is measured separately under the actual schedule.Source: §D.2 · p. 25, Table A8 · p. 25
  • Versus shared LoRA, MorphAct adds 25.00 ms chunk latency and 1.67 GiB memory on π0, and 25.29 ms and 1.65 GiB on π0.5.Table A8 reports end-to-end chunk latency and inference memory.Source: Table A8 · p. 25, §D.2 · p. 25
  • MorphAct’s measured control rates are 23.20 Hz on π0 and 21.20 Hz on π0.5, versus 28.09 Hz and 24.99 Hz for shared LoRA.Rates use the actual action-execution schedule.Source: §D.2 · p. 25, Table A8 · p. 25
  • Generation is paid per context rather than solver step; MorphAct reuses updates throughout chunk prediction, outside the refinement loop.Adapted-module costs still include computing and applying generated residuals.Source: §D.2 · p. 25
Loading 2610.20019v1…