PaperlayerView exact revision on arXiv

Source-linked AI summary

PhaseLoRA: Control-Regime-Conditioned Low-Rank Adaptation for Continuous-Action Vision-Language-Action Policies

Yufei Guo, Yinan Wu, Haoran Duan, Guiguang Ding, Jungong Han

arXiv:2608.15285v1cs.ROcs.AI

TL;DR

Continuous-action VLA rollouts span distinct control regimes, but standard adapters apply the same update throughout. PhaseLoRA conditions lightweight LoRA updates within trajectories, improving LIBERO macro-average success by 12.2 points over matched-parameter High-rank LoRA.

  • Problem

    Standard adapters apply the same update throughout a rollout, despite continuous-action manipulation spanning distinct regimes that may require different corrections.

  • Method

    PhaseLoRA conditions LoRA updates at each action-chunk prediction step using weakly supervised fine-control and event/boundary descriptors.

  • Results

    12.2 points: PhaseLoRA improves macro-average success rate over matched-parameter High-rank LoRA on LIBERO and outperforms stronger LoRA variants.

  • Takeaways & Limitations

    Within-trajectory control-regime conditioning is an effective lightweight PEFT axis for continuous-action VLA policies.

  • Takeaways & Limitations

    The method relies on action-derived weak proxy descriptors and may require alternatives when action statistics do not reflect fine-control or boundary events.

Abstract

from arXiv · show

Parameter-efficient fine-tuning (PEFT) is a natural way to adapt pretrained vision-language-action (VLA) policies, but most adapter designs apply temporally static updates throughout a control rollout, overlooking the phase-dependent nature of continuous-action manipulation. Such policies traverse distinct regimes, including approach, contact transition, grasping, transport, and placement, each requiring different adaptation behaviors. We propose \textbf{PhaseLoRA}, a lightweight LoRA parameterization that conditions adaptation at each action-chunk prediction step using two weakly supervised descriptors: fine-control tendency and event/boundary intensity. PhaseLoRA modulates the LoRA left factor in the action expert, allowing the effective low-rank update direction to vary over time while keeping the backbone largely frozen. On LIBERO, PhaseLoRA improves average success rate by 12.2 points over a matched-parameter high-rank LoRA baseline and outperforms stronger LoRA variants. Ablations show that random temporal modulation and scalar gating do not reproduce the performance of the full model, while update-direction analyses reveal structured temporal variation associated with the predicted control descriptors. These results establish within-trajectory conditioning as an effective lightweight PEFT axis for continuous-action VLA policies.

1 Introduction

PhaseLoRA addresses the temporal heterogeneity of continuous-action manipulation by conditioning LoRA updates within trajectories rather than applying a static adapter throughout execution. It uses weakly supervised control descriptors to vary low-rank update directions and shows benefits over standard, matched-parameter, random-modulation, and scalar-gating alternatives.

  • Method: PhaseLoRA modulates the LoRA left factor while preserving low-rank updates, allowing effective update directions to vary across control steps.Its parameterization uses descriptor-conditioned terms for the base, fine-control, event/boundary, and interaction effects.
  • Motivation: Static PEFT applies the same update throughout a rollout, despite manipulation traversing distinct regimes that require different policy corrections.These regimes include approach, contact or release transitions, grasping, transport, and precise placement.
  • Method: PhaseLoRA conditions LoRA updates at each action-chunk prediction step using fine-control tendency and event/boundary intensity.The descriptors represent precision-demanding moments and abrupt control changes such as grasping, placement, contact, and release.
  • Evaluation: On LIBERO, PhaseLoRA improves over standard LoRA, a matched-parameter high-rank LoRA baseline, and stronger LoRA variants.The evaluation tests whether within-trajectory conditioning provides benefits beyond increased parameter count or generic temporal variation.
  • Results: PhaseLoRA outperforms random-modulation and scalar-gating controls, while update-direction analyses relate its temporal variation to predicted descriptors and salient manipulation events.These analyses assess whether control-informed descriptors outperform random temporal modulation and whether scalar rescaling of a fixed direction is sufficient.

2 Related Work

Prior work spans the progression from multitask imitation and large-scale vision-language-action policies to efficient downstream adaptation, richer LoRA parameterizations, and explicit modeling of manipulation phases and contact regimes.

  • Generalist robot policies and vision-language-action models: Generalist robot learning progressed from multitask imitation policies such as BC-Z and PerAct toward large-scale, multimodal, multi-embodiment robot foundation models.RT-1 linked improved real-world robotic control to scaling data and model capacity, while PaLM-E and RT-2 connected pretrained models to embodied action generation.
  • Fine-tuning and parameter-efficient adaptation for robot foundation models: Efficient downstream adaptation has become a practical bottleneck as robot foundation models scale, motivating systematic studies of decoding, action representation, chunking, and training objectives.OpenVLA emphasized efficient fine-tuning for VLA deployment, while OpenVLA-OFT examined several adaptation choices.
  • LoRA variants, dynamic adapters, and conditional parameter generation: LoRA research has expanded through rank allocation, alternative parameterizations, expert mixtures, and structured methods that address heterogeneous adaptation demands.Examples include DyLoRA and AdaLoRA for rank or budget allocation, DoRA for alternative parameterization, X-LoRA and D-MoLE for expert mixtures, and MokA and HINT for structured adaptation.
  • Phase structure, skill decomposition, and contact-rich manipulation: Manipulation research increasingly treats control as phase-structured rather than homogeneous, using skill decomposition, intermediate representations, and explicit interaction-regime modeling.Examples include The Art of Imitation, TacUMI, RoboInter, and methods separating reaching from local interaction or conditioning policies on contact-related regimes.

3 Method

PhaseLoRA replaces temporally static LoRA updates with within-trajectory, control-regime-conditioned adaptation for continuous-action VLA policies. A lightweight router predicts fine-control tendency and event/boundary intensity, which modulate the action expert’s LoRA left factor while preserving a shared right factor and frozen backbone.

  • Motivation: Standard LoRA applies the same update throughout a rollout, whereas PhaseLoRA adapts to changing control regimes within trajectories.The targeted regimes include coarse motion, contact transitions, grasp execution, and precise alignment.
  • Control-regime conditioning: A lightweight router predicts two [0, 1]-valued descriptors at each query: fine-control tendency and event/boundary intensity.The descriptor pair is shared across all PhaseLoRA layers for the forward pass that predicts an action chunk.
  • PhaseLoRA parameterization: PhaseLoRA conditions only the action expert’s left factor, using B_t = B_0 + P̂_tB_P + Ê_tB_E + (P̂_tÊ_t)B_PE while sharing the right factor A across queries.For fixed predicted descriptors, the update remains rank at most r; the descriptor-dependent terms alter output directions while A defines a common input projection.
  • Weak supervision: Weak proxy targets are computed offline from low-level actions and gripper commands to supervise descriptors for fine-control tendency and event intensity.Motion magnitude weakly indicates fine-grained control, while action jerk captures abrupt changes; the targets are not exact symbolic phase labels.
  • Router and training: At inference, a one-layer GRU router with hidden dimension 128 uses visual-semantic features and recent action/gripper history, updated online from executed outputs.The history branch uses the previous K = 6 control steps and retains the first 5 executed low-level actions from each past action chunk.
  • Efficiency: Relative to standard LoRA, PhaseLoRA adds 3rdout trainable parameters per adapted layer plus a lightweight router, with only descriptor-routing overhead at inference.A layer uses rdin + 4rdout parameters instead of r(din + dout), and no additional full-capacity expert branches are introduced.

4 Experiments

Experiments show that PhaseLoRA improves continuous-action VLA adaptation over static and matched-capacity LoRA baselines on LIBERO and a real-world benchmark. Ablations and update-direction analyses indicate that aligned, complementary descriptors enable temporally varying low-rank update directions beyond arbitrary modulation or scalar gating.

  • Main results: 12.2 points: PhaseLoRA improves macro-average LIBERO success over matched-parameter High-rank LoRA and achieves the highest macro-average success rate.It also improves over standard LoRA by 30.6 points and outperforms DoRA, LoRA-MoE, and LoRA-SP.
  • Real-world benchmark: 18.4 points: PhaseLoRA improves average success over matched-parameter High-rank LoRA on the real-world tabletop manipulation benchmark.The benchmark uses four tasks on a Piper robotic arm, with 100 demonstrations per task and 10 trials per task for each of 3 seeds.
  • Ablations: Random descriptors substantially underperform PhaseLoRA, showing that arbitrary time-varying inputs are insufficient without alignment to rollout-dependent control variation.The random-descriptor control preserves time-varying scalar inputs and the PhaseLoRA parameterization while replacing P and E with Uniform(0, 1) values.
  • Ablations: Scalar gating underperforms PhaseLoRA, showing that descriptor-dependent rescaling of a fixed LoRA direction cannot recover the benefit of changing the effective update direction.The control uses the same router and predicted descriptors but constrains the update to a scalar multiple of a fixed low-rank direction.
  • Ablations: Removing proxy supervision or either descriptor reduces performance, while removing the interaction term causes a smaller drop, indicating complementary descriptor roles and a mainly additive contribution.Fine-control tendency and event/boundary intensity provide complementary information; the interaction term supplies an additional refinement.
  • Update-direction analysis: Update-direction changes are sparse, coherent across adapted layers, and concentrated near salient manipulation events, while timestep-level variation correlates with descriptor changes, especially event/boundary intensity.The qualitative pattern appears across LIBERO suites, with peaks near grasp, transport transition, and placement.

5 Conclusion

PhaseLoRA conditions LoRA updates on weakly supervised control descriptors and improves over standard, matched-parameter high-rank, and structured LoRA variants on LIBERO. Its lightweight design relies on action-derived proxies, with evaluation concentrated on LIBERO and limited real-world testing.

  • 5 Conclusion: PhaseLoRA conditions LoRA updates on fine-control tendency and event/boundary intensity, using trajectory-level control heterogeneity to vary adaptation.Ablations and update-direction analyses support descriptor-conditioned update direction.
  • 5 Conclusion: PhaseLoRA improves over standard LoRA, matched-parameter High-rank LoRA, and structured LoRA variants on LIBERO.The conclusion attributes these gains to descriptor-conditioned adaptation for continuous-action VLA policies.
  • Limitations and scope: PhaseLoRA uses action-derived weak proxy descriptors instead of manual phase labels or additional force/tactile sensing, keeping the method lightweight but potentially requiring alternative descriptors.Alternative descriptors may be needed when action statistics do not reflect fine-control or boundary events.
  • Limitations and scope: The main comparisons cover four LIBERO suites, detailed ablations focus mainly on LIBERO-Spatial, and real-world evaluation is limited to four tabletop tasks on one robot platform.These scope constraints limit how broadly the reported findings are evaluated.

A Implementation and Reproducibility Details · A.1 Backbone and action representation

The appendix specifies the implementation needed to reproduce the LIBERO experiments, including the π0.5 backbone, image and language inputs, and the action-chunk representation. The policy predicts 32-dimensional action chunks over horizon H = 10, while LIBERO control uses seven action dimensions with defined translational, rotational, and gripper components.

  • A Implementation and Reproducibility Details: The reproducibility appendix provides additional implementation details for the LIBERO experiments.
  • A.1 Backbone and action representation: All experiments use the π0.5 VLA backbone initialized from the public pi05_base checkpoint.The checkpoint path is gs://openpi-assets/checkpoints/pi05_base/params.
  • A.1 Backbone and action representation: The VLM uses gemma_2b_lora, while the action expert uses gemma_300m_lora.
  • A.1 Backbone and action representation: Images are resized to 224 × 224 during training and evaluation.
  • A.1 Backbone and action representation: For LIBERO evaluation, the policy receives base_0_rgb and left_wrist_0_rgb, while right_wrist_0_rgb is a zero-filled dummy view.
  • A.1 Backbone and action representation: Language instructions come directly from the LIBERO task language annotations.
  • A.1 Backbone and action representation: The model predicts action chunks with action dimension 32 and horizon H = 10, but LIBERO control uses the first seven dimensions.These comprise three translational dimensions, three rotational dimensions, and one gripper command.
  • A.1 Backbone and action representation: The first six LIBERO action dimensions use normalized delta commands for the robosuite v1.4.1 OSC_POSE controller.The translational components are scaled to 0.05 m per dimension, the rotation-vector components to 0.5 rad per dimension, and the gripper command lies in [−1, 1].

A.2 LIBERO benchmark and evaluation protocol · A.3 LoRA and PhaseLoRA configuration

The evaluation uses suite-specific policies across four standard LIBERO suites, with episode-level success averaged across suites. LoRA and PhaseLoRA adapt Gemma attention/feed-forward projections, while PhaseLoRA varies its low-rank update direction across control steps.

  • A.2 LIBERO benchmark and evaluation protocol: Four standard LIBERO suites—Spatial, Object, Goal, and 10—are evaluated using official datasets and language annotations.Each suite-specific policy is fine-tuned on its corresponding training split and evaluated on that suite.
  • A.2 LIBERO benchmark and evaluation protocol: Evaluation uses 10 trials per task and 3 seeds unless otherwise stated, with success measured at the episode level.The suite success rate is defined over total evaluation episodes.
  • A.2 LIBERO benchmark and evaluation protocol: The main-table average is computed across the four LIBERO suite scores.This aggregates the suite-level episode success rates into one reported average.
  • A.3 LoRA and PhaseLoRA configuration: LoRA wraps Gemma attention and feed-forward linear projections, including query, key, value, output, gate, and linear projections.The implementation follows the LoRA implementation in the Gemma modules of the π0.5 codebase.
  • A.3 LoRA and PhaseLoRA configuration: Standard LoRA and PhaseLoRA use VLM rank 48 with α = 48 and action-expert rank 96 with α = 96.Trainable-parameter percentages are measured against the full π0.5 backbone, and LoRA dropout is not used.
  • A.3 LoRA and PhaseLoRA configuration: The matched-parameter High-rank LoRA baseline uses VLM rank 80 with α = 80 and action-expert rank 160 with α = 160.These settings provide the stated matched-parameter comparison configuration.
  • A.3 LoRA and PhaseLoRA configuration: PhaseLoRA uses four left factors and one shared right factor per adapted layer, conditioned on router-predicted descriptors at control step t.The effective left factor depends on the predicted descriptor pair (P̂_t, Ê_t).
  • A.3 LoRA and PhaseLoRA configuration: For fixed descriptors, PhaseLoRA remains low-rank while its effective left factor varies across control steps.The shared component follows standard LoRA initialization, while descriptor-dependent left factors start at zero and learn deviations during fine-tuning.

A.4 Router architecture · A.5 Weak descriptor targets

The phase router predicts fine-control and event/boundary descriptors at every control step from multimodal context and recent action history. Offline weak targets derive these descriptors from normalized motion, gripper changes, and action jerk, with bounded target construction.

  • A.4 Router architecture: At each control step, the phase router predicts scalar fine-control and event/boundary descriptors.It uses a visual-language prefix summary plus recent action and gripper history.
  • A.4 Router architecture: The visual-language branch summarizes multimodal prefix embeddings by masked mean pooling, producing a 2048-dimensional summary in the gemma_2b_lora VLM.The summary is one of three router input branches.
  • A.4 Router architecture: The action-history branch retains the previous 6 chunks and their first 5 executed actions, yielding 30 low-level action steps.This history is combined with gripper information for router conditioning.
  • A.4 Router architecture: Each historical low-level step uses 23 features spanning actions, first- and second-order differences, gripper state, and gripper change.The branches are projected to dimension 128, and the temporal history is processed by a single-layer unidirectional GRU with hidden dimension 128.
  • A.4 Router architecture: At episode start, unavailable history is zero-padded; training uses offline history, whereas evaluation updates history online from previously executed policy actions.With no history available, the router receives an all-zero history window.
  • A.5 Weak descriptor targets: The fine-control target combines robustly normalized translational and rotational motion magnitudes with gripper-command change, weighted 0.5 and 0.1 respectively.The event/boundary target is derived from action jerk.
  • A.5 Weak descriptor targets: Trajectory statistics use median and median absolute deviation; sigmoid inputs are clipped to [−60, 60], targets are clamped to [0, 1], and gripper commands remain continuous.The normalization uses ϵ = 10−6 for numerical stability.

A.6 Training objective and optimization · A.7 Baseline fairness

Training uses a flow-matching mean-squared-error objective with a jointly supervised descriptor router, while optimization follows fixed AdamW settings. Baseline comparisons hold the training setup and adapted module family constant, with High-rank LoRA matched to PhaseLoRA’s trainable-parameter budget.

  • A.6 Training objective and optimization: The policy is trained with a flow-matching mean-squared-error objective over the full predicted action tensor.The target velocity is defined as sampled noise minus the action target, without manually tuned translation, rotation, or gripper weights.
  • A.6 Training objective and optimization: The router is supervised using weak descriptor targets through a weighted Huber-and-binary-cross-entropy loss.The loss is L_router = 0.5 Huber(P̂_t, P_t) + 0.2 BCE(Ê_t, E_t), with Huber delta 0.1 for fine-control targets.
  • A.6 Training objective and optimization: The learning rate warms up linearly for 10,000 steps to 5 × 10^-5, then remains constant through 30,000 optimization steps.Training uses batch size 32, bfloat16 precision, and no exponential moving average.
  • A.7 Baseline fairness: LoRA-based methods keep the backbone, data split, preprocessing, action representation, objective, batch size, schedule, steps, and adapted module family fixed.This controls the comparison across Standard LoRA, PhaseLoRA, High-rank LoRA, and LoRA-SP.
  • A.7 Baseline fairness: High-rank LoRA increases ranks to approximately match PhaseLoRA’s trainable-parameter budget, while Standard LoRA and PhaseLoRA share base LoRA ranks.LoRA-SP is applied to the same family of Gemma attention and feed-forward modules.

A.8 Compute resources

LIBERO experiments used single RTX 5090 GPUs with bfloat16 precision and batch size 32, while PhaseLoRA peaked at approximately 22106 MiB per run. The reported training runs required approximately 2160 GPU-hours in total, with about 100GB of local SSD storage used for datasets, checkpoints, logs, and cached artifacts.

  • Hardware and per-run cost: PhaseLoRA training used a single NVIDIA RTX 5090 GPU per run with bfloat16 precision and batch size 32.Peak GPU memory usage was approximately 22106 MiB per run.
  • Hardware and per-run cost: 24 GPU-hours was the approximate duration of one 30,000-step suite-specific LIBERO training run, depending on the method and suite.The reported duration varied by method and suite.
  • Aggregate compute: 2160 GPU-hours was the approximate total compute required by the reported LIBERO training runs.This total includes the main LIBERO comparison and additional LIBERO-Spatial ablations.
  • Storage and evaluation: 100GB of local SSD storage was used for LIBERO datasets, the π0.5 pretrained checkpoint, training checkpoints, logs, and cached artifacts.Real-world evaluation used the robot platform and required no additional large-scale model training beyond fine-tuned checkpoints.

B Real-world Benchmark Details … E Existing assets and licenses

The supplementary sections detail a controlled real-world benchmark, additional LIBERO update-direction evidence, deployment considerations, and the assets and licenses underlying the experiments. The physical evaluation uses matched training and control settings on one robot platform, while visualizations show event-associated temporal variation in adaptation updates.

  • B Real-world Benchmark Details: PhaseLoRA is evaluated on an AgileX Piper arm with a two-finger gripper in closed-loop tabletop manipulation, using the same pretrained VLA backbone and method-specific adapters as simulation.The robot is mounted on a tabletop workspace, and policies are initialized consistently across real-world and LIBERO experiments.
  • B Real-world Benchmark Details: The policy uses two 224 × 224 RGB camera views and language instructions, without proprioceptive, force, or tactile inputs.Observations come from a wrist-mounted camera and a fixed external camera.
  • B Real-world Benchmark Details: The benchmark includes four tabletop tasks covering grasping, transport, release, and placement, with 100 demonstrations per task for fine-tuning.Tasks involve moving or removing objects across specified plates or regions.
  • B Real-world Benchmark Details: H = 50 action chunks are executed receding-horizon style: only the first 10 low-level actions run before replanning, with chunking fixed across methods.Each policy query predicts 50 actions, while execution replans every 10 control steps.
  • B Real-world Benchmark Details: The real-world comparison controls backbone, observations, instructions, demonstrations, chunking, optimizer, schedules, training steps, and approximately matched trainable-parameter budgets.The High-rank LoRA baseline approximately matches PhaseLoRA’s trainable-parameter budget.
  • B Real-world Benchmark Details: The benchmark is limited to four tabletop tasks on one robot and workspace, testing whether the matched-capacity LIBERO trend appears under physical execution rather than broad robustness.It does not establish generalization across robots, camera placements, object categories, or environments.
  • C Additional Update-Direction Visualizations: Additional LIBERO visualizations show sparse, coherent update-direction changes across adapted layers, concentrated near grasping, transport transitions, and placement events.Figure 5 covers LIBERO-Object, LIBERO-Goal, and LIBERO-10, complementing the LIBERO-Spatial example.
  • D Broader impacts: PhaseLoRA may reduce adaptation costs, but real-world deployment should include task-specific safety checks, human oversight, and deployment constraints.The experiments use no personally identifiable or human-subject data, while broader robotic deployment may raise labor, safety, and accountability concerns.
Loading 2608.15285v1…