PaperlayerView exact revision on arXiv

Source-linked AI summary

Hierarchical Residual Policy Learning for Real-World Mobile Manipulation with Sparse Human Guidance

Zhefei Gong, Hefeifei Jiang, Zehao Wang, Jiachen Li

arXiv:2610.20607v1cs.ROcs.AIcs.LG

TL;DR

Real-world fine-tuning of mobile manipulation is challenging because high-dimensional actions and coupled modules complicate adaptation. HiRe-MoMa combines residual corrections, hierarchical base-arm conditioning, and expert-guided value alignment; across three TIAGo++ tasks it improves performance across two base-policy architectures with 25–40 minutes of online interaction.

  • Problem

    Fine-tuning mobile manipulation remains largely unexplored despite high-dimensional actions, coupled module dynamics, and the difficulty of generalizing pretrained policies to stochastic real-world deployment.

    Source: §1 · pp. 1–2

  • Method

    HiRe-MoMa fine-tunes a frozen base policy with additive residuals, hierarchical base-to-arm conditioning, and value alignment from sparse human interventions.

    Source: §1 · p. 2, §4.1 · p. 4

  • Results

    HiRe-MoMa consistently improves performance across three TIAGo++ tasks and two base-policy architectures with 25–40 minutes of online interaction.

    Source: §1 · p. 2

  • Takeaways & Limitations

    An asynchronous actor-learner setup supports real-world on-robot fine-tuning of mobile manipulation on a single robot.

    Source: §1 · p. 2

    Evaluation is limited to one TIAGo++ platform with 20 trials per condition and no seed sweeps, and the policies are not yet reliable for unsupervised deployment around people.

    Source: §6 · p. 9

Abstract

from arXiv · show

Mobile Manipulation (MoMa) is a cornerstone of autonomous robotic systems, yet real-world fine-tuning remains a formidable challenge because of high-dimensional action spaces and intricate coupling between heterogeneous modules. In this paper, we propose HiRe-MoMa (Hierarchical Residual Mobile Manipulation), a systematic framework for stable and sample-efficient real-world fine-tuning of complex MoMa systems. At its core, HiRe-MoMa adopts a Hierarchical Residual Architecture that fine-tunes a frozen base policy. A Policy-Agnostic Residual learns additive corrections atop the base policy, while Hierarchical Decoupling splits the residual into mobile-base and manipulator-arm heads with unidirectional conditioning, mitigating gradient interference and aligning corrections with the platform's physical hierarchy. To improve sample efficiency in data-expensive real-world settings, we propose Expert-Guided Value Alignment, which leverages sparse human interventions with under 20% training coverage to reshape the critic's value landscape, effectively guiding autonomous exploration through expert priors. Extensive real-world experiments on the TIAGo++ platform demonstrate that HiRe-MoMa consistently enhances task performance with only around 30 minutes of online interaction, regardless of the base policy architecture. To our knowledge, this is among the first frameworks to enable efficient on-robot fine-tuning for mobile manipulation in the real world, providing a practical paradigm for deploying complex robotic systems in dynamic, human-centric environments. Further details can be found on our project website: https://hire-moma.github.io/

1 Introduction

Real-world mobile manipulation is difficult to fine-tune because its high-dimensional action space couples heterogeneous modules, while fine-tuning for this domain remains relatively unexplored. HiRe-MoMa combines policy-agnostic residual learning, hierarchical base-arm corrections, and sparse expert guidance in an asynchronous real-world framework.

1 Introduction

  • Mobile manipulation fine-tuning remains largely unexplored despite the need to adapt policies to stochastic real-world deployment and the challenges of high-dimensional actions and coupled module dynamics.Prior fine-tuning successes described here concern isolated manipulation or locomotion tasks.Source: §1 · p. 1
  • Residual learning adds corrections atop a frozen base policy, while hierarchical decoupling conditions the manipulator correction on the base correction and expert-guided value alignment reshapes the critic using sparse interventions.The framework uses an asynchronous actor-learner system for on-robot fine-tuning.Source: §1 · p. 2, Figure 2 · p. 3
  • HiRe-MoMa is validated on three real-world TIAGo++ tasks, with consistent improvements across Diffusion Policy and ACT using 25–40 minutes of online interaction.The framework also uses sparse expert guidance covering no more than 20% of collected data.Source: §1 · p. 2

2 Related Work

Related work addresses real-world robot fine-tuning, residual reinforcement learning, mobile-manipulation coupling, and human-in-the-loop corrections. HiRe-MoMa positions its contribution at the intersection of policy-agnostic adaptation and structured whole-body control.

2 Related Work

  • Prior residual and human-in-the-loop systems fine-tune fixed-base arms, whereas HiRe-MoMa targets on-robot fine-tuning of a whole-body mobile manipulator without simulation-pretrained latent action spaces.The cited literature also includes off-policy robot fine-tuning and residual correction methods.Source: §2 · p. 3
  • Flat residual methods provide additive corrections to frozen policies, but do not structure the cross-module coupling between a mobile base and arm.Policy post-training can also be hampered by intractable log-likelihoods.Source: §2 · p. 3
  • Mobile-manipulation methods often rely on simulation-pretrained priors or expensive skill discovery, while HiRe-MoMa operates in the raw action space of a pretrained policy under sparse success-only reward.Source: §2 · p. 3
  • HiRe-MoMa treats human corrections as constraints on the critic’s value surface, using them to reflect expert priorities rather than only as safety buffers, imitation data, or auxiliary supervision.Source: §2 · p. 3

3 Problem Formulation

The problem formulation considers online refinement of pretrained policies for complex real-world tasks. It models the setting as a partially observable decision process with sparse success rewards and an objective of maximizing discounted return.

3 Problem Formulation

  • Supervised fine-tuning is bounded by the coverage and quality of scarce demonstrations, whereas reinforcement learning can explore beyond static datasets.Online refinement is framed as especially important for high-dimensional mobile manipulators facing a deployment-time gap after zero-shot transfer.Source: §3 · pp. 3–4
  • The task is formalized as a POMDP with hidden environment states, high-dimensional actions, multimodal observations, and real-world physical transition dynamics.A sparse binary reward assigns 1 to task success and 0 otherwise.Source: §3 · p. 4
  • Policy refinement aims to maximize expected cumulative discounted return, a formulation directed toward higher task success rates and improved temporal efficiency in real-world execution.Source: §3 · p. 4

4 Method

HiRe-MoMa fine-tunes a frozen prior with residual corrections, separates base and manipulator corrections through unidirectional conditioning, and uses sparse human interventions to align the critic with expert preferences.

4 Method

  • HiRe-MoMa combines a policy-agnostic residual, hierarchical decoupling, and expert-guided value alignment for real-world adaptation.The residual learns additive corrections atop a frozen prior without structural assumptions about the base policy’s architecture.Source: §4 · p. 4, §4.1 · p. 4

4.1 Hierarchical Residual Architecture

  • The residual policy is optimized for expected return while entropy regularization keeps it localized around the base policy’s trajectory.The method uses off-policy value-based learning with temporal-difference updates and a conservative target derived from the minimum of a critic ensemble.Source: §4.1 · p. 4
  • Hierarchical decoupling splits the residual into module-specific corrections and conditions the manipulator policy on the mobile-base correction.This reflects the base-to-arm dependency and is intended to reduce gradient interference and improve training stability.Source: §4.1 · p. 5

4.2 Expert-Guided Value Alignment

  • Expert-guided value alignment uses human interventions from an intervention buffer to reshape the critic rather than directly imitating correction actions.The critic is encouraged to favor human actions while remaining pessimistic toward autonomous exploration.Source: §4.2 · p. 5

5 Experiments

Real-world TIAGo++ experiments evaluate HiRe-MoMa across three mobile-manipulation tasks, comparing fine-tuning baselines, component additions, and residual corrections. The full method reports stronger task performance, while ablations and visualizations examine hierarchical decoupling, value alignment, and correction behavior.

5.2 Comparative Evaluation: Performance and Generality

  • 95% Place Only, 90% Sweep Litter, and 40% Pick Place (up from 22.5% with the frozen base) were reached after 25–40 minutes of fine-tuning.The full method also consistently reduces episode length relative to the baselines.Source: Table 1 · p. 6, §5.2 · p. 7

5.1 Experimental Setup

  • +HG-DAgger learns from human corrections, while +Flat Residual applies off-policy RL to a monolithic residual vector; the build-up adds hierarchical decoupling and then alignment.Diffusion Policy is the default prior, and ACT is also evaluated.Source: §5.1 · pp. 6–7
  • On TIAGo++, evaluation covers Place Only, Sweep Litter, and Pick Place, with randomized initializations, 20 trials per task, and task-specific success metrics.The platform uses a 17-dimensional action space with torso and head fixed; interventions are collected online through teleoperation.Source: §5.1 · pp. 6–7

5.2 Comparative Evaluation: Performance and Generality

  • With ACT as the prior, HiRe-MoMa reports a 94% relative improvement in success rate, extending evaluation beyond the default Diffusion Policy.The reported gains across ACT and Diffusion Policy support policy-agnostic generalization; the performance ceiling remains tied to base-policy quality.Source: Table 3 · p. 7, §5.2 · p. 7

5.3 Build-Up Ablation: Hierarchical Decoupling and Value Alignment

  • Moving from +Flat Residual to +Hierarchical Decoupling raises average success from 50.0% to 85.0% and shortens episode length from 361.5 to 293.5 steps.The residual is split into base and arm heads, with the arm conditioned on the base correction to mitigate gradient interference and reflect the platform hierarchy.Source: Table 2 · p. 6, §5.3 · p. 8
  • Adding Expert-Guided Value Alignment raises average success to 92.5% and reduces episode length to 253.0 steps; decomposition supplies most of the gain, with conditioning adding a smaller, task-dependent improvement.The alignment loss penalizes critic deviations from expert preferences; the human-intervention ratio decreases monotonically when alignment is added.Source: §5.3 · p. 8, Figure 5 · p. 8

5.4 Qualitative Analysis: Interpretability and Visualization

  • On Place Only, residual corrections concentrate at decision boundaries, compensating for base inertial drift and arm kinematic inaccuracies within a bounded correction magnitude.The visualization follows base forward velocity and the arm’s downward placement motion, with corrections during base acceleration and the lay-down phase.Source: §5.4 · p. 8, Figure 6 · p. 8

6 Limitations and Future Work

The evaluation is limited to one TIAGo++ platform and 20 trials per condition, with Pick Place constrained by hardware and policies not yet reliable for unsupervised deployment around people.

6 Limitations and Future Work

  • Evaluation covers one TIAGo++ platform with 20 trials per condition, and Pick Place remains bounded by platform hardware.The shared sparse reward lacks explicit hierarchical credit assignment, and training requires a warm-up buffer of successful on-robot rollouts; each run also requires a supervising operator.Source: §6 · p. 9
  • Policies are not yet reliable for unsupervised deployment around people.Future work includes contact-rich tasks with multimodal sensing, richer inter-module structures, and validation on more platforms.Source: §6 · p. 9

7 Conclusion

HiRe-MoMa combines policy-agnostic residual learning, hierarchical decoupling, and expert-guided value alignment. Its reported gains include higher success from decomposition and further task-dependent improvement from conditioning.

7 Conclusion

  • Across three TIAGo++ tasks, HiRe-MoMa raises average success from 39.2% to 75% within 25–40 min of online interaction, shortens execution by 1.29×, and remains effective across base architectures with less than 20% human intervention.The framework combines a policy-agnostic residual, hierarchical decoupling with unidirectional conditioning, and expert-guided value alignment.Source: §7 · p. 9

A.1 Disentangling Decomposition and Conditioning

  • Decomposition raises average success from 50.0% to 80.0% (+30 pt) and reduces average episode length from 361.5 to 306.5 steps.Adding unidirectional conditioning yields a further +5 pt average success gain, concentrated on Sweep Litter (75% →85%) rather than Place Only (85% →85%).Source: §A.1 · p. 14, Table 4 · p. 14

A.2 Training Steps vs. Wall-Clock Time

HiRe-MoMa adds modest per-step compute over the flat residual while improving success rate and episode length. The section also describes its training algorithms, robot interface, and asynchronous training setup.

A.2 Training Steps vs. Wall-Clock Time

  • HiRe-MoMa uses roughly 1–2× the per-step compute of +Flat Residual while delivering consistent gains in success rate and episode length.HG-DAgger advances through training steps faster, but its higher throughput does not translate into higher success rates; SAC updates are roughly an order of magnitude slower per step than its single regression step.Source: §A.2 · p. 14, Figure 7 · p. 16

B Pseudo Code

  • The pseudocode specifies an SAC-style training loop with Expert-Guided Value Alignment and a separate hierarchical-decoupling design.The loop uses a warm-up buffer of successful on-robot rollouts and stores human interventions in an expert buffer.Source: B Pseudo Code · pp. 14–15

C.1 Hardware and Control Interface

  • TIAGo++ control outputs base and arm commands jointly, while human interventions preempt the residual policy with absolute priority.The experiments fix torso and head poses, leaving a 17-dimensional action space.Source: §C.1 · p. 16

C.2 Distributed Asynchronous Training

  • Training separates an on-robot rollout worker, a central learner, and an inference node that supplies frozen base-policy predictions.The rollout worker collects transitions at 10 Hz and syncs the latest actor checkpoint every 50 environment steps.Source: §C.2 · p. 16

C.3 Tasks and Evaluation Protocol

The evaluation uses three real-world TIAGo++ tasks of increasing complexity, with randomized trials and task-specific success criteria. It also specifies bounded residual actions and reports hardware-related failure limits for Pick Place.

C.3 Tasks and Evaluation Protocol

  • The three tasks are Place Only, Sweep Litter, and long-horizon Pick Place, with maximum episode lengths of 500, 500, and 800 steps, respectively.Pick Place requires approaching and grasping bread, transitioning between tables, and placing it on a target plate.Source: §C.3 · p. 17
  • Each task uses 20 randomized trials, and success is measured by task-specific criteria, including a phased score for Pick Place: 0.5 for grasping and lifting and 1.0 for placement.Trials exceeding the step limit or violating task-specific safety constraints count as failures.Source: §C.3 · p. 17

C.4 Failure Modes and Pick Place Ablation

  • HiRe-MoMa reduces frozen-DP failures from 10/20 to 1/20 on Place Only and from 11/20 to 2/20 on Sweep Litter.Pick Place failures mainly reflect hardware constraints: narrow arm workspace and sticking casters; the task is omitted from the build-up ablation.Source: §C.4 · p. 17

D.1 Observations and Actions

  • The residual correction magnitude is bounded by ∥ares∥≤ϵ with ϵ = 0.3, and continuous action dimensions are normalized to [−1, 1].The observation combines egocentric RGB images at 208 × 160 resolution with a task-dependent ∼20-dimensional proprioceptive vector.Source: §D.1 · p. 17

D.2 Network Architecture and Optimization

This section specifies the residual learner’s optimization, the frozen-policy pretraining and on-robot warm-up data, and the intervention protocol. It also situates HiRe-MoMa within real-world RL approaches to reward specification, autonomous learning loops, and sample efficiency.

D.2 Network Architecture and Optimization

  • The residual actor and critic use 256-hidden-unit MLPs with off-policy SAC, while the base policy stays frozen and the residual bound is ϵ = 0.3 across tasks.The auxiliary alignment-loss weight is λimi = 10−3; the bound trades correction capacity against exploration stability.Source: §D.2 · p. 17

D.3 Base-Policy Pre-Training and Warm-Up Buffer

  • Base policies are pretrained on human demonstrations, but fine-tuning uses a separate warm-up buffer of about 500–600 successful on-robot transitions, without access to base-policy training data.The warm-up buffer is collected by rolling out the frozen base policy, with operator takeover when needed.Source: §D.3 · p. 18

D.4 Human Intervention Protocol and Budget

  • Human intervention coverage stays below 20% on all tasks, falling from roughly 15–20% early in training to 0–3% near convergence.A single operator, blind to the method, intervenes when failure is imminent but recoverable; operator expertise is not systematically studied.Source: §D.4 · p. 18

E.1 Real-World Reinforcement Learning

  • HiRe-MoMa uses sparse binary success rewards, treats automated reset methods as complementary, and builds on off-policy RL to target mobile-manipulation coupling.The discussion distinguishes its residual fine-tuning focus from self-resetting approaches and notes the computational expense and limited adaptability of simulation-pretrained latent action spaces.Source: §E.1 · p. 18

E.2 Policy Post-Training and Residual RL

The related-work discussion contrasts policy post-training and residual RL with HiRe-MoMa’s structured approach to mobile manipulation. It also frames human corrections as information about value, not only as imitation data or a safety buffer.

E.2 Policy Post-Training and Residual RL

  • Generative base policies’ intractable likelihoods complicate standard policy updates, prompting surrogate objectives [13] [24] and alternative entropy measures [25] [26]; residual RL instead adds corrections to frozen policies [19] [28].Policy-agnostic RL is more general but can be unstable and over-conservative [27] [18]; HiRe-MoMa adds structured decoupling for mobile-manipulation coupling.Source: §E.2 · pp. 18–19

E.3 Mobile Manipulation

  • HiRe-MoMa works in the raw action space of pretrained policies and uses hierarchical dependencies under sparse success-only reward, unlike approaches relying on simulation or restricted latent actions [8] [36].Prior mobile-manipulation work also uses VLMs for high-level reasoning [31] [3] [1] and human video datasets to scale base capabilities [63].Source: §E.3 · p. 19

E.4 Human-in-the-Loop Robot Learning

  • HiRe-MoMa treats expert corrections as functional constraints on the critic’s value surface, complementing rather than replacing imitation-based uses of human input [38] [39].Prior intervention approaches also use corrections as safety buffers [20] or imitation data [38].Source: §E.4 · p. 19

F Limitations and Future Work

The limitations span the evaluated platform and trial scope, the spatial-only hierarchy and warm-up requirements, and the current sensing and module structure. Future work proposes adaptive regularization, richer sensing and inter-module structures, and validation across simulators and platforms.

F Limitations and Future Work

  • Evaluation is limited to TIAGo++, 20 trials per condition, no seed sweeps, and build-up ablations on two of three tasks; Pick Place results are hardware-bound.The spatial hierarchy shares one sparse reward, so arm failures can lower the value of correct base motion; human-assisted successes can inflate base-pose values, and sparse interventions limit but do not remove this bias.Source: F Limitations and Future Work · p. 19
  • HiRe-MoMa needs a small warm-up buffer of successful on-robot rollouts per task, despite not accessing base-policy training data.The current observation-conditioned MLP residual may miss history-dependent corrections, such as those needed by action-chunking base policies.Source: F Limitations and Future Work · p. 19

Adaptive Regularization

  • The residual bound ϵ = 0.3 and alignment weight λimi = 10−3 are fixed across tasks and may not be optimal for broader task distributions.Proposed directions include learnable weights, curriculum scheduling, or scheduling human priors based on performance and task complexity.Source: F Limitations and Future Work · p. 19

Multimodal Sensing Integration

  • The framework primarily uses visual and proprioceptive feedback, while fine-grained contact-rich tasks require precise force and tactile sensing for safety and dexterity.The authors identify visuotactile or force-torque fusion as a next step toward complex industrial applications.Source: F Limitations and Future Work · p. 20

Richer Inter-Module Structures

  • The two-level base-to-arm hierarchy fits evaluated tasks but does not directly extend to bimanual coordination, active torso and head control, or multi-stage temporal dependencies.Deeper or non-tree structures, latent representations, or graph-based conditioning could capture richer dependencies; obstacle-rich larger workspaces call for collision-aware base corrections.Source: F Limitations and Future Work · p. 20

Cross-Platform and Simulation Validation

  • Validation in high-fidelity simulators and on additional platforms such as Stretch or Fetch would clarify generalizability and platform-specific tuning needs.A wider arm workspace and more reliable base actuation could also lift the hardware ceiling observed on Pick Place.Source: F Limitations and Future Work · p. 20
Loading 2610.20607v1…