PaperlayerView exact revision on arXiv

Source-linked AI summary

HIDRA: Hierarchical Dual-Routing Attention for Replay-Free Lifelong Imitation Learning

Fanqi Yu, Matteo Tiezzi, Cigdem Beyan, Tommaso Apicella, Vittorio Murino

arXiv:2610.36374v1cs.ROcs.AI

TL;DR

Replay-free, single-pass imitation learning must adapt from multimodal demonstration streams without revisiting past data, yet sequential updates can disrupt expert routing. HIDRA separates instruction retrieval from context refinement and regularizes the routing space; on LIBERO, it improves overall performance and forward transfer while maintaining low forgetting.

  • Problem

    In replay-free, single-pass lifelong imitation learning, sequential updates can distort representations and disrupt expert selection, while past demonstrations may not be stored or revisited.

    Source: §1 · pp. 1–2

  • Method

    HIDRA combines dual attention that separates instruction-driven expert retrieval from context-driven refinement with key-level geometric regularization and expandable experts.

    Source: §1 · p. 2

  • Results

    On LIBERO, HIDRA achieves consistent gains in forward transfer and overall performance with low forgetting, outperforming all SOTA methods.

    Source: §5 · p. 10

  • Takeaways & Limitations

    HIDRA supports stable expert selection during sequential adaptation without replay, using an expandable expert set for continual learning.

    Source: §1 · p. 2

    Evaluation is on simulated LIBERO benchmarks, and scaling the expandable expert design to millions of tasks may require more efficient routing or expert management.

    Source: §A.10 · p. 21

Abstract

from arXiv · show

Robotic agents operating in real-world environments must adapt continuously from a stream of multimodal demonstrations, often under constraints that preclude storing or revisiting past data. In this replay-free, single-pass lifelong imitation learning setting, sequential updates can progressively distort the representation space, making expert-based approaches particularly sensitive to routing interference, causing expert selection to degrade over time. We propose HIDRA, a structured routing framework that mitigates this issue through a hierarchical dual-attention mechanism, which decouples instruction-level expert retrieval from context-dependent refinement. To further stabilize routing under representation drift, HIDRA introduces a key-level geometric regularization that enforces both alignment within tasks and separation across tasks. The resulting approach enables reliable expert selection while supporting an expandable set of experts for continual adaptation. We evaluate HIDRA on several LIBERO benchmark suites, including more challenging variants with heterogeneous task sequences and paraphrased language instructions. Our approach consistently outperforms both replay-free and replay-based baselines, improving AUC and forward transfer while maintaining low forgetting, with stronger gains under high interference and language variation. The code is available at: https://github.com/yfqi/HIDRA.

1 Introduction

Replay-free, single-pass lifelong imitation learning must handle multimodal streams without access to past demonstrations, while sequential updates can distort representations and disrupt expert routing. HIDRA separates instruction-driven retrieval from context refinement, adds geometric regularization, and uses expandable experts; the approach assumes known task boundaries.

1 Introduction

  • Real-world robots may receive an unbounded stream of demonstrations and observations while constraints prevent storing or revisiting past data.The paper frames replay-free, single-pass learning as central to lifelong robot learning.Source: §1 · p. 1
  • Representation entanglement and drift make reliable expert routing difficult: new tasks can disrupt prior expert assignments, and single-pass learning leaves little opportunity to correct errors.These constraints are especially relevant when pretrained backbones are adapted from heterogeneous demonstrations without replay.Source: §1 · p. 2
  • HIDRA decouples instruction-driven expert retrieval from context-driven refinement and adds key-level geometric regularization to stabilize expert selection under representation drift.The framework also supports an expandable expert set and multimodal streaming updates without storing past demonstrations.Source: §1 · p. 2
  • HIDRA follows the standard lifelong imitation learning protocol of known task boundaries; its streaming setting means single-pass, non-revisitable task-wise data arrival.Source: §1 · p. 2

2 Related Work

Prior lifelong imitation learning methods address retention and adaptation through replay, regularization, or modular architectures, each with constraints on data access, sharing, or routing. HIDRA builds on modular routing and targets interference when replay or repeated optimization cannot correct expert selection.

2 Related Work

  • Replay-based approaches can be effective, but depend on stored or generated past experience and may be unsuitable under memory, privacy, or deployment constraints.Approaches include raw trajectory, retrieval-based, latent, and generative replay.Source: §2 · p. 3
  • Regularization methods avoid storing past data but can struggle to retain prior behaviors and adapt across long task sequences with shared parameters.Multimodal distillation stabilizes representations and actions but does not explicitly support selective reuse of modular knowledge.Source: §2 · p. 3
  • Task-conditioned adapters can aid retention but often require task identity at inference or restrict parameter sharing, while observation-conditioned expert libraries can face routing ambiguity from entangled representations.These trade-offs limit flexibility, forward transfer, or expert separability in replay-free settings.Source: §2 · p. 3
  • In strict single-pass, replay-free settings, routing errors cannot be corrected by stored prototypes or repeated optimization, motivating HIDRA’s hierarchical, regularized separation of task identity and context.The discussion situates HIDRA among modular methods including LOTUS [5], ISCIL [6], and DMPEL [26].Source: §2 · p. 3

3 Method

HIDRA adapts a pretrained policy to replay-free task streams by separating instruction-guided expert retrieval from context-dependent refinement and regularizing the geometry of expert keys. Its setup combines multi-task behavioral-cloning pretraining with one-pass adaptation to unseen tasks.

3.1 Preliminaries

  • Multi-task behavioral cloning precedes adaptation to disjoint unseen tasks whose demonstrations are observed once and not stored, with only a subset of policy parameters adapted.The observations include multimodal inputs, and the pretraining objective is supervised imitation.Source: §3 · p. 3, §3.1 · p. 4

3.2 HIDRA: Hierarchical Dual-Routing Attention

  • Instruction-level retrieval selects a language-semantic anchor and restricts context refinement to previously learned experts in its anchor-conditioned candidate pool.The context branch combines instruction and state-dependent evidence; experts introduced after the anchor are excluded from reuse.Source: §3.2 · p. 5
  • The instruction query captures invariant task semantics, while the context query pools recent visual and proprioceptive embeddings to represent momentary state.The temporal mean-pooling operation provides a non-parametric projection into the key space.Source: §3.2 · p. 5

3.3 Key-Space Regularization

  • For each task, HIDRA maintains a running-average context-query prototype, aligns that task’s context keys to it, and penalizes similarity between keys from different tasks.The alignment encourages task-relevant key structure, while the inter-task penalty encourages approximate orthogonality.Source: §3.3 · p. 6
  • The lifelong objective combines behavioral-cloning loss with weighted intra-task alignment and inter-task separation losses.The method summarizes this regularization as preserving key separability as the expert library expands.Source: §3.3 · p. 6

4 Experimental Analysis

HIDRA is evaluated in replay-free lifelong learning on standard and heterogeneous LIBERO task sequences, against replay-based and replay-free methods. The reported results favor HIDRA overall, while ablations and activation analyses examine how its routing design and regularization relate to performance and expert selection.

4 Experimental Analysis

  • LIBERO-MIX interleaves tasks from four suites; Inc1 is fully sequential, while Inc4 introduces four tasks per step and is more challenging.Evaluation also covers OBJECT, GOAL, SPATIAL, LIBERO-10/LONG, and LIBERO-INS; methods share pretrained initialization, and results are averaged over three seeds.Source: §4 · pp. 6–7

4.1.1 Comparison with SOTA Methods

  • HIDRA achieves the best AUC across all four standard LIBERO benchmarks, and leads FWT and AUC on both MIX settings plus overall performance on INS.For MIX Inc4, forgetting remains competitive against DMPEL and ISCIL.Source: Table 1 · p. 7, §4.1.1 · pp. 7–8, Table 2 · p. 8

4.1.2 Ablation Studies

  • Unstructured single-projection routing performs poorly; instruction routing beats context-only routing, and dual attention with both regularizers yields the best AUC and usually FWT.NBT improvements are less consistent.Source: Table 3 · p. 8, §4.1.2 · p. 8
  • Running Mean achieves the highest AUC across both benchmarks, while orthogonality-based inter-task regularization improves AUC with stable FWT and NBT.The comparisons cover prototype estimation and inter-task regularization on LIBERO-OBJECT and LIBERO-SPATIAL.Source: Figure 3 · p. 9, §4.1.2 · p. 9
  • Hierarchical routing achieves the best AUC; removing the preselected-expert mask significantly degrades performance, while heuristic mixing can reduce forgetting.The routing-strategy comparison varies how instruction- and context-level signals are combined and constrained.Source: Figure 3 · p. 9, §4.1.2 · p. 9

4.1.3 Expert Activation Patterns

  • HIDRA shows stronger diagonal expert activation than DMPEL; intra-task regularization sharpens it further, and adding inter-task regularization yields the most pronounced diagonal.Diagonal attention represents within-group routing, while off-diagonal attention represents cross-group activation.Source: §4.1.3 · p. 9, Figure 4 · p. 10

5 Conclusion

HIDRA addresses routing interference in replay-free, single-pass lifelong imitation learning with dual attention, geometric regularization, and expandable experts. On LIBERO it reports stronger overall performance and forward transfer with low forgetting, while identifying sensitivity on LIBERO-SPATIAL as a limitation.

5 Conclusion

  • HIDRA targets lifelong imitation learning in which agents adapt continuously from multimodal demonstration streams without revisiting past data.The conclusion identifies routing interference as a challenge because it degrades expert selection over time.Source: §5 · p. 10
  • Dual attention separates instruction retrieval from context refinement, while key-level geometric regularization and expandable experts support sequential adaptation without replay.Source: §5 · p. 10
  • On LIBERO, HIDRA reports consistent gains in forward transfer and overall performance with low forgetting, outperforming prior methods.Source: §5 · p. 10
  • LIBERO-SPATIAL remains a limitation: fine-grained spatial differences require precise context beyond instruction semantics, making routing sensitive to subtle perceptual variation.The conclusion also calls for evaluation on real robotic platforms.Source: §5 · p. 10

A Technical Appendices and Supplementary Material

The supplementary material extends the paper’s analyses and implementation details. It covers ablations, routing behavior, efficiency, multi-pass robustness, and benchmark task and instruction lists.

A Technical Appendices and Supplementary Material

  • Supplementary ablations vary regularization weights, selected-expert count, routing projection capacity, and demonstrations per task.Source: A Technical Appendices and… · p. 14
  • Additional materials analyze routing behavior, computational and storage efficiency, and robustness under multi-pass training.Efficiency measures include FLOPs, inference time, and per-task memory overhead.Source: A Technical Appendices and… · p. 14
  • The supplement provides full task and instruction lists for LIBERO-MIX and LIBERO-INS to support reproducibility.Source: A Technical Appendices and… · p. 14

A.1 Implementation Details

HIDRA combines pretrained multimodal representations with a two-branch router that fuses instruction-level and context-level evidence into task-specific expert scores.

A.1 Implementation Details

  • Language features condition visual and proprioceptive representations through FiLM, and a GPT-2 temporal decoder maps the fused features toward action prediction.CLIP encoders provide visual and language features; a dedicated policy head maps the decoder representation to actions.Source: §A.1 · p. 14
  • LoRA experts use fixed rank r = 16 on linear layers of the frozen encoders and temporal backbone.The low-rank constraint keeps per-task parameter growth minimal while supporting task-specific adaptation.Source: §A.1 · p. 14
  • Instruction keys start from task language embeddings, while context keys are learned to capture environmental cues under key-space regularization.At inference, the instruction query comes from the current language embedding and the context query from the fused multimodal state.Source: §A.1 · p. 14
  • Separate instruction and context projections are concatenated and passed through an MLP with sigmoid activation to produce a fused routing score.The task-specific projection uses hidden dimension du = 8 to combine both kinds of routing evidence.Source: §A.1 · p. 14

A.2 Extended Ablation Study

The extended ablations examine regularization weights, expert count, projection capacity, demonstration volume, prototype estimation, and routing choices, revealing trade-offs between performance and stability.

Effect of regularization weights

  • λintra = 1.0 and λinter = 1.0 provide the best overall balance, achieving the highest AUC across both benchmarks with competitive forward transfer and moderate forgetting.Small λinter raises FWT but worsens forgetting, while intermediate λinter reduces NBT at the cost of overall performance; reducing λintra degrades FWT and AUC.Source: §A.2 · pp. 14–15, Table 4 · p. 15

Effect of the number of selected experts

  • K = 3 achieves the best AUC and forward transfer across both benchmarks, while K = 2 reduces forgetting and K = 4 provides no further gains.Performance remains robust across the tested expert counts, with moderate selection offering the best balance.Source: §A.2 · p. 15, Table 5 · p. 15

Effect of the task-specific routing projection dimension du

  • A hidden dimension of 8 achieves the highest AUC on LIBERO-OBJECT and strong performance across metrics, while smaller dimensions reduce performance and larger ones yield no consistent overall gain.The results indicate a non-monotonic capacity trade-off; dimension 8 is used in the final model.Source: §A.2 · p. 15, Table 6 · p. 16

Effect of the number of demonstrations per task

  • Increasing demonstrations improves FWT and AUC across both benchmarks, with the largest gains when increasing from 10 to 20 demonstrations per task.NBT does not improve monotonically with more demonstrations, although forgetting remains generally well-controlled.Source: §A.2 · pp. 15–16, Table 7 · p. 16

Prototype estimation strategies

  • The running-mean prototype strategy achieves the best AUC and FWT across benchmarks, while EMA-based methods yield lower NBT.The running mean offers a better overall performance–stability trade-off than batch-level or momentum-based alternatives.Source: §A.2 · p. 16, Table 8 · p. 16

Inter-task regularization strategies

  • Orthogonality loss achieves the best AUC and FWT across benchmarks, while InfoNCE yields lower NBT, indicating a performance–stability trade-off.The results support geometric structure in the routing space, with stability not uniformly maximized by the best-performing objective.Source: §A.2 · p. 16, Table 9 · p. 16

Routing strategies

  • Structured routing with hierarchical selection, masking, and learned fusion achieves the best AUC and FWT; removing masking significantly degrades performance.Heuristic combinations can lower forgetting, while learned fusion provides a better overall performance–stability trade-off.Source: §A.2 · pp. 16–17, Table 10 · p. 17

A.3 Router Selection Patterns

Routing statistics on LIBERO-MIX show that the proposed regularization makes expert selection more semantically aligned with query tasks.

A.3 Router Selection Patterns

  • The analysis measures same-group and cross-group expert-selection counts and attention mass, alongside entropy as a measure of routing dispersion.The benchmark groups are OBJECT, SPATIAL, GOAL, and LONG.Source: §A.3 · p. 17
  • Adding intra-task regularization increases same-group selection and attention while reducing cross-group statistics; inter-task regularization strengthens this pattern further.The highest same-group selection and attention occur after adding inter-task regularization.Source: §A.3 · p. 17
  • DMPEL shows higher cross-group selection and attention and higher routing entropy than the proposed method, indicating less semantically consistent and more dispersed decisions..Source: §A.3 · p. 17

A.4 Computational and Storage Efficiency

The efficiency analysis finds negligible computational overhead from regularization and lower per-task storage than DMPEL, which maintains past data.

A.4 Computational and Storage Efficiency

  • Forward Time measures single-action inference latency, FLOPs measure per-forward-pass computation, and Train-FLOPs estimate training cost as FLOPs × iterations × 3.Latency is measured on an NVIDIA A100 using 10 warm-up runs and 100 timed runs; reported values are mean and standard deviation.Source: §A.4 · pp. 17–18
  • Adding intra-task and inter-task regularization leaves forward latency, FLOPs, and Train-FLOPs nearly unchanged, with efficiency similar to DMPEL.The table reports computational cost on LIBERO-OBJECT.Source: Table 12 · p. 17, §A.4 · p. 18
  • Each task adds approximately 2773 KB of storage, while DMPEL requires approximately 16× more memory per task.HIDRA stores newly added task-specific components rather than past data; most of its storage overhead comes from adapter-based experts, and the router contributes a small fraction.Source: §A.4 · p. 18, Table 13 · p. 18

A.5 Comparison with SOTA Methods under Multi-pass Training

Under multi-pass training, HIDRA achieves the best AUC in most settings and maintains strong forward transfer, with performance improving further in the 5-pass regime across benchmarks.

A.5 Comparison with SOTA Methods under Multi-pass Training

  • Each task is trained for 3–5 epochs to assess robustness beyond the target single-pass setting.This follows prior work allowing limited data revisitation after large-scale pre-training.Source: §A.5 · p. 18
  • HIDRA achieves the best AUC in most settings and consistently strong forward transfer, with further gains across benchmarks in the 5-pass regime.Some baselines have lower NBT in specific cases, while HIDRA provides a better overall balance between adaptation and retention.Source: §A.5 · p. 18
  • The results indicate that HIDRA maintains strong performance when multiple passes over the data are allowed.The reported gains improve further in the 5-pass regime across all benchmarks.Source: §A.5 · p. 18

A.6 Further Analysis Related to Routing Mechanism

The routing analysis finds semantic structure in context-query embeddings and identifies potential expert-selection ambiguity. It also reports empirical routing stability and evaluation boundaries rather than formal guarantees.

A.6 Further Analysis Related to Routing Mechanism

  • Tasks with similar semantics cluster in UMAP context-query space, while similar task keys can produce similar attention scores and couple expert selection.This reduces expert discriminability and the diversity of expert combinations as tasks accumulate.Source: §A.6 · p. 18, Figure 5 · p. 19
  • Inter-task regularization encourages separation between context keys of different tasks while preserving their representational fidelity.The analysis identifies this as useful for addressing reduced expert discriminability.Source: §A.6 · p. 19

A.7 Task Instructions used in LIBERO-INS and LIBERO-MIX

A.8 Statistical Significance

  • Welch’s t-tests find significant AUC improvements over most baselines, especially on LIBERO-OBJECT and LIBERO-SPATIAL, while small gaps are not always significant.Table 17 reports approximate significance for HIDRA versus representative baselines.Source: §A.8 · pp. 20–21, Table 17 · p. 21

A.9 Routing stability under library expansion

  • Previously learned experts, routing projections, and keys remain frozen, but a new instruction key can still replace the selected anchor; LIBERO-OBJECT had zero observed retroactive switches.The authors describe this as empirical stability, not a formal guarantee.Source: §A.9 · p. 21
  • A new expert is initialized with an instruction key matching its task query and remained selected during adaptation in all experiments, but arbitrary task sequences are not guaranteed to do so.The new expert is not enforced through a hard inclusion constraint.Source: §A.9 · p. 21

A.10 Limitations and Future Work

  • HIDRA performs less well on tasks requiring fine-grained spatial reasoning, where expert selection relies more on context representations.The authors identify improving spatial representations and context modeling as future work; competing methods show similar or larger drops in these settings.Source: §A.10 · p. 21
  • Evaluation remains in simulated LIBERO environments, and scaling expandable experts to millions of tasks may require more efficient routing or expert management.Real-robot evaluation would assess robustness to sensor noise and actuation variability.Source: §A.10 · p. 21

Related papers

Loading 2610.36374v1…