PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-Repairing Recurrent Ensembles for Real-Time Recovery from Distribution Shift

Julian Lemmel, Pedro D. Wendel Garcia, Taisuke Kobayashi, Radu Grosu

arXiv:2610.03249v1cs.NEcs.ROcs.SYeess.SY

TL;DR

Pretrained controllers can lose the ability to act under sensor shifts, often when no expert is available to correct them. The paper uses a masked recurrent ensemble and self-supervised online updates; on simulated locomotion tasks, it recovers close to pre-shift performance after a sudden sensor shift, unlike an ensemble with full observation visibility.

  • Problem

    Sensor shifts can make a deployed controller unable to act, while expert corrective labels may be unavailable when the shift occurs.

    Source: Introduction · p. 1

  • Method

    A recurrent ensemble with randomly masked observations fuses members’ Gaussian actions into a consensus used as a self-supervised label for per-step fine-tuning with RFLO.

    Source: Introduction · pp. 1–2, Conclusion · p. 7

  • Results

    On simulated locomotion tasks, RTR-FTL recovers close to pre-shift performance after a sudden sensor shift, while an ensemble with full observation visibility collapses and does not recover.

    Source: Conclusion · p. 7

  • Takeaways & Limitations

    Replacing the consensus label with an expert action gives fully online interactive imitation learning without a separate training phase.

    Source: Conclusion · p. 7

    The approach remains untested on simultaneous mixed multisensor corruption, action-correlated shifts, and a physical platform.

    Source: Conclusion · p. 7

Abstract

from arXiv · show

Deploying a pretrained controller exposes it to conditions that are absent from its training data. Sensor drift, outright sensor failure and accumulating measurement noise all induce a distribution shift that can collapse an otherwise competent policy; typically at a point in time where no expert is available to supply corrective labels. We present a method that lets a policy recover from such shifts online and without supervision. Our controller is an ensemble of recurrent networks, each of which observes a randomly masked subset of the observation vector, and whose Gaussian outputs are combined through sequential Kalman fusion so that confident members dominate the consensus action. At deployment, we treat this consensus as a self-supervised label and fine-tune each member towards it, scaling each member's contribution proportional to the complement of its squared Kalman gain. Gradients are computed using RFLO, an efficient and biologically plausible approximation of Real-Time Recurrent Learning, so that a parameter update follows every environment step and the policy reacts to a shift as it unfolds. On a range of simulated continuous control tasks, our approach recovers close to the original performance after a sensor shift, while ensembles that see the full observation are unable to recover. The same framework subsumes fully online interactive imitation learning: when an expert is present, the consensus label is replaced by the expert action and the identical update rule refines the policy during teleoperation.

Introduction

The paper targets unsupervised recovery when sensor shifts disrupt a deployed controller and no expert is available. It uses a diverse recurrent ensemble’s fused consensus to guide online updates, and extends the update rule to expert-guided imitation.

Introduction

  • Sensor drift, failure, or rising noise can push a deployed controller outside its training distribution and cause it to lose the ability to act.The expected observation shape can remain unchanged even when its values are unfamiliar.Source: Introduction · p. 1
  • Randomly masked observation subsets let unaffected ensemble members continue predicting sensible actions when a shift corrupts dimensions used by other members.The paper uses this diversity to identify reliable members and move affected ones back towards them.Source: Introduction · p. 1, Figure 1 · p. 1
  • Sequential Kalman fusion weights confident Gaussian predictions more heavily, and the fused mean becomes a self-supervised label for fine-tuning members.Each member’s contribution is weighted by the complement of its own contribution to the fusion.Source: Introduction · p. 2
  • RFLO supplies recurrent-network gradients at every environment step, allowing adaptation as a shift unfolds rather than after the episode.It avoids the sequence buffering and replay required by conventional backpropagation through time.Source: Introduction · p. 2
  • Replacing the consensus label with an expert action makes the same update rule a fully online interactive imitation-learning method.The method requires no separate training phase.Source: Introduction · p. 2
  • Across simulated shifts, random observation masking is necessary for recovery: full-observation ensembles fail to recover under the same shift.The evaluation includes shifted readings, complete sensor failure, and gradually increasing noise.Source: Introduction · p. 2

Related Work

The related work situates the paper among sensor-robust training, ensemble-based imitation learning, test-time adaptation, and online recurrent learning. Its approach uses ensemble diversity for self-supervision and RFLO for real-time updates.

Robustness to sensor corruption

  • Prior sensor-corruption training yields robustness only to corruptions anticipated during training; here masking creates ensemble diversity, while online fine-tuning supplies recovery.The paper reports that masking alone, without adaptation, does not restore performance.Source: Related Work · p. 2

Ensemble policies and uncertainty

  • Ensembles are used in imitation learning to decide when a novice should defer to an expert (Menda, Driggs-Campbell, and Kochenderfer 2019; Kobayashi 2026), whereas this work uses an ensemble as a source of self-supervision.The fusion rule combines Gaussian predictions through sequential Kalman updates, weighting members by their own reported confidence.Source: Related Work · p. 2

Test-time adaptation

  • Unlike majority-vote adaptation for language models (Zuo et al. 2025), this loss fuses continuous action distributions and uses direct behavioral cloning for single-step gradients.The paper contrasts this with optimizing a proxy reward using reinforcement learning.Source: Related Work · p. 2

Online learning for recurrent networks

  • RFLO approximates RTRL by dropping non-local gradient terms, and the paper applies real-time recurrent gradients to unsupervised distribution-shift recovery.The authors state this is, to their knowledge, the first such use.Source: Related Work · p. 2

Background

The background reviews interactive imitation learning and recurrent-network gradient methods that underpin the paper’s online approach. It covers expert-labeled novice trajectories, safety-gated variants, recurrent models, and RFLO.

Background

  • The section reviews interactive imitation learning and safety-gated DAgger variants alongside recurrent models and online gradient computation.These are the foundations for RTR-IIL and the proposed recurrent approach.Source: Background · p. 3

Interactive Imitation Learning

  • DAgger collects expert labels on states visited by a novice, including recovery states, and alternates training with experience collection to improve robustness.SafeDAgger and HG-DAgger restrict control based on whether the novice can be trusted; EnsembleDAgger also gates on ensemble action variance.Source: Interactive Imitation Learning · p. 3

RNNs and Online Gradient Computation

  • The paper introduces CT-RNNs and linear recurrent units as recurrent-model families compatible with its approach and fast online gradients.Linear recurrent units describe latent dynamics through a linear system, enabling efficient state updates.Source: RNNs and Online Gradient Computation · p. 3
  • RFLO approximates RTRL by dropping non-local terms for a local, constant-cost trace update and using fixed random feedback; the paper uses it for online RTR-IIL and RTR-FTL updates.The cited description says the update is applied at every step from output error.Source: RNNs and Online Gradient Computation · p. 3

Methods

The methods combine streaming expert-guided recurrent imitation learning with an ensemble that fuses member predictions. For expert-free recovery, the fused action becomes a self-supervised label for online fine-tuning.

Methods

Real-Time Recurrent Interactive Imitation Learning

  • RTR-IIL queries the expert when ensemble variance or novice–expert action distance exceeds a threshold, then updates recurrent parameters using the output gradient and Jacobian trace.The procedure combines each module's output-gradient vector with its RNN Jacobian trace to compute parameter updates.Source: Real-Time Recurrent Interactive… · p. 4
  • The ensemble combines Gaussian action predictions sequentially using Kalman gains, which weight each member's confidence against the tentative fused distribution.The fusion order is fixed in this paper, although the authors note that order matters.Source: Real-Time Recurrent Interactive… · pp. 4–5

Recovery from Distribution Shifts via Self-Supervised Fine-Tuning

  • Random binary masks train each ensemble member to predict expert actions from a different subset of observation dimensions.The masking setup is intended to address single-sensor failure and increase diversity among policy functions.Source: Recovery from Distribution Shifts via… · p. 5
  • RTR-FTL uses the mean of the fused action distribution as a self-supervised label and scales each member's log-probability contribution by a coefficient based on its squared Kalman gain.This replaces the discrete majority-vote signal described for Zuo et al. (2025) with a fused continuous action distribution.Source: Recovery from Distribution Shifts via… · p. 5
  • RFLO computes the fine-tuning gradient and applies a parameter update after every environment step.The authors state that this lets the policy react to distribution shifts during operation.Source: Recovery from Distribution Shifts via… · p. 5

Experiments

The experiments test online recovery from sensor shifts, failures, and rising noise, as well as interactive imitation learning across continuous-control tasks. RTR-FTL recovers close to original performance in two masking settings after a sensor shift, and improves outcomes in the other tested corruption settings.

Experiments

  • The evaluation compares RTR-IIL with CT-RNN, LRU, and EnsembleDAgger variants on ant, halfcheetah, and humanoid tasks.PPO policies served as experts, and their trajectories were used to pretrain ensembles later fine-tuned with RTR-FTL.Source: Experiments · p. 5
  • For training from scratch, RTR-IIL with LRU is on par with EnsembleDAgger and performs well with CT-RNN except on humanoid, while remaining fully online.The comparisons include EnsembleDAgger variants using MLPs or RNNs trained with BPTT.Source: Experiments · pp. 5–6

Self-Repair Experiments

  • The self-repair experiments use pretrained ensembles and apply a parameter update at each inference step, rather than training from expert labels during fine-tuning.The ensembles were pretrained using 50 trajectories from an expert RL policy.Source: Self-Repair Experiments · p. 6
  • Policies with 50% and 70% observation masking recover close to original performance after a sensor shift, while all tested models remain unable to move without learning.The shift was applied at observation index 5; Figure 2 shows the recovery during fine-tuning.Source: Self-Repair Experiments · p. 6, Figure 2 · p. 6
  • RTR-FTL significantly improves results after the ant observation at index 10 is pinned to zero to simulate total sensor failure.Index 10 corresponds to the angle between the two links of the left back leg.Source: Self-Repair Experiments · p. 6, Figure 4 · p. 7
  • RTR-FTL improves halfcheetah distance traveled over a no-learning control under linearly increasing sensor noise during an unrestarted episode.The experiment applied rising Gaussian noise to observation indices [0, 1] after 50k steps.Source: Self-Repair Experiments · p. 7, Figure 5 · p. 7

Conclusion

The recurrent ensemble recovers from distribution shifts during deployment without expert supervision. Its masking and fusion support self-supervised updates at every environment step, while its tested scope leaves several shift conditions and physical deployment unresolved.

Conclusion

  • RTR-FTL uses the fused consensus as a self-supervised label and updates each member according to its contribution to fusion.RFLO makes updates available at every environment step, enabling adaptation as a shift unfolds.Source: Conclusion · p. 7
  • Replacing the consensus label with an expert action yields RTR-IIL, a fully online interactive imitation method requiring no separate training phase.The same update machinery serves both unsupervised recovery and expert-guided teleoperation.Source: Conclusion · p. 7
  • RTR-FTL recovers close to pre-shift performance after sudden sensor shifts, while full-observation ensembles collapse and do not recover.The comparison supports masking-induced diversity, rather than the ensemble or fusion rule alone, as the source of robustness.Source: Conclusion · p. 7
  • The approach was also evaluated under complete sensor failure and gradually accumulating noise, extending tests to instantaneous and episode-long failure modes.The authors identify simultaneous mixed multisensor corruption and shifts correlated with agent actions as untested.Source: Conclusion · p. 7
  • The approach remains untested for simultaneous mixed multisensor corruption, action-correlated shifts, and transfer to a physical platform.The authors identify these as future-work boundaries.Source: Conclusion · p. 7
Loading 2610.03249v1…