PaperlayerView exact revision on arXiv

Source-linked AI summary

Privileged Critic Training Enables Sensor-Free Thruster Fault Adaptation in End-to-End RL

Ricard Marsal I Castan, Miguel A. Olivares-Méndez

arXiv:2608.22976v1cs.RO

TL;DR

Thruster robots must adapt to continuous and multi-mode failures without dedicated fault sensing, while oracle policies require unavailable degradation-state access. The paper trains a PPO actor with task observations and a privileged critic, and reports 70.2% success at four failures while the actor remains sensor-free.

  • Problem

    Fault-tolerant thruster navigation needs adaptation to continuous, multi-mode failures, but classical pipelines require dedicated fault detection and oracle policies require true degradation sensing.

  • Method

    RAFT trains a PPO actor-critic asymmetrically, giving the critic Dgt during training while the actor receives only standard task observations.

  • Results

    70.2% success at four simultaneous failures closes 84% of the gap from the failure-naive 4.8% baseline to the Oracle 82.4%, while VAN-MLP-AC closes 79%.

  • Takeaways & Limitations

    The results identify privileged critic access as the primary fault-tolerance mechanism, with RAFT’s GRU adding 3.8 pp success rate.

  • Takeaways & Limitations

    Experiments are entirely simulated and evaluate only 2D navigation; sim-to-real transfer and 6-DOF tasks are not evaluated.

Abstract

from arXiv · show

Fault-tolerant navigation for thruster-actuated robots requires online adaptation to failures that are neither binary nor fully observable: thrusters may degrade continuously, fail dead, or jam stuck-open. Classical fault detection pipelines require dedicated sensors unavailable at deployment; oracle controllers that observe the true failure state are equally impractical. We show that privileged critic training is sufficient for sensor-free fault adaptation: giving the PPO value function access to the true degradation state dgt during training, while the actor receives only standard task observations, shapes a policy that compensates for failures at deployment without any dedicated fault sensing. We propose RAFT (Recurrent Asymmetric Fault Tolerant), a policy with recurrent memory trained with a privileged asymmetric critic. Evaluated on a floating-platform robot (8 thrusters, 1 reaction wheel) under up to four simultaneous thruster failures, RAFT achieves 70.2% success at four concurrent failures, closing 84% of the gap from a failure-naive baseline (4.8%) to an oracle policy that sees the full degradation state at deployment (82.4%). All code, checkpoints, and data are open-source.

I. INTRODUCTION

Thruster robots face continuous, multi-mode, simultaneous degradation that makes explicit fault detection and mode-specific recovery difficult. The paper asks whether privileged critic access during training can produce sensor-free fault tolerance at deployment.

  • Continuous degradation, dead failures, and stuck-open actuators require online detection and compensation across multiple thrusters.
  • Classical fault-tolerant control uses fault detection and isolation followed by switched recovery, but dedicated signatures and fault combinations make the pipeline brittle.
  • Oracle policies receive the true degradation vector, whereas vanilla policies train failure-free; neither is deployment-ready.
  • Asymmetric actor-critic training gives the critic privileged degradation access while restricting the actor to task observations, targeting sensor-free adaptation.
  • RAFT achieves 70.2% success at four simultaneous failures, closing 84% of the failure-naive-to-oracle gap without deployment fault sensors.
  • The work positions critic privilege as an alternative to explicit FDI, while distinguishing its jointly trained Observer extension from prior history-conditioned distillation approaches.

III. PROBLEM FORMULATION

The study formulates 2D go-to-position control for a floating platform with eight thrusters and one reaction wheel, subject to independently parameterized actuator failures. Failure count and mode are randomized during training, with a curriculum increasing the maximum number of failed thrusters.

  • A. Robot and Task: The robot has eight independently actuated thrusters plus one reaction wheel, with nine normalized control commands.
  • A. Robot and Task: The go-to-position task starts from a random spawn within 3 m and requires holding the fixed goal within 5 cm for 50 consecutive steps.
  • A. Robot and Task: The nominal observation contains 15 task variables covering relative goal position and orientation, body-frame velocity, and angular rate.
  • B. Failure Model: Each thruster follows a scale-plus-offset failure model with scale si and additive offset δi.
  • B. Failure Model: DEG models continuous degradation using δi = 0 and si sampled uniformly from [0, 1].
  • B. Failure Model: DEAD models total actuator loss with δi = 0 and si = 0, while STK models stuck-on thrust with si = 1 and uniformly sampled δi.
  • B. Failure Model: The 16-dimensional degradation state contains eight scale factors and eight offsets; each episode samples k failed thrusters and training curricula increase kmax from 0 to 4.

C. Reinforcement Learning Formulation

The reinforcement-learning formulation uses a Markov decision process with a shaped per-step reward for position, heading, speed, and boundary behavior. Success rate measures reaching and holding the goal, while final position error measures settling precision among successful episodes.

  • The task is modeled as a Markov Decision Process defined by state space, action space, transition model, reward, and discount factor.
  • The reward combines exponentially scaled position and heading terms with penalties for linear velocity, angular velocity, and workspace-boundary violations.
  • An episode succeeds when distance to the goal remains below 0.05 m for 50 consecutive steps.
  • Figure 1 contrasts task-only actors with an asymmetric critic receiving [otask ∥Dgt] and an optional history-based Observer extension.

IV. METHOD: PRIVILEGED ACTOR-CRITIC FOR FAULT ADAPTATION

The method trains a PPO actor-critic asymmetrically: the actor uses only standard task observations, while the training-only critic also sees the true degradation state. Fault-aware value estimates shape the actor toward fault-compensating behavior without deployment fault sensing.

  • The PPO critic receives privileged Dgt during training, while the actor receives only the standard task observation otask.
  • The actor outputs a Gaussian action distribution from otask at every step.
  • No fault information reaches the actor during training or deployment, so fault-compensating structure must emerge through policy gradients.
  • The critic forms its input by concatenating otask with the privileged ground-truth degradation state unavailable at deployment.
  • Fault-aware value estimates produce lower-variance advantages that shape policy gradients into fault-compensating behavior, with no deployment cost because only the actor executes.

B. Actor Architecture Variants

RAFT uses a GRU-64 actor with a privileged critic, while ablations separate recurrence, hidden-state size, and gating effects under the same critic setup.

  • RAFT and ablations: RAFT combines a GRU-64 actor with the privileged critic introduced in the paper.Its recurrent hidden state supports temporal fault adaptation without requiring an explicit history buffer.
  • RAFT and ablations: VAN-MLP-AC is a memory-less feedforward MLP designed to isolate the privileged critic’s effect without recurrence.It uses hidden dimensions [256, 128, 64].
  • RAFT and ablations: GRU-256-AC, LSTM-64-AC, and LSTM-256-AC vary hidden-state size and gating type while retaining the same privileged critic.These variants test whether recurrence design changes adaptation beyond critic privilege.
  • Reference policies: VAN is a failure-free-trained MLP lower bound, whereas Oracle matches RAFT’s GRU-64 architecture and receives Dgt at every step.Oracle therefore represents the success rate achievable with perfect failure sensing at deployment.

C. Observer Extension for Interpretability

The Observer adds an optional history-based fault-estimation pathway for interpretability without changing the privileged critic, but its default form is not trained to estimate degradation.

  • Observer design: The Observer is an interpretability extension motivated by explicit system identification, with the same critic architecture as the base method.It does not alter privileged critic training.
  • Observer design: A history buffer stacks the most recent L=32 task observations into H ∈ R^L×15 and flattens them to 480 features.This history is the Observer’s input representation.
  • Observer design: A three-layer MLP maps the flattened history to a 16-dimensional latent using hidden dimensions [128, 64], ELU activations, and a sigmoid output.The resulting estimate is concatenated with task observations after detachment from PPO gradients.
  • Default OBS: In OBS (λ=0), the observer receives no supervised gradient and remains a fixed random projection of the history buffer.E6 reports comparable performance to VAN-MLP-AC: 64.4 % versus 66.4 %, p>0.05.
  • OBS-MSE: OBS-MSE trains the observer to estimate Dgt, making its output human-readable for inspecting implicit fault beliefs at deployment.This interpretability variant incurs a reported 11 pp SR cost in E1.

D. Training Protocol

The study trains and evaluates policies across mixed-mode thruster failures using a curriculum, repeated seeds, and experiments targeting scalability, severity, online adaptation, recurrence, and critic privilege.

  • Training setup: Training uses 5 000 PPO iterations, 4 096 parallel environments, 24-step rollouts, and approximately 5 × 10^8 environment steps.The failure curriculum increases kmax from 0 to 4 over the first 5 × 10^7 steps, with three independent seeds.
  • Evaluation plan: Six experiments address oracle proximity, failure-mode scalability, continuous severity, mid-episode injection, recurrence, and asymmetric-critic contributions.The questions are grouped across E1–E6.
  • Evaluation plan: Five methods are compared under mixed-mode failures for k ∈ {0, ..., 4}: VAN, Oracle, RAFT, OBS, and OBS-MSE.VAN-MLP-AC is evaluated separately in E6.
  • Main comparison: 70.2 % success at k=4 lets RAFT close 84 % of the VAN-to-oracle gap, from 4.8 % to 82.4 %, while VAN collapses at four failures.All privileged-critic variants maintain 100 % success at k≤1.

B. E2 — Failure Severity Sweep

The severity sweep finds stable success across continuous DEG and STK severities at one failure, while recurrence improves precision; across pinned modes, DEAD is hardest and RAFT remains ahead of OBS.

  • Severity sweep: At k=1, severity is swept independently across seven DEG scales and seven STK offsets.The sweep spans scale values 0.0–1.0 for DEG and offset values 0.0–1.0 for STK.
  • Severity sweep: 100 % success is maintained by RAFT and OBS across the full DEG and STK severity spectra at k=1.No severity cliff is observed for either method across the tested levels.
  • Precision: RAFT’s final position error is consistently roughly half OBS’s across severity levels, except at STK severity 1.0 because of a single-seed outlier.The table reports final position error in centimeters, with three-seed means.
  • Precision: GRU recurrence provides more precise failure compensation even when RAFT and OBS have identical success rates.This separates precision from the binary success metric.
  • Failure-mode scalability: DEG is easier than STK, which is easier than DEAD for RAFT, OBS, and Oracle under mode-pinned evaluation.The ranking persists even for the oracle with full degradation-state access.
  • Failure-mode scalability: At k=4, RAFT reaches 96.2 % on DEG, 72.7 % on STK, and 52.2 % on DEAD, outperforming OBS in each mode.The corresponding RAFT leads are +3.0 pp, +5.4 pp, and +7.9 pp, respectively.

D. E4 — Mid-Episode Failure Injection

Mid-episode failures expose the importance of privileged critic training: recurrence without asymmetric critic access does not provide reliable fault adaptation, while a memory-less actor with the privileged critic closely matches RAFT.

  • E4 — Mid-Episode Failure Injection: At step 100, k thrusters are atomically degraded without a reset signal, testing online adaptation during 400-step episodes.Oracle receives the updated degradation state instantly; RAFT uses recurrent hidden state, VAN-MLP-AC is memory-less, and OBS uses a sliding history buffer.
  • E4 — Mid-Episode Failure Injection: VAN-MLP-AC drops 5.4 pp at k=4, closely matching RAFT’s 5.5 pp mid-episode drop.VAN-MLP-AC falls from 66.4 % reset-time success to 61.0 %, while RAFT falls from 70.2 % to 64.7 %.
  • E4 — Mid-Episode Failure Injection: RAFT’s GRU adds 3.7 pp beyond VAN-MLP-AC, narrowing the Oracle gap at k=4 to 10.8 pp versus 14.5 pp for VAN-MLP-AC.OBS has a 26.3 pp gap, while VAN collapses to 6.8 % at k=4.
  • E5 — Recurrent Policy Ablation: At k=4, recurrent baselines score 4.0 % or less and collapse at high failure counts, tracking VAN-MLP rather than RAFT.GRU-64, GRU-256, LSTM-64, and LSTM-256 achieve 4.0 %, 1.9 %, 3.3 %, and 3.0 %, respectively, versus VAN-MLP at 4.8 %.

F. E6 — Asymmetric Critic Ablation

E6 isolates privileged critic training from actor architecture: critic access to degradation state drives most fault tolerance, while compact recurrence adds modest gains. Larger recurrent hidden states underperform, and interpretability introduces a quantified performance tradeoff.

  • Critic privilege: 66.4 ± 2.9% at k=4: memory-less VAN-MLP-AC gains 61 pp over VAN-MLP, showing privileged critic access alone recovers most fault tolerance.The critic receives Dgt during training while the actor does not, so policy gradients encode fault-compensating behavior without explicit failure state input.
  • Actor architecture: 70.2 ± 1.0%: RAFT/GRU-64-AC exceeds VAN-MLP-AC by 3.8 pp, while LSTM-64-AC reaches 69.5 ± 0.9%.Compact recurrence adds temporal context on top of the privileged critic; both variants share the same critic.
  • Actor architecture: 256-dimensional recurrent actors underperform their 64-dimensional counterparts in both GRU and LSTM families.The result is consistent with overparameterization on this task.
  • Mechanism: Removing privileged critic access collapses the best recurrent baseline to 4.0% at k=4, whereas restoring it recovers 66.4% for a memory-less actor.This comparison supports the critic-side mechanism rather than recurrence as the dominant source of adaptation.
  • Interpretability: OBS-MSE makes fault estimates human-readable but costs 11 pp success rate, creating a choice between RAFT performance and observer interpretability.The observer objective constrains its representation to align with Dgt, which can bottleneck control-relevant features.
  • Scope and observability: STK faults are plausibly unobservable from passive motion history alone, unlike DEG and DEAD faults that visibly alter motion dynamics.The paper identifies active excitation as a likely requirement for closing this observability gap.
  • Scope and observability: Simulation-only experiments, 2D navigation, and a non-converged Transformer baseline constrain conclusions about real-world transfer and 6-DOF tasks.The paper leaves sim-to-real transfer and broader platform settings for future work.

VII. CONCLUSION

The paper concludes that privileged critic training enables sensor-free thruster fault adaptation, with RAFT achieving strong four-failure performance while the actor never observes degradation at deployment. Observer-based interpretability offers a quantified tradeoff, and future work targets broader dynamics, transfer, and observability.

  • Conclusion: VAN-MLP-AC closes 79% of the gap, while RAFT’s GRU adds 3.8 pp; recurrent policies without privileged critic access reach at most 4.0%.These comparisons isolate privileged critic training as the primary mechanism and recurrence as an incremental contributor.
  • Conclusion: OBS matches VAN-MLP-AC at 64.4%, whereas human-readable OBS-MSE estimates cost 11 pp success rate.This quantifies the performance-versus-interpretability choice for practitioners.
  • Future work: Future work will extend evaluation to 6-DOF platforms, test sim-to-real transfer, and investigate observer objectives addressing the STK observability gap.These directions follow the paper’s stated scope boundaries and observability concern.
Loading 2608.22976v1…