PaperlayerView exact revision on arXiv

Source-linked AI summary

XAI-Guided Fourier Domain Generalisation for Pure-Simulation Sim-to-Real Deep Reinforcement Learning

Wei Yao, Chuyao Wang, Ziwei Wang, Nabil Aouf

arXiv:2610.16478v1cs.AIcs.CV

TL;DR

Visual policies trained in simulation can degrade after real-world deployment, while existing sim-to-real approaches may require target-domain data or use unguided randomisation. XFD uses attribution-guided Fourier interventions and robust policy optimisation to construct training domains entirely in simulation. It improves performance over SAC and random domain randomisation across evaluated tasks and deployment settings.

  • Problem

    Simulation-trained visual policies can degrade after deployment, while target-informed adaptation and policy-agnostic randomisation have limitations for target-free sim-to-real transfer.

    Source: §I · pp. 1–2, B. Policy-Relevant Domain Construction · p. 3

  • Method

    XFD uses teacher attribution maps to guide Fourier-domain appearance perturbations while preserving task structures, then trains a student with robust policy optimisation in simulation.

    Source: §I · p. 2

  • Results

    XFD outperforms standard SAC and random domain randomisation across lane-following and obstacle-avoidance tasks; on Map 2 lane-following, travelled distance is 11.00 versus 5.88 for base SAC and 10.22 for DR.

    Source: A. Simulation Evaluation · p. 9, §VI · p. 13

  • Takeaways & Limitations

    Attribution analyses show more stable, task-focused visual evidence across simulation and real-world observations, alongside improved closed-loop robustness and transfer in unseen environments.

    Source: §VI · p. 13

    Attribution quality does not substitute for closed-loop performance, and the analysis does not establish a universal causal relation between explanation and performance.

    Source: C. XAI Attribution Analysis · p. 13

Abstract

from arXiv · show

Deep reinforcement learning has shown strong potential for vision-based autonomous driving, but policies trained in simulation often suffer severe performance degradation when deployed in the real world. Existing sim-to-real methods commonly address this gap through image-transfer-based domain adaptation or policy-agnostic domain randomisation. However, the former requires laborious acquisition of target-domain data and is often limited to the deployment domain, whereas the latter introduces unguided perturbations that may not reflect the underlying sim-to-real discrepancy, leading to suboptimal transfer performance. To overcome these limitations, we propose XFD, an explainable artificial intelligence (XAI)-guided Fourier domain generalisation framework for pure-simulation sim-to-real reinforcement learning. The proposed framework extracts explainability attribution maps from simulated observations, and converts these attributions into policy-conditioned Fourier-domain interventions. By perturbing appearance-sensitive spectral components while preserving task-defining spatial structures, XFD constructs robustness-enhancing visual domains entirely within simulation. Subsequently, the proposed robust policy optimisation guides the policy towards attending to task-critical features while suppressing sensitivity to irrelevant visual information. Extensive experiments in simulation and zero-shot real-world deployment demonstrate that XFD consistently improves traveled distance and reward over standard policy and randomly domain-randomised baseline, especially on unseen maps with cluttered backgrounds. Attribution-based analysis further proves that the proposed policy preserves more consistent task-relevant visual evidence across simulation and real-world observations.

I. INTRODUCTION

Simulation enables scalable visual-control training, but policies can degrade on deployment because real-world appearance, sensing, and dynamics differ. XFD uses policy attributions to guide Fourier interventions and robust policy optimisation, constructing visual domains in simulation for target-free transfer.

I. INTRODUCTION

  • XFD uses DeepLIFT attributions to guide spectral perturbations that shift appearance while preserving task-defining spatial structure, and trains the student with teacher-guided robust policy optimisation.The teacher supplies attribution guidance and stabilisation, not imitation targets or a performance upper bound.Source: §I · p. 2
  • XFD consistently improves transfer on lane-following and obstacle-avoidance tasks, including unseen maps with substantial background clutter, without target-data acquisition or real-world fine-tuning.The evaluation includes simulation and zero-shot real-world deployment.Source: §I · p. 2

A. Sim-to-Real Visual Domain Generalisation

  • Target-informed adaptation and image transfer use target-domain evidence, while pixel-level transformations may alter action-critical lane, boundary, and obstacle structures [14],, [24]–.These methods may also depend on target observations or statistics.Source: A. Sim-to-Real Visual Domain… · p. 2
  • Fourier methods separate appearance-related amplitude from structure-preserving phase, but prior approaches use target spectra or generic mixing disconnected from closed-loop policy evidence, [17], [18].This motivates conditioning spectral intervention on action-relevant visual evidence.Source: A. Sim-to-Real Visual Domain… · p. 2

B. Policy-Relevant Domain Construction

  • Existing objective-guided randomisation evaluates domains through simulator parameters, discrepancy, rollouts, or task outcomes, without specifying which visual evidence drives action selection.XFD addresses this gap by using action-related visual evidence to guide simulation-generated shifts.Source: B. Policy-Relevant Domain Construction · p. 3

C. Explanation-Guided Training

Explanation-guided learning has mostly used explanations for interpretation, debugging, or supervision rather than constructing training domains. XFD uses DeepLIFT-guided Fourier shifts in a teacher-student SAC framework, with auxiliary teacher guidance stabilising student optimisation.

C. Explanation-Guided Training

  • DeepLIFT attribution is used as a training-time signal for policy-conditioned domain construction, extending explanation use beyond interpretation, debugging, or explanation-level supervision [20],, [37].DeepLIFT supplies dense input-level contributions suited to identifying fine-grained evidence influencing continuous actions.Source: C. Explanation-Guided Training · p. 3

A. Overview

  • A frozen, domain-randomised SAC teacher supplies attributions while a student trains from scratch on Fourier-perturbed observations for simulation evaluation and zero-shot Duckiebot deployment.The attribution map regulates appearance shifts to protect task-critical regions and permit stronger changes in less relevant regions.Source: A. Overview · p. 3
  • A reward-gated curriculum introduces Fourier perturbations after basic driving behavior emerges, and auxiliary teacher-guided terms stabilise the student while SAC drives policy improvement.The teacher-guided terms do not replace the SAC objective.Source: A. Overview · p. 3

B. SAC-Based Visual Driving Policy

  • The visual control backbone uses SAC [8], with a stochastic wheel-control actor and two clipped-double-Q critics using separate visual encoders.Observations combine a front-view RGB image with a five-dimensional navigation state; actions are left- and right-wheel commands.Source: D. Student Training (RL with XAI… · p. 4

C. Attribution Extraction

A frozen teacher’s DeepLIFT map estimates which image locations contribute to wheel-control outputs. XFD uses that map to regulate Fourier appearance shifts, preserving task-relevant structure, and introduces stronger intervention after basic control competence develops.

C. Attribution Extraction

  • DeepLIFT maps from a frozen, pretrained domain-randomised SAC teacher identify image evidence contributing to its continuous wheel-control output.Contributions across both action dimensions are aggregated into a spatial map, where larger values indicate stronger contribution.Source: C. Attribution Extraction · p. 4
  • The attribution map serves as a training-time domain-construction signal: high-attribution regions are treated as control-relevant, while low-attribution regions allow stronger appearance perturbation.This separation guides subsequent Fourier intervention to preserve structures important for closed-loop control.Source: C. Attribution Extraction · p. 4

D. XAI-Guided Fourier Domain Intervention

  • A reward-gated curriculum delays strong visual perturbations because applying them before the student acquires a reliable driving prior may destabilise exploration.The schedule increases intervention after basic task competence is acquired.Source: D. XAI-Guided Fourier Domain… · p. 5
  • The intervention modifies Fourier amplitude for appearance changes and preserves phase for lane geometry, boundaries, and obstacle layout, with teacher attribution regulating spatial perturbation strength.The approach shifts illumination, contrast, texture, and color statistics while retaining task-relevant structure.Source: D. XAI-Guided Fourier Domain… · p. 5
  • High-attribution regions receive less perturbation freedom and low-attribution regions more; blended observations change domain-dependent appearance while retaining lane, boundary, and obstacle structures.The student is trained on these visually shifted, structurally consistent observations.Source: D. XAI-Guided Fourier Domain… · p. 5

E. Target-Free Robust Policy Optimisation

The student is trained without target-domain images, real-world rollouts, or demonstrations, using a teacher as a local behavioral reference alongside reinforcement learning and attribution regularisation. A gate switches training from original simulation observations to attribution-guided Fourier observations and activates auxiliary regularisation under visual shifts.

E. Target-Free Robust Policy Optimisation

  • The student is optimised without target-domain images, real-world rollouts, or demonstrations, and reinforcement learning improves control rather than merely imitating the teacher.The frozen teacher serves as a local behavioral reference and is not updated.Source: E. Target-Free Robust Policy… · pp. 5–6
  • Behavioral consistency compares the student’s response to a perturbed observation with the teacher’s response to the corresponding clean observation.The teacher’s deterministic mean action is the reference for this consistency term.Source: E. Target-Free Robust Policy… · pp. 5–6
  • The outside-attribution loss penalizes student attribution in regions weakly supported by the teacher, discouraging reliance on visually perturbed regions.The complementary mask is defined as Mt = 1 − AT(It).Source: E. Target-Free Robust Policy… · p. 6
  • The attribution-alignment term encourages the student to preserve the teacher’s overall spatial attribution structure under perturbed observations.Source: E. Target-Free Robust Policy… · p. 6
  • The final student actor objective combines reinforcement learning, behavioral consistency, and attribution regularisation.The auxiliary terms apply when the intervention gate is active; before activation the student uses original simulation observations, and afterward it uses attribution-guided Fourier observations.Source: E. Target-Free Robust Policy… · pp. 5–6

F. Obstacle-Aware Semantic Replay

Obstacle-aware semantic replay organizes transitions from randomized obstacle-placement training into categories and samples mini-batches according to a progress-dependent category distribution. The schedule shifts from near-obstacle and failure cases toward successful avoidance while retaining lane-following samples.

F. Obstacle-Aware Semantic Replay

  • One replay buffer contains both lane-tracking transitions and sparse obstacle interactions from a single obstacle-aware policy trained with randomized obstacle placements.Source: F. Obstacle-Aware Semantic Replay · p. 6
  • Transitions are assigned to normal lane-following, near-obstacle, successful-avoidance, or failure-related categories.Normal samples preserve lane-tracking competence, while the other categories increase exposure to sparse safety-critical cases.Source: F. Obstacle-Aware Semantic Replay · p. 6
  • Early sampling increases near-obstacle and failure cases; as training proceeds, sampling shifts toward successful avoidance while retaining normal lane-following samples.The schedule is progress-dependent and is intended to support obstacle-aware control without sacrificing lane-following behavior.Source: F. Obstacle-Aware Semantic Replay · p. 6
  • The replay mechanism is independent of Fourier intervention and corrects the long-tailed transition distribution induced by randomized obstacle placement.It operates as a task-level sampling strategy within the same SAC optimisation framework.Source: F. Obstacle-Aware Semantic Replay · pp. 6–7

G. Reward Formulation

The reward is designed to encourage stable lane-following and discourage stopping, heading deviation, and boundary-hugging; baseline and proposed methods use the same task reward. Its components include forward progress, motion, stopping, heading, and edge terms, with a fixed penalty for invalid lane poses.

G. Reward Formulation

  • Both the baseline and proposed method use the same task reward, which is intended to encourage stable lane-following and suppress stopping, excessive heading deviation, and boundary-hugging.The reward formulation is described as separate from the proposed Fourier intervention and attribution-guided policy regularisation.Source: G. Reward Formulation · p. 7
  • Invalid lane poses receive a fixed reward penalty of −5.For valid lane poses, the reward is defined by the task’s reward formula.Source: G. Reward Formulation · p. 7
  • The progress reward encourages forward motion along the lane, while a small motion reward discourages stationary behavior.The progress term uses normalized speed and heading projection onto the lane direction.Source: G. Reward Formulation · p. 7
  • Stopping is penalized below normalized speed 0.10, while heading and edge penalties address heading error and driving close to the lane boundary.The supplied constants include kangle = 0.2, θdead = 30°, kedge = 4.0, and ethr = 0.10.Source: G. Reward Formulation · p. 7

IV. EXPERIMENTS

The experiments evaluate the proposed framework across task and map configurations, baselines, training protocols, and evaluation. The metrics are intended to assess closed-loop robustness, cross-map generalisation, and zero-shot sim-to-real transfer under visual domain shifts.

IV. EXPERIMENTS

  • The experimental protocol covers task and map configurations, baseline methods, training protocols, and evaluation.Source: §IV · p. 7
  • The evaluation metrics are designed to assess closed-loop robustness and cross-map generalisation.Source: §IV · p. 7
  • The evaluation also targets zero-shot sim-to-real transfer under visual domain shifts.Source: §IV · p. 7

A. Environment Configurations

The experiments use Map 2 for simulation training, Maps 1 and 3 for unseen simulation evaluation, and Maps 4 and 5 for zero-shot real-world deployment.

A. Environment Configurations

  • Map 2 is the simulation training layout, where one obstacle-aware policy is trained with randomized obstacle placements.Training requires lane-following while responding appropriately to obstacle interactions.Source: A. Environment Configurations · p. 7
  • Maps 1 and 3 are unseen simulation layouts used to assess cross-map generalization.Source: A. Environment Configurations · p. 7
  • Maps 4 and 5 are real-world deployment layouts corresponding to evaluation Maps 1 and 3, and the same policies are tested without target-domain fine-tuning or data acquisition.The layouts are shown in Fig. 4.Source: A. Environment Configurations · p. 7

B. Baseline Methods

The study compares Base, DR, and XFD policies using the same SAC visual-control backbone and obstacle-aware Map 2 training setting. It describes the teacher-student training process and evaluates the same trained policy across tasks and simulation and real-world maps.

B. Baseline Methods

  • Base is standard SAC without sim-to-real techniques, DR adds target-free visual randomization, and XFD adds DeepLIFT-guided Fourier intervention and target-free robust policy optimization.All methods share the SAC-based visual-control backbone and Map 2 obstacle-aware training setting.Source: B. Baseline Methods · pp. 7–8
  • Each method’s same trained policy is evaluated on both lane-following and obstacle-avoidance tasks, focusing comparison on robustness of a single obstacle-aware policy.The comparison does not rely on task-specific policy training.Source: B. Baseline Methods · pp. 7–8

C. Training Protocols

  • Policies are trained entirely in simulation on Map 2 with randomized obstacles, using cropped front-view RGB images and a five-dimensional navigation state as input.The action is a two-dimensional continuous command for left and right wheel velocities.Source: C. Training Protocols · p. 8
  • XFD starts with clean-observation SAC training, then activates attribution-guided Fourier intervention after a reward-gated criterion and progressively increases perturbation strength.The converged DR policy is a frozen teacher that provides attribution guidance; only the XFD student is updated during this phase.Source: C. Training Protocols · p. 8
  • The same checkpoint is evaluated without further adaptation on simulation Maps 1–3 and corresponding real-world Maps 4–5 for both tasks.Source: C. Training Protocols · p. 8

D. Evaluation Metrics

Evaluation covers lane-following and obstacle avoidance in simulation and real-world settings, using shared protocols and four complementary measures. Implementation uses a common visual actor-critic architecture, with deployment requiring only the trained student actor.

D. Evaluation Metrics

  • Lane-following uses ten runs per map; obstacle avoidance uses two obstacle configurations with five runs each, pooled into ten trajectories per map.A completed circuit counts as success; leaving the drivable lane or colliding with an obstacle terminates a run.Source: D. Evaluation Metrics · p. 8
  • Real-world pose is motion-capture recorded, projected onto the calibrated map frame, and resampled at 10 Hz to compute trajectory-based measures.Reported results are means and standard deviations across ten runs; higher smoothness indicates less abrupt yaw variation.Source: D. Evaluation Metrics · p. 8
  • The four evaluation metrics are travelled distance, reconstructed reward, lateral deviation, and yaw-based smoothness.Reward combines forward speed, heading alignment, and centerline deviation, with a terminal failure penalty for an incomplete lap.Source: D. Evaluation Metrics · p. 8
  • All methods use the same task settings, maps, obstacle placements, and reward reconstruction procedure, so metric differences reflect the method rather than protocol or reward changes.Source: D. Evaluation Metrics · pp. 8–9

E. Implementation Details

  • The actor and critic use separate MobileNetV2 visual encoders, while navigation state is processed separately before feature concatenation and policy or value prediction.The actor outputs a two-dimensional Gaussian wheel-command policy, and twin Q-value heads form the critic; training hyper-parameters are summarized in Table II.Source: E. Implementation Details · p. 9, Table II · p. 9
  • At deployment, Fourier perturbation and DeepLIFT computation are unnecessary; only the trained student actor is exported to ONNX, adding no inference cost.Source: E. Implementation Details · p. 9

V. RESULTS

XFD outperforms the base and domain-randomised policies in simulation and zero-shot physical deployment, including on unseen maps. Its gains span lane following and obstacle avoidance, with longer trajectories and higher task rewards.

A. Simulation Evaluation

  • On unseen Map 3, lane-following distance was 10.72 for XFD, versus 4.04 for the base policy and 9.09 for DR, which had large variance.XFD's reward on this map was 799.75.Source: A. Simulation Evaluation · p. 10
  • On unseen Map 3, obstacle-avoidance distance was 10.96 for XFD, versus 4.13 for the base policy and 8.30 for DR; XFD's reward was 759.08.All policies were trained on Map 2 and evaluated on Maps 1–3 without adaptation.Source: A. Simulation Evaluation · pp. 9–10
  • Across three simulation maps, XFD's average distance gains over DR were 10.9% for lane following and 15.5% for obstacle avoidance; reward gains were 16.5% and 25.2%, respectively.The average obstacle-avoidance distance gain over the base policy was 55.8%.Source: A. Simulation Evaluation · p. 10

B. Real-World Deployment

  • On Map 5's consecutive turns, lane-following distance was 9.63 for XFD, compared with 0.77 for the base policy and 2.91 for DR.The base and DR policies terminate early on this more challenging physical layout.Source: B. Real-World Deployment · p. 11
  • On physical Map 5, obstacle-avoidance distance reached 9.77 for XFD versus 0.98 for the base policy and 2.85 for DR; Fig. 6 shows longer, more continuous XFD trajectories.The reported trajectory comparison also describes baseline instability after obstacle encounters.Source: Figure 6 · p. 11, Table III · p. 11, B. Real-World Deployment · p. 12
  • Across two physical maps, XFD's average lane-following distance was 9.95 versus 1.07 for the base policy and 3.49 for DR; obstacle-avoidance distance was 9.86 versus 1.20 and 2.81.Table III reports mean ± standard deviation over ten evaluation trajectories.Source: Table III · p. 11, B. Real-World Deployment · p. 12

C. XAI Attribution Analysis

XFD outperforms standard SAC and random domain randomisation across the evaluated tasks, while attribution analysis shows more stable, task-focused visual evidence across simulation and real-world observations. The authors caution that attribution quality cannot substitute for closed-loop performance.

C. XAI Attribution Analysis

  • In lane following, baseline attributions shift away from lane structure in real observations, while domain-randomised attributions remain scattered across background or texture-dependent regions.The paper says these patterns indicate sensitivity to appearance changes and background cues.Source: C. XAI Attribution Analysis · p. 12
  • In Fig. 7, brighter regions indicate stronger contributions to control; XFD maintains concentrated attribution around lane boundaries and road geometry across simulated and real observations.The analysis describes this as preserving control-relevant visual evidence under domain shifts.Source: C. XAI Attribution Analysis · pp. 12–13
  • Attribution quality is not a substitute for closed-loop performance, and a visually plausible map does not guarantee successful control.The authors describe the alignment between stable, task-focused evidence and stronger distance and reward as empirical, not a universal causal relation.Source: C. XAI Attribution Analysis · p. 13

VI. CONCLUSION

  • DeepLIFT attributions regulate Fourier perturbations to generate visual domains in simulation while preserving task-relevant spatial information.The teacher provides attribution guidance and training stabilisation while the student is optimised through reinforcement learning.Source: §VI · p. 13
  • XFD consistently outperforms standard SAC and random domain randomisation across lane-following and obstacle-avoidance tasks in simulation and zero-shot real-world deployment.The conclusion also reports that XFD surpasses the DR teacher policy in simulation.Source: §VI · p. 13
  • The authors identify complex dynamic environments, larger-scale deployment, and online attribution-guided adaptation as future extensions.Attribution analyses found more stable and task-focused visual evidence across simulation and real-world observations.Source: §VI · p. 13

Related papers

Loading 2610.16478v1…