Off-Dynamics Reinforcement Learning: Training for Transfer with Domain Classifiers

1 INTRODUCTION

• Target-domain trial-and-error can be too costly for safety-critical tasks, while experience in a structurally similar source domain is cheaper to collect.The paper frames domain adaptation as modifying source-domain training so the resulting policy works in a given target domain.Source: §1 · p. 1
• Prior RL adaptation of observations has not addressed domain adaptation of the dynamics.The paper identifies dynamics differences as the transfer challenge addressed by its method.Source: §1 · p. 2
• DARC adapts to dynamics changes by modifying rewards with classifiers that distinguish source- from target-domain transitions, rather than estimating transition probabilities.The paper formally analyzes conditions for near-optimal target policies and evaluates the approach on discrete and continuous control tasks.Source: §1 · p. 2

2 RELATED WORK

• DARC uses two classifiers to estimate conditional transition-density ratios and adds the log ratio to rewards; direct ratio weighting was numerically unstable and performed poorly in experiments.The method applies the logarithm of the ratio rather than weighting source samples by the ratio itself.Source: §2 · p. 2
• System identification typically requires an environment model, domain randomization is sensitive to parameter choices, and observation adaptation ignores differing dynamics.The paper contrasts these requirements and limitations with an approach that does not require learning a dynamics model.Source: §2 · pp. 2–3
• Several prior transfer and meta-learning approaches assume many same-distribution source tasks, a specified task curriculum, locally matching dynamics, or shared useful actions; DARC does not require these assumptions.The broken-robot experiments include a target domain where a disabled joint makes some source-domain actions unhelpful.Source: §2 · p. 3
• DARC is applicable to stochastic environments with continuous states and actions, unlike approaches that require learning a dynamics model.The paper says its algorithm can be used with any maximum-entropy RL algorithm and uses SAC in its experiments.Source: §2 · p. 3

3 PRELIMINARIES

• The source and target MDPs share state and action spaces, rewards, and the initial-state distribution; their transition dynamics may differ.Source: §3 · p. 3
• Domain adaptation for RL uses source-MDP interactions plus a small number of target-MDP interactions to acquire a policy with high target reward.Source: §3 · p. 3
• Every transition possible in the target must also be possible in the source, but source-possible transitions need not be target-possible.Without this support condition, target-optimal behavior might be impossible to learn by practicing in the source domain.Source: §3 · p. 3

4 A VARIATIONAL PERSPECTIVE ON DOMAIN ADAPTATION IN RL

• DARC minimizes reverse KL between the agent’s source-domain trajectory distribution and the desired target-domain trajectory distribution.The distributions are assumed to share the same initial-state distribution.Source: §4 · p. 4
• The reward correction is the log target-to-source transition-probability ratio, penalizing transitions likely in the source but unlikely in the target.When source and target dynamics are equal, the correction is zero.Source: §4 · p. 4

4.1 THEORETICAL GUARANTEES

• The guarantee assumes a target-optimal policy has similar expected reward in both domains; it does not require the source-optimal policy to perform well in the target.If multiple target-optimal policies exist, the assumption need hold for only one of them.Source: §4.1 · p. 4
• Under that assumption, DARC attains near-optimal entropy-regularized reward on the target domain.DARC maximizes the modified entropy-regularized reward in the source domain.Source: §4.1 · p. 5

5 DOMAIN ADAPTATION IN RL WITH A LEARNED REWARD

• DARC estimates the dynamics-based reward correction without learning an explicit dynamics model.The practical algorithm applies the correction to source-domain rewards.Source: §5 · p. 5
• A transition classifier and a state-action classifier provide predictions that combine into an estimate of the reward correction.The correction compares domain evidence conditioned on (s_t, a_t, s_{t+1}) with evidence conditioned only on (s_t, a_t).Source: §5 · p. 5
• The authors interpret the method as making source-domain dynamics observed by the policy imitate target-domain dynamics, without assuming expert experience.They relate the correction to how much transition information helps predict whether data came from the source or target domain.Source: §5 · p. 6
• Each iteration collects source transitions and less-frequent target transitions, updates both classifiers, corrects source rewards, and applies MaxEnt RL.The paper uses SAC but states DARC applies to any MaxEnt RL algorithm.Source: §5 · p. 6

Illustrative example

• In the gridworld, a source-trained route goes through a target-only obstacle, whereas DARC’s modified reward yields a route around it that succeeds in the target.The gridworld comparison also finds DARC succeeds in 80% of trials versus 0% for MATL when domains differ in action-conditioned dynamics despite matching next-state marginals under random actions.Source: §6 · pp. 6, 9

Scaling to more complex tasks

• On three of four tasks, DARC matches or surpasses target-domain RL’s asymptotic performance while using 5–10× less target-domain experience; it also outperforms MATL on all four.The tasks include crippled robots and a target-only obstacle; the figure caption reports that direct learning on crippled robots requires an order of magnitude more experience than DARC.Source: Figure 6 · p. 7, §6 · pp. 7–8

Visualizing the reward modification in stochastic domains

• For the windy source and still-air target, the modified reward favors a policy that performs well indoors, although it does not yield high reward outdoors.The outdoor optimum compensates for wind by aiming left (θ = −0.8), while the indoor optimum aims straight ahead (θ = 0).Source: §6 · p. 7

Scaling to more complex tasks

• The continuous-control tests use three robots crippled in the target and a half-cheetah task with a target-only obstacle; DARC succeeds even on the obstacle task, which violates an assumption.The target-only obstacle makes some transitions possible only in the target domain, contrary to the stated assumption.Source: §6 · p. 7

Ablation Experiments

• Removing the SA classifier makes DARC perform considerably worse, while classifier input noise helps at a suitable level; too little or too much noise performs poorly.The authors attribute the second classifier’s role to correcting source–target shifts in state and action visitation; the noise value of 1 was used throughout and not tuned.Source: §6 · p. 8, Figure 7 · p. 8

Safety emerges from domain adaptation to the termination condition

• With zero task reward, DARC learns to keep a humanoid standing nearly the entire episode when falling terminates target-domain episodes, suggesting safety may emerge without manual reward design.Source episodes instead have a fixed length of 300 steps, so termination dynamics differ across domains.Source: §6 · p. 9, Figure 9 · p. 9

Comparison with Prior Transfer Learning Methods

• On a gridworld where source and target effects of actions differ but next-state marginals match under random actions, DARC succeeds in 80% of trials and MATL in 0%.The comparison highlights the importance of conditioning the classifier on actions when using experience from stochastic policies.Source: §6 · p. 9

7 DISCUSSION

• Control-task experiments show DARC can use source-domain experience to learn policies that work well in the target domain with only a handful of target transitions.Source: §7 · p. 9
• The method requires sufficiently stochastic source dynamics; it worked best when the source allowed strategies that were incompatible with the target dynamics.The authors note that adding noise or ensembling sources can usually satisfy the stochasticity assumption.Source: §7 · p. 9

A.1 CODING THEORY

• Under a target-optimized replay-buffer compression assumption, negative reward correction measures the additional bits needed to encode source transitions relative to target transitions.Maximizing the correction favors transitions encoded efficiently in the source replay buffer.Source: §A.1 · p. 15

A.2 MUTUAL INFORMATION

• The expected reward correction is the target-versus-source difference in conditional mutual information about the next state, favoring transitions that indicate the target domain.The authors describe this as rewarding transitions informative about the target and penalizing those that hint at the source.Source: §A.2 · p. 15

A.3 LOWER BOUND ON THE RISK-SENSITIVE REWARD OBJECTIVE.

• DARC maximizes a lower bound on a risk-sensitive reward objective, but whether to optimize that bound or the original objective remains an open trade-off.The authors suggest the bound may avoid risk-seeking behavior induced by risk-sensitive objectives.Source: §A.3 · p. 15
• The reward correction includes the log ratio of target to source transition probabilities, added to the task reward.Source: §A.3 · p. 15

A.4 A CONSTRAINT ON DYNAMICS DISCREPANCY

• DARC’s objective can be viewed as restricting MaxEnt RL to policies that avoid exploiting dynamics discrepancies; adaptively weighting the correction remains future work.The authors propose adjusting its weight according to how much the policy is exploiting the discrepancy.Source: §A.4 · p. 16

• Maximizing modified reward in the source domain is equivalent to maximizing unmodified reward subject to a constraint against exploiting dynamics.The proof describes this constraint as limiting policies to the no-exploit set.Source: B PROOFS OF THEORETICAL GUARANTEES · p. 16
• Policies in the no-exploit set have similar expected entropy-regularized rewards across source and target domains.The proof states that these policies satisfy the same condition as the optimal policy under Assumption 1.Source: B PROOFS OF THEORETICAL GUARANTEES · p. 16
• If the target-optimal policy satisfies Assumption 1, DARC receives near-optimal entropy-regularized reward in the target domain.Source: B PROOFS OF THEORETICAL GUARANTEES · p. 17

C THE SPECIAL CASE OF AN OBSERVATION MODEL

• When state comprises latent state and observation, the objective separates into MaxEnt RL, observation-adaptation, and dynamics-adaptation terms.Source: C THE SPECIAL CASE OF AN OBSERVATION… · p. 17
• DARC’s reward correction addresses both observation and dynamics adaptation, and the authors leave combining it with prior approaches for future work.Source: C THE SPECIAL CASE OF AN OBSERVATION… · p. 17

D ADDITIONAL EXPERIMENTS

• Across all environments, ablations support the importance of using two classifiers and input noise.The target-domain reward does not start increasing until DARC is used.Source: D ADDITIONAL EXPERIMENTS · p. 18

E EXPERIMENT DETAILS AND HYPERPARAMETERS

• DARC uses two 256-unit hidden-layer networks with ReLU; the SAS classifier adds the SA network's output, and both classifiers use Gaussian input noise.The SAS classifier propagates gradients through both networks; most experiments use one target-domain step per ten source-domain steps, and the ratio was not tuned.Source: E EXPERIMENT DETAILS AND HYPERPARAMETERS · p. 18

E.1 BASELINES

• The RL on Source and RL on Target baselines omit the reward correction; Importance Weighting estimates transition-density ratios with exp(∆r) to weight SAC losses.RL on Target (10x) takes ten gradient steps per environment interaction.Source: §E.1 · p. 19

E.2 ENVIRONMENTS

• Broken-joint tasks disable torque at a selected target-domain joint; the broken Reacher, Half Cheetah, and Ant use joints 2, 0, and 3, respectively.The selected joint was the one for which RL on Source performed worst relative to RL on Target on the target domain.Source: §E.2 · p. 19
• The Half Cheetah Obstacle adds a wall about three meters behind the agent, while Humanoid uses fixed-length source episodes and fall-based target termination.The Humanoid source terminates after 300 steps; its target uses the unmodified fall termination.Source: §E.2 · p. 20

E.3 FIGURES

• Learning curves average three random seeds and also show each seed; Figure 13 overlays target-domain reward on a copy of Figure 8.The seed protocol applies unless otherwise noted.Source: Figure 13 · p. 18, §E.3 · p. 20

E.4 ARCHERY EXPERIMENT

• The archery experiment models a target 70 m north of the agent with east-west wind, and trains a classifier on 10,000 source and 10,000 target episodes.The classifier uses one 32-unit hidden layer; rewards for Figure 4 are aggregated as J(θ) = log Ep(s′|θ)[exp(r(s′))].Source: §E.4 · p. 20