PaperlayerView exact revision on arXiv

Source-linked AI summary

DAMPER: Return-Prioritized Gradient Control for Smooth Policies

Seokmin Ko, Taewon Goo, Kihyuk Hong

arXiv:2609.38903v1cs.LGcs.AI

TL;DR

Actor–critic policies can produce highly oscillatory actions, and auxiliary smoothness losses may be weak relative to the native actor gradient or complicate loss balancing. DAMPER combines conflict-conditioned projection with adaptive magnitude control for the temporal gradient. It improves oscillation scores over native agents across all evaluated task–backbone pairs, while return is not uniformly preserved.

  • Problem

    Auxiliary smoothness losses can have negligible influence when their gradients are small relative to the native actor gradient, and multiple losses complicate balancing without necessarily improving the return–smoothness trade-off.

    Source: §1 · p. 1

  • Method

    DAMPER combines the native actor and temporal-consistency gradients by removing the return-opposing temporal component and controlling the retained direction's magnitude relative to the actor gradient norm.

    Source: §1 · pp. 1–2

  • Results

    Oscillation scores improve over native agents in all 12 task–backbone pairs, with the best score among compared methods in eight; return is not uniformly preserved.

    Source: §1 · p. 2, §7 · p. 11

  • Takeaways & Limitations

    DAMPER reduces reported action oscillation across the evaluated task–backbone pairs, but return trade-offs remain task-dependent.

    Source: §1 · p. 2, §7 · p. 11

    Return is not uniformly preserved, and the theoretical analysis covers a single actor update rather than the full training process.

    Source: §7 · p. 11

Abstract

from arXiv · show

Actor-critic methods achieve strong performance in continuous control, but their policies can produce highly oscillatory actions. A common remedy is to add auxiliary smoothness losses. However, their contribution can be negligible when their gradients are small relative to the native actor gradient. Moreover, existing methods often combine multiple auxiliary losses, complicating loss balancing without necessarily improving the return-smoothness trade-off. We introduce DAMPER (Direction-Aware Magnitude-Controlled Projection with Explicit Return Priority), which combines the native actor gradient with a temporal-consistency gradient through conflict-conditioned projection and adaptive magnitude control. It removes the auxiliary component opposing the actor gradient and scales the retained temporal direction relative to the actor gradient norm, preserving positive alignment with the native actor gradient. Experiments with TD3 and SAC on six continuous-control tasks show reduced action oscillation relative to the native agents in all 12 task-backbone pairs and the best oscillation score among the compared methods in eight, with task-dependent return trade-offs.

1 Introduction

High episodic return does not ensure smooth actions, a concern for physical deployment. DAMPER combines the native actor and temporal-consistency gradients with conflict-conditioned projection and adaptive magnitude control; it reduces oscillation across evaluated task–backbone pairs, though return trade-offs remain.

1 Introduction

  • Auxiliary smoothness losses can have limited effect, while combining multiple such losses complicates balancing without necessarily improving the return–smoothness trade-off.Rapid action changes can waste energy, amplify unmodeled dynamics, and increase actuator wear on physical platforms.Source: §1 · p. 1
  • DAMPER removes the temporal-gradient component opposing the native actor gradient while leaving the native gradient unchanged.Aligned temporal components are retained rather than discarded.Source: §1 · pp. 1–2
  • Its bounded η parameter controls the retained temporal direction relative to the native actor-gradient norm, interpolating between cap-only and norm-matched endpoints.Unlike a scalar regularization coefficient, η reconciles observed gradient magnitudes after differentiation.Source: §1 · p. 2
  • Oscillation improves over native agents in all 12 task–backbone pairs, and DAMPER has the best score among compared methods in eight.The improvement is more consistent for action stability than return, with several locomotion settings showing a smoothness–return trade-off.Source: §1 · p. 2
  • DAMPER is introduced as a return-prioritized actor update combining conflict-conditioned projection with norm-ratio interpolation.The paper also establishes finite-step descent for a fixed actor surrogate and characterizes temporal progress at matched first-order primary progress.Source: §1 · p. 2

2 Related Work

Prior smoothness methods regularize policy outputs, sensitivity, or critic gradients, while gradient-combination methods often treat objectives symmetrically. DAMPER instead keeps the native actor objective primary and adjusts the temporal auxiliary gradient by alignment and relative scale.

2.1 Policy Smoothness in Reinforcement Learning

  • DAMPER uses a temporal surrogate but modifies its gradient according to alignment and scale relative to the native actor gradient, rather than adding it with a fixed coefficient.The temporal loss form is shared with prior consecutive-output regularization, while CAPS combines temporal and spatial consistency [19].Source: §2.1 · pp. 2–3
  • Policy-sensitivity and critic-regularization methods encode richer notions of regularity, but multi-component scalarized objectives add relative weights.DAMPER focuses on how a temporal auxiliary gradient enters an actor update whose native objective remains primary.Source: §2.1 · p. 3
  • Architectural and action-space approaches can structure smooth behavior, whereas DAMPER changes training-time actor gradients without changing policy architecture or execution interface.Examples include LipsNet, temporally correlated exploration noise, and action-rate parameterizations [2].Source: §2.1 · p. 3

2.2 Gradient Combination and Objective Priority

  • Unlike symmetric multi-objective methods, DAMPER treats the native actor objective as primary and never alters its gradient to accommodate temporal consistency.The related methods include MGDA [24], GradNorm [1], PCGrad [28], and CAGrad [17].Source: §2.2 · p. 3
  • DAMPER combines conditional removal of the primary-opposing temporal component with bounded norm-ratio magnitude control.Its fixed hierarchy, conflict-conditioned projection, and interpolation distinguish the combination from related priority-aware methods [20] [18] [22] [12].Source: §2.2 · p. 3

3 Preliminaries

The preliminaries frame policy learning as actor–critic optimization with a native actor loss and a temporal-consistency penalty. The conventional auxiliary-loss formulation combines their gradients using a scalar weight.

3 Preliminaries

  • Actor–critic methods use a learned action-value function to guide policy optimization, with the underlying algorithm's native actor loss denoted LR(θ).The paper uses a loss-minimization convention; algorithm-specific loss definitions are deferred to Appendix E.Source: §3 · pp. 3–4
  • The temporal-consistency loss compares deterministic actions from the same current policy at consecutive states and encourages similar actions along environment trajectories.The states are drawn from an observed transition distribution.Source: §3 · p. 4
  • The conventional auxiliary-loss formulation combines the actor-loss gradient gR with the temporal gradient gT as gR + λgT.Here, “return gradient” means the critic-based actor-loss gradient, not a direct gradient of measured episodic return.Source: §3 · p. 4

4 Method

DAMPER adds temporal consistency to the native actor update while prioritizing the actor objective. It first handles directional conflict, then controls the temporal contribution's magnitude and passes the merged gradient to the native optimizer.

4 Method

  • DAMPER combines a temporal-consistency objective with the native actor objective, which remains primary.The two operations address directional conflict and scale imbalance in incorporating temporal consistency.Source: §4 · p. 4

4.1 Conflict-Conditioned Projection

  • When the temporal gradient conflicts with the native actor gradient, DAMPER projects away its opposing component; otherwise, it leaves that gradient unchanged.The native gradient is never modified, and the retained temporal gradient has nonnegative inner product with it, preserving first-order decrease in native actor loss when added with a nonnegative multiplier.Source: §4.1 · p. 4

4.2 Adaptive Magnitude Control

  • Magnitude control addresses temporal gradients that are too small to influence smoothness or large enough to dominate the actor update.The target combines a temporal contribution of magnitude s with the unchanged actor gradient, then makes the smallest Euclidean adjustment to preserve first-order actor-loss progress.Source: §4.2 · p. 5
  • The cap-only target uses min(nT, nR), while the norm-matched target sets the temporal magnitude to nR and amplifies it when nT < nR.A bounded η geometrically interpolates between these choices; increasing η strengthens the temporal gradient when nT < nR.Source: §4.2 · p. 5

4.3 DAMPER Actor Update

  • For nonzero gradients, Algorithm 1 computes both gradients, applies projection and magnitude control, and sends the merged gradient to the native actor optimizer.Source: §4.3 · p. 5

5 Analysis of the Primary–Auxiliary Update

The analysis describes DAMPER’s actor update and establishes that it preserves descent on the native actor loss. At matched primary progress, it also establishes a temporal-gain advantage over fixed-weight scalarization when the temporal gradient is sufficiently small.

5 Analysis of the Primary–Auxiliary Update

  • DAMPER computes native and temporal gradients, then combines the native direction with a norm-scaled temporal direction before applying the actor optimizer.The update uses the native actor gradient norm to control the temporal component’s magnitude.Source: §5 · p. 6

5.1 Descent on the Native Actor Loss

  • Under differentiability and global smoothness assumptions, every Algorithm 1 step strictly decreases the fixed native actor loss.For sufficiently small steps, DAMPER incorporates the temporal gradient while preserving descent on the native actor loss.Source: §5.1 · p. 6

5.2 Temporal Progress at Matched Primary Progress

  • For noncollinear gradients and fixed η > 0, DAMPER provides greater additional temporal gain than any fixed-positive-weight scalarization when the temporal gradient is sufficiently weak, at matched primary progress.The comparison rescales updates to match their predicted decrease in native actor loss, isolating temporal benefit from overall update scale.Source: §5.2 · pp. 6–7

6 Experiments

Across six continuous-control tasks and TD3/SAC, DAMPER lowers action oscillation relative to unregularized agents in all 12 task–backbone pairs, though return effects vary by task. The experiments compare its overall performance, projection and magnitude-control variants, and fixed-weight scalarization.

6.1 Experimental Setup

  • Five seeds and ten evaluation episodes per seed are used to compare episodic return and FFT-based action-oscillation score across six tasks and two backbones.The baselines are unregularized TD3/SAC and five smooth-control methods: CAPS, Grad-CAPS, ASAP, L2C2, and PAVE.Source: §6.1 · p. 7

6.2 Overall Performance

  • DAMPER lowers action oscillation versus unregularized agents in all 12 task–backbone pairs and achieves the lowest score among compared methods in eight, with task-dependent return effects.Figure 2 shows smaller consecutive action changes on Walker under both backbones.Source: §6.2 · p. 7, Figure 2 · p. 8, Table 1 · p. 8
  • TD3 on LunarLander improves both metrics, increasing return from 205.5 to 266.4 and reducing sm from 1.815 to 0.313.On Hopper and Walker, sm falls from 3.061 to 0.240 and from 1.937 to 0.346, respectively, alongside lower returns than unregularized TD3.Source: §6.2 · p. 7

6.3 Ablation Studies

  • Conflict-conditioned projection achieves the highest mean return on three TD3 tasks; unconditional summation sharply reduces Hopper return from 3078 to 317 despite lower oscillation.Unconditional projection also lowers Walker return from 4275 to 3525 with greater variability; the ablation supports removing conflicting components while retaining aligned temporal information.Source: §6.3 · pp. 8–9, Table 3 · p. 9
  • Against a PEGrad-based baseline, DAMPER lowers mean oscillation by 49–62% across three tasks, with higher mean returns on LunarLander and Hopper but slightly lower return on Walker.The baseline limits temporal-gradient magnitude but lacks a parameter for strengthening weak temporal gradients relative to the native actor gradient.Source: §6.3 · p. 9, Table 4 · p. 9

6.4 Comparison with Fixed-Weight Scalarization

  • Across six TD3 tasks, DAMPER has lower mean oscillation than every tested fixed weight on five; on Hopper, its sm is 0.240 versus 0.714 for the smoothest scalarized policy.Tuned scalarization nevertheless achieves competitive returns and improves both mean metrics over multiple specialized smoothness baselines on LunarLander, Ant, and Hopper.Source: §6.4 · p. 10, Figure 3 · p. 10

6.5 Effect of the Interpolation Parameter

  • All three TD3 tasks reach their lowest mean oscillation score at η = 1, with a temporary increase at η = 0.6 on Walker.Oscillation decreases monotonically with η on LunarLander and Hopper; scores are normalized within each task's sweep, so absolute scores remain task-dependent.Source: Figure 4 · p. 10, §6.5 · p. 11

7 Conclusion

DAMPER combines conflict-conditioned temporal-gradient projection with norm-ratio magnitude control while leaving the native actor gradient unchanged. It reduces oscillation across all evaluated task–backbone pairs, but returns are not uniformly preserved.

7 Conclusion

  • DAMPER combines conflict-conditioned temporal-gradient projection with norm-ratio magnitude control in a primary–auxiliary actor update.The native actor gradient is left unchanged.Source: §7 · p. 11
  • Action-oscillation scores fall across all 12 evaluated task–backbone pairs, but return is not uniformly preserved.The theoretical analysis covers a single actor update, not the full training process.Source: §7 · p. 11
  • Physical-system evaluation is identified as a next step for understanding when smoother control can be obtained without sacrificing task performance.Source: §7 · p. 11

AI Use Statement

This block documents the authors’ AI use and gives an optimization characterization and descent proof for the combined gradient. The characterization relates the temporal contribution’s pre-projection magnitude to its attenuated post-projection magnitude.

AI Use Statement

  • AI tools assisted with writing, coding, and checking mathematical derivations, while authors reviewed the text and code and retained responsibility for the final work.The tools were not used to originate research hypotheses or design the methodology or experiments.Source: AI Use Statement · p. 11

A Optimization Characterization of the Combined Gradient

Then the optimization problem

  • The norm-ratio rule sets the temporal magnitude before projection; under conflict, projection makes its final contribution smaller.Using the original temporal-gradient norm in the denominator preserves that attenuation instead of scaling the projected gradient back up.Source: Then the optimization problem · pp. 14–15

B.1 Proof of Theorem 1

  • The proof establishes strict decrease in the native actor loss under its stated step-size condition, including when the numerical guard is inactive.When the guard activates, the update is the native actor gradient; the proof bounds the step-size condition in the other case.Source: §B.1 · p. 15

B.2 Proof of Theorem 2

The proof interprets G(d) as additional first-order temporal decrease per unit primary decrease relative to the native-gradient update. It establishes positive primary progress for the scalarized direction and an asymptotic advantage for noncollinear gradients under fixed positive parameters.

B.2 Proof of Theorem 2

  • G(d) measures additional first-order temporal decrease per unit primary decrease relative to the native-gradient update.Source: §B.2 · p. 16
  • The scalarized direction also has positive primary progress.Source: §B.2 · p. 16
  • For fixed η > 0 and λ > 0, the lower bound diverges as q approaches zero, establishing the claimed asymptotic advantage for noncollinear gradients.The accompanying bounds use constants independent of q and c.Source: §B.2 · pp. 16–17

C Additional Fixed-Weight Scalarization Results

The fixed-weight sweeps show that tuned temporal-consistency losses can outperform smoothing baselines, but the return–smoothness trade-off depends on the task. Comparisons use reported means and sweep-selected weights, so they do not establish statistical significance.

C Additional Fixed-Weight Scalarization Results

  • Each task sweep evaluates 20 fixed weights with five training seeds and ten episodes per seed, plotting mean return against FFT-based oscillation score.The highlighted point is the sweep’s lowest mean oscillation score; axis ranges vary by task.Source: C Additional Fixed-Weight… · p. 17
  • Fixed-weight scalarization’s lowest sm is 0.681 at λ = 0.80 on LunarLander and 0.825 at λ = 0.85 on Pendulum, versus DAMPER’s 0.313 and 0.463.Source: Figure 5 · p. 17, Figure 6 · p. 17
  • Reacher’s lowest scalarization score, 0.0325 at λ = 0.95, is below DAMPER’s 0.048; Ant’s 1.327 at λ = 0.90 and Walker’s 0.674 at λ = 0.95 are not below DAMPER.Source: Figure 7 · p. 18, Figure 8 · p. 18, Figure 9 · p. 18
  • On Ant, λ = 0.75 scalarization improves return and oscillation over all five baselines, with (re, sm) = (5486, 1.574).Source: C Additional Fixed-Weight… · p. 18
  • On LunarLander and Hopper, selected scalarization weights improve both metrics over CAPS, ASAP, L2C2, and PAVE.LunarLander uses λ = 0.80 with (259.8, 0.681); Hopper uses λ = 0.80 with (3338, 0.714).Source: C Additional Fixed-Weight… · p. 18
  • On Pendulum, λ = 0.85 gives (−154.6, 0.825), improving both metrics over ASAP (−155.3, 2.013) and L2C2 (−156.0, 3.293).Source: C Additional Fixed-Weight… · p. 19
  • The gains are task dependent: CAPS remains smoother on LunarLander, while CAPS and PAVE remain smoother on Pendulum.Reacher and Walker’s lowest scalarization scores improve on all five baselines but have lower returns; comparisons use reported means and sweep-selected weights and do not establish statistical significance.Source: C Additional Fixed-Weight… · p. 19

D Experimental Details

The section documents baseline and DAMPER configurations, the native actor objectives used for TD3 and SAC, and update-time measurements on Hopper-v5. With η = 0, DAMPER adds measured update overhead, while its SAC runtime is lowest among the compared smoothness methods.

D Experimental Details

  • CAPS, Grad-CAPS, and ASAP use the same baseline settings for TD3 and SAC, while PAVE uses backbone-specific loss weights.Settings follow the PAVE appendix [16]; Grad-CAPS is labeled GRAD in the original tables.Source: D Experimental Details · p. 19, Table 5 · p. 19
  • PAVE weights mixed-partial regularization, vector-field consistency, and curvature preservation with λ1, λ2, and λ3, respectively.Its perturbation scale σ = 0.01 and curvature floor δ = 1.0 are fixed across tasks and backbones.Source: D Experimental Details · p. 19
  • L2C2 uses σ = 1.0, λ = 0.01, λ = 1.0, and β = 0.1 for both backbones on all six tasks; DAMPER’s η is task–backbone specific.The parameter sweeps for DAMPER ablations are reported separately.Source: D Experimental Details · p. 19, Table 6 · p. 20
  • The training budgets and network sizes follow the PAVE protocol [16], and experiments used NVIDIA GeForce RTX 2080 and RTX 5060 Ti GPUs.The cited passages do not specify the individual budget or network-size values.Source: Table 7 · p. 20, D Experimental Details · p. 20

E Native Actor Objectives

  • TD3 and SAC use their respective native actor losses and deterministic action maps, with critic parameters fixed during actor optimization and gradients passing through the critic’s action input.The temporal-consistency action is the TD3 deterministic actor output or SAC’s squashed mean action, including action-bound rescaling.Source: E Native Actor Objectives · p. 20

F Computational Cost

  • Runtime tests used 1,000 timed update iterations after 50 warmup iterations, with batch size 256 and logging, diagnostics, and target-network copies excluded.TD3 performed 500 actor updates and SAC 1,000 during the timed interval; times were normalized to each unregularized backbone.Source: F Computational Cost · p. 21
  • 6.57 ms per iteration with TD3 and 14.26 ms with SAC (η = 0), DAMPER overheads of 44.8% and 31.3%, respectively, over the unregularized backbones.On SAC, DAMPER has the lowest measured runtime among compared smoothness methods; the measurements are update computation, not end-to-end training time.Source: Table 8 · p. 21, F Computational Cost · p. 21

Related papers

Loading 2609.38903v1…