Source-linked AI summary
Unthrottling the Tanh Jacobian in SAC: A Negative Result on Bang-Bang Control and MetaDrive
Faiq Shamass
TL;DR
The paper asks whether tanh’s vanishing Jacobian prevents SAC from learning extreme actions and tests an added detached critic-gradient bypass. Across a bang-bang control task and warm-started MetaDrive, the bypass does not help: vanilla SAC is already near the bound, while bypass variants saturate or otherwise degrade performance.
Problem
The paper asks whether tanh’s vanishing Jacobian starves SAC of critic signal when tasks require extreme actions near the bounds.
Method
The paper adds a no-gain actor-loss term whose gradient on the pre-tanh mean is the detached action-gradient of Q, using ungated and shoulder-gated variants.
Results
Across minimum-time control and warm-started MetaDrive, the bypass does not improve results; vanilla SAC is already near the bound, while bypass variants saturate or degrade performance.
Takeaways & Limitations
The tanh Jacobian is a real gradient factor, but undoing it is not a free speedup and is not helpful on the studied tasks.
Takeaways & Limitations
The result concerns a specific intervention, with driving evidence limited by mostly single paired seeds, unmatched additional no-entropy seeds, and incomplete ablations.
Abstract
from arXiv · showhide
Soft Actor-Critic (SAC) represents a continuous policy as an unbounded Gaussian that is squashed by tanh. The Jacobian of that map is $\partial a/\partial u = 1-a^2$, which vanishes as $|a|\to 1$. A natural concern is that this throttle starves the actor of critic signal exactly where extreme actions (full brake, full throttle) are optimal. We test a minimal intervention that restores the missing signal: one extra term in the actor loss whose gradient on the pre-tanh mean is the detached action-gradient of $Q$, with no gain parameter. On a minimum-time double integrator whose optimum is bang-bang at the action bounds, vanilla SAC already reaches near-optimal return ($-31.6$ vs. a calibrated optimum of $-30.3$) across ten paired seeds. An ungated bypass does saturate the policy (99% of eval steps with $|a|\ge 0.9$) and collapses return to $-195.5$. A gated bypass that fires only on the flat shoulder $|a|\in[0.9,0.999]$ also fails, and does so without leaving a saturated policy. Warm-started MetaDrive fine-tuning shows the same pattern: the bypass does not improve return, and where collision rate falls it is typically traded for out-of-road departures. Auto-tuned entropy coefficient rises against the bypass, which is a push toward the tails. The Jacobian effect is real. Treating it as a bug to be undone is not free, and on the tasks studied here it is not helpful. Saturating a bound is not the same as solving a problem whose optimum lives on that bound.
1 Introduction
The paper tests whether tanh’s vanishing Jacobian harms SAC on tasks requiring extreme actions, finding that restoring the attenuated critic gradient fails on both control and driving domains.
- 1 Introduction: The motivating concern is that tanh attenuates critic signal near action bounds, where extreme actions may be optimal.
- 1 Introduction: The intervention adds an actor-loss term whose pre-tanh mean gradient is the detached action-gradient of Q rather than its tanh-throttled version.No step-size knob is introduced, and the relative extra strength grows as 1/(1 −a2) near saturation.
- 1 Introduction: Vanilla SAC solves the bang-bang control task near the calibrated reference, while bypass variants fail through saturation or destabilization.The ungated bypass locks the policy to the wall, whereas shoulder gating avoids saturation without recovering performance.
- 1 Introduction: The bypass does not improve warm-started MetaDrive return, and collision reductions are accompanied by higher out-of-road rates.
- 1 Introduction: The conclusion is limited to this specific intervention and does not establish tanh as the optimal bound-handling method.Alternative action distributions remain open.
2 Background
SAC squashes Gaussian actions with tanh, whose Jacobian attenuates near the bounds; however, this attenuation slows optimization without necessarily changing the standard objective’s stationary point.
- 2 Background: The tanh Jacobian appears in both the likelihood correction and the reparameterized actor gradient, with the latter vanishing near action bounds.
- 2 Background: Entropy regularization pushes pre-tanh actions away from the tails, creating tension with interventions that drive them toward saturation.
- 2 Background: Multiplying the standard actor gradient by 1 −a2 slows movement along a coordinate but does not by itself move the objective’s stationary point.
- 2 Background: Adding an unthrottled action-gradient copy changes the actor’s first-order condition and biases the pre-tanh mean toward the bound indicated by Q.Its usefulness depends on whether moving farther toward the wall is actually the task’s error.
3 Gradient bypass
The bypass adds a detached critic-gradient term to the SAC actor loss, optionally activates it near action saturation, and evaluates it under multiple entropy treatments.
- 3 Gradient bypass: The bypass uses the detached critic action-gradient on the pre-tanh mean, with an optional per-dimension mask and no scalar gain.Its relative strength is 1/(1 −a2) wherever the mask is active.
- 3 Gradient bypass: The ungated mask activates the intervention in every dimension, while the gated mask activates it only for means in the shoulder 0.9 ≤ |aµ,d| < 0.999.MetaDrive further combines the gate with a blame bit from collision or out-of-road failure windows.
- 3 Gradient bypass: Default experiments auto-tune entropy, with additional runs using α = 0 or a frozen checkpoint value because tail-driving can oppose entropy optimization.
- 3 Gradient bypass: The bypass arm is implemented by adding the second actor-loss term while keeping collection, replay, critics, targets, batch size, and other choices identical.
4 Toy: minimum-time double integrator
On the bang-bang double integrator, vanilla SAC already approaches the calibrated optimum, while both Jacobian bypasses substantially worsen performance for different reasons.
- Setup and baseline: −31.6 final return matches the c = 0.9 reference, and p95(|a|) = 0.97 shows vanilla SAC already reaches near-bound actions.The calibrated uncapped optimum is −30.3, while the c = 0.9 reference is −31.5.
- Results: −195.5 return and 2% reach rate follow ungated bypass saturation at 99% of eval steps, leaving the policy on the wall without switching.Its mean |a| reaches 1.00, but the resulting behavior is equivalent to an effective cap of approximately 0.3.
- Results: −108.6 return and 58% reach rate show that the gated bypass avoids wall-locking but produces an unsaturated, low-quality policy.Saturation falls from 0.32 to 0.05 relative to baseline, so the intended mechanism does not appear at evaluation.
- Results: No bypass seed reaches the −35 return threshold within 40k steps, whereas every baseline seed does.This rejects the hypothesis that the bypass reaches the same destination by a shorter path.
5 MetaDrive fine-tuning
Warm-started MetaDrive fine-tuning does not produce a return gain from the gated bypass. Apparent benefits are configuration- and metric-dependent, with collision reductions traded against out-of-road departures.
- Experimental setting: The MetaDrive evidence is limited by mostly single paired seeds, unmatched extra no-entropy bypass seeds, warm-starting, and the gated blame-based intervention.The study did not complete from-scratch MetaDrive, throttle-only, or multi-seed frozen-α evaluations.
- Experimental setting: At density 0.3, all arms decay from the warm checkpoint, so the comparison measures decay rates rather than improvement on an unsolved task.The checkpoint is already strong at density 0.1, with held-out success approximately 0.92 and return approximately 301.
- Results: Entropy-on bypass runs are slightly worse on return and success, while lower collision rates are offset by similarly higher out-of-road rates.The authors characterize this as a mode trade rather than an improvement in driving.
- Results: With frozen α at density 0.1, return ties across training while collisions fall from 0.086 to 0.052 and out-of-road departures rise from 0.140 to 0.244.Both arms remain below the t=0 checkpoint, with success declining from 0.92 to approximately 0.70–0.77.
- Results: MetaDrive does not rescue the toy: the bypass shows no return gain across the warm-start comparisons, and the only locally positive cell uses α = 0 on one seed.The α = 0 cell is explicitly described as the least standard SAC setting.
6 Discussion
The experiments show that vanilla SAC was not prevented from reaching extreme actions, while Jacobian bypasses changed the objective and failed to improve control. In MetaDrive, any collision reductions were accompanied by higher out-of-road rates rather than clean return gains.
- Vanilla SAC already commands p95(|a|) ≈0.97 and matches a 0.9-thrust reference on the task most favorable to the Jacobian-starvation hypothesis.
- Entropy rises against the bypass, and removing entropy discards a stabilizer that the baseline needs.The bypass pushes the actor toward the tails while auto-tuned α compensates.
- A different bound representation, such as Beta policies or a residual around a saturated switching controller, is more plausible than adding another gate.
- In MetaDrive, the bypass never produces a clean return win, while collision reductions track higher out-of-road rates in three of four settings.
7 Limitations
The MetaDrive evidence is constrained by limited pairing and evaluation coverage, while the toy experiment provides the stronger and cleaner negative result.
- Driving results are mostly one paired seed, and extra no-entropy seeds lack matched baselines.Fine-tuning starts from a strong checkpoint, often uses shifted traffic density, and early evaluations use 20 episodes.
- The MetaDrive bypass differs from the ungated toy term, and several comparisons were not completed.The study did not complete from-scratch MetaDrive, throttle-only ablations, or a multi-seed frozen-α grid.
8 Conclusion
Restoring tanh's attenuated critic gradient does not help on the tested extreme-action tasks. The Jacobian affects the actor gradient, but undoing it is not a free speedup.
- The bypass does not improve return: vanilla SAC is already near the bound, while ungated and gated variants fail differently on the double integrator and MetaDrive fine-tuning shows no return gain.
- The tanh Jacobian is a real factor in the actor gradient, but this evidence does not identify it as the reason these policies fail.