Source-linked AI summary
A Geometric Approach to Soft Actor-Critic with Zonotopes for Locomotion Learning
Panagiotis Roditis, Panagiotis P. Filntisis, Petros Maragos
TL;DR
Off-policy actor–critic methods use a fixed minimum rule for twin critics, despite disagreement that can have richer structure. GeZo-SAC uses critic-side zonotopes to adapt pessimism and critic combination; across four MuJoCo-v5 tasks, it achieves top returns on two tasks and strong control-quality measures.
Problem
Off-policy actor–critic methods can amplify positive critic errors through policy improvement, while twin critics use a fixed minimum rule that does not exploit richer disagreement structure.
Source: §I · p. 1Method
GeZo-SAC equips each critic with a state–action-dependent zonotope whose directional supports provide pessimistic value offsets and disagreement signals for combining critics.
Source: §I · p. 1Results
Takeaways & Limitations
Across four MuJoCo-v5 locomotion tasks, GeZo-SAC combines strong returns with low measured mechanical effort and near-zero measured overestimation.
Source: §V · p. 8
Abstract
from arXiv · showhide
Off-policy actor--critic methods control overestimation bias by taking the minimum of two critics. This uses the same aggregation rule everywhere, regardless of how the critics disagree. We propose \textbf{GeZo-SAC}, which uses auxiliary geometric representations to adapt critic pessimism to the state and action. Alongside its scalar value, each critic predicts a set of generators defining a zonotope. Probing this zonotope along sampled directions provides a geometric width, "subtracted from each critic value as a pessimistic offset, and a measure of disagreement between the two critics, aggregated with log-sum-exp. This disagreement controls how the critics are combined, moving from a width-weighted average toward the usual minimum as disagreement increases. At inference, the deployed policy is an unmodified SAC actor, since the generators are used only on the critic side during training.Across four MuJoCo-v5 locomotion benchmarks and six off-policy baselines, GeZo-SAC achieves the highest mean return on Ant-v5 and Hopper-v5 and remains competitive with other methods on the remaining tasks. Our analysis further shows that GeZo-SAC achieves the lowest average actuator work and action effort per metre among the evaluated methods, while maintaining near-zero measured overestimation frequency across all four environments.
I. INTRODUCTION
GeZo-SAC addresses the limits of fixed twin-critic aggregation by using state–action-dependent zonotopes to adapt pessimism and critic combination. Across four MuJoCo-v5 benchmarks, it performs strongly on returns and control-quality measures.
I. INTRODUCTION
- In off-policy actor–critic methods, critic errors can be amplified by policy improvement, destabilizing training and degrading final policy quality.The actor uses critic values for both temporal-difference targets and policy-update gradients, and may select actions exploiting positive Q errors.Source: §I · p. 1
- Twin-critic methods such as TD3 and SAC [5] use the minimum of two scalar predictions, without adapting aggregation to richer disagreement structure.The passage says this fixed rule does not exploit richer structure in how the critics disagree.Source: §I · p. 1
- GeZo-SAC adds a state–action-dependent zonotope to each critic; directional probes supply pessimistic width offsets and disagreement signals for critic aggregation.As disagreement increases, the combination shifts from a weighted average toward the minimum.Source: §I · p. 1
- The design draws on tropical-geometry connections and uses temperature-controlled log-sum-exp to smooth the maximum sampled directional discrepancy.The cited background relates ReLU networks to polyhedral and zonotopic representations,, [8],,.Source: §I · p. 1
- GeZo-SAC achieves the highest mean return on Ant-v5 and Hopper-v5 and the lowest average actuator work and action effort per metre, with near-zero measured overestimation.It remains competitive on the other tasks among six off-policy baselines.Source: §I · p. 1
II. RELATED WORK
Related work addresses optimistic value errors through conservative targets, penalties, uncertainty estimates, and structured return models. GeZo-SAC draws on zonotopic representations and support functions to adapt critic pessimism and aggregation.
II. RELATED WORK
- Overestimation countermeasures include Double DQN [11], which decouples action selection and evaluation; TQC, which truncates optimistic quantiles; GPL, adaptive critic-target penalties; TDC-λ, lower-confidence Bellman targets; and CQL, conservative Q regularization.Under stated assumptions, CQL’s expected policy value under the learned critic lower-bounds the true policy value.Source: §II · p. 2
- Uncertainty-aware approaches include OAC, distributional SAC refinements, and SUNRISE, which use uncertainty or return distributions for exploration or critic updates.OAC uses an upper confidence bound to guide exploration, while SUNRISE reweights Bellman backups and selects exploratory actions using upper-confidence estimates.Source: §II · p. 2
- Prior work connects ReLU networks with zonotopic representations, [8], and approximation error with distances between geometric objects; GeZo-SAC uses support functions and temperature-controlled LSE to compare learned zonotopes.LSE provides a smooth approximation of the maximum discrepancy over sampled directions.Source: §II · p. 2
III. METHOD
GeZo-SAC extends SAC with state–action-conditioned zonotopes predicted alongside scalar critic values. Directional geometric signals adjust pessimism and how the twin critics are aggregated.
III. METHOD
- Each critic predicts a scalar value and K generators in a fixed auxiliary space, enabling geometric modulation to vary with state and action.The auxiliary-space dimension sets the representation's capacity.Source: §III · p. 2
- Generator heads are trained with geometric regularization, while TD learning influences them indirectly through the shared trunk.The regularizers include squared-band, decorrelation, and cross-critic band-symmetry penalties.Source: §III · p. 3
- Band strength starts at zero for 104 critic updates, then rises linearly toward λzono over the next 2×10^5 updates.During warm-up, band offsets and generator penalties are inactive, but geometric aggregation remains active.Source: §III · p. 3
- Directional support probes estimate zonotope bands and critic disagreement at target and actor-update inputs.The band approximates average directional support radius and is subtracted from critic values as a pessimistic offset.Source: §III · pp. 3–4
- Similar critic supports favor a weighted average of pessimistic values, while large directional discrepancies shift aggregation toward the conservative minimum.The disagreement is aggregated with temperature-controlled log-sum-exp.Source: §III · pp. 3–4
- An empirical target-inclusion controller adapts κ toward an inclusion rate of 0.8, which the paper cautions is not statistical coverage of true policy value.The critic loss also penalizes excessive value magnitudes and positive TD residuals.Source: §III · p. 4
A. Experimental Setup
The experiments compare GeZo-SAC with six off-policy baselines on four MuJoCo-v5 locomotion tasks under a shared evaluation protocol. Evaluation covers returns, value-estimation diagnostics, and mechanical and action costs, including cross-environment normalized metrics.
Baselines and environments
- Six off-policy baselines are evaluated alongside GeZo-SAC on Ant-v5, Hopper-v5, Walker2d-v5, and Humanoid-v5.The methods share the environment and evaluation protocol while retaining their own critic structures and aggregation mechanisms.Source: A. Experimental Setup · pp. 4–5
Training and evaluation
- Each method uses five independent training runs; every 50 training episodes, checkpoints are evaluated on 10 deterministic episodes, and the highest-mean-return checkpoint is evaluated over 300 deterministic episodes.Source: A. Experimental Setup · p. 5
Metrics
- Evaluation includes episodic return, overestimation frequency, critic–return discrepancy, actuator work per metre, forward distance, action variation, and action effort.Mechanical metrics are normalized within each environment before averaging across tasks; lower values indicate lower cost for Work/m, TV/m, and Effort/m.Source: A. Experimental Setup · p. 5
B. Results
GeZo-SAC has the highest mean return on Ant-v5 and Hopper-v5 and remains competitive on Humanoid-v5 and Walker2d-v5. Its cross-task return and mechanical-cost rankings are favorable overall, while task-level trade-offs and generator-count sensitivity vary by environment.
Performance and training dynamics
- GeZo-SAC achieves the highest mean evaluation return on Ant-v5 and Hopper-v5 and remains competitive on Humanoid-v5 and Walker2d-v5.All four environment results use K = 15 generators.Source: B. Results · pp. 5–6
Mechanical cost and control quality
- Across tasks, GeZo-SAC combines the highest average return with the lowest average actuator work, action variation, and action effort per metre.It ranks second in forward distance; Table II reports environment-normalized mechanical metrics averaged across tasks.Source: B. Results · p. 6
Sensitivity to generator count
- Figure 4 evaluates sensitivity to zonotope-generator count using mean evaluation return over five training seeds, with shaded regions and error bars showing ± one standard deviation; bold labels mark the highest observed mean in each environment.Source: Figure 4 · p. 6
Performance–energy trade-offs
- Task-level work–return trade-offs differ: GeZo-SAC is especially favorable on Ant-v5 and Humanoid-v5, while on Hopper-v5 it uses more work than SAC and SUNRISE.On Walker2d-v5, GeZo-SAC and SUNRISE have similar returns at the lowest work per metre.Source: B. Results · p. 6
Locomotion and mechanical cost
- In illustrative one-second Ant-v5 rollouts, GeZo-SAC travels farther than TQC and GPL-SAC while expending less absolute actuator work.The figure pairs time-matched poses with cumulative actuator work versus forward displacement.Source: B. Results · p. 6, Figure 3 · p. 6
C. Value Estimation and Policy Performance
GeZo-SAC pairs value-estimation diagnostics with policy returns across four locomotion tasks. Its return ranking does not always track absolute critic–return agreement.
C. Value Estimation and Policy Performance
- GeZo-SAC has the highest mean Monte-Carlo return on Ant-v5 and Hopper-v5, while TQC achieves a higher return on Humanoid-v5.Figure 5 compares policy return with two value-estimation diagnostics across four tasks.Source: C. Value Estimation and Policy… · pp. 6–7, Figure 5 · p. 7
- Measured overestimation frequency is near zero across all four environments; on Ant-v5, GeZo-SAC is below SAC and TQC.On Hopper-v5, GeZo-SAC, SAC, TD3, and TQC also show near-zero measured frequency, while GPL-SAC is higher.Source: C. Value Estimation and Policy… · p. 6, Figure 5 · p. 7
- GeZo-SAC has the largest mean critic–return discrepancy on Ant-v5, Humanoid-v5, and Walker2d-v5, despite strong returns on Ant-v5 and Humanoid-v5.The passage notes that an action-independent offset can change absolute value errors without changing action preferences.Source: C. Value Estimation and Policy… · p. 7
D. Ablation: Number of Zonotope Generators
The effect of generator count varies by task: larger sets favor Ant-v5 and Hopper-v5, whereas smaller sets favor Humanoid-v5 and Walker2d-v5.
D. Ablation: Number of Zonotope Generators
- The ablation varies K across {1, 2, 4, 6, 8, 10, 15, 20}, holding other hyperparameters fixed and using five training seeds per configuration.Source: D. Ablation: Number of Zonotope… · p. 7
- Ant-v5 and Hopper-v5 attain their highest mean returns at K = 15 and K = 20, while Humanoid-v5 and Walker2d-v5 peak at K = 1 and K = 2.Larger generator sets also generally show less variability across seeds on Ant-v5 and Hopper-v5.Source: D. Ablation: Number of Zonotope… · p. 7
- Increasing generator count does not uniformly improve performance, and the main comparisons use shared K = 15 across all four environments.Source: D. Ablation: Number of Zonotope… · p. 7
E. Ablation: Fixed versus Adaptive Pessimism
The fixed-versus-adaptive pessimism ablation finds different return outcomes across tasks and shows κ changing during Hopper-v5 training.
E. Ablation: Fixed versus Adaptive Pessimism
- The comparison uses fixed κ = 1, near-zero κ = 10^-6, and adaptive κ, with K = 15, 3 × 10^6 environment steps, and three seeds per environment.Source: E. Ablation: Fixed versus Adaptive… · p. 7
- Adaptive κ has higher displayed returns on Ant-v5 and Humanoid-v5 and the highest final evaluation return on Hopper-v5, while near-zero κ performs best on Walker2d-v5.Hopper-v5 training curves overlap despite the adaptive setting's highest final evaluation return.Source: E. Ablation: Fixed versus Adaptive… · p. 7
- On Hopper-v5, the adaptive controller decreases κ substantially early in training and later stabilizes at an intermediate level.Source: E. Ablation: Fixed versus Adaptive… · p. 7
F. Computational Cost
GeZo-SAC adds critic capacity while keeping the deployed actor's architecture and inference latency the same as SAC.
F. Computational Cost
- GeZo-SAC's training cost is comparable to TQC and substantially below larger ensemble-based methods such as GPL-SAC and SUNRISE.Its critic capacity is higher than SAC's.Source: F. Computational Cost · p. 7
- The additional zonotope representation is confined to the critics.Source: F. Computational Cost · p. 7
- The deployed actor has the same architecture and inference latency as SAC, so the extra computational cost is incurred only during training.Source: F. Computational Cost · p. 7
V. CONCLUSIONS
GeZo-SAC augments SAC’s twin critics with zonotope generators whose directional supports adapt pessimism and critic aggregation. Across four MuJoCo-v5 tasks, it achieves the highest mean return on Ant-v5 and Hopper-v5 while also attaining the lowest average work and action effort per metre and near-zero measured overestimation frequency.
V. CONCLUSIONS
- GeZo-SAC achieves the highest mean return on Ant-v5 and Hopper-v5, outperforms SAC and TD3 on every task, and attains the lowest average work and action effort per metre with near-zero measured overestimation frequency.These results span four MuJoCo-v5 tasks.Source: §V · p. 8
- GeZo-SAC augments SAC’s twin critics with auxiliary zonotope generators whose directional supports provide heuristic geometric bands and a disagreement signal for adapting pessimism and critic aggregation.Source: §V · p. 8
- The authors plan to extend and evaluate GeZo-SAC on robotic manipulation tasks.Source: §V · p. 8