Source-linked AI summary
Mildly Conservative Q-Learning for Offline Reinforcement Learning
Jiafei Lyu, Xiaoteng Ma, Xiu Li, Zongqing Lu
TL;DR
Offline RL must control overestimation caused by distribution shift while avoiding the excessive pessimism that limits value generalization. MCQ actively trains OOD actions with pseudo Q targets, and its theory and D4RL experiments show improved guarantees and performance, including offline-to-online transfer. The method still requires tuning a weighting coefficient.
Problem
Offline RL must control OOD-action overestimation from distribution shift without the excessive pessimism that suppresses value-function generalization and performance.
Method
MCQ combines the practical Mildly Conservative Bellman operator with active OOD-action training using pseudo target values and a CVAE-integrated SAC algorithm.
Results
MCQ surpasses strong baselines on many non-expert D4RL MuJoCo datasets, remains competitive on expert datasets, and shows superior offline-to-online generalization.
Takeaways & Limitations
MCQ provides a milder form of conservatism that retains pessimism while improving generalization and policy performance in offline reinforcement learning.
Takeaways & Limitations
MCQ requires tuning the weighting coefficient λ, with automatic tuning left for future work.
Abstract
from arXiv · showhide
Offline reinforcement learning (RL) defines the task of learning from a static logged dataset without continually interacting with the environment. The distribution shift between the learned policy and the behavior policy makes it necessary for the value function to stay conservative such that out-of-distribution (OOD) actions will not be severely overestimated. However, existing approaches, penalizing the unseen actions or regularizing with the behavior policy, are too pessimistic, which suppresses the generalization of the value function and hinders the performance improvement. This paper explores mild but enough conservatism for offline learning while not harming generalization. We propose Mildly Conservative Q-learning (MCQ), where OOD actions are actively trained by assigning them proper pseudo Q values. We theoretically show that MCQ induces a policy that behaves at least as well as the behavior policy and no erroneous overestimation will occur for OOD actions. Experimental results on the D4RL benchmarks demonstrate that MCQ achieves remarkable performance compared with prior work. Furthermore, MCQ shows superior generalization ability when transferring from offline to online, and significantly outperforms baselines. Our code is publicly available at https://github.com/dmksjfl/MCQ.
1 Introduction
Offline RL avoids costly or unsafe environment interaction but must manage distribution shift between learned and behavior policies. MCQ addresses the resulting conservatism–generalization trade-off by actively training OOD actions, with theory and experiments supporting improved performance.
- Offline RL learns from a static dataset, avoiding environment interactions that may be expensive, difficult, or dangerous.
- Distribution shift makes OOD action evaluation vulnerable to extrapolation and bootstrapping errors, so conservative value estimation is necessary.
- Existing policy-regularization and value-penalization methods can overly restrict learning, limiting generalization and causing pessimism when behavior data are poor.
- MCQ introduces the MCB operator, which actively trains OOD actions with queried values while preserving conservatism where the dataset supports the optimal policy.
- MCQ surpasses strong baselines on most D4RL MuJoCo tasks, especially non-expert datasets, and shows superior offline-to-online generalization.
2 Preliminaries
Offline RL uses a fixed dataset and therefore cannot rely on new environmental interaction during training. Because learned-policy actions may fall outside the behavior policy’s support, MCQ actively trains OOD actions with pseudo targets to improve generalization while retaining pessimism.
- An MDP specifies states, actions, rewards, initial-state distribution, transition probabilities, and a discount factor.
- RL seeks a policy maximizing expected discounted cumulative rewards, while Q(s, a) measures discounted return from a state-action pair.
- The optimal policy’s Q function is characterized by the Bellman optimal operator.
- Offline RL accesses only a previously collected dataset, with behavior policy μ describing how its actions were generated.
- Policy actions outside the behavior-policy support can have arbitrarily wrong values, so MCQ constructs pseudo targets to train OOD actions and retain pessimism with better generalization.
3 Mildly Conservative Q-Learning
MCQ combines the Mildly Conservative Bellman operator with practical behavior-policy modeling to train OOD actions using pseudo-target values. Its theory guarantees convergence, no erroneous overestimation, and policy performance at least matching the behavior policy under stated conditions.
- Algorithm: A corrected Equation 5 was released after the authors identified a bug; the modification has marginal theoretical influence and no effect on the practical MCQ algorithm.The update credits Jiayi Guan for identifying the issue.
- Mildly Conservative Bellman Operator: MCB actively trains OOD actions by replacing their value estimates with pseudo-target values based on in-support actions, preserving pessimism while improving generalization.For OOD actions, the operator uses maxa′∼Support(µ(·|s)) Q(s, a′) − δ, with δ > 0 arbitrarily small.
- Mildly Conservative Bellman Operator: The MCB operator is a γ-contraction in the behavior-policy support, so repeated application converges from any initial Q function to a unique fixed point.The practical MCB operator retains this contraction property.
- Mildly Conservative Bellman Operator: Within the behavior-policy support, the fixed point satisfies Qµ ≤ QMCB ≤ Qµ∗, ensuring the induced policy behaves at least as well as the behavior policy.The result also indicates approximation to the optimal batch-constraint policy.
- Mildly Conservative Bellman Operator: MCB provides milder pessimism than value penalization or explicit policy-constraint methods while offering a tighter lower bound for consistent policy improvement.The comparison includes CQL and explicit policy-constraint methods such as TD3+BC.
- Practical MCB Operator: The practical operator fits an empirical behavior policy from the dataset, samples N actions from it, and uses their maximum value to construct OOD pseudo-targets.A CVAE is the default behavior-policy model; CGAN is suggested for highly multimodal datasets.
- Practical MCB Operator: Under a comparatively well-fitted empirical behavior policy, the practical operator avoids erroneous overestimation despite possible distributional shift between the fitted and true behavior policies.The pseudo-target approximates the maximum in-support Q-value with high probability when the fitting error is sufficiently small.
- Algorithm: MCQ combines MCB with SAC, adding an auxiliary critic loss that actively trains OOD actions and can also be plugged into other off-policy algorithms.The method does not require state or value-function normalization and is also combined with TD3 as a deterministic variant.
4 Experiments
MCQ is evaluated on D4RL MuJoCo benchmarks, parameter sensitivity, Q-value estimation, and offline-to-online fine-tuning. It generally outperforms baselines, especially on non-expert datasets, while maintaining stable value estimates and adapting effectively online.
- Results on MuJoCo Datasets: MCQ is compared with BC, SAC, CQL, UWAC, TD3+BC, and IQL on 15 MuJoCo-v2 datasets spanning five dataset types.The experiments use four random seeds and normalized average scores.
- Results on MuJoCo Datasets: MCQ achieves the best performance on 11 of 15 datasets, with a total average score of 72.8.It also substantially outperforms prior methods on many random and medium-replay datasets and remains competitive on expert datasets.
- Parameter Study: Smaller λ causes significant performance drops, while λ balances in-distribution and OOD-action training.λ = 1 reduces MCQ to SAC, whereas very small λ values allow OOD actions to overwhelm critic training.
- Parameter Study: MCQ is insensitive to N across a wide range and therefore uses N = 10 by default.N regulates the balance between pessimism and generalization when OOD actions are sampled from an approximate behavior policy.
- Q Function Estimation: No overestimation appears even at λ = 0.95, while small λ values produce severe Q-value underestimation.Q estimates remain similar across different N values when λ is fixed appropriately.
- Offline-to-online Fine-tuning: MCQ consistently outperforms prior offline RL methods and AWAC during online fine-tuning on random and medium-replay datasets.Training uses 1M offline gradient steps followed by 100K online gradient steps across four random seeds.
5 Related Work
Related work covers model-free and model-based offline RL, along with offline-to-online adaptation. Existing methods use policy constraints, value penalties, uncertainty estimation, model learning, or other mechanisms to address distribution shift and adaptation.
- Model-free offline RL: Model-free offline RL methods restrict OOD actions through importance sampling, policy constraints, latent actions, value penalties, adaptive methods, or uncertainty quantification.Representative methods include CQL, TD3+BC, and uncertainty-based approaches.
- Model-based offline RL: Model-based offline RL methods learn dynamics models and use them for policy optimization, incorporating uncertainty, conservative values, representation learning, behavior cloning, or sequential modeling.The listed approaches differ in how they control model uncertainty and policy behavior.
- Offline-to-online RL: Offline-to-online RL improves an offline-trained policy through online interactions, using techniques such as model ensembles and explicit policy constraints.The goal is faster adaptation and stable policy improvement.
- Offline-to-online RL: Overly pessimistic value functions or policies can make offline-to-online fine-tuning difficult and lead to a suboptimal policy.This motivates methods that preserve offline performance while enabling adaptation.
6 Conclusion
The paper presents MCQ as a method for reducing excessive pessimism in offline RL while preserving conservative value estimation. It reports strong benchmark and offline-to-online results, but requires tuning λ.
- Conclusion: MCQ actively trains OOD actions with pseudo target values guided by the practical MCB operator.The method is designed to alleviate over-pessimism in existing offline RL algorithms.
- Conclusion: MCQ's induced policy is theoretically at least as good as the behavior policy, and the practical MCB operator avoids erroneous overestimation.The paper also reports tighter theoretical policy-improvement guarantees than several conservative alternatives.
- Conclusion: MCQ surpasses baselines by large margins on many non-expert datasets, remains competitive on expert datasets, and generalizes well from offline to online learning.The conclusion identifies mild conservatism as critical for offline learning.
- Conclusion: MCQ requires tuning the weighting coefficient λ, although 0.7 ≤ λ < 1 usually provides satisfying performance.Automatic tuning of λ is left for future work.
Checklist
The checklist records affirmative reporting of theoretical assumptions and proofs, experimental reproducibility details, ethical-scope disclosures, and error-bar and compute reporting.
- Reporting and Ethics: The paper reports that its main claims accurately reflect its contributions and that its limitations and scope are described.The checklist also marks potential negative societal impacts as not applicable.
- Theory: The theoretical-results checklist marks the full assumptions and complete proofs as included.This concerns the paper's theoretical results and their supporting proofs.
- Experiments: The experiments checklist marks reproducibility materials and training details as included.The reported materials cover code, data, instructions, data splits, hyperparameters, and selection details.
- Reproducibility and Assets: The checklist reports error bars, compute and resource information, and citations and licenses for existing assets.It also records that no new assets were included and that human-subject items were not applicable.
A Missing Proofs
The MCB operator contracts on behavior-policy support, yielding convergence and a fixed point bounded between behavior and batch-optimal values. Its induced policy matches or exceeds behavior performance while avoiding severe OOD overestimation under a behavior-policy fit assumption.
- The MCB operator is a γ-contraction on the behavior-policy support, so repeated application converges from any initial Q function to a unique fixed point.
- The fixed point satisfies Qµ ≤ QMCB ≤ Qµ∗ on behavior-policy support, placing it between behavior and batch-optimal Q functions.
- MCB assigns pseudo target values to OOD actions using the maximum supported-action Q value minus a small positive δ.
- The induced MCB policy executes only actions within the behavior-policy support and therefore achieves return at least J(µ).
- MCB can improve policy performance with a tighter lower bound than explicit policy constraints or value-penalization methods such as CQL.
- If the empirical behavior policy is sufficiently close to the true behavior policy, the OOD extrapolation error becomes arbitrarily small and avoids severe overestimation.
B Deterministic MCQ
The deterministic MCQ variant combines the practical MCB operator with TD3, using a CVAE to generate actions and auxiliary critic targets for OOD actions. It is evaluated on D4RL MuJoCo benchmarks with normalized scores over repeated evaluations.
- Deterministic MCQ incorporates the practical MCB operator into TD3, demonstrating that the operator is compatible with off-policy online RL algorithms.
- The method trains a CVAE generative model alongside twin critics and an actor to support action generation and Q-learning.
- For each offline minibatch, deterministic MCQ trains the CVAE, computes in-distribution and OOD targets, updates critics, and applies policy-gradient and target-network updates.
- The experiments use D4RL MuJoCo-v2 tasks and report normalized average scores over the final 10 evaluations across four random seeds.
C.1 D4RL Benchmarks
The evaluation covers D4RL MuJoCo locomotion environments, dataset qualities, baselines, and value-estimation analyses. MCQ performs especially strongly on non-expert data, remains competitive on expert data, and shows stable value estimates under suitable settings.
- D4RL Benchmarks: D4RL evaluation spans halfcheetah, hopper, and walker2d with random, medium, medium-replay, medium-expert, and expert datasets.
- D4RL Benchmarks: D4RL normalized score maps random-policy performance to 0 and expert-policy performance to 100 using environment-specific reference returns.
- Baseline Comparisons: MCQ is compared with BC, SAC, CQL, UWAC, TD3+BC, IQL, BCQ, BEAR, MOPO, Decision Transformer, CDC, and PBRL.
- Baseline Comparisons: MCQ outperforms baseline methods on most non-expert datasets, often by a large margin, and is competitive on expert datasets.
- Value Estimation: Smaller λ causes Q-value collapse and performance degradation, whereas MCQ is comparatively insensitive to N across a wide range.
- Value Estimation: Across datasets, MCQ provides stable Q estimation without observed overestimation at λ = 0.95, supporting the claimed conservative behavior.
F Computation Cost and Compute Infrastructure
The paper reports MCQ’s training cost and introduces standard VAE/CVAE background used by its generative component. MCQ generally trains faster than CQL in the reported MuJoCo comparison.
- Computation Cost: MCQ generally requires 8 to 12 hours to train on MuJoCo tasks, compared with approximately 12 to 15 hours for CQL.
- Computation Cost: On halfcheetah-medium-v2, the paper compares MCQ and CQL by runtime per epoch and GPU memory on a single RTX3090 GPU.
- Compute Infrastructure: The experiments use documented compute infrastructure for running MCQ and baseline algorithms.
- Generative Models: A VAE models data through a latent variable, encoder, and decoder, optimizing a variational lower bound involving reconstruction and KL-divergence terms.
- Generative Models: A CVAE models conditional data distributions pθ(X|Y) and samples by drawing z from p(z|Y) before decoding.
H Experimental Results on Other Datasets
MCQ is evaluated on maze2d and Adroit datasets from the D4RL benchmarks against several established offline RL baselines. The comparison uses normalized scores and experiments across multiple random seeds.
- MCQ is evaluated on 4 maze2d "-v1" and 8 Adroit "-v0" datasets from D4RL.
- The comparison includes BC, BEAR, CQL, BCQ, TD3+BC, and IQL.
- All methods are run over 4 different random seeds.
- Table 9 reports normalized scores where 0 corresponds to a random policy and 100 corresponds to an expert policy.
- Table 10 lists the detailed hyperparameters used for MCQ on the maze2d and Adroit datasets.