Source-linked AI summary
Online Decision Transformer
Qinqing Zheng, Amy Zhang, Aditya Grover
TL;DR
Offline RL sequence models are limited when pretrained policies must be finetuned through exploration-driven online interactions. ODT unifies offline pretraining and online finetuning with stochastic sequence modeling and trajectory-level entropy regularization, achieving competitive D4RL performance and stronger finetuning gains. Its convergence analysis is empirical and assumes convergence of the training algorithm.
Problem
Offline RL policies are limited by passive dataset quality, while extending supervised sequence modeling to exploration-driven online finetuning remains an open question.
Method
ODT combines Decision Transformer pretraining with stochastic autoregressive policies, sequence-level entropy regularization, and online replay-based finetuning.
Results
ODT achieves competitive absolute performance on D4RL and larger relative improvements during finetuning than other baselines.
Takeaways & Limitations
ODT provides a unified framework for sample-efficient policy optimization across offline pretraining and online finetuning.
Takeaways & Limitations
The convergence guarantee of ODT’s training algorithm remains an open question, and experiments find ODT preferred in some but not all environments.
Abstract
from arXiv · showhide
Recent work has shown that offline reinforcement learning (RL) can be formulated as a sequence modeling problem (Chen et al., 2021; Janner et al., 2021) and solved via approaches similar to large-scale language modeling. However, any practical instantiation of RL also involves an online component, where policies pretrained on passive offline datasets are finetuned via taskspecific interactions with the environment. We propose Online Decision Transformers (ODT), an RL algorithm based on sequence modeling that blends offline pretraining with online finetuning in a unified framework. Our framework uses sequence-level entropy regularizers in conjunction with autoregressive modeling objectives for sample-efficient exploration and finetuning. Empirically, we show that ODT is competitive with the state-of-the-art in absolute performance on the D4RL benchmark but shows much more significant gains during the finetuning procedure.
1 Introduction
Offline RL sequence modeling turns trajectory learning into supervised autoregressive prediction, but offline policies remain limited by dataset quality and online finetuning requires exploration-aware objectives. ODT unifies offline pretraining and online finetuning with stochastic policies, entropy regularization, and a compatible replay strategy.
- Offline RL trains autoregressive trajectory models by maximizing the likelihood of trajectories in a fixed dataset.
- Offline policies are limited by the quality of passive data, while online finetuning introduces exploration and a changing data distribution.
- ODT blends offline pretraining with online finetuning for sample-efficient policy optimization using the Decision Transformer architecture.
- ODT uses stochastic policies, trajectory-level entropy constraints, and a replay buffer designed for its architecture and training protocol.
- On D4RL, ODT reports competitive absolute performance and larger relative improvements from finetuning than other baselines.
2 Related Work
Related work frames offline RL as context-conditioned sequence modeling and explores methods for combining offline data with online finetuning. ODT differs by integrating exploration and positive sampling into an end-to-end sequence-modeling framework.
- Transformers for RL: Transformer-based offline RL predicts action sequences conditioned on task specifications instead of explicitly learning Q-functions or policy gradients.
- Transformers for RL: Traditional offline RL methods add conservative components to off-policy algorithms, often requiring extensive modifications and hyperparameter retuning.
- Offline RL with Online Finetuning: Naive reuse of offline or off-policy RL methods during online finetuning often fails to help and can hinder performance.
- Offline RL with Online Finetuning: Offline finetuning failures are attributed to off-policy bootstrapping error accumulation or excess conservatism carried over from offline training.
- Offline RL with Online Finetuning: Prior approaches use balanced replay, Q-function ensembles, positive sampling, or explicit online exploration to address offline-online training challenges.
3 Preliminaries
The paper models environments as MDPs and represents trajectories through states, actions, and return-to-go tokens. Decision Transformer predicts actions autoregressively from a fixed context of recent timesteps, with uniform subsequence sampling during training.
- Setup and Notation: The environment is modeled as an MDP with state and action spaces, transition dynamics, rewards, and discount factor.
- Setup and Notation: A trajectory’s return-to-go at timestep t is the sum of future rewards, represented alongside its state and action sequences.
- Decision Transformer: Decision Transformer interleaves RTGs, states, and actions as the input sequence for trajectory modeling.
- Decision Transformer: At each timestep, DT uses the latest K timesteps to generate an action through an order-K autoregressive GPT model.
- Decision Transformer: Training predicts action tokens under a standard ℓ2 loss using length-K subsequences from the offline dataset or replay buffer.
- Decision Transformer: During evaluation, the desired initial RTG is updated by subtracting observed rewards as the generated trajectory unfolds.
4 Online Decision Transformer
ODT modifies Decision Transformer for online finetuning by using stochastic policies, entropy-constrained sequence modeling, hindsight relabeling, and trajectory replay. Its training dynamics and empirical behavior are analyzed under an explicit convergence assumption.
- Motivation: Standard Decision Transformer learning is insufficient for online learning because offline policies can be suboptimal and collapse during finetuning.
- Probabilistic Objective: ODT replaces deterministic action prediction with a stochastic policy trained through a probabilistic likelihood objective.
- Probabilistic Objective: The stochastic formulation subsumes DT’s deterministic policies as a special case of diagonal, equal-variance Gaussian policies.
- Max-Entropy Sequence Modeling: ODT quantifies exploration with policy entropy and imposes a lower-bound constraint whose dual form regularizes the supervised NLL objective.
- Max-Entropy Sequence Modeling: During offline training, the cross-entropy term controls distribution mismatch; during online training, it becomes entropy and encourages exploration.
- Max-Entropy Sequence Modeling: Unlike SAC’s transition-level constraint, ODT averages entropy across K consecutive timesteps, producing a larger feasible policy set when K > 1.
- Optimization: ODT uses one-step alternating gradient updates and one-sample reparameterized Monte Carlo estimates for practical optimization.
5 Experiments
The experiments evaluate ODT on D4RL locomotion and AntMaze tasks, comparing offline and online performance and testing its design components. ODT shows larger finetuning gains than IQL, stable training relative to deterministic policies, and benefits from specific return-conditioning choices.
- Benchmark Comparisons: The evaluation covers dense-reward Gym tasks and sparse-reward AntMaze tasks using D4RL offline datasets, with comparisons against DT, IQL, SAC, and finetuning baselines.Experiments use Hopper, Walker, HalfCheetah, Ant, and AntMaze datasets, including medium, medium-replay, umaze, and umaze-diverse settings.
- Benchmark Comparisons: IQL outperforms ODT and DT on most tasks during offline pretraining, but ODT obtains substantially larger improvements after 200k online samples.The average relative finetuning improvement for ODT is approximately 9x that of IQL on Gym tasks and datasets.
- Benchmark Comparisons: ODT matches IQL’s absolute performance after finetuning on challenging Ant environments, while IQL can struggle to improve or degrade its initial performance.The degradation example is reported for AntMaze with the umaze-diverse dataset.
- Benchmark Comparisons: SAC performs substantially worse than ODT on all Gym tasks except HalfCheetah under a 200k-interaction budget and fails to learn non-trivial AntMaze policies.This comparison uses SAC as a representative purely online baseline.
- Ablation Study: Stochastic ODT remains stable and consistently performs better than the deterministic variant during Hopper finetuning, whose performance declines and fluctuates as online samples accumulate.The comparison averages five training instances across small- and large-capacity architectures; the deterministic variant begins decreasing after 600k online samples in the small-capacity setting.
- Ablation Study: For return-to-go conditioning, ODT uses a fixed online RTG twice the expert performance; this strategy outperforms curriculum RTGs, while excessively large values increase variance.The fixed, scaled strategy is reported as outperforming the curriculum heuristic, and returns are stable at 1–2x expert performance before decreasing slightly with larger values.
6 Discussion
ODT extends pretrained Decision Transformers to online finetuning, broadening their use to settings that combine offline data with online interactions. The authors identify open questions about purely online sequence modeling and the limits of supervised-learning approaches for RL.
- ODT finetunes a pretrained Decision Transformer online, extending sequence-modeling RL to mixed offline and online interaction settings.
- The authors leave purely online reinforcement learning with supervised learning approaches as an open direction for future work.
- The discussion notes that ODT is preferred in some environments but not all.
- Sequence-modeling approaches may create delusions by treating an agent’s own actions as task evidence, motivating causal-intervention treatments.
A Additional Ablation Study
The ablation compares selecting the highest-return offline trajectories with random replay-buffer initialization. Using the top trajectories slightly improves performance.
- The replay buffer Treplay is initialized with the top N highest-return trajectories from the offline dataset.The alternative is random selection of N trajectories.
- The top N initialization strategy slightly outperforms random initialization across the compared environments.
- The comparison uses a replay buffer size of 1000 and averages results over 10 training instances with different seeds.
B Additional Design Choices for ODT
The additional studies examine context length, positional embeddings, architecture, optimization, and pretraining choices that affect ODT’s long-horizon policy modeling and finetuning performance.
- Evaluation Context Length: Evaluation context length controls how many past states and RTGs influence the present action; length 1 makes ODT policies Markovian.
- Evaluation Context Length: For online finetuning, context length 5 generally performs well, while Ant prefers context length 1; offline-training preferences are mixed.
- Positional Embedding: For Walker online finetuning, training without positional embeddings consistently outperforms training with them across tested learning rates and weight decays.
- ODT uses a four-layer transformer with four attention heads and embedding dimension 512, with larger capacity motivated by stochastic online adaptation.
- Policy parameters use LAMB, while the transformed dual parameter log λ uses Adam with fixed learning rate 0.0001.
- Embedding dimension 1024 performs better than 512 for antmaze-umaze-diverse finetuning but worse during offline training, and extensive pretraining can hurt exploration.
D Comparison of the Exploration RTG gonline
The study tests exploration rollouts initialized with target returns g˚ and 2g˚. After collecting 200k online samples, the resulting returns are comparable.
- ODT exploration rollouts use initial RTG tokens set to g˚ or 2g˚.Here, g˚ denotes the expert return.
- After 200k online samples, returns obtained with the two exploration RTGs are comparable across the evaluated environments.Results are averaged over 10 training instances with different seeds.
E Training Stability Comparison
ODT improves smoothly during online finetuning, while IQL remains near its pretraining performance and SAC fluctuates. The appendix also describes a length-proportional trajectory sampler followed by uniform subsequence sampling.
- Training stability: Both ODT and IQL remain stable during online finetuning, while SAC fluctuates.
- Training stability: ODT’s return improves smoothly during online finetuning, whereas IQL remains stable and SAC exhibits high variability.The comparison uses Hopper with the medium dataset and averages results over five training instances with different seeds.
- Sampling strategy: The sampling strategy first selects a replay-buffer trajectory with probability proportional to its length, then samples subsequences uniformly.
- Sampling strategy: The sampling strategy is compared with importance sampling, which assigns probability proportional to trajectory return.For many non-negative dense-reward D4RL datasets, trajectory return is highly correlated with trajectory length.
- Sampling strategy: For each sampled trajectory, the procedure computes return-to-go values, samples a timestep uniformly, and outputs a length-K subsequence of actions, states, and return-to-go values.
G Training Dynamics: the convergence of λ
ODT alternates updates to the policy parameters and entropy-constraint multiplier in a Lagrangian objective. Under convergence, the multiplier can converge to a common value between 0 and 1 across different initializations when the constraint is tight.
- Optimization dynamics: ODT alternates updates to θ and λ while optimizing a Lagrangian that combines the objective with an entropy-constraint term.
- Optimization dynamics: If training converges and λ settles in [0, 1), the overall loss converges to the desired constrained formulation.
- Constraint behavior: λ can be zero when the entropy constraint is inactive or positive when the constraint is tight.
- Constraint behavior: Across five Walker medium-dataset runs initialized with λ values of 1, 2, 5, 10, and 100, λ converges to the same value between 0 and 1.The associated entropy estimate and λ trajectories are shown during both pretraining and finetuning.
H Negative results
The authors tested several modeling and optimization heuristics but found them of limited utility in preliminary experiments, so they were excluded from the final design.
- Negative results: Several positional-embedding, dropout, context-length, learning-rate, and adaptive-update heuristics showed limited utility in preliminary experiments.The tested heuristics were therefore not included in the final design.