PaperlayerView exact revision on arXiv

Source-linked AI summary

DEAS: DEtached value learning with Action Sequence for Scalable Offline RL

Changyeon Kim, Haeone Lee, Younggyo Seo, Kimin Lee, Yuke Zhu

arXiv:2510.07730v1cs.LGcs.AIcs.RO

TL;DR

Offline RL methods struggle with complex, long-horizon decisions, and existing horizon-reduction approaches can depend on explicit goals. DEAS learns values over action sequences with detached value learning, outperforming baselines on OGBench and improving VLA performance in simulation and real-world manipulation.

  • Problem

    Offline RL methods still struggle with complex long-horizon tasks, while prior horizon-reduction approaches rely on explicit goals that may be unavailable in practice.

    Source: §1 · p. 1

  • Method

    DEAS treats action sequences as options for SMDP value learning and detaches critic training from the policy to steer estimates toward high-return dataset actions.

    Source: §1 · p. 2, §4.1 · p. 4, §4.2 · p. 5

  • Results

    DEAS consistently outperforms baselines on challenging OGBench tasks and improves large-scale VLA performance on RoboCasa Kitchen and real-world manipulation tasks.

    Source: §1 · pp. 2–3, §6 · p. 9

  • Takeaways & Limitations

    The results support DEAS’s practical potential for scaling offline RL to real-world scenarios.

    Source: §6 · p. 9

    DEAS uses fixed action-sequence lengths despite task-dependent optimal lengths.

    Source: C LIMITATIONS AND FUTURE WORK · p. 19

Abstract

from arXiv · show

Offline reinforcement learning (RL) presents an attractive paradigm for training intelligent agents without expensive online interactions. However, current approaches still struggle with complex, long-horizon sequential decision making. In this work, we introduce DEtached value learning with Action Sequence (DEAS), a simple yet effective offline RL framework that leverages action sequences for value learning. These temporally extended actions provide richer information than single-step actions and can be interpreted through the options framework via semi-Markov decision process Q-learning, enabling reduction of the effective planning horizon by considering longer sequences at once. However, directly adopting such sequences in actor-critic algorithms introduces excessive value overestimation, which we address through detached value learning that steers value estimates toward in-distribution actions that achieve high return in the offline dataset. We demonstrate that DEAS consistently outperforms baselines on complex, long-horizon tasks from OGBench and can be applied to enhance the performance of large-scale Vision-Language-Action models that predict action sequences, significantly boosting performance in both RoboCasa Kitchen simulation tasks and real-world manipulation tasks.

1 INTRODUCTION

DEAS addresses the difficulty of long-horizon offline RL by learning values over action sequences with a detached critic. It outperforms baselines on OGBench and improves large-scale VLA performance on simulation and real-world manipulation tasks.

1 INTRODUCTION

  • Prior horizon-reduction methods rely on explicit expert goals, which may be unavailable, while high-n TD updates can increase bias and bootstrap error without them.Existing offline RL methods also primarily target short-horizon tasks with dense rewards and struggle with complex long-horizon scenarios.Source: §1 · p. 1
  • Action sequences can cause severe value overestimation in actor-critic methods, while value-only CQN-AS incurs discretization errors and cannot use expressive policy classes.Offline distribution shift can exacerbate extrapolation errors.Source: §1 · pp. 1–2
  • DEAS treats action sequences as temporally extended options for horizon reduction and detaches critic training from the actor to favor high-return dataset actions.It also incorporates distributional RL to mitigate instability from accumulated multi-step-return bias.Source: §1 · p. 2
  • DEAS significantly outperforms baselines across 30 diverse OGBench scenarios and improves VLA results on RoboCasa Kitchen and real-world tasks over expert-only policies.The paper also reports consistent improvements on challenging OGBench tasks.Source: §1 · pp. 2–3

2 RELATED WORK

Prior offline RL work addresses distribution shift through several forms of regularization and in-sample learning. Work on action sequences spans imitation and RL, but expanded action spaces create value-estimation challenges that DEAS targets.

Offline reinforcement learning

  • Offline RL methods address distribution shift through weighted regression, conservative or behavioral regularization, and in-sample distribution maximization.DEAS builds on IQL and removes the actor-output-based critic update, while allowing different policy-extraction methods.Source: §2 · p. 3

BC/RL with action sequence

  • Behavior cloning uses action sequences to capture temporal dependencies that single-step actions miss, and RL research has also incorporated sequences into learning.Q-Chunking places sequences in actor-critic frameworks without policy-class constraints.Source: §2 · p. 3
  • Expanded action spaces increase overestimation risk under limited offline data, while CQN-AS avoids an actor but incurs discretization errors that limit complex-task performance and expressive policies.DEAS combines sequence-based horizon reduction with detached value learning.Source: §2 · p. 3

3 PRELIMINARIES

The preliminaries set offline RL in an MDP with a fixed trajectory dataset, introduce options as temporal abstractions, and summarize IQL’s in-sample value learning.

Problem formulation

  • Offline RL is framed as learning a reward-maximizing policy from a fixed dataset of trajectories without further environment interaction.The formulation specifies an MDP with state and action spaces, transitions, rewards, an initial-state distribution, and a discount factor.Source: §3 · p. 3

Options framework

  • A Markovian option is defined by an initiation set, an intra-option policy, and a termination function, and following options induces an SMDP.The options framework provides flexible temporal abstractions for decision making.Source: §3 · p. 3

Implicit Q Learning (IQL)

  • IQL uses in-sample expectile regression to approximate values within dataset-supported action distributions; with τ > 0.5, it penalizes overestimated out-of-distribution values.This lets V and Q be approximated in the region of in-distribution actions.Source: §3 · pp. 3–4

4 METHOD

DEAS treats temporally extended action sequences as options for goal-free offline RL, then stabilizes sequence-value learning with detached value learning and dual discounts.

4 METHOD

  • DEAS uses a critic over H-step action-sequence options and a flexible policy that outputs H-step sequences.The critic estimates expected returns for options under the data-collection policy.Source: §4 · p. 4, §4.1 · p. 4

4.1 OPTIONS FRAMEWORK FOR ACTION SEQUENCE RL

  • The option-based SMDP update aggregates rewards across H steps and propagates values over extended transitions, reducing the effective planning horizon like n-step TD.The update extends standard Q-learning to temporally extended transitions.Source: §4.1 · p. 4

4.2 DEAS: DETACHED VALUE LEARNING WITH ACTION SEQUENCE

  • Detached value learning steers the critic toward high-return dataset actions, reducing exploitation of critic errors and supporting stable learning for longer sequences.Action sequences expand the action space, making accurate Q-value estimation harder and increasing the risk of overestimation.Source: §4.2 · pp. 4–5
  • DEAS uses categorical critic/value distributions and IQL-weighted classification, with truncated-normal targets centered on Bellman targets, to manage variance from long sequences.The distributions use a fixed support from vmin to vmax, discretized into m bins.Source: §4.2 · p. 5
  • Experiments find lowering intra-option γ1 and raising inter-option γ2 stabilizes training, especially as action sequences lengthen.The two discounts separately weight rewards within and across action sequences.Source: §4.2 · pp. 5–6
  • Because value learning need not query the policy, critic training is decoupled from policy updates, allowing weighted behavior cloning, DPG, best-of-N, and flow-matching approaches.The framework supports separate policy extraction after value-function training.Source: §4 · p. 4, §4.2 · pp. 5–6

5 EXPERIMENTS

The experiments evaluate DEAS on OGBench tasks, test its use with VLAs in RoboCasa and real-world manipulation, and examine key design choices through ablations.

5.1 OGBENCH EXPERIMENTS

  • DEAS consistently achieves the best performance across all six OGBench task categories and dataset sizes, particularly on puzzle and cube-quadruple tasks.It matches or outperforms QC-FQL across tasks.Source: §5.1 · pp. 6–7, Table 1 · p. 7
  • Across three representative OGBench tasks, DEAS achieves the highest success rates at every evaluated dataset size.Figure 3 reports means and stratified bootstrap confidence intervals over four runs.Source: Figure 3 · p. 7, §5.1 · p. 7

5.2 VLA EXPERIMENTS

  • The VLA experiments test whether offline RL can improve task performance using suboptimal demonstrations alongside limited expert data.The study includes four hard RoboCasa Kitchen tasks and real-world experiments with a Franka Emika Research 3 arm.Source: §5 · p. 6, §5.2 · p. 7

5.2.1 ROBOCASA KITCHEN EXPERIMENTS

  • In RoboCasa Kitchen, DEAS achieves the highest success rates on three of four tasks and improves over the base model on the fourth.The experiments use GR00T N1.5 and compare offline IL/RL methods on selected tasks.Source: §5.2.1 · pp. 7–8, Table 2 · p. 8

5.2.2 REAL-WORLD EXPERIMENTS

  • In real-world pick-and-place, DEAS achieves the highest success rates across all three tasks and remains stable with limited data.The tasks use peach, milka, and hichew; the reported gains are especially notable for the deformable milka object.Source: §5.2.2 · p. 8, Table 3 · p. 9

5.3 ABLATION STUDIES AND ANALYSES

  • Single-step and two-step sequences fail to achieve meaningful performance, while sequences longer than eight require proportionally larger actor networks.The results indicate a tradeoff between sequence length and computational efficiency.Source: §5.3 · p. 9
  • Value models need sufficient capacity, but excessive unregularized capacity destabilizes value estimates; the sequence discount γ1 is set to 0.9.γ1 controls the temporal horizon for value estimation within action sequences.Source: §5.3 · p. 9
  • Combining detached value learning with distributional estimation improves results, whereas using distributional RL or standard regression alone shows limited performance.The ablation suggests both components are important for stable training with action sequences.Source: §5.3 · p. 9, Table 4 · p. 9

6 CONCLUSION

DEAS uses action sequences as temporally extended actions for scalable offline value learning, with detached value learning addressing value overestimation. Experiments show consistent improvements on OGBench and successful application to large-scale VLAs.

6 CONCLUSION

  • DEAS leverages action sequences for scalable learning in complex offline RL tasks.The method treats temporally extended actions through the options framework and SMDP Q-learning for horizon reduction.Source: §6 · p. 9
  • Detached value learning addresses value overestimation while DEAS uses action sequences for horizon reduction.The approach models temporally extended actions through the options framework.Source: §6 · p. 9
  • DEAS consistently improves over baselines on OGBench and is successfully applied to large-scale VLAs.The conclusion describes this as showing practical potential for scaling offline RL to real-world scenarios.Source: §6 · p. 9

A.1 OGBENCH EXPERIMENTS

The OGBench experiments use six state-based, goal-free UR5 manipulation environments, while the VLA experiments fine-tune GR00T N1.5 using specified training setups. The appendix also documents implementation, baseline, and hardware choices.

A.1 OGBENCH EXPERIMENTS

  • Six goal-free UR5 environments each contain five subtasks, with state observations, Cartesian-arm actions, and a semi-sparse reward based on unfinished subtasks.Episodes are capped at 1,000 steps.Source: §A.1 · p. 15
  • The OGBench implementation builds on FQL, uses a smaller value network than critic network, and applies BRO regularization for cube experiments.Distributional-RL support bounds are selected either from dataset-return quantiles or theoretical reward and horizon bounds.Source: §A.1 · p. 15
  • OGBench training uses 1M or 100M play datasets, with 1M or 2.5M gradient steps and success rates averaged across the final three evaluation epochs.For the 10M dataset, the first ten 1M-transition files are selected; policy-extraction BC coefficients are swept per task and baseline, with a stated exception for cube −double.Source: §A.1 · p. 15
  • Baseline implementations use the cited FQL and Q-Chunking codebases and the original CQN-AS implementation, with Q-Chunking's discount factor matched to γ2.The CQN-AS setup also modifies its BC loss for suboptimal data; rewards are normalized to [−1, 0], with vmin = −200 and vmax = 0.Source: §A.1 · pp. 15–16

A.2 VLA EXPERIMENTS

  • VLA experiments use A100 GPUs; GR00T N1.5 is fine-tuned without future tokens using batch size 32 for 30K RoboCasa or 10K real-robot steps.Fine-tuning uses AdamW with learning rate 1 × 10−4 and cosine annealing; the reported runtimes are about four hours for fine-tuning and ten hours for DEAS and baselines.Source: §A.2 · p. 16

A.2.1 ROBOCASA KITCHEN EXPERIMENTS

The RoboCasa Kitchen evaluation focuses on four challenging long-horizon tasks and uses visual, proprioceptive, and language inputs. The setup also specifies policy candidates, demonstration training, and evaluation conditions.

A.2.1 ROBOCASA KITCHEN EXPERIMENTS

  • The evaluation selects four challenging RoboCasa Kitchen tasks from 24 atomic tasks, using sparse completion rewards.The tasks involve a mobile manipulator and delicate manipulation; observations include three camera views, proprioceptive states, and language instructions.Source: §A.2.1 · p. 16
  • The value function receives robot proprioception plus VLM-encoded camera views and instructions, and action selection ranks ten policy candidates by Q-value.Final actions use either greedy highest-Q selection or temperature-controlled softmax sampling, with the best result reported per task.Source: §A.2.1 · p. 16
  • Training uses 100 randomly sampled expert demonstrations and evaluates each task over 150 rollouts on unseen object instances.Models train for 30K steps with batch size 64 and Adam at learning rate 3 × 10−4.Source: §A.2.1 · p. 16

A.2.2 REAL ROBOT EXPERIMENTS

The real-robot experiments evaluate pick-and-place across three objects, with demonstrations from five initialization positions and rewards based on task stages. The appendix lists OGBench hyperparameter tables and relates DEAS to hierarchical RL and VLA methods.

A.2.2 REAL ROBOT EXPERIMENTS

  • Real-robot evaluation uses a Franka Research 3 arm, dual-camera perception, and teleoperated demonstrations recorded at 15 Hz.The pick-and-place task covers three objects and five demonstration initialization positions, with reward defined as the negative number of uncompleted stages across four stages.Source: §A.2.2 · p. 17
  • Real-robot action selection samples 50 policy candidates and follows the RoboCasa procedure for choosing the final action.The experiment implementation otherwise follows the RoboCasa Kitchen setup.Source: §A.2.2 · p. 17

A.3 HYPERPARAMETERS

  • Tables 5 and 6 list OGBench and baseline hyperparameters, while Table 7 lists policy-extraction hyperparameters and notes Q-normalization use.Q-normalization is excepted for cube-double tasks.Source: §A.3 · p. 18, Table 5 · p. 18, Table 6 · p. 18

B EXTENDED RELATED WORK

  • DEAS treats action sequences as primitive options to reduce the planning horizon without explicit goals or manual subtask specification.The related-work discussion contrasts DEAS's detached sequence value learning with VLA approaches using single-step values and with CO-RFT's demonstrations and architecture choices.Source: B EXTENDED RELATED WORK · p. 19

C LIMITATIONS AND FUTURE WORK

DEAS has limitations in adapting action-sequence lengths, scaling unified value functions, and selecting distributional RL parameters; the authors also describe their limited use of LLMs in manuscript preparation.

C LIMITATIONS AND FUTURE WORK

  • Fixed action-sequence lengths do not adapt to task complexity, although optimal lengths vary significantly across tasks.The authors suggest investigating adaptive mechanisms, potentially using hierarchical policies.Source: C LIMITATIONS AND FUTURE WORK · p. 19
  • Scaling unified value functions beyond the current 3–4 simultaneously trained tasks remains a challenge for applications involving hundreds or thousands of tasks.The authors identify scalable architectures and training procedures that maintain sample efficiency and avoid catastrophic forgetting as future work.Source: C LIMITATIONS AND FUTURE WORK · p. 19
  • Fixed distributional RL support ranges and discretization parameters can significantly affect performance, limiting robustness across domains and reward scales.The authors call for methods that adapt to reward distributions or provide principled parameter-setting approaches.Source: C LIMITATIONS AND FUTURE WORK · p. 19

D USE OF LARGE LANGUAGE MODELS

E FULL EXPERIMENTAL RESULTS

The full OGBench offline RL results are presented in Table 8, with success rates and confidence intervals reported across 30 tasks.

E FULL EXPERIMENTAL RESULTS

Related papers

Loading 2510.07730v1…