PaperlayerView exact revision on arXiv

Source-linked AI summary

D$^2$-VLA: Dual-Memory Dual-Frequency Vision-Language-Action Model For Long Dynamic Manipulation

Zijian Ye, Chengqi Wei, Wei Huang, Anlin Zheng, Chunyu Zou, Liangyu Wu, Zikang Zhao, Zhenjie Peng, Yushuo Yang, Shuman Zhao, Zhongrui Wang, Xiaojuan Qi

arXiv:2609.34792v2cs.CV

TL;DR

Long-horizon manipulation asks robots to retain cues no longer in view while responding to moving objects, but VLA policies commonly rely on the latest observation and costly VLM refreshes. D2-VLA combines consumer-specific historical KV reads with slow VLM updates and fast visual refinement, and achieves 29.3% success on DOMINO and 60.0% on DOMINO-Long.

  • Problem

    Long-horizon manipulation requires remembering task cues absent from the current view while responding to moving objects, yet longer histories raise attention costs and frequent VLM refreshes are expensive.

    Source: §1 · p. 1

  • Method

    D2-VLA reuses causal KV history, selects distinct historical reads for the VLM and action expert, and refines the latest history-conditioned KV block with fresh visual features between VLM updates.

    Source: §1 · p. 2

  • Results

    D2-VLA achieves 29.3% success on DOMINO and 60.0% on DOMINO-Long, with consistent improvements across simulation and real-robot evaluations.

    Source: §6 · p. 9

  • Takeaways & Limitations

    D2-VLA's results demonstrate the benefits of combining historical context with timely visual feedback for long-horizon dynamic manipulation.

    Source: §6 · p. 9

Abstract

from arXiv · show

Long-horizon manipulation requires robots to remember cues that are no longer in view while responding to moving objects. Yet vision-language-action (VLA) policies often rely on the latest observation, and refreshing their visual context typically requires another costly vision-language model (VLM) pass. We present D$^2$-VLA, which combines dual memory and dual-frequency control at the KV-cache interface of a pretrained VLA. D$^2$-VLA uses block-wise causal KV caching to encode observations incrementally and, guided by distinct temporal attention patterns, constructs separate historical KV read views for the VLM and action expert. Between periodic VLM updates, a gated adapter incorporates fresh visual features into the latest history-conditioned KV block, while a short fast-memory queue supports action replanning. We introduce DOMINO-Long, a ten-task benchmark requiring robots to use earlier visual cues when manipulating moving objects. D$^2$-VLA achieves complete-task success rates of 29.3\% on DOMINO, compared with 9.6\% for $\pi_{0.5}$ and 17.2\% for PUMA, and 60.0\% on DOMINO-Long, compared with 35.4\% and 20.6\%, respectively. It improves success rates on eight real-robot tasks and reaches 97.5\% on LIBERO-Long and 74.3\% on RoboTwin 2.0.

1 INTRODUCTION

Long-horizon manipulation requires remembering cues outside the current view while responding to moving objects, but VLA policies often couple latest-frame context with costly VLM refreshes. D2-VLA separates history access and update rates for the VLM and action expert, and is evaluated on dynamic-manipulation benchmarks including DOMINO-Long.

1 INTRODUCTION

  • Long-horizon memory and fast responsiveness are coupled challenges because task cues can leave view while manipulated objects continue moving.Longer VLM histories increase attention costs, whereas frequent VLM refreshes are expensive.Source: §1 · p. 1
  • Prior memory and real-time methods largely address the two temporal dimensions separately, and existing KV approaches do not tailor historical reads to the VLM and action expert.Fast controllers are typically conditioned on the latest slow representation.Source: §1 · pp. 1–2
  • The VLM relies on broader history for semantic reasoning, while the action expert attends mostly to recent observation tokens containing motion information.This temporal asymmetry motivates decoupling KV-cache selection and refresh rates.Source: §1 · p. 2
  • D2-VLA reuses historical KV states, selects separate history reads for the VLM and action expert, and refines the latest history-conditioned block with fresh visual features between VLM updates.A bounded fast-memory queue conditions the action expert on short-range dynamics.Source: §1 · p. 2
  • Across DOMINO, DOMINO-Long, static long-horizon benchmarks, and real-robot experiments, D2-VLA consistently improves over single-frame and memory-augmented baselines, achieving 29.3% complete-task success and a 40.6 manipulation score on DOMINO.It also outperforms π0.5 and MemoryVLA on all four reported dynamic real-robot tasks.Source: §1 · p. 2
  • DOMINO-Long tests retaining earlier spatial configurations, presentation orders, or color cues while manipulating constantly moving objects.The benchmark targets long-term task information and dynamic interaction together.Source: §1 · pp. 2–3

2 RELATED WORK

Related work addresses long-horizon memory and dynamic control through distinct families of methods. D2-VLA instead studies consumer-specific historical KV selection and lightweight refinement of a history-conditioned VLA.

Long-horizon manipulation and memory-augmented policies

  • Memory-oriented methods use sequential policies, multi-frame training, retrieved perceptual or cognitive memories, short-term video plus long-term text, or selected keyframes.D2-VLA instead tailors historical KV reads to the VLM and action expert while preserving complete stored blocks.Source: §2 · p. 3

Dynamic manipulation and real-time control

  • Real-time approaches overlap execution, stream inference, separate fast visual control from slow VLM context, or add high-frequency force feedback to reused VLM states.These works address adaptive update schedules and sensory conditioning in different ways.Source: §2 · p. 3
  • D2-VLA explores refreshing a history-conditioned VLA from current visual observations through a lightweight KV refinement interface.This is presented alongside prior approaches to dynamic control and efficient sensory conditioning.Source: §2 · p. 3

3 METHOD

D2-VLA adds temporal KV reuse to a single-frame VLA, giving the VLM and action expert separate historical reads and update rates. It combines periodic VLM refreshes with fast replanning and fresh visual adaptation between refreshes.

3.1 PRELIMINARIES

  • The policy receives multi-view images, an instruction, and proprioceptive state, while single-frame conditioning trades historical context against costly VLM refreshes.The state is supplied directly to the action expert rather than included in VLM observation blocks or cached history.Source: §3.1 · pp. 3–4

3.2 CAUSAL TEMPORAL MODELING

  • Block-wise causal attention enables bidirectional reads within each observation block, causal attention across blocks, and reuse of retained history without re-encoding.The current visual and instruction KV block is appended to the retained history.Source: §3.2 · pp. 4–5, Figure 2 · p. 4

3.3 DUAL-MEMORY: ASYMMETRIC HISTORY SELECTION

  • The VLM and action expert show different temporal preferences and concentrate attention on few tokens, motivating independent historical KV reads.The VLM reads selected history to encode the current observation; the expert combines its selected history with the complete current block. The KV-Cache prototype improves success over single-observation π0.5 on LIBERO-Long and RoboTwin 2.0, but increases latency and memory.Source: §3.3 · p. 5

3.4 DUAL-FREQUENCY: MULTI-RATE VISUAL CONDITIONING

  • D2-VLA refreshes the VLM every V environment steps and replans every E steps, with a gated adapter refining the latest slow anchor from current visual features between refreshes.The adapter produces compact queries from current image features and temporal offset; its refinements preserve the complete slow anchor's KV layout, and intermediate replanning does not rerun the VLM or modify slow memory.Source: §3.4 · pp. 5–6

4 DOMINO-LONG DATASET

DOMINO-Long extends DOMINO to test long-term retention of task-relevant information during dynamic interaction. Its tasks separate early cue acquisition from later action and include moving objects.

4 DOMINO-LONG DATASET

  • The benchmark addresses DOMINO's relatively weak demands on long-term memory and extended decision-making by testing retention across long temporal intervals under dynamic interaction.Tasks require early-observed cues, such as object locations, presentation order, or color instructions, for later decisions.Source: §4 · p. 6
  • DOMINO-Long contains 10 multi-stage tasks, 500 demonstrations, and 227,208 observation–action records, with demonstrations averaging 454.4 interaction steps.Each demonstration spans 201–911 interaction steps.Source: §4 · p. 6
  • Tasks include restoring earlier object configurations, following previously observed sequences, and selecting moving objects using cues no longer visible.Several tasks require combining retained history with current visual feedback.Source: §4 · p. 6

5 EXPERIMENTS

Experiments evaluate D2-VLA on dynamic, long-horizon, static, and real-world manipulation, testing whether it retains task context while incorporating fresh visual feedback. Results include gains on dynamic benchmarks and static tasks.

5.1 EXPERIMENTAL SETUP

  • Training uses episode-level samples with a slow anchor, historical observations, and three action branches; modules are trained end-to-end with flow-matching loss.The configuration retains π0.5 Base’s PaliGemma-3B VLM and shared flow-matching expert; inference uses adapter updates between VLM refreshes.Source: §5.1 · pp. 6–7
  • Experiments span dynamic simulation, real-world manipulation, and static simulation; DOMINO has 35 tasks, and DOMINO-Long adds ten tasks using earlier observations to guide later actions.Real-world evaluation uses eight Unitree G1D tasks; static tests include LIBERO-Long and 50 RoboTwin 2.0 tasks.Source: §5.1 · p. 7

5.2 EVALUATION METRICS

  • Complete-task SR requires satisfying all task requirements, including the full sequence, correct object identities, ordering, and final configuration.DOMINO’s MS also measures progress toward the target while penalizing unsafe or undesirable behaviors.Source: §5.2 · p. 7

5.3 EVALUATION ON DYNAMIC MANIPULATION BENCHMARKS

  • On DOMINO-Long, average complete-task SR was 60%, exceeding π0.5 (35.4%) and PUMA (20.6%) by 24.6 and 39.4 percentage points, respectively, in long-horizon dynamic manipulation.Source: §5.3 · p. 8
  • 29.3% SR and 40.6 MS on DOMINO; D2-VLA exceeds π0.5 by 19.7 SR points and 14.4 MS points, and PUMA by 12.1 and 5.6 points, respectively.Evaluation uses clean demonstrations and clean L1 with predictable low-order motion.Source: §5.3 · p. 7, Table 1 · p. 8

5.4 REAL-WORLD EVALUATION

  • On eight Unitree G1D tasks, D2-VLA improves over π0.5 and MemoryVLA on dynamic tasks by varying margins.The evaluation reports complete-task success; Figure 4 shows representative executions and comparisons.Source: Figure 4 · p. 8, §5.4 · p. 9

5.5 EVALUATION ON STANDARD STATIC BENCHMARKS

  • 97.5% SR on LIBERO-Long exceeds π0.5 (92.4%) and MemoryVLA (93.4%), while 74.3% SR on RoboTwin 2.0 exceeds π0.5 (57.0%).The high-rate pathway is disabled; full-KV training and online dual-memory selection are retained.Source: Table 2 · p. 9, §5.5 · p. 9

5.6 ABLATION STUDIES

  • On DOMINO, adding dual-memory raises SR from 9.6% to 16.1%, and adding dual-frequency control raises it further to 29.3%.The first stage uses separate VLM and expert history reads; the second adds fast memory between VLM refreshes.Source: §5.6 · p. 9

6 CONCLUSION

D2-VLA combines causal KV reuse, consumer-specific historical reads, and dual-frequency control; the paper reports gains on DOMINO, DOMINO-Long, and dynamic real-robot tasks. Its implementation uses incrementally encoded observation blocks, separate history selection for each consumer, and a fast pathway between VLM refreshes.

6 CONCLUSION

  • 29.3% SR on DOMINO and 60.0% on DOMINO-Long; D2-VLA also improves dynamic real-world manipulation and extends memory-based design to static benchmarks.The conclusion describes these outcomes as benefits of combining historical context with timely visual feedback.Source: §6 · p. 9

A.1 MODEL ARCHITECTURE AND OBSERVATION–ACTION INTERFACES

  • The VLM uses a PaliGemma-3B backbone with SigLIP and Gemma language components, while the action expert uses Gemma-300M; three camera views yield 768 visual tokens.The robot state is separately projected into a token for the action expert and is not stored in the visual-language cache.Source: §A.1 · p. 14

A.2 CAUSAL MODELING AND CACHE CONVENTIONS

  • Observation blocks use bidirectional attention internally and causal attention across time; retained historical blocks are reused rather than re-encoded at each slow update.The slow cache may retain blocks whose representations already include information propagated from earlier observations.Source: §A.2 · pp. 14–15

A.3 ONLINE CONSUMER-SPECIFIC HISTORICAL KV SELECTION

  • Historical KV selection uses independent online attention statistics for VLM prefill and action denoising, with configurable visual-token budgets for each historical block.Reported experiments use a historical visual-token retention ratio of 0.7 for active consumers.Source: §A.3 · pp. 15–16

A.4 FULL-BLOCK ADAPTER AND DUAL-RATE INFERENCE

  • The VLM refreshes every V environment steps while replanning occurs every E steps; between refreshes, refined blocks enter a fast FIFO that is cleared at the next slow refresh.The fast queue retains up to two complete blocks, and slow and fast caches are not combined into one expert condition.Source: §A.4 · pp. 17–18

A.5 TRAINING OBJECTIVES AND OPTIMIZATION

Training groups one slow branch with two subsequent fast branches and optimizes action prediction through flow matching. The experiments use full fine-tuning, ten-step Euler inference, and separate training budgets by evaluation setting.

A.5 TRAINING OBJECTIVES AND OPTIMIZATION

  • Each grouped sample supplies one slow branch and two chronological fast branches with H-action targets; missing history, unavailable branches, and episode-boundary targets are masked.The anchor observation is not inserted as a valid fast-memory entry, and slow history is encoded once per grouped sample.Source: §A.5 · pp. 18–19
  • Training constructs noisy action targets from normalized actions and Gaussian noise, then minimizes a masked per-component velocity error; the adapter receives no separate loss.The adapter learns through the action objective, while online KV selection is disabled during training.Source: §A.5 · p. 19
  • Full fine-tuning trains the VLM, action expert, adapter, and state/action projections, while high-rate SigLIP features are detached.The visual encoder is therefore updated only through the slow path.Source: §A.5 · p. 19

A.6 EXPERIMENT-SPECIFIC CONFIGURATIONS

  • Inference starts from Gaussian noise and uses 10 Euler steps; each call executes an H-step action chunk before replanning, while DOMINO-Long trains for 50,000 updates on all 500 demonstrations.Inference uses EMA model weights.Source: §A.5 · p. 19, §A.6 · p. 19

B SIMULATION EVALUATION PROTOCOLS

The appendix specifies protocols for four simulation benchmarks and reports DOMINO metrics and the memory demands of DOMINO-Long. DOMINO-Long contains ten tasks that combine earlier visual cues with manipulation of moving objects.

B SIMULATION EVALUATION PROTOCOLS

  • The appendix presents protocols for four simulation benchmarks and separates settings by evaluation context; training budgets are 75,000 updates for DOMINO and 50,000 each for DOMINO-Long and real-world tasks.The shared settings are listed for DOMINO, DOMINO-Long, and real-world tasks.Source: B SIMULATION EVALUATION PROTOCOLS · pp. 19–20, Table 5 · p. 20

B.1 DOMINO

  • DOMINO evaluation covers 35 tasks in the L1 + clean setting and reports complete-task success rate and Manipulation Score.Per-task results are separate from the aggregate comparison in the main text.Source: §B.1 · p. 20, Table 6 · p. 21

B.2 DOMINO-LONG

  • DOMINO-Long comprises 10 simulated tabletop tasks requiring retained object locations, identities, motion directions, presentation order, or color–region associations.Its task types include disappearing cues, objects that reappear in new arrangements, and scenes altered by earlier actions.Source: Figure 5 · p. 21, §B.2 · p. 22, Table 7 · p. 22

B.2.1 TASK DEFINITIONS

The ten-task suite tests whether robots can retain earlier cues, identities, locations, and presentation order while manipulating moving objects. The section also describes benchmark settings, demonstrations, and real-world long-horizon tasks.

B.2.1 TASK DEFINITIONS

  • Bottle Return, Can Grasp, Motion Memory, and Color Memory test retaining an object's original location, identity, prior motion, or a color-to-location association across changing scenes.The tasks require using earlier information to choose a moving object or its destination.Source: §B.2.1 · pp. 22–23
  • Stack Two, Two Color Cue, Can Placement, Tray Swap, Screen Cue, and Stack Three test remembering sequences, distinct cue roles, destinations, source locations, or object-selection cues.Stack Three extends ordered recall to three objects and placements.Source: §B.2.1 · pp. 22–23

B.3 RANDOMIZATION AND DEMONSTRATION COLLECTION

  • Randomization varies task-relevant factors such as identity, cue, side, order, motion direction, and lane, but does not rule out every alternative cue.The dataset has 50 demonstrations per task, 500 total, all used for training.Source: §B.3 · p. 24

B.4 LIBERO-LONG

  • LIBERO-Long evaluates all ten tasks with the high-rate pathway disabled, full-KV conditioning in training, and online historical selection at inference.Results are reported in Table 2.Source: §B.4 · p. 24

B.5 ROBOTWIN 2.0

  • RoboTwin 2.0 evaluates all 50 clean-setting tasks with the high-rate pathway disabled and online historical selection at inference.Complete-task success rates are reported per task in Table 8.Source: §B.5 · p. 24, Table 8 · p. 24

C REAL-WORLD SETUP, TASKS, AND EVALUATION

  • Eight real-world tasks comprise four long-horizon tasks and four long-horizon dynamic tasks, designed to assess ordered subgoals, object assignments, and use of task context with updated vision.Experiments use a Unitree G1D with stereo head images and left/right wrist views.Source: C REAL-WORLD SETUP, TASKS, AND… · pp. 24–25, §C.1 · p. 25

C.2 LONG-HORIZON STATIC TASKS

  • The four static tasks evaluate ordered cup stacking, cross-plate object swaps, ordered retrieval, and returning blocks to their original layout after rearrangement.The setup uses seven joint dimensions per arm and two gripper dimensions; joint actions are relative and gripper actions absolute.Source: §C.2 · p. 25, §C.1 · p. 25

C.3 LONG-HORIZON DYNAMIC TASKS

The real-world dynamic tasks pair remembered locations, colors, or object order with manipulation of moving objects. The section also outlines evaluation examples and analyzes historical KV attention patterns.

C.3 LONG-HORIZON DYNAMIC TASKS

  • Object Return, Match and Pick, Sequential Pick, and Color Sorting require recalling earlier locations, colors, order, or color-to-compartment mappings while handling conveyor-borne objects.The tasks combine memory for prior cues with timely manipulation of moving targets.Source: §C.3 · pp. 25, 27

C.4 EVALUATION PROTOCOL AND ADDITIONAL EXAMPLES

  • Table 9 reports complete-task success rates for four long-horizon static real-world tasks, with Memory-VLA and D2-VLA identified as the compared methods.The caption does not provide the table's cell values.Source: Table 9 · p. 27

D MEMORY ANALYSIS AND ABLATION STUDIES

  • The analysis examines historical attention patterns, accuracy–memory–latency trade-offs, retention ratios, and high-rate pathway update frequency.The stated goal is to assess how intermediate visual updates contribute to dynamic long-horizon control.Source: D MEMORY ANALYSIS AND ABLATION STUDIES · p. 27

D.1 HISTORICAL KV ATTENTION ANALYSIS

  • LIBERO and RoboTwin visualizations separate VLM and action-expert reads at two observation chunks; they are diagnostic, not an offline selector prior.The selector ranks tokens within each retained historical block and has no separate recent-frame-only rule.Source: §D.1 · p. 27

D.2 ACCURACY–MEMORY–LATENCY TRADE-OFF OF KV CACHING

Selective historical KV reads outperform uncompressed reads across three benchmarks while reducing inference latency and peak memory. The comparison preserves complete stored blocks and changes only temporary consumer-specific read views.

D.2 ACCURACY–MEMORY–LATENCY TRADE-OFF OF KV CACHING

  • Consumer-specific selection constructs separate historical read views from online attention statistics while retaining complete KV blocks in persistent storage.Selection changes temporary reads, not the stored blocks; the comparison suggests filtering redundant context may contribute to higher success, but does not isolate that mechanism.Source: §D.2 · pp. 27, 29
  • Compared with single-observation π0.5, D2-VLA raises LIBERO-Long success from 92.4% to 97.5%, while latency rises from 141.03 ms to 174.92 ms and peak memory from 9,157 MiB to 9,654 MiB.On RoboTwin 2.0 and DOMINO, success improves by 17.3 and 19.7 percentage points, respectively, with peak memory only about 1–2% higher than the single-observation baseline.Source: §D.2 · pp. 27, 29
  • Historical KV activation distributions are non-uniform, with stronger magnitudes concentrated in subsets of tokens and feature dimensions.This concentration motivates selective compression as a potential way to reduce memory and computation with limited performance loss.Source: Figure 8 · p. 29
  • On LIBERO-Long, D2-VLA raises success from 94.5% to 97.5%, lowers latency from 184.69 ms to 174.92 ms, and reduces peak memory from 10,738 MiB to 9,654 MiB versus Full KV Cache.On RoboTwin 2.0 and DOMINO, success is also higher and latency lower; peak memory falls by approximately 4.1% and 6.4%, respectively.Source: §D.2 · pp. 27, 29

D.3 CONSUMER-SPECIFIC SELECTION ANALYSIS

The section evaluates historical visual-token retention ratios and frames their relationship to DOMINO success as part of the KV-caching trade-off analysis.

D.3 CONSUMER-SPECIFIC SELECTION ANALYSIS

  • The retention study tests ratios of 1.0, 0.9, 0.8, 0.7, 0.6, 0.5, 0.4, and 0.3.Within each experiment, VLM prefill and slow and fast denoising use the same ratio.Source: §D.3 · p. 29
  • Figure 9(b) presents DOMINO success rate under different historical visual KV-retention ratios.The supplied caption identifies the comparison but does not state its outcome.Source: Figure 9 · p. 30

D.4 UPDATE-FREQUENCY ABLATION

The ablation compares slow-only inference with two dual-frequency settings on DOMINO. Both enabled settings outperform slow-only inference, and the main configuration uses V = 3E and H = E = 25.

D.4 UPDATE-FREQUENCY ABLATION

  • 27.6% and 29.3% SRs for enabled settings exceed 16.1% without the high-rate pathway on DOMINO.The comparison is between (V, E) = (50, 25) and (75, 25) versus slow-only inference.Source: Table 10 · p. 30, §D.4 · p. 30
  • With E = 25, enabled settings vary V between 50 and 75, giving period ratios of 2 and 3 and one or two intermediate adapter calls per slow cycle.The settings differ in slow-refresh interval and intermediate updates, not absolute action-replanning rate.Source: §D.4 · p. 30, Table 10 · p. 30
  • The main configuration uses V = 3E and H = E = 25.Source: §D.4 · p. 30

Related papers

Loading 2609.34792v2…