PaperlayerView exact revision on arXiv

Source-linked AI summary

PonderPounce: A Pretrained MLLM as an Episode Context Engine for Robot Control

Suhwan Choi, Jaeyoon Jung, Sungkyung Kim, Yunsung Lee, Youngjae Yu

arXiv:2608.24115v1cs.ROcs.AI

TL;DR

VLA models generally do not use pretrained MLLMs’ contextual capacity as episode memory, despite robot control requiring information absent from the current observation. PonderPounce reuses an MLLM’s native causal context and asynchronously routes cognition to a VLA, achieving higher aggregate RoboMME success and modestly higher RoboCasa-DC success than the cited baselines. The evidence is limited to two simulated benchmarks, one cognition carrier per query, and comparisons confounded by unequal reasoning supervision.

  • Problem

    VLA models generally do not carry pretrained MLLMs’ ability to integrate long histories and examples into episode memory for context-dependent control.

  • Method

    PonderPounce jointly trains a System 2 MLLM that accumulates episode context and a System 1 VLA that asynchronously receives only the newest cognition token and its age.

  • Results

    PonderPounce reaches 60.83% and 75.54% on RoboMME at base and 9× data scales versus FrameSamp+Modul’s 44.51% and 57.88%, and 12.5% versus 11.6% on RoboCasa-DC.

  • Takeaways & Limitations

    Pretrained MLLM context functions as a strong general-purpose memory substrate, although task-level results are not uniformly best across memory demands.

  • Takeaways & Limitations

    Evaluation is limited to two simulated benchmarks and one cognition carrier per query, while baseline comparisons can reflect unequal reasoning supervision.

Abstract

from arXiv · show

Multimodal large language models (MLLMs) can integrate long visual histories, reason under partial observability, and infer behavior from a few examples. Yet vision-language-action (VLA) models generally inherit pretrained representations without using this contextual capacity as episode memory. Memory-dependent policies address this gap through purpose-built history mechanisms. PonderPounce instead reuses an MLLM's native causal context as robot memory. Ponder, a System2 MLLM, accumulates episode observations, demonstrations, and prior cognition in its native causal context and can generate subgoal text and demonstration reasoning for internal use. Pounce, a System1 VLA, receives the current observation, instruction, and proprioception directly; through the Ponder--Pounce interface, it asynchronously receives only the newest continuous cognition token and its age. Both are jointly trained end to end without a purpose-built memory module or separate bridge pretraining. Optimized serving achieves p50 latencies of 78ms for cognition refresh and 25ms for action-model invocation, supporting 20Hz action playback. On RoboMME with base-scale training data, PonderPounce reaches 60.83% with 9B and 50.04% with 0.8B under the same Pounce architecture and interface, versus 44.51% for FrameSamp+Modul and 17.93% for the current-observation π_{0.5}. With 9x data, it reaches 75.54% versus 57.88% for FrameSamp+Modul. On RoboCasa-DC, the same interface learns from action supervision alone and reaches 12.5% versus 11.6% for the strongest published demonstration-conditioned baseline, falling to 8.6% when cognition is replaced by a learned null state.

1 INTRODUCTION

PonderPounce addresses the gap between MLLMs’ contextual reasoning and VLAs’ limited episode memory by routing accumulated context to control through an asynchronous cognition interface. It improves RoboMME and RoboCasa-DC performance while preserving a separately scalable fast controller.

  • Motivation: 17.93% current-observation π0.5 success on RoboMME motivates making earlier observations and demonstrations actionable for control.Adding past actions raises success only to 19.73%, versus 90.50% for humans.
  • Approach: PonderPounce reuses a pretrained MLLM’s native causal context as episode memory and sends continuous cognition to a VLA asynchronously.Ponder accumulates observations, demonstrations, and prior cognition, while Pounce conditions on the newest cognition and its age.
  • Design: The design jointly trains both pretrained components without a purpose-built memory module or separate bridge pretraining.Ponder’s subgoal text and demonstration reasoning remain internal to System 2, and optimized serving supports 20 Hz playback.
  • Results: 60.83% base-scale and 75.54% 9×-data RoboMME success exceed FrameSamp+Modul’s 44.51% and 57.88%.The comparison uses the reported RoboMME success rates at the corresponding data scales.
  • Results: 12.5% RoboCasa-DC success exceeds the 11.6% strongest published demonstration-conditioned baseline, while learned-null cognition reduces success to 8.6%.The same interface supports demonstration-conditioned control on RoboCasa-DC.

2 RELATED WORK

Related work separates retaining context from exposing it to action models, using purpose-built history mechanisms, test-time demonstration conditioning, or intermediate text and representations. PonderPounce combines persistent MLLM context, decoupled clocks, and recurrent continuous routing.

  • Purpose-built episode memory: Purpose-built memory policies retain history through sampling, segmentation banks, retrieval, recursive language memory, or per-observation summaries.Examples include FrameSamp+Modul, SAM2Act+, MemER, MEM, and MemoryVLA.
  • Demonstrations as test-time context: Demonstration-conditioned systems infer tasks from examples through autoregressive sequences, cross-attention, embodiment-agnostic skills, or joint demonstration-action modeling.The cited systems extend one-shot imitation with richer test-time context.
  • Routing context to control: Context-to-control interfaces may transmit subgoals, language motions, coordinates, or internal representations, but continuous channels can collapse toward instruction-level information.PonderPounce instead routes one cognition token and its age asynchronously from persistent MLLM context.
  • Routing context to control: PonderPounce combines persistent MLLM context, decoupled clocks, and recurrent continuous routing, which the paper identifies as absent from prior work.Helix is described as the closest asynchronous analogue but does not recurrently transmit episode or demonstration context.

3 PONDERPOUNCE

PonderPounce connects a pretrained System 2 MLLM and System 1 action model through recurrent cognition carriers updated on asynchronous clocks. The interface preserves causal context in Ponder while exposing only the newest cognition and its age to Pounce.

  • Architecture: Ponder updates K carrier states from instruction, episode history, demonstrations, and prior cognition, then Pounce conditions action chunks on the newest ready state.Both pretrained components are assembled and trained end to end.
  • PONDER (System 2): Ponder’s append-only context retains observations, demonstrations, internal subgoal text, demonstration reasoning, and earlier cognition without explicit action history.Transition tokens gate optional text generation before carrier states are formed.
  • POUNCE (System 1): Pounce receives instruction, the current observation, proprioception, newest cognition, and its age; its action head predicts A1:h for 20 Hz playback.The default System 2-to-System 1 interface transmits only cognition and age.
  • Training: Joint training combines action flow-matching MSE with token cross-entropy for transition, subgoal-text, and demonstration-reasoning targets where annotations exist.Grounding enters through Ponder’s language-model head, while the action loss reaches Ponder through cognition carriers.
  • Serving: StaticCache avoids re-encoding causal context, while fused kernels reduce Pounce latency from 142 to 25 ms and support 20 Hz playback.The cache is described as native transformer KV caching rather than a separate episode-memory store.

4 EXPERIMENTS

Experiments evaluate memory-dependent control on RoboMME and demonstration-conditioned control on RoboCasa-DC across reported baselines, data scales, and cognition interventions. PonderPounce leads the aggregate RoboMME comparisons and modestly exceeds the published RoboCasa-DC baseline, with task-level variation and low absolute RoboCasa-DC success.

  • Benchmarks: RoboMME tests remembering earlier hidden information across 16 tasks in Counting, Permanence, Reference, and Imitation at 1× and 9× data scales.RoboCasa-DC instead evaluates five held-out demonstration-conditioned tasks.
  • Configuration: The study uses pretrained Qwen3.5-9B Ponder with one cognition carrier and jointly trains it with π0.5 on RoboMME or GR00T N1.5 on RoboCasa-DC.Subgoal and demonstration-reasoning annotations supervise Ponder where available.
  • Baselines: RoboMME comparisons include current-observation and past-action π0.5, SAM2Act+, MemER, and FrameSamp+Modul, while RoboCasa-DC compares published demonstration-conditioned systems.The data-matched 9× FrameSamp+Modul comparison changes only the training set.
  • RoboMME: 60.83% base-scale and 75.54% 9×-data RoboMME success exceed FrameSamp+Modul by 16.32 and 17.66 percentage points.The reported averages cover four task families.
  • RoboMME: PonderPounce leads RoboMME on Permanence and Reference, while FrameSamp+Modul remains stronger on Imitation and 9× Counting.This shows that aggregate gains coexist with family-level differences.
  • RoboMME: Episode context preserves a transient PickHighlight cue after its visual markers disappear, consistent with benefits for hidden-state tracking and reference resolution.The same architecture gains 14.71 percentage points with expanded training data.
  • RoboCasa-DC: 12.5% RoboCasa-DC success exceeds SeeTraceAct’s 11.6%, while learned-null cognition reaches 8.6% and no-demonstration control reaches 9.0%.PonderPounce and its cognition-disabled condition are averaged over five runs; the no-demonstration control uses one run.

5 ANALYSIS OF THE COGNITION CHANNEL

The analysis tests how supervision, interface design, refresh timing, and pretrained context capacity affect PONDERPOUNCE control. Results show that grounding and timely within-subgoal cognition refreshes matter, while larger pretrained context engines improve performance through the same interface.

  • 5.1 SUPERVISION AND THE COGNITION INTERFACE: Removing demonstration-reasoning targets lowers average RoboMME success from 60.83% to 48.21%, while removing all LM-head grounding lowers it to 27.96%.Because LM-head supervision and cognition carriers share a transformer trunk, these ablations do not distinguish richer cognition from improved gate or subgoal generation.
  • 5.1 SUPERVISION AND THE COGNITION INTERFACE: 60.83% versus 59.96% shows continuous cognition is accuracy-competitive with a separately adapted transition-only subgoal-text reference.The 0.87 pp gap is smaller than the 2.63 pp spread across base-scale evaluation runs.
  • 5.1 SUPERVISION AND THE COGNITION INTERFACE: 78 ms p50 supports per-query cognition refresh, whereas a forced 45-token subgoal-text decode would take 0.82 s per fire.RoboCasa-DC also drops from 12.5% to 8.6% when cognition is replaced by a learned null state, despite lacking subgoal or demonstration-reasoning annotations.
  • 5.2 COGNITION REFRESH AND STALENESS: 1.83% versus 60.83% shows the reported checkpoint depends on within-subgoal cognition refreshes, not only transition-time updates.The intervention reuses the last transition-time state while other inputs remain fixed, but its stale age reporting and missed transitions prevent ruling out training-matched sparse delivery.
  • 5.2 COGNITION REFRESH AND STALENESS: At 4.3 s, normalized loss is 9.22× for 1 s-refresh training, compared with 3.19× and 1.14× for 2 s- and 4 s-refresh checkpoints.Slow-refresh training improves stale-cognition robustness but raises absolute loss at 0.3 s from 0.117 to 0.213 and 0.232.
  • 5.3 DOES PRETRAINED CONTEXT CAPACITY TRANSFER TO CONTROL?: 60.83% versus 50.04% shows that increasing the pretrained context engine from 0.8B to 9B improves RoboMME success by 10.79 pp with unchanged POUNCE architecture and interface.The larger context engine remains off the fast action path.
  • 5.3 DOES PRETRAINED CONTEXT CAPACITY TRANSFER TO CONTROL?: 0.00% success after random 9B initialization shows that the current end-to-end recipe depends on pretrained context-model initialization.Because training does not converge, this does not isolate pretraining’s benefit under matched optimization.

6 CONCLUSION

PONDERPOUNCE uses pretrained MLLM contextual capacity as robot memory, routing asynchronous continuous cognition to a VLA. Across RoboMME and RoboCasa-DC, context improves control, but the substrate is not universal or compute-free.

  • 6 CONCLUSION: 60.83%/75.54% versus 44.51%/57.88% shows PONDERPOUNCE outperforming FrameSamp+Modul on RoboMME at 1× and 9× data.With the same controller architecture and interface, increasing the pretrained context engine from 0.8B to 9B improves success by 10.79 pp.
  • 6 CONCLUSION: PONDERPOUNCE reuses native MLLM context and routes one asynchronous cognition token to a VLA without a purpose-built episode-memory store or retrieval path.Continuous cognition remains competitive with a separately trained transition-only subgoal-text interface.
  • 6 CONCLUSION: 12.5% versus 8.6% shows cognition affects RoboCasa-DC control without LM-head grounding, while timely within-subgoal refreshes remain necessary for the reported checkpoint.The conclusion characterizes pretrained MLLM context as a strong but not universal or compute-free memory substrate.

7 LIMITATIONS

The evaluation has important scope and cost constraints: comparisons mix architecture with unequal reasoning supervision, cover only two simulated benchmarks, and use limited cognition transmission.

  • Supervision and evaluation scope: RoboMME comparisons conflate architecture with supervision because simulator-derived gates, subgoals, and demonstration-reasoning targets are unavailable to published baselines.The production cost of these targets is not measured.
  • Supervision and evaluation scope: The evaluation covers two simulated benchmarks and one cognition carrier per query (K=1), limiting evidence about broader transfer.RoboCasa-DC provides initial interface-transfer evidence rather than broad cross-embodiment generalization.
  • Systems constraints: Pairing a 9B context model with a 3–3.6B controller incurs additional training and inference cost.The latency profiles characterize batch-1 calls rather than concurrent throughput, and append-only context is limited to 16K tokens.
  • Open mechanisms: The representation and use of cognition by POUNCE remain open despite interventions establishing that cognition contributes to control.Further limitations include teacher-forced staleness diagnostics and no matched pretraining-convergence comparison.

8 FUTURE WORK

Future work targets fairer supervision comparisons, broader validation, and deeper analysis of cognition transmission, representation, and system-level efficiency.

  • Evaluation and supervision: Future evaluations should compare architectures under matched supervision and annotation cost, including label quality and production cost for generated labels.The paper suggests testing labels propagated from vision-language models rather than relying only on simulator annotations.
  • Evaluation and supervision: Counterfactual history pairs, richer context-type breakdowns, additional embodiments, and real-robot experiments would broaden evaluation beyond current simulated settings.These directions are proposed to test history dependence and generalization more directly.
  • Efficiency and scaling: System-level studies should compare energy and concurrent throughput under matched compute while evaluating smaller, distilled, or quantized context models.This directly addresses the cost and batch-1 throughput boundaries identified in the limitations.
  • Cognition analysis: Closed-loop refresh sweeps, sparse-delivery training, representation probes, cognition-width sweeps, and broader scale controls could clarify channel encoding and controller use.The proposed studies target both temporal delivery and internal representation.
  • Cognition analysis: Longer, temporally coherent vision–action data may be required as soft-token learning scales.The paper identifies this as a possible requirement rather than an established limitation.

B.1 ROBOMME

RoboMME evaluates memory-dependent and demonstration-conditioned control across 16 tasks using paired demonstrations, multimodal trajectories, simulator-derived reasoning labels, and multi-scale training protocols.

  • Task suite and representation: RoboMME contains 16 tasks across Counting, Permanence, Reference, and Imitation, with observation-only demonstrations paired to action-labeled executions.Each trajectory includes front and wrist 256×256 RGB, 8D state and action, aligned at 20 Hz.
  • Dataset scales: The base dataset contains 1,587 episodes, while the 9× dataset is a fresh oracle collection of 14,400 episodes rather than duplicated base data.The larger collection provides 900 solvable episodes per task.
  • Reasoning annotations: Reasoning annotations add deterministic simulator-template labels for demonstration facts and episode reasoning, attached at the first execution transition.Episodes without demonstrations contain no demonstration-reasoning target.
  • Training: RoboMME training uses end-to-end optimization from Qwen3.5-9B and π0.5 components, with a 20-step action-chunk policy trained for 4,320 steps.The configuration uses eight B200 GPUs, global batch 32, and a 2×10−5 learning rate.
  • Evaluation: Evaluation averages 50 episodes per task across 800 episodes, reports three-run means, and fixes synchronized 1 Hz model clocks with a 300 ms delay.Reported overall spread is 2.63 pp at 1× and 1.63 pp at 9×, with task variation up to 16 pp.
  • RoboCasa-DC: RoboCasa-DC trains on 19 tasks and evaluates five category-balanced, non-overlapping held-out tasks with approximately 1,900 training and 500 held-out pairs.Evaluation follows a cross-embodiment protocol using paired PandaOmron executions and GR-1 demonstrations.

C SELECTED CONTEXT-TO-CONTROL ARCHITECTURES

The selected architectures span discrete text hierarchies, continuous internal representations, recurrent or latent context mechanisms, and varied backbone-sharing and training choices.

  • Adjacent mechanisms: Adjacent mechanisms include recurrent hidden states, learned latent action or motion tokens, and fast-weight history compression at deployment.These methods process recurrent or latent context but do not expose the same decoded or continuous activation-level interface.
  • Comparison dimensions: Selected methods are compared by contextual-model scale, control channel, training regime, episode or demonstration context, and channel analysis.Table 9 organizes these dimensions across representative context-to-control architectures.
  • Architecture coupling: The comparison also distinguishes shared or coupled parameters and backbone roles across methods such as FiS, LaST0, and the listed text-hierarchy systems.Table 9 records whether contextual and action components share training or parameters.

D SERVING MEASUREMENTS

Serving measurements compare optimized and unoptimized POUNCE and PONDER paths, showing that caching keeps PONDER latency nearly flat as context grows while fused kernels accelerate POUNCE.

  • Throughput: Table 10 compares target and sustainable output rates per clock, with per-call p50 latency shown in parentheses.The optimized paths use fused POUNCE kernels and session-resident StaticCache with torch.compile for cache-only PONDER appends.
  • POUNCE: 1.8× faster fused kernels than the best torch.compile mode on H100 and 1.4× on A100 accelerate POUNCE serving.The comparison uses bf16, batch 1, and warmup with CUDA synchronization around each call.
  • POUNCE: Removing cognition changes POUNCE latency by less than 0.2 ms.The production configuration uses three 224 px views, a 60-token prompt, 10 flow-matching steps, a 20-step chunk, and K=8 prefix.
  • PONDER: Cached PONDER latency stays nearly flat from 0.8K to 14K tokens, whereas re-encoding latency grows with context.The sweep uses 10 demonstration frames, two 256 px cameras, and a 16K cache across cognition-only and forced subgoal fires.
  • Latency breakdown: Tables 11 and 12 report p50 invocation latency separately for POUNCE and per-fire latency separately for PONDER across serving paths.Bold entries identify the fastest path in each table.
Loading 2608.24115v1…