PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-Supervised Keyframe Discovery for Horizon-Invariant Behavior Cloning

Prabin Kumar Rath, Omkar Patil, Nakul Gopalan

arXiv:2610.10857v1cs.AIcs.RO

TL;DR

Behavior cloning in non-Markovian settings must use long-term context, while recurrent and attention-based policies have retention or scaling limits. Keyframe Mnemonics discovers decision-relevant observations and trains BC policies conditioned on them. The paper reports performance across synthetic, grid, and robot-memory evaluations, with horizon-invariant behavior under stated assumptions.

  • Problem

    Behavior cloning in non-Markovian environments needs long-term context, but recurrent and attention-based policies have limitations in scaling or retaining history.

    Source: §1 · p. 1

  • Method

    Keyframe Mnemonics uses randomly sampled past observations to learn a reward for selecting decision-relevant keyframes, then trains a BC policy conditioned on them.

    Source: §1 · p. 2, §2.1 · p. 3

  • Results

    Across diverse memory-intensive benchmarks, the method achieves state-of-the-art performance; Transformer policies improve absolute SR by 13.9% on MIKASA-Robo.

    Source: §1 · pp. 2–3, §5 · p. 10

  • Takeaways & Limitations

    Under task-structure assumptions, mnemonic-conditioned BC policies can retain context and generalize beyond the training horizon.

    Source: §1 · p. 2

    Each demonstration must contain at least one keyframe, limiting the current algorithm in multi-task or play-demonstration settings.

    Source: §5 · p. 10

Abstract

from arXiv · show

Behavior cloning (BC) in non-Markovian environments is a challenging problem because policies have to reason over contextual information over long horizons. Existing policy architectures rely on recurrent or attention-based mechanisms to capture long-term dependencies. However, recurrent models suffer from hidden-state collapse and gradient instability under backpropagation through time, while attention-based models are fundamentally limited by context length. To address these issues, we propose Keyframe Mnemonics, a novel self-supervised method that $\textit{discovers}$ a set of information-critical observations ($\textit{mnemonics}$) by learning an objective from randomly sampled past observations and using it as a reward for keyframe selection. We then train a BC policy that conditions on the discovered keyframes to model the action distribution. Under certain task-structure assumptions, our formulation provides context retention guarantees over an infinite horizon, while maintaining a small set of decision-relevant keyframes in the policy's working memory. We evaluate our method on synthetic memory domains, where mnemonic-conditioned BC policies achieve $100$% success rates (SR) and generalize to horizons orders of magnitude beyond training without performance degradation. Additionally, we evaluate on memory-intensive robot manipulation benchmark, achieving a $13.9$% average absolute SR improvement over the strongest baseline across $23$ tasks and retaining $80$% SR at $20\times$ longer horizons on a real robot. Code and videos are available at https://keyframe-mnemonics.github.io.

1 Introduction

Long-horizon behavior cloning must retain decision-relevant history, but recurrent and attention-based policies face limits on scaling or retaining that context. Keyframe Mnemonics discovers informative observations for a BC policy, with theoretical retention guarantees under task-structure assumptions and reported performance across memory domains.

1 Introduction

  • Recurrent policies face BPTT scaling challenges and hidden-state collapse, while attention-based policies have finite context and can incur quadratic compute as context grows.The introduction also notes that hybrid memory methods can degrade beyond training horizons.Source: §1 · pp. 1–2
  • Keyframe Mnemonics discovers observations relevant to the action distribution, trains a selector to identify them, and conditions a BC policy on those keyframes.The method is presented as self-supervised and designed to work without explicit keyframe annotations.Source: §1 · p. 2
  • Under task-structure assumptions, the method provides information-retention guarantees and supports generalization beyond training horizons.The paper describes these as theoretical intuition and guarantees under certain assumptions.Source: §1 · p. 2
  • MLP policies achieve 100% SR in synthetic and grid memory domains, while Transformer policies improve absolute SR by 13.9% on MIKASA-Robo and a flow-matching policy retains 80% SR at 20× longer test horizons on a real robot.These results are reported as evidence of horizon-invariant performance.Source: §1 · pp. 2–3

2 Methodology

Keyframe Mnemonics learns a keyframe-selection objective from randomly sampled past observations, then uses selected observations to condition behavior cloning. Under stated structural assumptions, an optimal selector retains keyframes over an infinite horizon.

2 Methodology

  • The method assumes sparse keyframes approximately suffice for the belief state and provide more action-disambiguating information than background frames; it defines mnemonics as such keyframes.The selector aims to place these observations in a bounded memory buffer for the behavior-cloning policy.Source: §2 · p. 3

2.1 Leveraging Simplicity-bias for Keyframe Discovery

  • Randomly masking up to k past observations creates combinatorially large training support; under realizability and simplicity-bias assumptions, SGD favors invariant keyframe hypotheses over more complex background-based ones.The proxy policy is trained on masked buffers to predict expert actions.Source: §2.1 · pp. 3–5
  • Minimizing a frozen invariant proxy's loss over candidate buffers minimizes the action-information gap and recovers keyframes when their information margin exceeds predictor approximation error.If the keyframe set is unique under the memory budget, the objective recovers that set.Source: §2.1 · p. 5

2.2 Selector Optimization for Keyframe Identification

  • To avoid exhaustive buffer search, a selector scores observations in a priority-queue MDP, rewarded for lowering the proxy objective and weighted toward high-action-uncertainty timesteps.PPO trains the selector; training uses FIFO eviction and inference uses LIFO eviction.Source: §2.2 · pp. 5–6

2.3 Training BC Policy with Keyframe Mnemonics

  • For an optimal selector minimizing frozen objective Jθ*(Bt), when a unique keyframe set Xkey has the highest mutual information with action and fits memory budget k, priority ordering retains Xkey without eviction for the entire horizon T.Source: §2.3 · p. 6
  • When background observations may still affect actions, the BC policy is trained on both the current observation and the selected-keyframe buffer.The paper notes this possibility in complex domains such as robot manipulation.Source: §2.3 · p. 6

3 Experiments and Result Analysis

The experiments test keyframe discovery and whether mnemonic-conditioned behavior-cloning policies generalize beyond training horizons across synthetic, grid, and robot-manipulation settings. KM performs strongly on memory-intensive benchmarks, including extended-horizon evaluations.

3 Experiments and Result Analysis

  • The evaluations cover synthetic sequence tasks, grid memory environments, and robot-manipulation benchmarks, comparing KM with sequence-model and memory-based baselines.MIKASA-Robo evaluations compare against ELMUR and other baselines; the horizon-retention comparison includes ELMUR and PTP.Source: §3.1 · p. 7, §3.2 · p. 8

3.1 Synthetic Domain Benchmarks and Convergence Analysis

  • Across three synthetic tasks, the proxy objective decreases as keyframes are added; with sufficiently large calibrated support, active regret reaches zero while passive regret increases with support size.These observations support Corollary 1 and Theorem 1, respectively.Source: §3.1 · pp. 7–8, Figure 2 · p. 7

3.2 Robot Manipulation Benchmarks and Real-world Experiment

  • Keyframe Mnemonics improves average absolute success rate by 13.9% over ELMUR across 23 MIKASA-Robo tasks.The Figure 4 caption reports wins on 17 of 23 tasks, while the main-text results report 21 of 23.Source: Figure 4 · p. 8, §3.2 · p. 9

3.1 Synthetic Domain Benchmarks and Convergence Analysis

  • KM maintains near-perfect performance across horizons orders of magnitude beyond training on all evaluated synthetic and grid tasks, while baselines collapse beyond the training range.KM retained equivalent in-distribution performance on three synthetic tasks at horizons 10^5 and 10^6.Source: Figure 3 · p. 7, §3.1 · p. 8

3.2 Robot Manipulation Benchmarks and Real-world Experiment

  • In the real-world RememberColor2 task, KM retains average 80% success rate at test horizons 20× longer than training.The evaluation used 30 demonstrations and tested delays up to 20× longer than the approximately 5-second training delays.Source: Figure 6 · p. 9, §3.2 · p. 10
  • On RememberColor3-∆T, KM sustains its training-time success rate of approximately 70% through ∆T = 250, while ELMUR and PTP degrade at longer delays.All three methods were trained at ∆T = 15.Source: §3.2 · p. 9, Figure 5 · p. 9
  • On Markovian Robomimic tasks Lift, Can, and Square, KM matches the diffusion-policy baseline trained without keyframes.The reported result indicates no performance deterioration on these three benchmarks.Source: §3.2 · p. 9, Figure 5 · p. 9
  • KM trains faster than PTP, whereas ELMUR is the fastest to train; additional ELMUR training did not yield success beyond the training horizon.After approximately 10 hours of training, ELMUR remained at 0% SR for ∆T > 15.Source: §3.2 · p. 10, Table 1 · p. 10

4 Related Work

The related work covers memory mechanisms for behavior cloning, keyframe-conditioned policies, and learned context management. KM is distinguished by self-supervised keyframe identification for non-Markovian behavior cloning.

4 Related Work

  • Prior BC memory methods use embedding caches, learned compression, temporal aggregation, or hybrid approaches, while KM preserves information-rich keyframes across horizons.Other approaches based on 2D tracking or 3D workspace mapping are described as limited by their underlying vision foundation model.Source: §4 · p. 10
  • Earlier keyframe-conditioned BC methods rely on manual annotations [31] [50] [51] or task-specific VLM prompts [52], whereas KM requires no task-level oracle annotations.The paper therefore excludes comparisons against methods requiring additional data beyond expert demonstrations.Source: §4 · p. 10
  • Prior RL-based keyframe selection was studied in simple grid and MuJoCo settings, leaving its scalability to complex robot manipulation unclear.The paper describes KM as the first self-supervised keyframe-identification method for non-Markovian behavior cloning.Source: §4 · p. 10

5 Conclusion

The paper presents Keyframe Mnemonics for horizon-invariant behavior cloning in partially observable, memory-intensive environments. It concludes that learned mnemonics support compact context retention, while identifying memory-budget and demonstration-coverage constraints.

5 Conclusion

  • KM trains a reward model from randomly sampled past observations and optimizes a selector to discover decision-relevant keyframes without supervision for keyframe locations.The discovered keyframes condition a behavior-cloning policy.Source: §5 · p. 10
  • The authors report state-of-the-art performance across memory-intensive benchmarks, from synthetic tasks to robot manipulation.They frame compact, decision-relevant context over extended horizons as a step toward more reliable long-horizon behavior cloning.Source: §5 · p. 10
  • The memory budget k is difficult to predetermine, and each demonstration must contain at least one keyframe, limiting current use in some multi-task or play-demonstration settings.Allowing a larger inference-time priority queue can provide richer policy context when information-equivalent buffer states arise.Source: §5 · p. 10

A Proofs

The proofs connect masked sampling and simplicity bias to keyframe recovery and infinite-horizon retention under stated information and uniqueness conditions. The buffer-queue discussion explains how training and inference eviction rules support discovery and retention.

A Proofs

  • Masked sampling creates a combinatorially large training-pair support, making the expected complexity of background-based hypotheses exceed that of keyframe-based hypotheses under the information-gap assumption.The argument assumes sufficiently large support so its linearly scaling information-gap term dominates constant overheads.Source: A Proofs · pp. 17–19
  • With a positive keyframe information advantage and simplicity-biased SGD, the invariant hypothesis is increasingly preferred as the masked-sampling support grows.The proof’s preference need not hold if a lower-complexity non-invariant shortcut exists.Source: A Proofs · pp. 19–20
  • A frozen loss objective recovers keyframes up to information-equivalent subsets when the positive action-information margin exceeds proxy approximation error.If the keyframe set is unique within memory budget k, the minimizer is that set; the result also extends to MSE under fixed-variance Gaussian actions.Source: A Proofs · pp. 20–21
  • Under the stated assumptions, the optimal selector retains the unique keyframe set without eviction as the horizon tends to infinity.The result assumes keyframes fit within memory and have the highest mutual information with actions.Source: A Proofs · p. 21

B.1 Buffer priority queue

  • Training FIFO eviction ages out even uniformly high-priority frames, while inference LIFO eviction preserves older high-priority entries and uses a no-repeat threshold to retain diversity.With a near-optimal selector and enough memory for all keyframes, the queues behave identically in synthetic-domain experiments.Source: §B.1 · pp. 22–23

B.2 Proxy policy training

Masked-sampling training and a frozen proxy objective provide the basis for selector optimization. The selector is trained in an RL environment, then paired with a BC policy using a keyframe-augmented dataset at inference.

B.2 Proxy policy training

  • Proxy training uses fresh online masks for MIKASA-Robo and cached offline masked-sampling pairs for synthetic and grid domains.Offline passes are increased until cached pairs exceed the support size required by Theorem 1.Source: §B.2 · p. 23
  • Masked-sampling and a random uniform-priority queue induce the same distribution over buffer states, aligning selector optimization with the proxy’s training search space.The equivalence uses a rejection threshold of 0.5, which yields the masked-sampling cardinality distribution.Source: §B.2 · p. 23

B.3 Selector RL environment

  • The selector RL environment updates the priority queue from each emitted priority and scores the resulting buffer with the frozen proxy’s surprise-weighted log-reward.Surprise scaling is enabled for synthetic and grid domains but disabled for robot manipulation, where unscaled reward gives a more stable PPO signal.Source: §B.3 · p. 24

B.4 End-to-end training and inference

  • Training proceeds from proxy fitting to selector PPO and then BC on selector-augmented demonstrations; inference updates a LIFO buffer before the BC policy acts.LIFO is preferred for grid and robot domains, while either eviction rule worked in synthetic domains where the retention theorem holds strongly.Source: §B.4 · p. 24

C.1 Synthetic environments

The synthetic and grid benchmarks require policies to preserve sparse task cues amid irrelevant observations and use them at later decision points. Their evaluation increases sequence length, corridor length, or distractor count.

C.1 Synthetic environments

  • In Add, the policy must sum two flag-marked values at a joint emit step; the marked observations and emit step are keyframes.Evaluation lengthens sequences by adding unmarked background observations.Source: §C.1 · pp. 25–26
  • In Scattered-Copy, the policy recalls relevant source tokens in order at delimiter positions, so both relevant tokens and delimiters are keyframes.Evaluation increases noise tokens and blank positions.Source: §C.1 · p. 26
  • In Active T-Maze, the agent must remember the oracle’s goal cue and use it to choose an arm at the junction; those cue and junction observations are keyframes.Evaluation stretches the ambiguous corridor between cue and junction.Source: §C.1 · p. 26

C.2 Grid environments

  • MiniGrid-Memory requires matching the start-room object to one of two goal arms, with the cue as the keyframe and corridor observations as background.Evaluation stretches the corridor.Source: §C.2 · p. 27
  • LTMB-Hallway requires matching the start-room target’s shape and color to one object behind a door; evaluation adds distractor hallways.The target cue is the keyframe, while corridor observations are background.Source: §C.2 · p. 27

C.3 Training dataset

The appendix describes dataset variants, model and baseline training choices, and a parameter-budget comparison. KM’s three-module design remains below most baseline parameter counts, with most of its budget in the proxy and BC policy.

C.3 Training dataset

  • Training datasets contain expert demonstrations only, and baselines are tested using the better-performing of small- and large-distribution training variants.The large variant scales trajectory horizon and variation count; some baselines benefit from it, while others generalize better from the small dataset.Source: §C.3 · pp. 27–28, Table 2 · p. 28

C.4 Training Keyframe Mnemonics policies

  • KM uses proxy, selector, and BC networks, with memory budgets chosen empirically and architectures varying by benchmark.Synthetic proxies use MLPs, while grid-world models use CNNs and Transformers; the selector uses PPO settings summarized per benchmark.Source: §C.4 · pp. 28–29, Table 3 · p. 28, Table 4 · p. 29

C.5 Baselines and benchmarking details

  • Baselines share an AdamW trainer with warm-up cosine scheduling and gradient clipping, and their training recipes were tuned for the benchmark domains.Baseline design choices derive from the RATE open-source implementation.Source: §C.5 · p. 29

C.6 Parameter-budget comparison

  • Despite using three networks, Keyframe Mnemonics’ total parameter count remains well below most baselines, with the selector accounting for under 1% on synthetic tasks and roughly 15% on grid tasks.Most of the KM parameter budget sits in the proxy and BC policy.Source: Table 5 · p. 29, §C.6 · pp. 29–30

C.7 Benchmark result analysis

The benchmark analysis attributes long-horizon failures to information blur in recurrent or external memories and context-window limits, while Keyframe Mnemonics retains task-relevant observations. The accompanying robot-domain details describe its offline three-stage training pipeline and implementation.

C.7 Benchmark result analysis

  • In T-Maze and Add, recurrent baselines that fit training data can collapse at longer horizons, whereas Keyframe Mnemonics retains task-relevant keyframes across horizon changes.In T-Maze, LSTM (BPTT) and TrXL reach train SR 1.0 but fall near chance on longer horizons; the analysis reports a similar trend in Add.Source: §C.7 · p. 30
  • Scattered-Copy requires recalling multiple numbers in observed order; Keyframe Mnemonics stores selected tokens separately, helping the BC policy copy them while ignoring noise.The analysis contrasts this buffer with hidden-state or external-memory updates that blur information over time.Source: §C.7 · p. 30
  • Decision Transformer and DMamba lack explicit memory and degrade toward near-chance performance when sequences exceed their training context window.The analysis says relevant information falls out of context once sequence length exceeds that window.Source: §C.7 · pp. 30–31

C.8 Hardware and wall-clock

  • The three-stage pipeline takes about 3 hours per seed across the five synthetic and grid benchmarks, with grid tasks accounting for most of the runtime.All runs used one NVIDIA RTX 3090 GPU.Source: §C.8 · p. 31

D.2 Pipeline overview

  • The robot pipeline trains a proxy, a PPO keyframe selector, then a BC policy using offline expert demonstrations; the simulator is not queried during proxy or selector training.The memory budget is k = 3/2 for proxy/BC, so the BC policy receives the current observation and two mnemonics.Source: §D.2 · p. 31

D.3 Architectures and parameter counts

  • Robot modules share an ImageNet-initialized ResNet-18 visual encoder and use overhead and wrist cameras alongside 25-dimensional proprioception.Camera frames are 128×128 RGB; the proprioceptive state combines TCP pose, joint positions, and joint velocities.Source: §D.3 · p. 32

D.5 Stage-1: Proxy policy fθ; D.6 Stage-2: Selector policy πs

  • The proxy reconstructs expert action chunks from randomly masked past observations, while the selector assigns observation priorities using the proxy objective as its reward.The proxy uses L1 loss for action chunks of length H = 8; the robot selector uses PPO for 5 × 10^6 environment steps and omits the surprise coefficient.Source: §D.5 · p. 32, §D.6 · p. 32

D.8 Camera-image data augmentation

  • The BC policy is trained on augmented camera images, with independent color and affine transformations; proprioceptive state is left unchanged.The authors report the augmentation as crucial for avoiding causal confusion from the larger context size.Source: §D.8 · p. 33

D.9 MIKASA-Robo all results

On MIKASA-Robo, Keyframe Mnemonics improves average success rate over several baselines across the 23 non-zero tasks. The appendix also reports the training cost for these robot tasks.

D.9 MIKASA-Robo all results

  • +13.9% average absolute SR over ELMUR across 23 non-zero MIKASA-Robo tasks, with improvements of +30.6% over RATE, +36.6% over diffusion policy, +37.1% over BC, +40.4% over Decision Transformer, and +42.5% over Conservative Q-Learning.The passage identifies ELMUR as the strongest baseline among the listed comparisons.Source: §D.9 · p. 34

D.10 Hardware and wall-clock

  • About 13 hours are required per SST task, while each of the four Intercept* flow-matching tasks takes roughly a full day.The SST estimate includes about 1.5 hours for proxy training, 10 hours for selector training, and 1.5 hours for BC; flow matching adds about 10 hours at the BC stage.Source: §D.10 · p. 34

E Experimental Details

  • The random baseline is omitted from the reported comparison because its test success rate remains low across every benchmark-horizon cell.The passage says this is consistent with its performance in Table 7.Source: E Experimental Details · p. 34

E.1 Horizon-invariance experiments on synthetic and grid domains

Keyframe Mnemonics maintains horizon-invariant performance on the synthetic and grid domains, including tests far beyond the training horizons. Baselines can perform strongly within their training range but collapse at substantially longer horizons.

E.1 Horizon-invariance experiments on synthetic and grid domains

  • The evaluation varies task horizons across five settings for synthetic tasks and uses increasing environment sizes or corridor lengths for the grid benchmarks.Synthetic horizons include 10, 20, 30, 1000, and 10000; MiniGrid-Memory sizes range from 17 to 1001, and LTMB-Hallway lengths from 8 to 1000.Source: §E.1 · pp. 34–35
  • Keyframe Mnemonics generalizes up to five orders of magnitude beyond training horizons on synthetic tasks, while baseline methods collapse when test horizons exceed training horizons by even one order of magnitude.The reported result supports Theorem 2 in regimes where the underlying assumptions hold strongly.Source: §E.1 · p. 35
  • LSTM (BPTT), Decision Transformer, and Transformer-XL reach 1.0 SR on T-Maze through h = 30 within the training horizon range.The passage contrasts these in-range results with baseline collapse beyond the training horizon.Source: §E.1 · p. 35

E.2 Horizon-invariance experiment on RememberColor3-∆T

In RememberColor3-∆T, Keyframe Mnemonics retains success as cue-to-action delays increase without retraining, unlike the compared baselines. Ablations indicate that learned selection and longer-horizon selector optimization matter, while surprise scaling is task-dependent.

E.2 Horizon-invariance experiment on RememberColor3-∆T

  • The horizon study trains policies at ∆T = 15 and evaluates them at delays up to ∆T = 250 without retraining; training demonstrations use a 10-step hover phase.The reported results average three environment seeds and 100 episodes per seed.Source: §E.2 · p. 35
  • ≈0.65 success rate at ∆T = 250, about 17× the training delay of ∆T = 15, with no architectural change or fine-tuning; ELMUR falls to zero by ∆T ≥ 35 and PTP drops below 10% by ∆T = 200.The evaluation reinstantiates the simulator at each delay without retraining or fine-tuning.Source: §E.2 · pp. 35–36, Table 13 · p. 36

F.1 Learned selector against a random selector

  • Replacing the learned selector with a random selector worsens performance at every delay interval on RememberColor3-∆T.The passage describes the learned selector as critical for the memory task.Source: §F.1 · p. 36, Table 14 · p. 36

F.2 Selector discount factor

  • Greedy selector optimization at γ = 0 fails on every domain except active T-Maze, with performance recovering as γ increases.The passage attributes active T-Maze’s exception to the action changing immediately after the keyframe; other domains have a multi-step gap.Source: §F.2 · pp. 36–37, Table 15 · p. 37

F.3 Surprise coefficient

  • κ = 0.2 has the widest usable range and is the default surprise-scaling temperature for Add, Scattered-Copy, and MiniGrid-Memory.The procedure adds surprise scaling only when it improves performance over the unscaled reward.Source: §F.3 · p. 37, Table 16 · p. 37

F.4 Current observation in proxy training

The appendix examines proxy-training inputs, selector behavior, memory capacity, and policy use of discovered keyframes. Results show that some training and capacity choices matter, while learned keyframes can capture task cues and support action disambiguation.

F.4 Current observation in proxy training

  • Including the current observation in proxy training collapses performance on MiniGrid-Memory and RememberColor3.The authors attribute this to proxy overfitting to the current observation, which creates a shortcut for keyframe identification.Source: §F.4 · p. 37, Table 17 · p. 37

F.5 Queue tie-break rule

  • FIFO training followed by LIFO inference outperforms the reverse on RememberColor3-∆T at 9 of 10 retention intervals; all four tie-break combinations perform identically on synthetic and grid domains.The queue rule matters when observations tie at the lowest priority and a buffer slot must be evicted.Source: §F.5 · pp. 37–38, Table 19 · p. 38, Table 18 · p. 38

F.6 Memory buffer size

  • Halving buffer capacity degrades performance across domains, while doubling it helps some domains and hurts others.Larger buffers can make the KM policy harder to train; default capacities were chosen independently by domain as minimum-capacity floors.Source: §F.6 · p. 38, Table 20 · p. 38

G.1 Interpretability of discovered keyframes

  • Discovered buffers qualitatively capture task-relevant cues and belief-state information, using expert state-action demonstrations without temporal keyframe annotations.The authors present inspectable buffer contents as a way to understand which prior observations the policy relies on.Source: §G.1 · p. 38

G.2 Selector priority behavior

  • The trained selector’s priorities match oracle keyframes across synthetic benchmarks, averaging 1.00 on keyframes and 0.00 on background frames.The evaluation spans three increasing horizons, with the longest extending one order of magnitude beyond training.Source: Figure 11 · p. 39, §G.2 · p. 39

G.3 BC Policy activation map

  • BC activation maps show task-specific cues in keyframes and current observations, including object identity, initial location, and task progress, helping disambiguate actions under partial observability.In TakeItBack, a second keyframe captures task progress just before pushing, alongside the initial cube-location cue in the first keyframe.Source: §G.3 · p. 39, Figure 12 · p. 40

G.4 Regret on low memory budget k

  • Active regret varies with buffer capacity and task: insufficient capacity can prevent retaining all keyframes, while a selector may favor cues with higher mutual information with actions.Passive regret stays low below the true keyframe count and decreases again at twice the default buffer size, when random selection is more likely to retain keyframes.Source: §G.4 · p. 40, Figure 13 · p. 41

G.5 PPO reward plots

  • Selector training converges within 200K steps on synthetic domains and 5M steps on robot tasks, while ChainOfColors has a noisy curve with modest reward improvement.The authors associate noisy learning with an undertrained proxy policy or assumptions that do not hold well in the environment.Source: §G.5 · p. 40

G.6 Limitation discussion

This section discusses the method’s limitations, including restrictive task assumptions, challenges when many keyframes matter, and a robot-domain compute bottleneck, while positioning temporal masking and keyframes in related work. It also notes that human-subject/crowdsourcing research and core-method LLM use are not applicable.

G.6 Limitation discussion

  • Assumption 1 restricts the method to tasks where keyframes are individually identifiable from single observations, making it best suited to sparse, frame-localizable memory tasks.When multiple keyframes represent the same information, several equivalent buffer states can achieve similar proxy-policy objectives, making it difficult to retain one mnemonic set.Source: §G.6 · p. 41
  • Assumption 2 requires action ambiguity to correlate with sparse keyframes; when actions are mostly local and low-variance, the mutual-information gap can become negligible and proxy-policy convergence may fail.The authors observed this failure on VizDoom-Two-Colors [22], where a color cue determines a later target but navigation actions remain largely local.Source: §G.6 · p. 41
  • Learning the invariant hypothesis becomes difficult when decision-critical actions depend on many keyframes or demonstrations are very long and keyframes sparse.The authors suggest increasing the training memory budget to improve relevant-frame sampling and support size M, or composing specialized selectors trained on shorter low-capacity tasks.Source: §G.6 · p. 41
  • Robot-domain PPO training is the main computational bottleneck because SB3 incurs CPU-GPU overhead; the authors propose GPU-native PPO to accelerate selector training.The selector environment mainly requires proxy-policy inference, which could in principle run entirely on the GPU.Source: §G.6 · p. 41

H Extended Related Works

  • The authors present Keyframe Mnemonics as an early method explicitly leveraging neural-network simplicity bias for behavior cloning in POMDPs, using temporal-history masking to encourage invariant hypotheses.They describe keyframes as complementary to language-based state tracking: keyframes address partial observability, while language supports abstract reasoning over past events.Source: H Extended Related Works · p. 42

NeurIPS Paper Checklist

  • The authors state that the work has no direct societal interaction in its current introductory stage and uses artificial benchmarks and simulated robot tasks; they report no human data collection.They characterize improving neural-network memory for eventual long-horizon robot tasks as a prospective societal application.Source: NeurIPS Paper Checklist · pp. 46–47
  • The checklist marks human-subject and crowdsourcing research as not applicable and says LLMs were not used to develop the core idea or experimental design.Source: NeurIPS Paper Checklist · p. 49
Loading 2610.10857v1…