PaperlayerView exact revision on arXiv

Source-linked AI summary

REKI: Registering Knowledge in VLAs

Juliette Marrie, Paul Pacaud, Adrien Ramanana Rahary, Shizhe Chen, Moritz Böhle

arXiv:2610.22178v1cs.ROcs.CVcs.LG

TL;DR

REKI targets the difficulty of reproducing and attributing VLA results when pretraining data and pipelines are proprietary, while memory methods can add overhead. It combines a public-data VLM and flow-matching action expert with cached register memory. It leads compared public-data VLAs out of distribution, and memory improves success while making oscillations rarer.

  • Problem

    Strong VLAs partly use proprietary pretraining data and unreleased pipelines, making results difficult to reproduce and gains difficult to attribute.

    Source: §1 · p. 1

  • Method

    REKI pairs a VLM with a flow-matching action expert and caches action-expert registers that summarize each frame's context.

    Source: §1 · p. 1

  • Results

    REKI leads compared VLAs pretrained on public data out of distribution, with public pretraining adding 13.8 points on Clean2Rand.

    Source: §8 · p. 12

  • Takeaways & Limitations

    The released full pipeline and checkpoints provide an efficient open baseline for comparing VLA architectures, data mixtures, and training objectives.

    Source: §8 · p. 12

Abstract

from arXiv · show

We introduce REKI, a 2.4B vision-language-action model (VLA) pretrained exclusively on public data that sets a new Pareto frontier between performance and model size: on LIBERO-Plus, RoboTwin Clean2Rand and RoboCasa365, it is outperformed only by models about twice its size. REKI consists of a vision-language model (VLM) followed by a flow-matching action expert, and is pretrained on a curated mixture of public robot, egocentric human and vision-language data. Additionally, we endow it with a memory that gives the policy access to its recent past at negligible cost: a few register tokens in the action expert summarise the output of the VLM at each frame and persist in the expert's cache. This memory improves success in simulation and on real robots, and makes oscillations between poses about four times rarer. REKI uses a simple model design and training paradigm, and is optimised to be efficient at training and inference. We release the full code across data curation, training, inference and evaluation, deployment, and all pretrained and finetuned model weights: https://github.com/kyutai-labs/reki.

1 INTRODUCTION

REKI addresses the opacity and memory overhead of existing VLA approaches with a public-data model and a lightweight action-expert memory. The paper presents it as a strong performance–size trade-off and reports that memory improves success and reduces pose oscillations.

1 INTRODUCTION

  • The strongest VLAs partly rely on unreleased proprietary data and pipelines, making their results difficult to reproduce and gains hard to attribute.REKI releases its pipeline from data curation to deployment.Source: §1 · p. 1
  • REKI uses a few action-expert registers to summarize past context, cache it across frames, and avoid extra memory modules or sequential training.Each frame is encoded only once.Source: §1 · p. 1
  • Memory consistently improves performance, with the largest gains from the first past frame, and makes pose oscillations about four times rarer.Without memory, the policy often oscillates between poses across action chunks.Source: §1 · p. 1
  • REKI sets a new performance–size Pareto frontier, scoring highest on LIBERO-Plus and being outperformed on RoboTwin Clean2Rand and RoboCasa365 only by models about twice its size.The full results are reported in Fig. 4.Source: Figure 1 · p. 2, §1 · p. 2

2 RELATED WORK

Prior VLA work offers both open pretraining pipelines and varied ways to incorporate temporal context. REKI builds on open-pretraining efforts and places a compact memory in the action expert.

2 RELATED WORK

  • OpenVLA, Octo, RDT-1B, SmolVLA, MolmoAct, and MolmoAct2 released public-data pipelines, enabling researchers to compare models under shared conditions.Models using larger in-house datasets and newer designs have nonetheless retained a lead on most benchmarks.Source: §2 · p. 2
  • Existing memory approaches stack or compress past frames in the VLM, use recurrent states or fast weights, or store past features in an extra module.These choices incur longer VLM inputs, sequential training, or dedicated memory components, respectively.Source: §2 · p. 2
  • REKI stores memory as action-expert registers, encoding each frame once and training a window in one batched pass with losses on every frame.Figure 2 contrasts this design with input-history, recurrent, and added-module approaches and shows which frames receive supervision.Source: §2 · p. 2, Figure 2 · p. 3

3 BACKGROUND

The paper formulates robot control as predicting an action chunk from context, then describes flow matching and the conditioning signals used by the action expert.

3 BACKGROUND

  • At each decision step, the policy predicts the next H actions from context containing the instruction, camera observations, and robot state.The context may include all observations so far; available views and state/action dimensions vary across embodiments and control modes.Source: §3 · p. 3

3.1 DIFFUSION POLICY

  • The action expert generates action chunks by flow matching, training to regress the constant velocity from a noisy interpolant toward the clean chunk.At inference, the initial noise is integrated from flow time 1 to 0 in ten Euler steps.Source: §3.1 · p. 3

3.2 CONDITIONING

  • The expert reads final-layer VLM hidden states for prompt and camera views through cross-attention, while a pooled robot-state embedding modulates it through adaptive normalisation.The state embedding is computed once per frame for all Euler steps because it does not depend on flow time.Source: §3.2 · p. 4

4 REKI MODEL

REKI gives its action expert a cached register memory of prior frames while retaining the VLM-plus-expert design. Training supervises each frame using context available up to that frame, and inference reuses cached registers across action-generation steps.

4.1 PRESERVING PAST CONTEXT

  • REKI gives the action expert access to prior-frame observations through registers whose keys and values persist in a rolling cache.Registers attend to encoded observations and earlier registers, but not action tokens; the memory window covers K=3 frames.Source: §4.1 · p. 4, Figure 3 · p. 5

4.2 TRAINING AND INFERENCE

  • Each training window averages flow-matching losses across its frames, conditioning each action chunk on contexts up to that frame.Frames are sampled at a 1 s stride, and later frames reading earlier registers gives those registers gradients from multiple frame losses.Source: §4.2 · p. 5
  • Eight independent flow-time and noise draws per frame share that frame’s encoded context and registers, reducing the cost of estimating the training expectation.The encoder runs once for the shared context, while the expert processes the eight noisy chunks.Source: §4.2 · p. 5
  • At inference, cached registers are reused across Euler steps and frames; at K=3, a chunk takes 21.8 ms on one H100.The rolling cache drops the oldest frame when a new one enters; reading cached registers adds 0.1 ms.Source: §4.2 · p. 6

5 EXPERIMENTAL SETUP

REKI is pretrained on public robot, egocentric human, and vision-language data, then finetuned separately for each benchmark. The setup combines dataset-specific action handling with visual augmentations intended to improve robustness.

5.1 PRETRAINING DATA

  • Robot pretraining combines public teleoperation, simulation, and bimanual demonstrations spanning varied platforms and settings.Datasets retain their native control conventions; actions are normalized per dataset, padded to a common width, and curated to remove static or near-zero-variance episodes and correct sign conventions.Source: §5.1 · p. 6
  • Egocentric human episodes and spatially focused vision-language data supplement the robot demonstrations during pretraining.The human datasets contribute about 350k episodes and 170M frames; vision-language data is mixed in at a small ratio.Source: §5.1 · p. 6

5.2 MODELS

  • The Qwen3.5-2B backbone and 10-block action expert are pretrained for 200k steps on 32 H100 GPUs, then finetuned separately per benchmark.The model uses 32 registers per frame and a K=3 memory window; 10% of training samples are vision-language data.Source: §5.2 · pp. 6–7

5.3 VISUAL DATA AUGMENTATION

  • Visual and state augmentations materially affect robustness: disabling camera dropout or image augmentation lowers viewpoint-change success by more than 20 points, while disabling state dropout lowers initial-state-change success by 8.1 points.Mirroring is also used when finetuning on RoboTwin, RoboCasa365, and RoboCasa-GR1.Source: §5.3 · p. 7

6 EXPERIMENTS

REKI leads public-data VLAs on the out-of-distribution LIBERO-Plus and RoboTwin Clean2Rand benchmarks, while remaining competitive with larger models on several in-distribution benchmarks. The experiments also find that memory improves task performance, reduces oscillations, and can be used with fast compiled inference.

6.1 RESULTS

  • 38.4 on RoboCasa365 exceeds RLDX-1 (36.0) and Qwen-RobotManip (35.9), while 58.6 on RoboCasa-GR1 is near RLDX-1 (58.7) and ABot-M0 (58.3).On RoboTwin, REKI reaches 85.9 and 84.6, below InternVLA-A1.5 (93.3 / 93.0).Source: §6.1 · p. 7
  • 90.4 on LIBERO-Plus and 65.8 on Clean2Rand put REKI first among public-data VLAs; it is competitive with larger models in distribution.It leads the next public-data model by 5.6 points on LIBERO-Plus and 10.1 points on Clean2Rand.Source: §6.1 · p. 7, Figure 4 · p. 8
  • 13.8 points on Clean2Rand and 4.3 on LIBERO-Plus are the out-of-distribution gains from public-data pretraining.The reported gains are 30.3 points under clutter and 22.4 under the Hard condition on Clean2Rand.Source: §6.1 · p. 8

6.2 ABLATION

  • 90.4, 73.4 and 38.4 with a three-frame window exceed the memoryless scores of 88.9, 67.9 and 33.0 on LIBERO-Plus, RoboTwin and RoboCasa365.Most of the gain comes with one past frame; evaluation without memory performs close to finetuning at K=1, indicating gains mainly come from reading past frames at inference.Source: §6.2 · p. 9
  • 14.8% of episodes oscillate when past frames are removed at evaluation, versus 4.1% with memory; one past frame brings the share to 4.6%.Trained without memory, REKI oscillates in 18.6% of episodes, and in half of those that fail.Source: §6.2 · p. 9
  • 45% of seeds succeed with memory where the memoryless policy oscillates; the most frequent two-arm cycle is an arm swap (65 versus 1 of 1,000 episodes).The paper attributes oscillation to perceptual aliasing: two poses can each appear to continue toward the other.Source: §6.2 · pp. 9–10

6.3 INFERENCE LATENCY

  • 21.8 ms per chunk makes compiled REKI the fastest compared model, ahead of X-VLA (25.5 ms), π0.5 with openpi (33.8 ms), ABot-M0 (46.1 ms) and InternVLA-A1.5 (51.3 ms).The protocol measures one call from raw observation to unnormalised actions, including preprocessing and host-device transfers.Source: §6.3 · p. 10
  • Figure 8 reports task scores for REKI with and without memory and π0.5 on two shorter- and three longer-horizon real-robot tasks, over 10 episodes each.Source: Figure 8 · p. 11

7 REAL ROBOT EXPERIMENTS

On five real-robot manipulation tasks, REKI outperforms π0.5, and its memory advantage is concentrated in longer-horizon tasks. The evaluation uses a UR5 arm and compares REKI with and without memory.

7 REAL ROBOT EXPERIMENTS

  • The robot is a 6-DoF UR5 with a 15 Hz action loop, observed by a fixed RGB camera over a 1 m×1 m workspace.Observations and actions use 7D absolute joint states.Source: §7 · p. 11
  • The evaluation covers pick, place, stack, and sweep tasks, with 50 training episodes and 10 test episodes per task.Test episodes use randomized initial object placements and distractors.Source: §7 · p. 11
  • REKI and π0.5 are finetuned for 3,000 optimiser steps with a global batch of 360; REKI is tested at K=3 and K=1, while π0.5 has no memory.Their action chunks are 30 steps for REKI and 50 for π0.5; only the last four vision-encoder blocks are trained.Source: §7 · p. 11
  • REKI outperforms π0.5 by 7 to 37.5 percentage points across the five real-robot tasks.Removing memory leaves shorter-horizon task performance unchanged (65 against 65 and 57.5 against 57.5), but lowers performance on longer-horizon tasks by 12 to 17.5 percentage points.Source: §7 · p. 11

8 DISCUSSION

REKI is a public-data, single-stage VLA intended as an open reference for studying architecture, data, and training choices. Its memory and efficient release support benchmarking, while the adequacy of short memory for tasks requiring distant past events remains untested.

8 DISCUSSION

  • REKI competes with and often outperforms similarly sized VLAs pretrained on in-house data, and leads compared public-data VLAs out of distribution.Pretraining adds 13.8 points on Clean2Rand and 4.3 on LIBERO-Plus; REKI uses no proprietary data, world model, or additional training stages.Source: §8 · p. 12
  • The released pipeline and benchmark checkpoints make REKI an open reference for controlled comparisons, with finetuning costing 8 H100 GPU-hours on real-robot tasks and 40 to 400 on benchmarks.Memory adds no training cost and less than 1% of inference time; the released code predicts a LIBERO-input action chunk in 21.8 ms on one H100.Source: §8 · p. 12
  • At 21.8 ms per chunk, REKI is presented as an open starting point for post-training that repeatedly queries policies during rollouts.The paper notes that inference time is multiplied across reinforcement-learning rollouts in simulation or learned world models.Source: §8 · p. 12
  • Three-frame memory raises success by 1.5 to 5.5 points across LIBERO-Plus, RoboTwin, and RoboCasa365, and by 12 to 17.5 points on longer real-robot tasks.Most gains come from one past frame; whether registers carry much longer context for events far in the past remains to be tested.Source: §8 · p. 12

A MODEL DETAILS

  • The prompt is a structured user turn with one field per line, including embodiment, instruction, speed, frame rate, and control mode.The speed field is omitted at evaluation and encodes episode length in frames divided by 100 and rounded.Source: A MODEL DETAILS · p. 21
  • Register i occupies slot i and action step j occupies slot nr + j, giving each frame’s registers their own temporal encoding.The frame and slot axes reuse the VLM’s temporal and spatial axes for multimodal rotary embedding.Source: A MODEL DETAILS · p. 21

A.1 INFERENCE COST AND CACHING

At inference, REKI computes register tokens for each frame once and caches their keys and values across action-chunk Euler steps. The cache keeps recent register context available with a small measured latency and memory cost.

A.1 INFERENCE COST AND CACHING

  • One chunk uses an encoder pass followed by N Euler steps; the first step also computes the current frame’s registers.Across the steps, the encoded context and its projected keys and values, current-frame register keys and values, and prior-frame register keys and values are reused.Source: §A.1 · p. 21
  • At inference, the first registers in a window carry memory from earlier registers, whereas training has no such carried-over memory.Figure 10 shows current-frame registers reading context and cached past registers; action tokens read the registers, and past frames are not encoded again.Source: §A.1 · p. 21, Figure 10 · p. 22
  • At K=3, a chunk takes 22.8 ms, versus 22.1 ms without registers; even K=32 takes 24.6 ms.At K=3, the cache uses 4 MiB on a 4.7 GiB policy.Source: §A.1 · p. 22

B.1 ROBOT EMBODIMENTS

  • Robot datasets may use different action conventions for the same robot, so state projections follow state layout and control convention, and sources retain their own action statistics.Joint-delta actions from Berkeley RPT and RoboSet are converted to end-effector deltas by forward kinematics.Source: §B.1 · p. 23

B.2 DATA CURATION

The appendix describes curation and sampling choices for heterogeneous robot data, as well as the construction of egocentric action targets and the vision-language data pool. Sampling weights account for frame counts and language and vision diversity.

B.2 DATA CURATION

  • Robot-data curation addresses heterogeneous noise by selecting commanded channels, discarding static recordings, detecting sudden jumps, harmonising rotations, normalising action dimensions, and correcting sign conventions.Whole episodes are removed for sudden jumps only from InternData-A1, where simulator collisions cause them.Source: §B.2 · pp. 23–24

B.3 SAMPLING WEIGHTS

  • Dataset sampling weights use frame counts and language and vision diversity, with additional downweighting for RoboCOIN and InternData-A1 and a cap of four exposures per dataset.The two diversity measures are effective ranks of embedding sets; datasets from the same source are grouped before weighting.Source: §B.3 · p. 24

B.4 EGOCENTRIC DATA

  • Egocentric targets encode each hand’s wrist motion and 10 articulation coefficients, and are composed relative to the frame opening each action chunk.Because egocentric data lack proprioception, the policy uses a learned null state; human motion is slowed using a per-source pace.Source: §B.4 · p. 24

B.5 VISION–LANGUAGE DATA

  • The vision-language pool retains 60 datasets and 7.64M samples, covering manipulation-relevant objects, spatial relations, counting, grounding, captioning, scene text, and knowledge and commonsense VQA.It samples one question–answer pair per sample, uniformly over samples; additional pointing and counting questions are made from RoboPoint, RefSpatial, and PixMo-Points splits.Source: §B.5 · p. 25

C TRAINING CONFIGURATION

REKI’s training setup combines public robot and egocentric data with camera and image augmentations, optimization settings, and optional mirroring. On Clean2Rand, longer pretraining improves mean success, with the largest early gains under clutter and the Hard condition.

C.1 CAMERA SELECTION

  • Pretraining draws on robot corpora and egocentric human-video datasets, with sampling weights based on frame count, language diversity, and vision diversity.The listed robot sources include OXE, DROID, AgiBot, RoboMIND, and others; the human-video sources are EgoDex, EgoVerse, and Xperience.Source: §C.1 · p. 26
  • Camera dropout, shuffled camera order, colour jitter, and random crops vary visual inputs; pretraining camera rules retain at least one non-overhead camera.The pretraining rules remove redundant side cameras and, on two-arm robots, the wrist camera of a static arm.Source: §C.1 · pp. 25–26
  • Optimization uses a 1×10−4 expert learning rate, a 1×10−5 to 3×10−5 VLM rate, gradient clipping at 1.0, and EMA with decay 0.999.The schedule includes warmup and, for finetuning, decay over the final 5,000 steps.Source: §C.1 · p. 26

C.2 MIRRORING

  • Mirroring is used in RoboTwin, RoboCasa365, and RoboCasa-GR1 finetuning, reflecting images, state, and actions and swapping arms for two-arm robots.The mirrored sample prompt is marked, and its state uses a separate projection and normalization; evaluation is in the unmirrored world.Source: §C.2 · pp. 26–27

C TRAINING CONFIGURATION

  • Table 3 specifies pretraining and five finetuning-run setups, evaluation rollouts, and separate Clean-only versus Clean-plus-Randomised RoboTwin runs.The table’s starred sampling weight is multiplied by language- and vision-diversity terms.Source: C TRAINING CONFIGURATION · p. 25, Table 3 · p. 27

D FULL BENCHMARK RESULTS

  • The benchmark comparisons cover LIBERO-Plus perturbations, RoboTwin in-distribution and Clean2Rand settings, and RoboCasa365’s 50-task multi-task setting.LIBERO-Plus Overall pools 10,030 perturbed episodes weighted by dimension episode counts; Clean2Rand tests clean demonstrations against environment randomisations.Source: D FULL BENCHMARK RESULTS · p. 27, Table 4 · p. 28, Table 5 · p. 28

D.1 PRETRAINING STEPS

  • 53.7 without pretraining rises to 61.9 after 50k steps, then gains 3.4 points more by 200k on RoboTwin Clean2Rand.The first 50k steps add 19.5 points under clutter and 11.7 points under the Hard condition.Source: §D.1 · p. 27

D.2 FINETUNING ABLATIONS

Finetuning ablations show that mirroring has task-dependent effects, visual augmentation supports camera robustness, and state dropout supports robustness to initial robot state. The compared training budgets do not establish compute-matched success rates, and inference timing follows a specified single-GPU protocol.

D.2 FINETUNING ABLATIONS

  • On RoboCasa365, mirroring adds 4.9 points on average, while on LIBERO-Plus it improves four dimensions but lowers Overall by 0.6.The LIBERO-Plus camera dimension drops by 6.5 points; the proposed explanation is that viewpoint changes may confuse inference about whether the world is mirrored.Source: §D.2 · p. 28, Table 8 · p. 30
  • Disabling camera dropout or image augmentations lowers the LIBERO-Plus camera-dimension score by 21.5 and 23.6 points, respectively.These augmentations also yield small gains on sensor-noise and object-layout dimensions; the text suggests they improve camera-viewpoint robustness.Source: §D.2 · p. 28
  • Disabling state dropout lowers the LIBERO-Plus robot dimension by 8.1 points, consistent with its stated role in robustness to initial robot state.Source: §D.2 · p. 29
  • Lowering the backbone learning rate from 0.3 to 0.1 times the expert rate lowers LIBERO-Plus Overall by 3.3 points.The largest dimension drops are camera by 11.9 points and robot by 4.6 points.Source: §D.2 · p. 29

D.3 TRAINING BUDGETS OF THE COMPARED METHODS

  • Published training budgets omit several methods, and none reports wall-clock time or accelerator-hours, so success rates are not matched for training compute.The table reports optimiser steps, global batch size, and samples seen where available.Source: §D.3 · p. 29

D.4 INFERENCE LATENCY OF THE COMPARED METHODS

  • Inference timing uses released checkpoints and code on one H100 at batch size one, with synthetic seeded observations and separate warm-up and timed chunks.The reported end-to-end timing runs from numpy observations to unnormalised numpy actions; compiled-path results indicate what models can reach, not necessarily typical user experience.Source: §D.4 · pp. 29–30

E.1 TASKS AND SCORING

Real-robot evaluation covers five manipulation tasks, scoring partial subtask completion while requiring correct order for full success. REKI and π0.5 are finetuned on these tasks, with rollout examples reported for REKI.

E.1 TASKS AND SCORING

  • The five tasks range from shorter single-object actions to ordered manipulation of two or three objects, where tracking completed steps matters.Partial credit is assigned for completed subtasks, but full success requires every subtask in the correct order.Source: §E.1 · p. 30

E.2 FINETUNING DETAILS

  • REKI and π0.5 start from pretrained checkpoints and are jointly finetuned on the five tasks using the real-robot experiment hyperparameters.REKI uses a 30-step, 2-second action chunk; π0.5 uses 50 steps, or 3.3 seconds, fixed by its pretraining.Source: §E.2 · p. 31, Table 11 · p. 32
  • Figure 12 presents one REKI rollout from start to end for each of the five tasks.Source: §E.2 · p. 31, Figure 12 · p. 32

Related papers

Loading 2610.22178v1…