PaperlayerView exact revision on arXiv

Source-linked AI summary

REFACTOR-VLA: Unsupervised Library Learning of Typed Motor Programs

Riyaaz Shaik, Chandru Venkataraman

arXiv:2609.01215v1cs.LGcs.AIcs.RO

TL;DR

Existing VLA skill discovery lacks a dynamics-aware account of when continuous action sequences are behaviorally equivalent, limiting reusable abstractions for long-horizon control. REFACTOR-VLA addresses this with a wake/sleep system that clusters fragments using a world-model Behavioral-Equivalence Kernel and learns typed library programs. On LIBERO, objective shaping beats capacity scaling: InfoNCE improves clustering and yields 4-of-4 baseline wins, while a task-language library rewrites all sampled demonstrations.

  • Problem

    Existing VLA models and skill-discovery methods lack a dynamics-aware way to identify behaviorally equivalent continuous action sequences for reusable abstractions.

  • Method

    REFACTOR-VLA alternates BEK-based fragment clustering and typed-lambda library learning with wake-phase program emission and library-conditioned rectified-flow decoding, gated by MDL and return preservation.

  • Results

    InfoNCE improves BEK NMI on all four LIBERO suites and beats the strongest published baseline 4-of-4 with mean Δ= +0.184, while 430M scaling worsens NMI 4-of-4.

  • Takeaways & Limitations

    The LIBERO results identify world-model objective shape rather than parameter count as the binding lever for clustering, and the compiler produces a usable task-language library.

  • Takeaways & Limitations

    Library learning remains limited to the language slot: the motor subspace admits zero abstractions at every tested MDL threshold.

Abstract

from arXiv · show

Most vision-language-action (VLA) models -- OpenVLA, $π_0$, RT-2, RDT-1B -- are monolithic: they emit raw motor commands or short action chunks without organizing behavior into reusable abstractions, so they degrade on long-horizon tasks and resist interpretation. Existing skill-discovery methods sidestep the core question of when two action sequences are behaviorally equivalent, either clustering contrastive embeddings or delegating the judgment to a language model uncalibrated to the robot's dynamics. We introduce REFACTOR-VLA, a wake/sleep system for learning reusable skills. Its sleep phase clusters motor-program fragments under a Behavioral-Equivalence Kernel (BEK) computed from rollouts of a learned latent world model $M_φ$; its wake phase emits typed lambda terms over a Hindley--Milner-inspired vocabulary, consumed by a library-conditioned rectified-flow action decoder. Abstractions are admitted only if they pass Minimum Description Length and return-preservation gates. On LIBERO we report two findings. First, enlarging the world model from 188M to 430M parameters worsened performance on 4 of 4 suites, so capacity alone does not help. Second, the training objective matters far more: adding an auxiliary supervised contrastive (InfoNCE) loss during world-model warmup substantially improves sleep-phase clustering, giving Normalized Mutual Information at $n=3$ seeds of $0.462 \pm 0.021$ (object), $0.867 \pm 0.025$ (spatial), $0.915 \pm 0.013$ (goal) and $0.754 \pm 0.010$ (LIBERO-10), and beating the strongest published baseline on all 4 suites by a mean $Δ= +0.184$. Across providers ($n=12$) the 95% bootstrap confidence interval for mean pairwise NMI is $[0.683, 0.729]$ (mean $0.705$). The sleep phase also yields the first real-LIBERO task-language library: the decoder uses 2 of 3 admitted abstractions and rewrites all 256 sampled demonstrations.

1 Introduction

REFACTOR-VLA addresses the difficulty of discovering behaviorally equivalent skills in continuous-action robotics by combining a dynamics-aware wake/sleep library-learning framework with typed program generation. Its LIBERO results show that training-objective shape matters more than world-model capacity for sleep-phase clustering.

  • Motivation: Long-horizon manipulation is difficult because sparse rewards and fast low-level sampling make policy search and temporal credit assignment impractical.Hierarchical skill coordination can avoid some low-level control detail.
  • Problem: Existing methods sidestep dynamics-aware equivalence, potentially separating trajectories with the same physical effect and grouping trajectories with different dynamical consequences.Prior systems use expert routing, keyframe segmentation, VLM annotation, or ungrounded language-model synthesis.
  • Method: BEK extends state-level bisimulation ideas to continuous trajectory fragments using rollouts from a learned latent world model.The kernel measures behavioral similarity through value and transition-distribution information.
  • Method: REFACTOR-VLA clusters fragments with BEK, distills a Siamese encoder, and compiles common structures into typed-lambda programs admitted by MDL and return-preservation gates.The wake phase uses a Typed Program Emitter and library-conditioned rectified-flow decoder.
  • Findings: 4-of-4 suites worsened when Mϕ grew from 188M to 430M parameters, while auxiliary supervised InfoNCE during warmup produced a 4-of-4 baseline win with mean Δ= +0.184.The result challenges capacity alone as a sufficient route to better skill representations.
  • Findings: The full LIBERO evaluation reports a 95% bootstrap interval of [0.683, 0.729] across 12 providers and a first real-LIBERO task-language library.The library contains 3 abstractions / 1211 nats and is used by the continuous decoder.

2 Related Work

Prior skill-discovery systems define equivalence through gates, keyframes, language-model annotations, or symbolic structure, while REFACTOR-VLA uses a learned-dynamics divergence to cluster physical trajectory fragments. It then combines that clustering kernel with typed program induction and wake/sleep library learning.

  • Skill discovery for VLAs: AtomicVLA, AtomSkill, BLADE, and LRLL use expert routing, VLM-specified keyframes, LLM-generated conditions, or LLM trajectory classification as their equivalence mechanisms.These methods represent distinct alternatives for discovering reusable skills.
  • Typed program induction: DreamCoder, LILO, and symbolic program induction grow typed libraries through syntactic anti-unification, which is suited to discrete token identity.That assumption does not directly resolve equivalence between physically different action paths.
  • Classical options and segmentation: Classical options and segmentation methods operate on temporally extended policies, recurrent segments, goal-conditioned skills, or tokenized subsequences.Their equivalence operators are based on single-state values or single-token segmentations.
  • Bisimulation and behavioral metrics: State-level bisimulation metrics combine reward differences with Wasserstein transition distances, but prior formulations use them as representation losses rather than clustering kernels.REFACTOR-VLA extends the divergence to trajectory fragments inserted at the same call site.
  • REFACTOR-VLA: REFACTOR-VLA uses Dϕ(τ, τ′) from learned-world-model rollouts as the sleep-phase clustering kernel for behavioral equivalence.This links dynamics-aware behavioral comparison to subsequent library compression.

3 Method

REFACTOR-VLA alternates wake control learning, sleep-phase BEK distillation, and library refactoring. The method combines a latent world model, typed program generation, rectified-flow decoding, and gated abstraction admission.

  • System overview: The system contains a VLA policy, latent world model Mϕ, typed program library Ft, and posterior ρt over library use.An outer driver alternates wake training with sleep-phase library extension.
  • Latent world model: Mϕ is a DreamerV3-style hierarchical model with a frozen DINOv2 encoder, posterior and prior heads, and feature-space reconstruction.The default model has 188.16M total parameters, with 101.58M trainable.
  • Behavioral-Equivalence Kernel: BEK combines expected-return differences with k-step latent-rollout Wasserstein distance and is used for fixed-k KMeans clustering.The method does not claim positive-semidefinite or Mercer properties.
  • BEK amortization: A Siamese amortizer replaces quadratic Monte Carlo pair evaluation with L2-normalized embeddings whose cosine distance approximates BEK.The encoder is distilled from frozen Mϕ representations without task labels.
  • BEK implementation: The default legacy_cosine BEK mode achieves higher NMI on all suites than separable_VPk while requiring about half the wall-clock time.Both modes share the distillation objective.
  • Wake phase: The Typed Program Emitter uses Hindley–Milner-style type checking over primitive, lambda, sequencing, repetition, and branching constructs.The LCAD consumes typed terms and state to produce 16-step action chunks through rectified-flow matching.
  • Wake/sleep loop: The alternating loop trains wake components, distills BEK during sleep, and admits anti-unified abstractions only after BEK soundness, return-preservation, and MDL-gain checks.The MDL threshold is 4 nats, while return preservation uses ε=0.05 and Kv=32 verifier rollouts.

4 Experiments

Experiments on the four-suite LIBERO matrix show that supervised InfoNCE improves BEK clustering and baseline comparisons, whereas scaling the world model reduces NMI despite lower training loss. The learned language library rewrites all sampled demonstrations while preserving decoder performance closely.

  • Experimental setup: The evaluation uses real LIBERO data across four suites, with 188M and 430M world-model configurations trained under the same distributed setup.The 430M model serves as a capacity-falsification probe.
  • Library-conditioned decoding: The 3-abstraction grammar rewrites 256/256 demonstrations using 2/3 abstractions, with TPE accuracy 1.000 and velocity error within ±0.02 of the no-library baseline.This demonstrates end-to-end use of the discovered language library.
  • BEK headline: The auxiliary InfoNCE objective improves every suite by mean +0.252 NMI absolute, with per-suite σ ≤0.025 and 26–98.5% recovery of the supervised upper-bound gap.The mean head-to-head gain over the strongest baseline is +0.184 NMI at n=3.
  • Cross-provider evaluation: The 12-pair cross-provider evaluation has mean NMI 0.705 with 95% bootstrap CI [0.683, 0.729], though the lower bound is 0.017 below the 0.70 gate.The point estimate clears the gate while remaining threshold-borderline.
  • Capacity probe: Scaling Mϕ from 188M to 430M reduces Phase C NMI on all four suites, including 0.285→0.245 on libero_object and 0.736→0.646 on libero_10.The larger supervisor has lower Phase A loss, so the downstream decrease is not attributed to undertraining.
  • Library learning: The sleep-phase compiler discovers 3 abstractions / 1211 nats in the language slot, while the motor-primitive subspace yields zero abstractions across five MDL thresholds.The top language abstraction matches the canonical pick-up-and-place template.
  • Baseline comparison: BEK+InfoNCE wins 4-of-4 against the strongest published baseline at n=3 multi-seed.The strongest baselines are AtomSkill for object and spatial, and AtomicVLA for goal and LIBERO-10.
  • Baseline comparison: The OpenVLA frozen-extractor cell reaches NMI 0.4094 on libero_10, 0.345 below BEK+InfoNCE at roughly 1/37 the parameter count.The comparison is reported on a clustering metric.

5 Discussion and Limitations

The results identify objective shape, rather than model capacity, as the stronger lever for BEK clustering, while exposing limits in label-free sampling, kernel design, motor abstraction, and cross-provider reliability.

  • Capacity and objective: 430M scaling lowers Phase C NMI on all 4 suites, whereas supervised InfoNCE raises it on all 4 with mean improvement Δ= +0.252.The resulting mean improvement over the strongest baselines is Δ= +0.184.
  • Label-free contrastive learning: Window-conditional positives restore libero_10 performance to NMI 0.761, while label-free InfoNCE passes all 4 suites with higher seed variance up to σ=0.055.Whole-episode positives fail on libero_10, where episodes mix pre- and post-grasp phases.
  • Kernel design: Removing the Wasserstein term cuts libero_object NMI by −0.175, while the default k=4 lies on an NMI plateau for k∈[4, 8].The separable head also trails legacy_cosine by −0.062 NMI over a six-point wE sweep.
  • Abstraction limits: Symbolic program induction admits 0 motor abstractions at every tested MDL threshold, leaving library learning limited to the Lang slot.DTW pre-alignment does not rescue motor abstraction because BEK clusters semantically while strict anti-unification requires syntactic alignment.
  • Reliability: The cross-provider mean pairwise NMI is 0.705 with 95% CI [0.683, 0.729], so the point estimate clears 0.70 but its lower bound does not.Only libero_goal cleanly passes the threshold; libero_object and libero_spatial do not.

6 Conclusion

The conclusion presents REFACTOR-VLA as wake/sleep library learning governed by BEK-based clustering and gated typed abstractions. Its LIBERO results favor objective shaping over parameter scaling, while the formal claims rely on explicit model, separation, sampling, clustering, Lipschitz, margin, and independence assumptions.

  • Conclusion: REFACTOR-VLA combines BEK-based fragment clustering, typed-lambda program generation, and MDL-gated, return-preserving library admission.Its stated result is that objective shape, not parameter count, is the binding lever.
  • Assumptions: The formal foundation assumes bounded Mϕ return error, class separation, sufficient held-out fragments, correctly specified cluster count, and BEK Lipschitzness.These conditions support the concentration and clustering arguments.
  • Scope boundary: On LIBERO, η is unmeasurable because the benchmark has no reward column, limiting empirical assessment of the return-error assumption there.The reported ηsup estimate comes from RecursivePourEnv rather than LIBERO.
  • Assumptions: The margin-density assumption requires near-boundary probe fragments to vanish as η→0, but its falsification is suite-conditional.The random-1-step null collapses NMI to 0.089 on libero_object and 0.357 on libero_10.
  • Assumptions: The theoretical setup also assumes i.i.d. probe fragments under the visitation distribution used to compute Dϕ and includes a zero-mean conditional residual assumption.The zero-mean conditional residual condition is used by the martingale refinement rather than Theorem 2.

A.2 Theorem 1 (Exact Recovery)

Theorem 1 gives exact KMeans recovery under a strict BEK separation condition, while empirical LIBERO results show that condition is not robust and rate-based analysis is used instead. The appendix further links InfoNCE shaping to improved NMI through task-discriminative latent geometry, while noting a weaker guarantee for Wasserstein dynamics information.

  • Exact recovery: Under Δmin > 4ε∗+4ζ, fixed-k KMeans recovers the true equivalence classes with probability at least 1−δ for n≥n0(δ).The result assumes the strict separation hypothesis and assumptions (A1)–(A2).
  • Exact recovery: The proof uses within-class concentration, between-class separation, and Hoeffding control of empirical centroid distances.The KMeans objective is uniquely minimized by the true partition up to permutation under the gap hypothesis.
  • Empirical qualification: Empirically, the exact-recovery gap is not robustly positive on LIBERO, so Theorem 2’s rate result supports the reported NMI analysis.The multi-seed RecursivePourEnv estimate is η=0.205±0.039.
  • InfoNCE mechanism: InfoNCE improves Phase-C clustering by making task-discriminative directions explicit in the latent representation rather than incidental reconstruction features.The auxiliary loss tightens a mutual-information lower bound between zT and the task label and changes the latent geometry used for clustering.
  • InfoNCE mechanism: The Wasserstein component receives weaker direct protection because InfoNCE constrains marginals, although low distillation error indicates the BEK kernel remains well fitted.Across four InfoNCE Phase-C runs, reported distillation errors remain in the 3–13.2×10−4 band.

B.7 Honest Limitations

REFACTOR-VLA’s supervised objective depends on task_index labels available in LIBERO but absent from the open-corpus setting it ultimately targets. Its empirical claims also leave residuals and boundary conditions around objective transfer, model capacity, and cross-provider robustness.

  • Label availability: task_index labels are available in LIBERO but will not be available for the open-corpus Phase-A pretraining setting targeted by REFACTOR-VLA.The paper proposes self-supervised temporally adjacent fragments and VLM-emitted pseudo-labels as substitutes; the former recovers 50–82% of the supervised gap on shorter suites but fails on libero_10.
  • Label availability: The label-free episode_contrast substitute recovers 50–82% of the supervised gap on three shorter suites but fails on libero_10.The paper attributes this boundary to whole-episode positives straddling phase boundaries on the longest-horizon suite.
  • Causal isolation: The Phase-A to Phase-C L2 distillation may not capture prior-shaping effects, and the residual has not been isolated experimentally.The reported 4/4 multi-seed win is described as consistent with the residual being small, not as eliminating it.
  • Capacity and objective: The 430M LR-retuned model achieves lower Phase-A loss than the 188M baseline yet regresses on all four LIBERO suites at fixed objective shape.The retuned run reaches 0.3555 versus 0.4176 on libero_object, while the downstream capacity comparison remains negative across 4/4 suites.
  • Capacity and objective: At fixed capacity, adding InfoNCE lifts all four suites, whereas the 430M scale-up alone regresses across all four suites.Figure 3 presents this as the paper’s capacity-versus-objective twin claim.

C.5 Empirical η on RecursivePourEnv (5 Seeds)

RecursivePourEnv supplies the only ground-truth-return environment for empirically measuring the world model’s sup-norm return error. Across five model-init seeds, the estimate is variable enough to make the original bound formally valid but numerically loose, while error increases with recursion depth.

  • Five-seed estimate: ηsup = 0.205 ± 0.039 across five model-initialization seeds on the 1400-fragment RecursivePourEnv held-out set.RecursivePourEnv is the only submitted environment with ground-truth returns; LIBERO has no reward column for measuring η.
  • Return bound: (ε + 2η)/(1 −γ) ≈46.0 at ε = 0.05 and γ = 0.99 for normalized rewards in [0, 1].The bound is formally meaningful but numerically loose; tightened distributional and martingale versions reduce it to approximately 15 and 7.
  • Uncertainty reporting: The five-seed mean ± standard deviation replaces a degenerate single-seed bootstrap interval for ηsup.Bootstrapping a sup statistic cannot exceed the original maximum, causing the earlier upper quantile to saturate at the point estimate; the per-seed range was [0.173, 0.269].
  • Depth dependence: η4 −η1 = +0.094, exceeding every per-depth standard deviation and indicating increasing return error with recursion depth.The paper connects this gradient to deeper recursion propagating more world-model residual through rolled-out values.

D Ablation Matrix A1–A8

The ablations test the BEK kernel, parser and gating choices, clustering robustness, and cross-provider stability. Results show that dynamics-aware distances and InfoNCE supervision are central, while parser and MDL gates control library formation and growth.

  • Kernel and representation: 0.013 ± 0.004 NMI without BEK versus 0.462 ± 0.021 with BEK, showing that non-dynamics distance fails to recover task identity.The absolute effect size is −0.449 NMI.
  • Parser and emitter: 0 abstractions without the parser versus 3 abstractions at 1211 nats with the parser, while TPE collapses to one primitive per task.The parser is required for non-degenerate sleep-step abstraction discovery in the real-LIBERO sweep.
  • Admission control: 47 candidates without the gate versus 3 with the gate, while held-out NMI remains statistically indistinguishable from baseline within ±0.012.The gate limits library growth and raises description-length-per-demo by 1.8× when removed.
  • BEK components: 0.110 NMI from the value-difference component alone versus 0.462 ± 0.021 baseline, indicating that the W2 term supplies the primary signal.Because LIBERO lacks reward annotations, value differences alone are uninformative.
  • Clustering sensitivity: 0.089, 0.418, 0.462, and 0.451 NMI for k ∈ {1, 4, 8, 16}, with performance plateauing at k ∈ [4, 8].The headline runs use k = 8.

F.5 Methodological Considerations

The methodological analysis identifies scope, sampling, supervisor, and synthetic-evaluation boundaries alongside supplementary library-learning results. These qualifications make the cross-provider and compositionality claims narrower than an unrestricted generalization.

  • Cross-provider protocol: Each cross-provider seed uses 2000 BEK steps rather than the 4000-step single-seed Phase-C schedule, so the resulting NMI values are not directly comparable.The disjoint-thirds partition also provides less task coverage per seed than single-seed runs.
  • Supervisor scope: The protocol shares one 188M libero_object-trained world-model supervisor across all four suites, leaving per-suite supervisors as an untested extension.Changing supervisors would alter the interpretation of the cross-provider test.
  • Statistical framing: The combined n = 12 mean is 0.7046 with 95% CI [0.6826, 0.7294], so the strict 0.70 lower-CI threshold is not met.The mean exceeds 0.70, but the combined confidence interval remains threshold-borderline.
  • Synthetic compositionality: Excluding degenerate d = 1 evaluation, BEK achieves NMI ≥0.86 for recursive pour depths d = 2/3/4.The d = 1 score is a placeholder because the evaluation slice contained only one label.
  • Label-free contrastive alternative: episode_contrast recovers 50–82% of the supervised gap on three shorter suites but fails on libero_10_image.Whole-episode positives can straddle phase boundaries on the longest-horizon suite.
  • Library learning: 3 abstractions at 1211 nats produce a real-LIBERO task-language library whose decoder uses 2 abstractions and rewrites 256/256 sampled demonstrations.This is the reported end-to-end library-plumbing result.
  • Mechanism alternatives: Four of five mechanism-alternative interventions fail to recover the 430M+InfoNCE embedding, while only the InfoNCE intervention rescues it.The alternative evidence is reported in Table 13.
  • Motor-primitive scope: 0 motor-primitive abstractions are admitted across all tested MDL thresholds because the typed-lambda vocabulary lacks the lifting structure needed for non-trivial anti-unification.The negative result is attributed to symbolic anti-unification rather than the BEK kernel.

H OpenVLA-7B Head-to-Head Reproducibility

The OpenVLA-7B head-to-head comparison remains incomplete because only libero_object_image is finished. The reported +0.184 comparison therefore uses the strongest closed baseline rather than a completed four-suite rerun.

  • Reproducibility boundary: The OpenVLA-7B Phase E run is complete only on libero_object_image; the remaining three suites are deferred to the camera-ready.The open rerun covers OpenVLA-7B, Octo, and Diffusion-Policy across all four suites.
  • Reported comparison: +0.184 mean improvement is measured against AtomicVLA’s closed libero_10 result of 0.584 ± 0.048 at n = 3.This comparison is the strongest closed-baseline head-to-head cell reported.

I Mechanism Alternatives at 430M (Falsifications)

At fixed 430M capacity, the mechanism alternatives isolate the training objective as the binding factor. Linear projection, nonlinear compression, state reconstruction, and smoothness regularization all fail to reproduce the InfoNCE result.

  • InfoNCE rescue: 0.9207 ± 0.016 NMI is achieved by the 430M+InfoNCE intervention, the only one surpassing the 188M+InfoNCE anchor.The result is measured across 3 seeds on libero_object_image.
  • Linear alternative: 0.193 NMI with PCA-64 falsifies the hypothesis that a clean linear skill basis is already recoverable from the 430M activations.The PCA probe replaces the contrastive head with a fixed 64-dimensional basis.
  • Nonlinear alternative: 0.218 NMI with a VAE-64 bottleneck shows that nonlinear compression without contrastive supervision is insufficient.The result is two orders of magnitude below InfoNCE.
  • Reconstruction alternative: 0.152 ± 0.009 NMI with state-vector reconstruction shows that reconstruction-target shape matters, not merely the presence of reconstruction.The associated teacher posterior becomes nearly constant, making distillation loss trivially zero.
  • Smoothness alternative: 0.124 NMI with VIB plus spectral normalization indicates that transition smoothness erases discrete-skill structure.The analysis characterizes smoothness as an inappropriate inductive bias in this setting.
  • Mechanism conclusion: Four of five alternatives fail, supporting training-objective shape rather than model capacity as the binding lever.The surviving intervention is InfoNCE applied to the image-reconstruction Phase-A posterior.
Loading 2609.01215v1…