Source-linked AI summary
RouteRLT: Learning When and Which RL Specialist Should Control a Vision-Language-Action Policy
Chongyu Zhu, Jaden Hinds, Hyegang Kim, Juan Sebastian Rojas, Ramy Elmallah, Chi-Guhn Lee
TL;DR
Precision-critical stages challenge generalist VLAs, while deciding when and which specialist should act without losing generalist behavior remains an open question. RouteRLT learns controller ownership from the frozen VLA and stabilizes handoffs across chunked outputs. It improves over the generalist in simulation and hardware, matching privileged-boundary routing in simulation.
Problem
Generalist VLAs can struggle during precision-critical stages, and it remains unresolved how to preserve their broader behavior while deciding when refinement is needed and which specialist should act.
Source: §I · pp. 1–2Method
RouteRLT predicts controller ownership from frozen VLA representations and combines a phase selector, switch stabilizer, and action-boundary manager for a bank of RL specialists.
Source: §I · p. 2Results
85.00% to 92.22% full-task success in simulation and 6.7% to 35.0% full-trajectory insertion success on hardware over the generalist baseline; simulation routing matched privileged boundaries.
Source: §I · p. 2Takeaways & Limitations
The results support learned routing as a practical way to retain generalist behavior outside critical phases while applying specialized control where additional precision is needed.
Source: §VI · p. 8The hardware evaluation covers one robot, connector, and port geometry, and the operator-aligned handoff does not establish fully autonomous multi-specialist composition.
Source: §V · p. 7
Abstract
from arXiv · showhide
Vision-language-action (VLA) models provide broad manipulation competence, but often struggle during the precision-critical stages that dominate contact-rich industrial tasks such as connector insertion and cable management. A common remedy is to refine a pretrained VLA with reinforcement learning (RL), enabling task-specific improvement beyond behavior cloning. However, how to preserve its generalist behavior while deciding when RL refinement is needed and which specialized policy should act remains an open question. In this work, we present RouteRLT, a routing framework that learns when and which RL specialist, an RL policy trained for a single precision-critical phase, should take control from a generalist VLA. A phase selector identifies the active controller, a stabilizer suppresses transient switches, and an action-boundary manager handles transitions between chunked policy outputs. We evaluate RouteRLT on multi-object pick-and-place tasks in LIBERO, as well as on a real-world cable pickup and port-insertion task with multiple precision-critical stages. In simulation, the learned routing improves over the base VLA and matches routing with privileged phase boundaries, without accessing those boundaries at deployment. The real-robot evaluation validates automatic routing to both the pickup and insertion specialists under an operator-aligned handoff protocol. Altogether, these results show that learned routing applies RL specialist control where precise adaptation is most valuable while preserving generalist VLA behavior, including recovery from failed execution attempts.
I. INTRODUCTION
Precision-critical failures can dominate contact-rich tasks even when a generalist VLA handles the rest of the trajectory. RouteRLT learns when to hand control to RL specialists while retaining the generalist outside those phases.
I. INTRODUCTION
- Failures in contact-rich tasks can concentrate in a small fraction of the trajectory, motivating RL specialists trained for individual precision-critical phases.Local specialization avoids spending scarce resources on segments the generalist already performs well and reduces the risk of overwriting its long-horizon behavior.Source: §I · p. 1
- Prior refinement methods set the scope using operator interventions or privileged simulation state unavailable at test time, whereas RouteRLT predicts ownership from the generalist’s internal state.The prediction is made at every control query and can be honored mid-chunk.Source: §I · pp. 1–2
- RouteRLT combines a phase selector, a stabilizer for transient switches, and an action-boundary manager that makes accepted ownership changes take effect immediately.The manager invalidates the unexecuted suffix of a chunk rather than waiting for the next replanning boundary.Source: §I · p. 2
- The framework extends to multi-object tasks and trajectories with multiple precision-critical stages, where routing must choose among several controllers.Its evaluations span simulation and a real-robot cable pickup and insertion task.Source: §I · p. 2
- In cable pickup and insertion, RouteRLT raised simulation full-task success from 85.00% to 92.22% and hardware full-trajectory insertion success from 6.7% to 35.0% over the generalist baseline, matching privileged-boundary routing.Source: §I · p. 2
II. RELATED WORK
The related work covers VLA policies, RL refinement, and controller handoffs; RouteRLT addresses handoffs between a pretrained generalist and separately trained, phase-specific RL policies.
A. Vision–Language–Action Models
- SmolVLA is the frozen generalist backbone, representing policies whose behavior-cloned action heads can struggle with precision-critical stages.The paper uses its compact footprint to keep online specialist training within a modest compute budget.Source: A. Vision–Language–Action Models · p. 2
B. RL Refinement of VLA Models
- VLA refinement methods update the policy, adapt a frozen policy’s latent space, or re-rank sampled actions; RLT instead uses a lightweight actor–critic for a predefined critical phase.RLT’s control scope remains fixed by a signal supplied during data collection.Source: B. RL Refinement of VLA Models · p. 2
- Prior controller-handoff work differs in its trigger signal, target controller, and decision resolution; RouteRLT decides at router queries and applies accepted switches at the next action.Priv. denotes a trigger using privileged or operator-supplied state unavailable at deployment; Res. denotes decision temporal resolution.Source: B. RL Refinement of VLA Models · p. 2
C. Generalist–Specialist Control
- RouteRLT studies handoffs between an existing generalist VLA and separately trained, phase-specific RL policies, unlike systems coordinating jointly trained experts.The cited work includes interactive imitation learning and multi-expert coordination for locomotion, manipulation, or humanoid motion skills.Source: C. Generalist–Specialist Control · p. 2
III. ROUTERLT
- RouteRLT learns when and which RL specialist takes control through a phase selector, stabilizer, and action-boundary manager.The framework handles transitions between chunked policy outputs.Source: §III · p. 2
A. Problem Formulation
RouteRLT uses a compact latent representation from a frozen VLA to identify controller ownership, mapping annotated phases to the generalist or a specialist during training.
A. Problem Formulation
- The frozen VLA encodes synchronized camera views, proprioception, and a language instruction before its action expert proposes an H-step action sequence.The proposal is queued in chunks, and the VLA and RL specialists may use different execution horizons.Source: A. Problem Formulation · pp. 2–3
- A learned token appended to the image–language prefix yields a compact latent state used by both the phase selector and the specialists.The encoder is frozen after representation learning; proprioception and the VLA reference action are provided only to specialists.Source: A. Problem Formulation · pp. 2–3
- The fixed phase-to-controller map sends non-critical phases to the VLA and each critical phase to its specialist, while the selector predicts controller probabilities from the latent.Training pairs annotated phases with controller labels; the objective is to recover privileged phase-routing performance without phase labels or privileged task state at deployment.Source: A. Problem Formulation · p. 3
B. RL Specialists
The RL specialists learn phase-specific chunked actions while balancing predicted return against deviation from the VLA proposal and smoothing consecutive actions.
B. RL Specialists
- Each specialist emits a C × da action chunk sampled from a diagonal Gaussian and conditioned on the base VLA’s reference chunk.During training, the reference is zeroed with probability pdrop to discourage the actor from collapsing into a perturbation of that reference.Source: B. RL Specialists · p. 3
- The actor balances predicted return against deviation from the VLA proposal and adds a penalty on variation between consecutive actions.The return and reference coefficients can be scheduled so the specialist is anchored early and can depart from the reference once its critic is informative.Source: B. RL Specialists · p. 3
- Specialists are trained offline-to-online with replay mixing prior trajectories and fresh experience, while the VLA remains frozen.Online fine-tuning uses an update-to-data ratio of 5, after warm-up rollouts that execute the VLA reference.Source: B. RL Specialists · p. 3
- Chunk-level transitions cover up to the next C executed actions, including overlapping windows and windows crossing a replanning boundary.Targets accumulate realized discounted reward and bootstrap with γ^L; tasks use sparse terminal success reward without shaping.Source: B. RL Specialists · p. 3
C. Latent-Space Phase Router
The latent-space router predicts controller ownership from compact VLA states, stabilizes phase decisions, and selects its classifier and stabilizer using validation data. At deployment, it updates controller predictions at fixed intervals and handles controller changes at action-chunk boundaries.
C. Latent-Space Phase Router
- The router uses only compact states as input, forming causal windows of two latents for its phase selector.The selector shares a projection layer across window positions and is trained with class-balanced loss.Source: C. Latent-Space Phase Router · pp. 3–4
- At deployment, the router updates predictions every query interval, stabilizes controller choice, clears queued actions on a change, and executes the front action.Router fitting uses training data and validation selection before the selected classifier and stabilizer are frozen.Source: C. Latent-Space Phase Router · p. 4
- For multiple specialists, the selector outputs a probability distribution over the full controller bank rather than a binary specialist-versus-VLA choice.The binary form applies when one specialist competes with the base VLA.Source: C. Latent-Space Phase Router · p. 4
- A three-phase stabilizer uses entry and exit thresholds with dwell counters to suppress transient controller switches.Its occupancy minimum constrains exits but does not delay entry.Source: C. Latent-Space Phase Router · p. 4
- Classifier weights and stabilizer parameters are selected using validation statistics before test episodes, with routing constraints and fewer switches as a tie-breaker.The classifier uses early stopping on validation AUPRC; weights and stabilizer parameters are fit per seed.Source: C. Latent-Space Phase Router · p. 4
D. Boundary-Aware Chunk Execution
The action-boundary manager makes controller changes effective within chunked execution by discarding the outgoing controller’s unexecuted actions and querying the incoming controller immediately.
D. Boundary-Aware Chunk Execution
- When controller ownership changes mid-chunk, the manager invalidates the unexecuted suffix and queries the incoming controller immediately.A VLA handoff queues the first C actions of a fresh proposal; a specialist handoff obtains a fresh VLA proposal from the same observation.Source: D. Boundary-Aware Chunk Execution · p. 4
- The environment executes only the front action, and specialist replay stores executed actions with the selected controller and next compact state.Evaluation episodes are checked for queued actions from superseded plans or inactive controllers.Source: D. Boundary-Aware Chunk Execution · p. 4
- The replay buffer keeps action ownership and learning targets consistent across controller transitions.Its transitions include the selected controller and next compact state.Source: D. Boundary-Aware Chunk Execution · p. 4
E. Training Protocol
Training proceeds from a demonstration-fine-tuned VLA through representation learning and specialist training, then uses frozen specialists to gather routing labels under privileged routing.
E. Training Protocol
- The base VLA is first fine-tuned on task demonstrations.Source: E. Training Protocol · p. 4
- The representation encoder is trained to reconstruct the frozen VLA prefix, then frozen to define the compact state.Source: E. Training Protocol · pp. 4–5
- Each specialist is fit offline on collected trajectories and fine-tuned online under its privileged phase boundary using mixed offline and online replay.Source: E. Training Protocol · p. 5
- After specialist training, the specialists are frozen and used to collect labeled routing data under privileged routing.Source: E. Training Protocol · p. 5
IV. EXPERIMENTS
On LIBERO Object, RouteRLT improves full-task performance while matching privileged-boundary routing without deployment-time phase information. The experiments also assess pickup performance, interaction efficiency, representation inputs, and control scope.
A. LIBERO Simulation
- Across 60 held-out episodes, RouteRLT was evaluated on three LIBERO Object tasks with matched initial conditions; the router was queried every environment step.The evaluation used a 300-step horizon and measured full-task success, pickup success, and successes per 1,000 steps.Source: A. LIBERO Simulation · p. 5
- At step 189, RouteRLT completes a held-out pudding pickup-and-place rollout in which SmolVLA@10 fails to secure the object and reaches the 300-step limit.The paired rollout uses identical initial conditions and matched environment steps.Source: Figure 4 · p. 5, A. LIBERO Simulation · p. 5
- The router reached every target pickup interval with entry recall 1.0 and zero-step median delay; 10 of 14 failures occurred because pickup never completed.Near-boundary entries were often successful: 47 of 49 early entries succeeded, while only eight episodes had a genuinely spurious entry or re-entry.Source: A. LIBERO Simulation · pp. 5–6
- On LIBERO multi-object pick-and-place, RouteRLT reaches 92.22%, retaining the specialist’s benefit while achieving the highest successes per 1,000 steps; learned routing recovers privileged-routing performance without longer trajectories.Source: A. LIBERO Simulation · p. 5, Table II · p. 6
- Image-only inputs preserved most pickup success but weakened full-task success and interaction efficiency; adding language tokens improved object identity in the compact state.The comparison indicates visual cues identify pickup timing, but language is needed to reliably retain the instructed object identity.Source: A. LIBERO Simulation · p. 6
- A single full-trajectory specialist improved over SmolVLA but remained weaker than RouteRLT in full-task success, pickup success, and interaction efficiency.The authors attribute the difference to the single specialist distributing capacity and sparse full-task reward across phases, including phases where the VLA is already strong.Source: A. LIBERO Simulation · p. 6
B. Trossen Real-Robot Insertion
On hardware, RouteRLT composes pickup and insertion specialists with the VLA under an operator-aligned handoff protocol. Learned routing reaches final insertion performance comparable to the oracle, while isolated tests show gains from both specialists and a remaining composition gap.
a) Evaluation protocol
- Full-trajectory evaluation uses pickup specialist π1 for grasping, insertion specialist π2 for alignment and seating, and SmolVLA for approach, transport, and retraction.Insertion execution begins after operator alignment, although specialist selection remains automatic.Source: B. Trossen Real-Robot Insertion · p. 6
b) End-to-end results
- The wrist-camera sequence marks handoffs from the VLA to π1 and π2, then returns control to the VLA after pickup and seating.These are annotated controller boundaries in a held-out router-validation trajectory.Source: Figure 5 · p. 7
- 35.0% final insertion with RouteRLT and 33.3% with Oracle-routed RLT compare with 6.7% for SmolVLA.RouteRLT reaches the fixture in 65.0% of trials, versus 80.0% for the oracle; conditional seating is 53.8% versus 41.7%.Source: B. Trossen Real-Robot Insertion · pp. 6–7
c) Specialist capability and composition gap
- The pickup specialist reaches 83.3% success versus 0.0% for SmolVLA@10, while the insertion specialist reaches 62.1% versus 8.0%.These isolated evaluations start from standardized phase-entry states with the router bypassed.Source: B. Trossen Real-Robot Insertion · p. 7, Figure 7 · p. 7
- After transport and routing are added, the connector reaches the fixture in 65.0% of trials, and 53.8% of those arrivals lead to seating.Composed performance is not the product of isolated specialist rates because each specialist receives states produced by the preceding controller; the remaining gap is attributed primarily to upstream transport and handoff-state consistency.Source: B. Trossen Real-Robot Insertion · p. 7
V. LIMITATIONS
The study validates modular routing, not general-purpose manipulation, and its evidence is bounded by the evaluated backbones, tasks, and hardware setting.
Scope of evaluation
- The study uses SmolVLA and does not compare RouteRLT with full-backbone online fine-tuning; stronger VLAs or full-model adaptation may change specialization’s relative benefit.The LIBERO setting is near saturation, limiting its ability to distinguish routing variants.Source: §V · p. 7
- Evaluation covers one RL specialist in simulation and two on hardware; routing larger controller banks with less distinct phase boundaries remains untested.Source: §V · p. 7
Hardware scope
- Hardware results are limited to one robot, connector, and port geometry, with grasp pose an uncontrolled source of insertion variation.The operator-aligned handoff does not establish fully autonomous multi-specialist composition.Source: §V · p. 7
Training and distribution shift
- Router and specialist learning are decoupled, and both components encounter out-of-distribution states during end-to-end execution.Routing labels use privileged boundaries, while specialists train within privileged or standardized phase windows.Source: §V · p. 7
VI. CONCLUSION
RouteRLT routes between a frozen generalist VLA and RL specialists using frozen VLA representations, stabilizing transitions and applying ownership changes across chunked outputs. It matches privileged-boundary routing on multi-object LIBERO tasks and achieves comparable final insertion to the oracle on hardware under an operator-aligned handoff protocol.
VI. CONCLUSION
- RouteRLT predicts controller ownership from frozen VLA representations, stabilizes phase transitions, and applies ownership changes immediately across chunked policy outputs.Source: §VI · p. 7
- On multi-object LIBERO tasks, learned routing preserves the gain from localized RL control and matches routing based on privileged phase boundaries.Source: §VI · p. 7
- On hardware, RouteRLT automatically selects pickup and insertion specialists and achieves final insertion performance comparable to Oracle-routed RLT under the operator-aligned handoff protocol.The results support learned routing as a way to retain generalist behavior outside critical phases while applying specialized control where additional precision is needed.Source: §VI · p. 8
- Future work will evaluate stronger VLA backbones and broader hardware geometries, and jointly adapt router and specialists to reduce handoff distribution shift.The authors also identify fully autonomous multi-specialist composition as a future goal.Source: §VI · p. 8