Source-linked AI summary
UniFS: Unified Fast-to-Slow Hierarchical Architecture for Vision-Language-Action Models
Lin Sun, Zhiwei Guan, Conglin Wang, Zihong Chen, Jianhai Yu, Zongsheng Li, Boyong He, Tao Sun, Jiale Cao, Lige Liu
TL;DR
VLA models face latency and interaction limits: fast-slow systems use fixed exchange frequencies and final-layer representations. UniFS provides multi-timescale VLM representations to the action expert and caches slow pathways. On LIBERO, it improves performance while reducing average inference latency.
Problem
VLA backbones constrain control frequency, and fast-slow systems restrict action experts to final-layer representations exchanged at a fixed frequency.
Source: §1 · p. 2Method
UniFS generates multi-timescale latent representations from a single VLM for the action expert and uses multi-level losses averaged across frequency levels.
Source: §1 · p. 3, §3.3 · p. 10Results
2.5% performance improvement on LIBERO accompanies a reduction in average inference latency from 36.5 ms to 17.8 ms.
Source: §1 · p. 4Takeaways & Limitations
Slow-frequency pathways can be cached and preserve historical dependencies as implicit memory without explicit memory buffers.
Source: §1 · p. 3Fast-to-slow architecture and latent vector inversion introduce distribution shifts that may interfere with pre-trained semantic knowledge, requiring careful fine-tuning.
Source: §5 · p. 15
Abstract
from arXiv · showhide
Mainstream Fast-Slow dual system vision-language-action models decouple a high-frequency action expert from a low-frequency vision-language model for efficiency, yet they face a fundamental frequency dilemma: large update gaps cause semantic drift from stale context, while small gaps erode the intended computational savings. Moreover, because the action expert receives only the VLM's final-layer representation at a single fixed frequency, rich intermediate features are discarded, limiting both information coupling and manipulation precision. Inspired by multi-timescale neural processing in the human brain, we introduce UniFS, a unified fast-to-slow architecture that resolves these challenges through three key designs. First, we stratify the VLM layers into groups with progressively decreasing update frequencies, enabling shallow layers to capture fast-changing dynamics while deeper layers cache stable semantic context. Second, a latent vector inversion mechanism re-routes the interaction order between multi-scale VLM features and the action expert, aligning fast-varying representations with fine-grained action decoding and slow-varying ones with coarse planning. Third, a multi-level supervision strategy enforces a coarse-to-fine learning hierarchy across temporal scales. Together, these designs enable richer cross-frequency information transfer within a single backbone, while the low-frequency pathways additionally preserve temporal context across steps. Experiments on LIBERO show that UniFS achieves state-of-the-art performance (98.3\% average success rate, a 2.5\% gain over VLA-Adapter baseline) while reducing average inference latency from 36.5~ms to 17.8~ms (2.1$\times$ speedup). Real-robot experiments on a Franka platform further validate its practical applicability. Code is opensourced at https://github.com/linsun449/UniFS.
1 Introduction
VLM backbones limit VLA control speed, while existing fast-slow systems also restrict how and when the action expert receives VLM features. UniFS varies update frequency across a single VLM’s layers to provide multi-timescale representations to the action expert.
1 Introduction
- VLM backbone latency leaves VLA end-to-end control frequencies at only 1–10 Hz, below levels typically required for stable closed-loop control in dynamic environments.Source: §1 · p. 2
- Existing dual-system paradigms rely on final-layer latents and fixed exchange frequencies, discarding multi-scale features and limiting adaptive responses to changing task demands.Source: §1 · p. 2
- UniFS generates multi-timescale latent representations directly from the VLM, varying frequency across the model rather than assigning it to separate modules.The design is motivated by layer-wise differences in how latent representations change over time.Source: §1 · pp. 2–3
- Slow-frequency pathways can be cached between inference steps, while preserving historical dependencies as implicit memory without explicit memory buffers.The unified pathways are also described as improving alignment between VLM and action-expert representations.Source: §1 · p. 3
- 2.5% performance improvement on LIBERO accompanies a reduction in average inference latency from 36.5 ms to 17.8 ms.The introduction reports the improvement and latency reduction as evidence of acceleration without sacrificing performance.Source: §1 · p. 4
2 Related Work
Related work traces VLA efficiency challenges to large VLM backbones and continuous action generation, then reviews fast-slow systems and multi-frequency latent fusion as responses.
2.1 The Rise of Large-Scale VLAs and Efficiency Challenges
- Large VLM backbones and multi-step continuous-action sampling restrict VLA inference to 5–10 Hz, below the >50 Hz threshold stated for stable real-time control.Source: §2.1 · p. 4
2.2 Fast-Slow Dual-Systems
- Fast-slow systems separate low-frequency VLM planning from high-frequency control, but fixed synchronization trades stale-context semantic drift against diminished acceleration, while module separation hinders end-to-end optimization.The cited systems use slow planning at 1–5 Hz and fast control at 20–50 Hz; shallow fusion also misses rich multi-scale temporal dependencies.Source: §2.2 · pp. 4–5
2.3 Towards Bio-Inspired Multi-Frequency Latent Fusion
- UniFS places frequency-specific pathways in one Transformer backbone, targeting deep fusion of fast dynamics and slow semantics rather than static latent fusion or segregated modules.The passage describes frequency-aware latent fusion for end-to-end VLAs as relatively underexplored.Source: §2.3 · p. 5
3 Method
UniFS generates multi-timescale VLM features for an action expert, with VLM layers updated at progressively slower frequencies. Its training and inference designs preserve frequency-specific features while reducing redundant computation.
3.1 Problem Formulation
- UniFS formulates action generation as multi-timescale VLM latents guiding an action expert, with intermediate VLM states supplying frequency-specific features.The VLM processes visual and language inputs to generate latent vectors spanning slow-to-fast features, which the expert uses to produce actions.Source: §3.1 · p. 6, §3.2 · p. 7
3.2 Framework
- UniFS stratifies VLM layers into groups with progressively slower updates and inverts latent interaction order so fast dynamics reach shallow layers and deeper features remain cacheable.The action expert fuses noisy action priors with corresponding VLM latents through cross-attention and projects refined queries into executable action chunks.Source: §3.2 · pp. 7–8
3.3 Training
- Frequency Feature Replacement preserves parallel training by computing all timestep features first, then reusing each layer’s block-start feature across its prescribed update interval.Temporal batch sampling selects ordered timesteps from a random window; multi-level losses across frequency groups encourage stable coarse plans at lower frequencies and fine motor corrections at higher frequencies.Source: §3.3 · pp. 8–10
3.4 Inference
- UniFS coordinates VLM and action-expert acceleration through cached latent vectors, yielding an overall theoretical speedup of about 2.6× while maintaining action quality and temporal responsiveness.Lower-frequency VLM layers reuse cached hidden states; in the action expert, slow semantic features enter early layers and fast sensory features enter later layers.Source: §3.4 · p. 10
4 Experiments
UniFS is evaluated on LIBERO, in component and frequency-configuration ablations, and on three real-robot tasks. The experiments report benchmark performance, inference latency, and the effects of its components and sampling frequencies.
4.1 Simulation Experiments on LIBERO
- On LIBERO-Spatial, the benchmark varies viewpoints and object positions; other suites test object generalization, goal-conditioned end states, and sequential mastery of 10 skills.LIBERO contains 130 tasks across four suites.Source: §4.1 · pp. 10–11
- 98.3% average success rate on LIBERO and a 2.5% improvement over the VLA-Adapter baseline are reported for UniFS.The benchmark spans four suites; Table 1 reports success rates across them.Source: §4.1 · pp. 11–12, Table 1 · p. 12
4.2 Ablation Studies
- With FSA, LVI, and MLS enabled, UniFS achieves a 2.5% overall improvement; FSA alone drops success to 70.3%, while adding LVI raises it to 94.3%.The authors attribute the FSA-only instability to frequency mismatch; they report better results across all tasks after introducing MLS.Source: §4.2 · pp. 12–13
4.1 Simulation Experiments on LIBERO
- 17.8 ms average inference latency is reported for UniFS in the efficiency comparison with OpenVLA, OpenVLA-OFT, and VLA-Adapter.The caption gives a 12.3 ms minimum and 32.6 ms maximum for UniFS.Source: §4.1 · p. 12, Table 2 · p. 13
4.2 Ablation Studies
- When the maximum-to-minimum sampling-frequency ratio is at most 24×, performance impact is negligible; above that, degradation accompanies only marginal latency gains.Uniform sampling also works without performance degradation.Source: §4.2 · pp. 13–14, Figure 5 · p. 13
4.3 Deployment on Real Robot
- On the Franka platform, UniFS achieves 75% success on Pick, 60% on Stack, and 50% on the short-horizon memory-aware Exchange Boxes task.Pick and Stack are reported as comparable to the baseline model without frequency.Source: §4.3 · p. 14
5 Conclusion
The conclusion presents UniFS as a unified fast-to-slow architecture with strong LIBERO performance and low inference latency. It also identifies training speed, batch-size demands, and possible shifts to pretrained representations as limitations.
5 Conclusion
- UniFS achieves 98.3% average success on LIBERO with 17.8 ms latency, and the conclusion describes the architecture as providing implicit short-term memory.The design combines Fast-to-Slow Architecture, Latent Vector Inversion, and Multi-Level Supervision to produce multi-frequency latent vectors for the action expert.Source: §5 · p. 14
- Training receives no computational speedup because Frequency Feature Replacement requires full parallel forward passes, and stable multi-frequency learning demands relatively large batches.The fast-to-slow architecture and latent vector inversion can shift distributions and interfere with pretrained semantic knowledge, so careful fine-tuning is needed.Source: §5 · pp. 14–15
A Detailed Architecture
The architecture assigns multiple frequencies to vision and language features and supplies selected hidden states to a 24-layer action expert. Its feature selection combines terminal group layers with intermediate LLM features.
A Detailed Architecture
- Both the vision encoder and LLM backbone use fast-to-slow structure because the vision encoder is more time-consuming in this implementation.The passage refers to module inference-time measurements in Table 4.Source: A Detailed Architecture · p. 19
- DINOv2 and SigLIP hidden states are extracted at 16f, 8f, and 4f, while Qwen2.5 hidden states use 4f, 2f, and f.The 4f frequency applies to both the vision encoder and LLM backbone.Source: A Detailed Architecture · p. 19
- The 24-layer action expert selects 24 hidden states across the vision encoder and LLM backbone as fast-to-slow latent vectors for action generation.Features are taken from terminal layers within each frequency group, with intermediate LLM features also included for their contribution to representation quality and performance.Source: A Detailed Architecture · p. 19
B More Analysis
UniFS analysis links layer depth to temporal feature stability and shows how selective layer updates reduce inference time. Its frequency schedule updates shallow and deep layers at different rates while reusing features.
Layer-wise temporal dynamics
- Table 5 describes the relative-frequency configurations for vision-encoder and LLM layers and the hidden states selected for the action expert.Source: Table 5 · p. 20
- UniFS shallow layers show higher median and wider cosine-distance distributions, while deeper layers have compact distributions consistent with stable semantic features.The analysis measures cosine distance between consecutive time steps; smaller values indicate more stable representations.Source: B More Analysis · pp. 19–20, Figure 6 · p. 20
Timing analysis
- 17.8 ms average inference time across t0–t15, versus 32 ms for full-layer inference.Block1 runs at 16f on every step in 13.3 ms; Block2–Block5 run at 8f, 4f, 2f, and f, adding approximately 4–5 ms per layer addition.Source: B More Analysis · p. 21
C Visualization of Results
The visualizations include LIBERO and real-world Franka experiments, with one LIBERO task illustrating how reused low-frequency states provide historical context in ambiguous manipulation.
C Visualization of Results
- In the cube-exchange task, UniFS has a higher success rate than VLA-Adapter, which does not reuse features, by reusing historical low-frequency hidden states.The ambiguity arises after moving a cube between containers, when current observations alone leave which cube to exchange unclear.Source: C Visualization of Results · p. 21