Source-linked AI summary
Latent Cluster Analysis for Vision-Language-Action Models
Theodor Wulff, Sergio Lanza, Tamara Bila, Angelo Cangelosi, Stefan Wermter, Igor Farkas
TL;DR
VLA models’ internal representations remain poorly understood despite their growing use in robotics. LAVLA performs layer-wise latent clustering of GR00T N1.5, using cross-attention-based weighting and concept extraction to analyze action-decoder representations. Weighted clustering improves reported quality metrics, while the analysis identifies disentangled spatiotemporal and kinematic features across the diffusion process.
Problem
VLA models have promising generalization capabilities, but limited research explains their internal representations, which matter for understanding behavior in robotics.
Method
LAVLA clusters layer-wise embeddings from GR00T N1.5, weights VLM tokens according to cross-attention influence, and assigns interpretable concepts to clusters.
Results
62.1% increase in Silhouette scores and 8% improvement in the Davies-Bouldin Index are reported for weighted clustering over baseline.
Takeaways & Limitations
LAVLA makes VLA latent spaces more transparent and reveals task-specific representations with spatiotemporal consistency across the diffusion trajectory.
Takeaways & Limitations
Experiments are limited to a single dataset and model, and the findings may not generalize to other datasets or architectures.
Abstract
from arXiv · showhide
Vision-Language-Action (VLA) Models are increasingly used in robotics for their ability to ground language and perception into action, yet the internal representations driving their behaviour remain poorly understood. We propose LAVLA, a framework for latent cluster analysis of VLA models, and conduct a layer-wise study of the state-of-the-art GR00T N1.5 model, with particular focus on its action decoder. To better characterise the latent space during action diffusion, we introduce a cross-attention-based embedding-weighting method that amplifies relevant features while suppressing less informative ones. Quantitative evaluation shows that weighted clustering consistently outperforms the baseline. To improve interpretability, we extract human-interpretable concepts for each cluster, linking latent representations to semantic descriptions. Our analysis shows that latent clusters progressively disentangle spatiotemporal and kinematic features, with representations becoming more refined in the middle layers and stabilising toward the output. As such, LAVLA advances the interpretability of language-driven robotic systems.
1 Introduction
LAVLA analyzes VLA latent spaces through layer-wise clustering, cross-attention-based embedding weighting, and human-interpretable concept extraction. Applied to GR00T N1.5, the framework improves clustering quality and reveals disentangled spatiotemporal and kinematic representations.
- Framework: LAVLA analyzes VLA embeddings from the final VLM backbone layer and DiT action-decoder layers using clustering.The framework is demonstrated on NVIDIA’s GR00T N1.5 model.
- Framework: Cross-attention-based weighting amplifies tokens prioritized by the action decoder and suppresses less influential features.The weighting module operates on visuolinguistic tokens used in action-decoder cross-attention.
- Results: 62.1% increase in Silhouette scores and 8% improvement in the Davies-Bouldin Index demonstrate consistent gains over baseline clustering.These improvements are reported across unsupervised clustering benchmarks.
- Interpretability: Human-interpretable concepts are assigned to clusters from their nearest samples and evaluated with CLIP.An external VLM labels patterns shared by representative samples.
- Analysis: Cross-attention layers focus on specific VLM tokens across diffusion timesteps, while clusters disentangle spatiotemporal and kinematic features.The findings suggest cross-attention incorporates visuolinguistic context during action diffusion.
2 Related Work
Related work positions VLAs as models mapping language and visual observations to robot actions, while latent-analysis methods seek more interpretable representations. Existing approaches include diverse action decoders, hierarchical modeling, clustering, and sparse autoencoders.
- Vision-Language-Action Models: VLAs are trained on broad trajectory and scenario collections to generalize across embodiments, tasks, and environments.They commonly map language instructions and visual observations to robotic actions.
- Vision-Language-Action Models: Action decoders generate continuous robot actions through flow matching, diffusion, or linear transformations.Some methods additionally model hierarchical subtasks for long-horizon behavior.
- Latent Analysis: Latent-analysis methods compress high-dimensional embeddings into representations that can be linked to human-interpretable concepts.Sparse autoencoders use sparse hidden activations to represent salient information.
3 Methodology
LAVLA collects layer-wise VLA embeddings, optionally weights them using cross-attention and reduces dimensionality, then clusters samples and labels representative clusters with concepts. The methodology also evaluates cluster quality and sample-assignment overlap.
- Pipeline: LAVLA collects selected-layer embeddings during forward passes, clusters them layer-wise, and optionally applies weighting and PCA before clustering.Embedding weighting precedes PCA when both are used.
- Embedding Collection: GR00T N1.5 embeddings are collected from the Eagle backbone’s 12th layer and every action-decoder cross-attention layer across four denoising steps.The decoder alternates self-attention and cross-attention while progressively refining noisy action tokens.
- Clustering: Agglomerative hierarchical clustering merges initially separate samples according to cluster proximity.DBSCAN and k-means were explored but agglomerative clustering was selected for superior early-stage performance.
- Embedding Weighting: Cross-attention weights are averaged across heads, normalized with softmax, and applied elementwise to visuolingual tokens.The resulting weighted tokens reduce less influential contributions and boost important ones for each layer and diffusion timestep.
- Evaluation: Cluster quality is evaluated with Silhouette, Calinski-Harabasz, and Davies-Bouldin metrics, while overlap compares assignments across experiments and layers.Higher Silhouette and CH values, and lower DB values, indicate better cluster separation.
- Concept Extraction: Human-interpretable concepts are generated by selecting top-n samples nearest each centroid and prompting a VLM to summarize their shared patterns.Each sample combines a textual command with a 10-frame action-video sequence from a 5-second clip.
4 Experiment Setup
The experiment collects 24,000 embeddings from a frozen GR00T N1.5 model across 24 tasks, using a fixed sampling and clustering setup on one NVIDIA A100 GPU.
- Data Collection: 24,000 embeddings are collected as 1,000 embeddings per task across 24 tasks in approximately four hours.Embeddings are extracted from a fully frozen GR00T N1.5 model on a single NVIDIA A100 GPU.
- Data Collection: Consecutive samples from the same task and episode are offset by 10 frames.This defines the temporal spacing used in the resulting embedding set.
- Clustering: Embeddings are clustered with agglomerative clustering using Euclidean distance and average linkage.The setup accounts for layer-specific changes in latent representations rather than assuming one universal per-token embedding.
5 Results
Across layers and diffusion timesteps, LAVLA reveals improving cluster structure, consistency, and separation, while embedding-weighting generally improves clustering quality and supports interpretable latent organisation.
- 5.1 Clustering Results: 62.1% higher Silhouette scores without PCA and approximately 8% better DB Index scores show gains from embedding-weighting.With PCA, Silhouette scores improve by 24.0%; DB Index scores improve by approximately 8% in each scenario.
- 5.1 Clustering Results: Later diffusion timesteps produce fewer clusters as increasingly structured representations consolidate disparate embeddings into cohesive groups.PCA reduces embeddings to 100 dimensions and significantly lowers the overall cluster count, while preserving the later-timestep decline.
- 5.1 Clustering Results: Embedding-weighting makes the timestep-based organisation trend less prominent by directing backbone embeddings toward certain clusters throughout diffusion.Without weighting, noisy action embeddings require additional denoising iterations before the action decoder resolves a structured latent space.
- 5.2 Cluster Consistency: Cluster assignments remain stable across diffusion timesteps within a layer and become more consistent nearer the final outputs across layers.Early unweighted configurations were excluded from some overlap heatmaps because singleton assignments inflated the corresponding metrics.
- 5.3 Latent Space Organisation: Action-diffusion clusters show stronger temporal separability than VLM-backbone clusters, with approximately 20% higher mean intra-cluster Wasserstein distance for arm-joint features.The increase is reported for both left- and right-arm joints and is less prominent for waist embeddings.
- 5.4 Concept Generation Results: PCA-based and weighted configurations tend to produce slightly more faithful concepts, although low visual diversity and short clips limit CLIPSIM performance.The generated concepts remain semantically close, mostly describing picking objects and returning them to a container.
6 Conclusion
LAVLA analyzes VLA latent spaces through clustering, using cross-attention weighting to clarify connections between backbone features and iterative action-decoder denoising. The experiments show clearer, task-specific representations across cross-attention layers and consistent spatiotemporal structure during diffusion.
- LAVLA analyzes VLA latent spaces through clustering, focusing on representations processed by the action decoder.
- Cross-attention weighting amplifies tokens prioritized during decoding, clarifying the connection between backbone feature extraction and iterative action denoising.
- Task-specific representations are distributed across distinct cross-attention layers and maintain high spatiotemporal consistency across the diffusion trajectory.
- LAVLA makes VLA latent spaces more transparent within the studied setting, supporting progress toward trustworthy language-driven robotic systems.
Ethical Considerations
The work examines VLA internals without introducing inherent privacy, inclusivity, or bias risks from its publicly available models and data. However, deploying insufficiently tested VLAs across real-world embodiments could have catastrophic consequences.
- The framework uses publicly available models and data and therefore poses no inherent data-privacy risk.
- Studying pretrained VLA latent spaces does not introduce stated inclusivity or bias risks and may help detect them.
- Untested deployment across real-world scenarios and embodiments could lead to catastrophic consequences.
Limitations
The study is bounded by a single dataset and model, short input horizons, challenging concept verification, static-image concept extraction, and agglomerative clustering assumptions. These constraints limit generalisation and may affect concepts for dynamic scenes or complex manifolds.
- Experiments use a single dataset and model, so results may not generalise to other datasets or architectures.
- Concept generation and unsupervised semantic verification remain challenging without ground-truth annotations.
- Static-image concept extraction makes temporal or dynamic scene aspects more difficult to capture.
- The short input horizon may limit extension of findings to longer-horizon behaviours.
- Agglomerative clustering assumptions may not fully capture complex, non-spherical manifolds, influencing resulting concept groupings.
A Data
The study collects GR00T embeddings from a sampled humanoid tabletop-manipulation dataset and analyzes clusters independently across layers and diffusion timesteps. Cluster alignment then enables tracing membership changes through the model, while visualizations and overlap measures characterize latent evolution.
- A Data: The dataset contains 240k trajectories across 24 tabletop-manipulation tasks, subsampled to 24k observation-action pairs because of memory constraints.
- A Data: Each observation combines an ego-perspective frame, a natural-language task instruction, and a robot joint-state vector.
- B Cluster Alignment: Clusters are computed independently for each layer and diffusion-timestep combination, so identifiers can switch across conditions.
- B Cluster Alignment: Optimal bipartite matching with the Hungarian algorithm aligns clusters using sample-overlap costs, enabling consistent tracking of memberships.
- C Formulas: Overlap Coefficient and Jaccard Index measurements quantify changes in internal representations across layers and diffusion steps.
D Cluster Visualisation
Embedding weighting structures the VLA latent space most clearly in early layers, where weighted embeddings form more coherent clusters than unweighted embeddings.
- Weighted embeddings form significantly more coherent clusters than unweighted embeddings, particularly in early layers.UMAP visualisations show the weighting mechanism’s strongest structuring effect during the initial diffusion stages.
- At the first cross-attention layer and diffusion timestep, barely processed noisy action tokens produce an unstructured, circular-shaped latent space.
E PCA Analysis
PCA and related visualisations reveal emerging separability and organisation in GR00T’s latent representations, while cluster analyses provide layer-specific feature and concept views.
- PCA Analysis: The first ten principal components carry the highest explained variance, but cumulative variance barely exceeds 60%.This indicates that individual components account for only a small portion of the information in a single embedding.
- PCA Analysis: Noisy action tokens show initial separability within the first diffusion timestep, with stronger separation emerging over time.
- PCA Analysis: Across layer 6, a single noisy cluster splits over time into 5 visually distinct clusters as principal-component variance increases.
- Cluster Visualisation: Wasserstein distances quantitatively compare selected-feature distributions across top clusters, while violin plots visualise organisation at different layers.
- Concept Extraction: Concept extraction uses cluster samples from 5 videos, each decoded into 10 sequential frames and paired with its task command.