Source-linked AI summary
Time-Frequency Geometric Cross-Attention for Chunked Vision-Language-Action Models
Shengye Dong, Haochen Niu, Hao Liu, Peiwen Lin, Chuang Wang, Shanmin Pang
TL;DR
Chunked VLA policies underuse the multi-scale and cross-phase geometric structure of action trajectories, especially relationships near orthogonality. TFGCA adds wavelet tokenization and dot-plus-wedge cross-attention as a drop-in module, improving performance over the same-source base across clean, OOD, randomized, and real-robot settings, with larger gains out of distribution.
Problem
Chunked VLA policies represent structured trajectories as generic per-timestep tokens, leaving multi-scale frequency content and near-orthogonal cross-phase relationships implicit.
Method
TFGCA uses per-dimension learnable stationary wavelet tokenization and cross-attention that blends dot-product similarity with wedge-product geometry, with identity initialization for pretrained attachment.
Results
+28.5 on the randomized RoboTwin six-task mean relative to π0.5, alongside consistent improvements across clean, OOD, and real-robot evaluations.
Takeaways & Limitations
TFGCA provides a drop-in way to expose time-frequency and cross-phase geometric structure while preserving the pretrained policy’s behavior at initialization.
Takeaways & Limitations
The frequency framing is weaker for short-horizon LIBERO, where T = 10 and one SWT level produce a near single trend/detail split.
Abstract
from arXiv · showhide
Modern vision-language-action (VLA) policies predict a whole chunk of actions: one to two seconds of coordinated motion emitted in a single forward pass. Yet an action chunk is essentially a short multivariate trajectory, but inside these models it is a sequence of generic per-timestep hidden tokens decoded by a linear head. This under-serves two motion structures. First, frequency: a chunk superimposes a smooth global trend and fine corrective motion across time scales, and a single token entangles them. Second, cross-phase geometry: motions of different phases (reach, contact, grasp adjustment, settling) unfold along very different, near-orthogonal directions in representation space, yet are tightly related for the task and arise across the time axis. Dot-product attention scores alignment by an inner product, so it favors aligned tokens and is least sensitive near orthogonality, leaving such relationships for the network to recover through a detour. We introduce Time-Frequency Geometric Cross-Attention (TFGCA), a drop-in module repairing both blind spots. TFGCA uses a per-dimension learnable stationary wavelet transform to decompose the action chunk into time-frequency tokens, and each time token retrieves information from them via a cross-attention that fuses the dot product (similarity) with the wedge-product magnitude (sensitive to near-orthogonality) through a learnable weight. A zero-initialized residual reproduces the base behavior at initialization, so it can be dropped onto a pretrained VLA and fine-tuned jointly. Relative to the same-source base, TFGCA improves in-distribution LIBERO by +1.5 on average, the OOD LIBERO-Plus by +6.3, the randomized average under RoboTwin domain randomization by +28.5, and the overall success rate on three real-robot AgiBot A2 tasks by +11.67 points, with larger gains out of distribution.
1 Introduction
Chunked VLA policies emit structured action trajectories but represent them as generic per-timestep tokens, leaving multi-scale frequency content and cross-phase geometric relationships implicit. TFGCA addresses both gaps with wavelet tokenization and geometry-sensitive cross-attention.
- Chunked VLAs predict a T × D trajectory in one forward pass, improving temporal consistency and short-horizon coordination.
- Generic hidden tokens and a linear action head treat the action chunk as unstructured, underusing how dimensions move together over time.
- Wavelet tokenization separates slow transport trends from faster contact corrections, grasp adjustments, and settling motions.
- RoboTwin analysis finds sequential near-orthogonal subspaces across ordered manipulation phases, which dot-product attention retrieves only indirectly.
- TFGCA combines per-control-dimension learnable stationary wavelet tokenization with time-to-time–frequency cross-attention using dot and wedge scores.
2 Related Work
Prior work uses action chunking, frequency-domain sequence modeling, and geometric attention, but TFGCA combines these ideas for continuous, non-periodic VLA trajectories. Its frequency representation is multi-resolution and time-localized rather than primarily for compression.
- VLAs include discrete action-token models and continuous chunk predictors based on diffusion or flow matching.
- Frequency-domain forecasting methods use seasonal–trend decomposition, Fourier attention, learned frequency representations, or stationary wavelets.
- FAST applies a DCT to action chunks for compact discrete autoregressive tokens, whereas TFGCA uses frequency as a multi-resolution, time-localized feature.
- Geometric-algebra networks provide broader multivector machinery, while TFGCA uses only wedge-product magnitude alongside inner product for lightweight orthogonality sensitivity.
3 Preliminaries
The pretrained flow-matching VLA predicts action-chunk velocities from hidden action tokens, while TFGCA inserts a transformation before the action head. Its attention combines similarity with a wedge score that is sensitive to orthogonality.
- Flow matching interpolates noise and actions over τ and trains the velocity field toward u = ϵ − a.
- The decoder transformer produces T hidden action tokens, which a linear head maps to the velocity prediction vθ.
- TFGCA transforms H into refined tokens before the action head and is applied during training and every inference denoising step.
- Dot-product attention favors aligned representations and is weak near orthogonality, requiring networks to re-encode such relationships as similarity.
- In Figure 1, frequency tokens provide keys and values, time tokens provide queries, and the result is written back residually before velocity decoding.
- The wedge-product magnitude complements the inner product: it vanishes for parallel vectors and is maximal for fixed-norm orthogonal vectors.
4 Method
TFGCA projects hidden action tokens into control space, decomposes each action dimension with a learnable stationary wavelet transform, and lets time tokens retrieve multi-resolution information through dot-plus-wedge cross-attention. A zero-initialized residual preserves the pretrained policy at attachment, while the wedge channel directly represents near-orthogonal relationships that dot-product attention alone cannot rank.
- Action-space projection: TFGCA projects abstract hidden tokens into per-dimension control space before modeling frequency structure in the policy’s action coordinates.The projection is aligned with embodiment-specific outputs, including joint-space actions or Cartesian and gripper channels.
- Wavelet tokenization: A stationary wavelet transform produces length-preserving, multi-resolution bands so frequency tokens remain aligned with original timesteps.The decomposition recursively separates approximation and detail bands; learnable per-dimension filters are initialized to db2 and can specialize to each action dimension.
- Wavelet tokenization: DC removal preserves amplitude differences between high-motion moments and stationary segments, while optional differencing can shift task-relevant energy into higher bands.The design avoids standard-deviation normalization because motion amplitude contrast is intended to guide attention.
- Geometric cross-attention: Time tokens query the full set of time-frequency tokens, whose values jointly encode all action dimensions at each time-band position.The cross-attention uses multi-head projections from time tokens as queries and frequency tokens as keys and values.
- Geometric cross-attention: The attention separately softmaxes dot and wedge scores, then blends their distributions with a learnable scalar controlling alignment versus orthogonality.This preserves a proper distribution for each channel rather than softmaxing a pre-summed score.
- Initialization and scope: Zero-initializing only the output projection makes the module reproduce the base prediction initially while retaining a generically nonzero gradient for learning.The supplied scope statement limits the theoretical separation to a fixed representation; learned projections could otherwise re-encode orthogonal relations as aligned ones.
- Theoretical mechanism: The wedge magnitude supplies a second relatedness coordinate: after normalization, dot and wedge scores satisfy ŝ^2 + ŵ^2 = 1, with wedge sensitivity increasing toward orthogonality.For sufficiently large β, the blended attention can favor a near-orthogonal key, a ranking unavailable to dot-product scoring alone.
5 Experiments
TFGCA is evaluated against same-source π0.5 baselines across in-distribution, OOD, randomized, and real-robot settings. Gains are modest on near-saturated LIBERO but larger under unseen perturbations and on the real robot.
- Evaluation scope: TFGCA improves π0.5 across LIBERO, LIBERO-Plus, RoboTwin, and AgiBot A2, with the strongest gains under OOD conditions.The evaluation covers four suites, seven LIBERO-Plus perturbation families, clean and randomized RoboTwin environments, and three real-robot tasks.
- LIBERO: 98.2% average LIBERO success rises from the reproduced π0.5 base’s 96.7%, a +1.5 gain on the near-saturated benchmark.The hardest suite, Long, increases from 94.6 to 97.0.
- LIBERO-Plus: 73.0 Total LIBERO-Plus success rises from the reproduced base’s 66.7, a +6.3 gain under unseen perturbations.TFGCA beats the base on six of seven perturbation families, with gains concentrated on Noise, Robot, and Camera.
- RoboTwin: +28.5 is TFGCA’s improvement over π0.5 on the six-task randomized RoboTwin average, compared with +3.7 on clean environments.The randomized average reaches 42.7, and randomization is unseen during training.
- Interpretation: The RoboTwin randomized gains are largest on contact and multi-stage tasks, but this relationship is reported as a correlation rather than causal evidence.One task regresses under randomization, and two clean regressions occur on near-saturated tasks.
- AgiBot A2: 61.67% overall AgiBot A2 success exceeds π0.5’s 50.00% across three real-robot tasks, an improvement of 11.67 percentage points.All three tasks improve, with the largest gain on pulling a tissue from a box.
6 Discussion and Limitations
The evidence supports empirical improvement over same-source bases, not a theoretical performance guarantee. The frequency interpretation is also weaker for short horizons with only one wavelet level, while several mechanism-focused diagnostics remain future work.
- Evidence boundary: Identity initialization guarantees pointwise agreement with the base at attachment, not that joint fine-tuning will outperform the base.Effectiveness is established empirically across four settings, with occasional small regressions on near-saturated tasks.
- Frequency limitation: The frequency framing is weaker on LIBERO because T = 10 and one SWT level produce nearly a single trend/detail split.The frequency interpretation is more strongly supported by the T = 50, two-level RoboTwin setting.
- Future diagnostics: Future work proposes testing whether wedge attention routes to token pairs with small dot products and large wedge magnitudes.The proposed Q/K routing analysis and OOD dot-only ablation would strengthen causal attribution of the mechanism.
- Future diagnostics: Future work also targets task-dependent replanning frequency and diagnostics for discontinuities where consecutive action chunks disagree.A boundary-discontinuity metric is suggested for evaluating chunk transitions.
7 Conclusion
The paper concludes that TFGCA augments chunked VLA policies with explicit frequency and geometric structure, while preserving the base output at initialization. Experiments span LIBERO, RoboTwin, and AgiBot A2, with dataset and embodiment details defining the evaluation scope.
- Conclusion: TFGCA adds wavelet-based time-frequency tokenization and similarity-plus-wedge cross-attention to recover trajectory structure that chunked VLAs under-use.A zero-initialized residual reproduces the base output exactly at initialization.
- Evaluation scope: LIBERO uses 256 × 256 RGB images, an 8-dimensional robot state, and a 7-dimensional Cartesian end-effector action with a binary gripper command.The action channels are not seven joint angles.
- Evaluation scope: RoboTwin uses three RGB views, a 14-dimensional robot state, and 14-dimensional dual-arm joint-space actions, with clean and randomized evaluation environments.Policies train only on clean demonstrations before testing unseen environmental perturbations.
- Evaluation scope: AgiBot A2 evaluation covers soap placement, plush-toy placement, and tissue extraction tasks on a real robot.The raw data and collection procedure are not released because of confidentiality constraints.
B Training and Evaluation Configuration
TFGCA and the reproduced π0.5 base use matched data, training budgets, and evaluation pipelines, with TFGCA-specific settings varying by benchmark. External-method comparisons are contextual because protocols may differ.
- Common configuration: TFGCA is attached between the VLA transformer output and action head, while the augmented model and π0.5 base share data, training budget, and evaluation pipeline.The common implementation uses LeRobot and base model lerobot/pi05_base.
- Per-benchmark configuration: TFGCA and the corresponding reproduced π0.5 base differ only in TFGCA-specific SWT and geometry configuration on each simulation benchmark.The alignment loss uses ground-truth actions with stop-gradient on the target branch.
- TFGCA-specific settings: SWT levels follow chunk length: one level for LIBERO and LIBERO-Plus, and two for RoboTwin.The settings are fixed from the data and base configuration rather than selected through a large benchmark-specific sweep.
- TFGCA-specific settings: The alignment-loss weight is 0.01 for LIBERO and LIBERO-Plus, and 0.1 for RoboTwin.Other optimizer, learning-rate, schedule, batch-size, denoising-step, and seed values are shared with the π0.5 base.
- Comparison protocol: External-method values in Table 7 and RoboTwin leaderboard values are reference comparisons because evaluation protocols may differ.π0.5 and TFGCA are directly comparable in the stated RoboTwin protocol, unlike the leaderboard-reported alternatives.
E LIBERO-Plus: Per-Suite Full Results
The LIBERO-Plus analysis reports full per-perturbation results across four suites and finds that TFGCA’s gains are concentrated on noise and camera perturbations. Performance is roughly flat or slightly lower on near-saturated light/background conditions.
- Reporting scope: TFGCA’s full LIBERO-Plus results are reported for Spatial, Object, Goal, and Long suites, with π0.5 values and deltas shown alongside each cell.The final average across the four suites corresponds to the two LIBERO-Plus rows in the main paper’s Table 2.
- Perturbation pattern: +19.1 on Spatial-Noise, +18.5 on Long-Noise, and +15.7 on Goal-Camera illustrate where TFGCA’s gains remain concentrated.These examples concern perturbations the base handles worst.
- Perturbation pattern: On Light and Background perturbations, TFGCA is roughly flat or slightly regresses within evaluation sampling variance.This pattern is consistent with the section’s average-level conclusion.
F Real-Robot Task Decomposition (AgiBot
The AgiBot A2 evaluation uses matched training and testing conditions and strict success criteria. Representative rollout keyframes decompose each real-robot task into stage-wise actions.
- Evaluation protocol: TFGCA and the π0.5 base use the same training data, training steps, task initialization, and testing conditions on AgiBot A2.Each model is tested 20 times per task.
- Evaluation protocol: A trial succeeds only when one continuous execution fully reaches the goal state without intervention, retries, dropped objects, or failed required sub-steps.The protocol therefore treats any failure of a required sub-step as a task failure.
- Task decomposition: Figures 4–6 use representative rollout keyframes to provide a stage-wise decomposition of the three AgiBot A2 tasks.The keyframes correspond one-to-one with Table 4 of the main paper.
G.1 Training Hardware and Wall-Clock
The supplied hardware section bounds what is reported about training resources and separately quantifies TFGCA’s architecture-determined parameter overhead. TFGCA adds about 4.23M parameters, roughly 0.10% of the approximately 4B-parameter π0.5 backbone.
- Resource reporting: Training resource records correspond to individual jobs rather than the cumulative cost of an entire benchmark.The section does not report peak memory, throughput, inference latency, or other unmeasured running-cost quantities.
- Resource reporting: Exact software versions are not stated, and AgiBot A2’s internal resource configuration is withheld for confidentiality.These boundaries prevent unsupported reconstruction of the complete hardware and software environment.
- Parameter overhead: The added parameters are dominated by four 1024 × 1024 Q/K/V/output projection matrices in geometric cross-attention.The per-dimension SWT encoder and action-space projection are almost negligible by comparison.
- Parameter overhead: About 4.23M parameters are added by TFGCA, roughly 0.10% of the approximately 4B-parameter π0.5 backbone.The count is determined exactly from the architecture and does not depend on training data or training steps.
H Temporal Orthogonal Division-of-Labor (TO-DoL): Definition and Data Analysis
TO-DoL formalizes temporal coordination in which manipulation phases occupy distinct directions, unfold in an ordered sequence, and remain coherent within phases. Data support the temporal-order condition most strongly, while also exposing limits: inter-phase orthogonality can be geometrically trivial, ordering varies by task, and coupling is population-level rather than per-trial feedback.
- Motivation and evidence: Same-instant orthogonal coupling is essentially absent, whereas orthogonal division-of-labor unfolds sequentially along the time axis.The analysis contrasts a nearly falsified simultaneous hypothesis with strong temporal-axis structure.
- Definition: TO-DoL requires inter-phase near-orthogonality, stable non-trivial temporal order, and within-phase directional coherence.The definition segments trajectories into ordered phases and imposes all three conditions simultaneously.
- Condition-by-condition verification: C1 holds numerically but is a trivial baseline: direction cosines are 0.0013 for aloha-agilex and 0.0014 for arx-x5.Because disjoint coordinate blocks are nearly orthogonal by geometry, the evidential weight rests on C2.
- Condition-by-condition verification: C2 holds strongly against an envelope-preserving order-shuffled control, with the real order structure 3.5–5× tighter rather than 9–37×.The larger range comes from a control that destroys the entire motion envelope and therefore inflates the effect size.
- Interpretation and limitations: The temporal coupling is population-level and script-level: randomly pairing episodes nearly preserves it, rather than demonstrating real-time cross-arm feedback.C2 remains significant and stable, but its interpretation is a shared sequential template across episodes.
- Robustness: TO-DoL is task-dependent: handover shows clear ordering, while symmetric stacking weakens C2 but still satisfies C1 and C3.Across embodiments, the two platforms give consistent conclusions, but strict ordering is not universal.