PaperlayerView exact revision on arXiv

Source-linked AI summary

TERRA: Learning Transportable Latent Actions through Temporal Effect Representation and Relational Alignment

Tianxingjian Ding, Mubarak Shah, Yu Tian

arXiv:2610.09509v1cs.CVcs.LGcs.RO

TL;DR

Latent actions must retain meaningful temporal information and remain consistent when reused in a different initial state. TERRA learns continuous latents from compact temporal effects and uses Effect-Anchored Transport to anchor decoded effects across contexts. It improves action alignment, distractor robustness, and transport fidelity, and reaches 93.4% average LIBERO success versus 91.8% for UniVLA at matched pretraining scale.

  • Problem

    Latent actions must preserve meaningful temporal distinctions while remaining consistent when reused, but endpoint differences omit within-window motion and reconstruction observes only matched state–transition pairs.

    Source: §1 · pp. 1–2

  • Method

    TERRA compresses net feature change and a low-order within-window dynamics component into a continuous latent, then uses Effect-Anchored Transport to anchor decoded effects in recipient states to the donor effect.

    Source: §1 · p. 2, §3.1 · pp. 3–4

  • Results

    TERRA's latents align more closely with actions, degrade more slowly under visual distractors, and preserve donor actions under transport; at matched pretraining scale, it reaches 93.4% average LIBERO success versus 91.8% for UniVLA.

    Source: §1 · p. 2, §6 · p. 10

  • Takeaways & Limitations

    After refinement, the frozen latent tokenizer provides latent-action targets for downstream policy training.

    Source: §3 · p. 3

    Transport is evaluated within a data source with shared embodiment and camera geometry; anchoring across embodiments would require a correspondence-aware descriptor.

    Source: §5 · p. 9

Abstract

from arXiv · show

Latent actions supervise robot policies with action-like codes inferred from visual transitions, and their usefulness hinges on two questions: what a code keeps from a transition, and whether it still means the same thing when reused in a different initial state. The first is a tension in time: an endpoint difference discards how motion unfolds, while the full sequence admits nuisance variation. The second is left open by reconstruction, which only ever observes a latent together with the state it came from. We argue that both questions can be answered in the same place. TERRA (Temporal Effect Representation and Relational Alignment) describes a transition by a compact temporal effect, its net feature change together with a low-order within-window dynamics component, and learns a continuous latent from this effect. The same effect space then serves as the reference for reuse: Effect-Anchored Transport (EAT) decodes a latent in other initial states and anchors the resulting effect to the one observed at its source, so that the latent is shaped by what it does across contexts rather than only by the transition it came from. With frozen linear readers, TERRA predicts actions more accurately than UniVLA and a LAPA-style baseline, degrades more slowly under visual distractors, and keeps transported transitions faithful to the donor action as the recipient context moves farther away; a same-budget control shows that these gains come largely from EAT. At matched pretraining scale, the complete system reaches 93.4% average success on LIBERO, compared with 91.8% for UniVLA.

1 INTRODUCTION

Latent actions must preserve how a transition unfolds while remaining consistent when reused in a different initial state. TERRA represents compact temporal effects and uses them to anchor cross-context decoding; its evaluations report stronger action alignment, transport stability, and matched-scale LIBERO success than UniVLA.

1 INTRODUCTION

  • Latent actions must preserve within-window motion that endpoint differences miss, without retaining the nuisance variation admitted by full sequences.Executions with the same endpoints but different rhythms receive the same code from endpoint differences alone.Source: §1 · p. 1, Figure 1 · p. 2
  • Reconstruction sees only matched state–transition pairs, so a latent that fits its source can decode to a different effect after the scene changes.This cross-context reuse is called transport.Source: §1 · pp. 1–2
  • TERRA compresses net feature change and a low-order within-window dynamics component into a continuous latent, then uses Effect-Anchored Transport to anchor decoded effects to the donor effect.EAT decodes the latent in recipient states from the same data source; relative-effect consistency, transition-distribution matching, and nuisance invariance regularize refinement.Source: §1 · p. 2
  • 93.4% average LIBERO success for TERRA versus 91.8% for UniVLA at matched pretraining scale.The reported gains also include lower action NMSE with a dynamics component, improved action-trend recoverability, and more faithful transport as recipient distance grows.Source: §1 · p. 2
  • The paper presents a temporal-effect representation, EAT for cross-context supervision, and controlled evaluations of action alignment, nuisance robustness, transport stability, and LIBERO success.The complete system's matched-scale comparison is against UniVLA.Source: §1 · p. 2

2 RELATED WORK

Related work studies latent-action representations, cross-context grounding, and continuous policy prediction, largely as separate problems. TERRA uses one effect space for temporal representation and cross-context supervision.

Latent action representations

  • Latent-action work spans inverse-dynamics labels, discrete codes for generation and policies, and continuous representations for richer motion structure.Examples include LAPA and UniVLA's quantized codes, CoMo's frame-pair feature differences, and RotVLA's temporally composed continuous latents.Source: §2 · pp. 2–3

Cross-context grounding and latent consistency

  • Cross-context work orients or regularizes latent identity, while analyses examine distractor sensitivity, action recoverability, and links between latent-quality proxies and downstream control.Olaf-World uses temporal feature differences, cycle-based approaches regularize identity through generated transitions, and LAOM studies action-correlated distractors.Source: §2 · p. 3

Continuous VLA prediction

  • Continuous VLA prediction includes continuous action heads, flow-matching action experts, jointly modeled latent and physical actions, and latent-policy learning.The cited examples are OpenVLA-OFT, π0, RotVLA, and LAFP.Source: §2 · p. 3

Synthesis

  • TERRA differs by using one effect space both to define what a latent preserves and to supervise its behavior in other states.The related work treats temporal latent construction, cross-context consistency, and continuous policy prediction largely as separate problems.Source: §2 · p. 3

3 TERRA: TEMPORAL EFFECT REPRESENTATION AND RELATIONAL ALIGNMENT

TERRA learns continuous latent actions from compact temporal evidence and uses that same effect space to guide how latents behave when decoded in different initial states.

3.1 TEMPORAL EFFECT REPRESENTATION (TER)

  • TER represents a transition with net feature change and a low-order dynamics component, retaining coarse within-window variation without encoding the full increment sequence.The net component captures accumulated change, while the dynamics component captures coarse within-window variation.Source: §3.1 · pp. 3–4

3.2 RELATIONAL ALIGNMENT (RA) VIA EFFECT-ANCHORED TRANSPORT

  • EAT decodes a donor latent in a recipient state and aligns the resulting effect direction with the donor’s observed effect.The tokenizer encodes only the donor transition, and the shared conditional decoder receives the recipient initial state without its future frames.Source: §3.2 · p. 5
  • RA combines effect anchoring with relative-effect consistency, which stabilizes comparisons among latent effects across recipient contexts without requiring generated transitions to be state-independent.EAT also uses transition-distribution matching and nuisance invariance; its scope is a same-source visual-effect regularizer, not supervision for a ground-truth counterfactual physical action.Source: §3.2 · pp. 5–6

4 EXPERIMENTS

The experiments evaluate TERRA at matched pretraining scale and probe its representations with frozen linear readers. TERRA outperforms retrained UniVLA on average LIBERO success, while temporal-effect modeling and EAT improve action alignment and transport stability.

4.1 EXPERIMENTAL SETUP

  • Models are pretrained on approximately 8.19M clips from 29 Open X-Embodiment sources, then evaluated using shared frozen DINOv2 features and linear readers.The system comparison retrains TERRA and UniVLA at matched data scale; representation analyses also include a LAPA-style discrete baseline.Source: §4.1 · pp. 6–7

4.2 CLOSED-LOOP MANIPULATION

  • 93.4% average LIBERO success for TERRA compares with 91.8% for retrained UniVLA at matched latent-action pretraining scale.TERRA also has higher success on all four suites, with the largest gains on Object (+2.6) and Long (+1.9).Source: Table 1 · p. 7, §4.2 · p. 7

4.3 TEMPORAL EFFECT REPRESENTATION

  • On clean transitions, TERRA achieves action NMSE .723 versus .785 for UniVLA and .819 for the LAPA-style baseline; its distractor-induced NMSE rise is .041.The corresponding rises are .076 for UniVLA and .107 for the LAPA-style baseline; UniVLA reads the action profile slightly better (.875 versus .909).Source: Figure 3 · p. 8, §4.3 · p. 8
  • Adding A1 lowers profile NMSE from .983 to .909 and action NMSE from .817 to .807, while net-action NMSE shifts from .703 to .734.The matched 65k tokenizers differ in whether A1 is provided; nuisance drop also falls from 33.6% to 22.4%.Source: Table 2 · p. 8, §4.3 · p. 8

4.4 CROSS-CONTEXT REFINEMENT WITH EAT

  • From the nearest to farthest recipient quartile, donor-action NMSE rises by .087 with full EAT versus .267 with reconstruction-only training; full EAT lowers NMSE on 22 of 29 sources.At equal additional-update budget, reconstruction-only updates lower 12-step action NMSE from .807 to .784, while EAT reaches .723.Source: Figure 4 · p. 9, §4.4 · p. 9

5 DISCUSSION AND LIMITATIONS

The paper's transport evaluation is limited to settings where embodiment and camera geometry are shared, and its compact effect representation and closed-loop evaluation have additional scope boundaries.

5 DISCUSSION AND LIMITATIONS

  • Transport is evaluated within a data source with shared embodiment and camera geometry; cross-embodiment anchoring would require a correspondence-aware descriptor.The transport metric tests whether donor-action semantics survive decoding in another state, not whether the result matches a unique counterfactual future.Source: §5 · p. 9
  • The effect space retains only two low-order temporal components on a coarse spatial grid, favoring robustness over fine temporal detail.Source: §5 · p. 9
  • Closed-loop results are limited to LIBERO at reduced pretraining scale; larger-scale pretraining and real-robot deployment remain future work.Source: §5 · p. 9

6 CONCLUSION

TERRA uses one compact effect space to define what a latent action retains and how it behaves in new contexts. Its latents align more closely with actions, remain more robust to visual distractors, and preserve donor actions under transport.

6 CONCLUSION

  • The complete system improves LIBERO success over UniVLA at matched pretraining scale, while its latents align more closely with actions, degrade more slowly under visual distractors, and preserve donor actions under transport.Source: §6 · p. 10
  • Temporal Effect Representation adds within-window dynamics to endpoint differences, while EAT anchors effects decoded in other states to the source effect.Together, they use one compact effect space to define what a latent keeps and how it behaves in new contexts.Source: §6 · pp. 9–10
  • The authors report using generative AI for drafting, code assistance, experiment orchestration, and language editing, with code and quantitative claims reviewed and checked.They take responsibility for the final content and artifacts.Source: AI USE STATEMENT · p. 11

A METHOD DETAILS

The method details specify how TERRA constructs its temporal representation and how EAT refines it using effect-based and complementary objectives. The appendix also describes normalization, nuisance handling, and the training schedule.

A.1 TEMPORAL EFFECT REPRESENTATION

  • The tokenizer forms 48 structured tokens from five observations, representing initial features, net change, and within-window dynamics on a 4 × 4 grid.A six-layer Transformer aggregates the tokens through four learned queries projected to a 4 × 64 latent.Source: §A.1 · p. 13
  • Feature increments use one fixed global RMS scale estimated from real training transitions, and generated increments are rescaled before temporal evidence is evaluated.The same normalization scalar is reused in training and evaluation.Source: §A.1 · p. 13

A.2 EFFECT-ANCHORED TRANSPORT

  • EAT represents effects with net and dynamics components, then applies a fixed random projection and frozen per-coordinate standardization to compare them.The projection is initialized once, and the same transform is applied to real and generated transitions.Source: §A.2 · pp. 13–14
  • Matching the projected effect descriptor constrains per-cell image-space effects up to projection distortion and diagonal rescaling, not equality of a state-independent physical action.The descriptor is defined on the pooled feature grid.Source: §A.2 · pp. 13–14
  • Relative-effect consistency uses detached EMA-tokenizer latents from a FIFO donor queue, so its loss regularizes the shared decoder directly and reaches the online tokenizer indirectly.The target tokenizer is updated with an EMA coefficient of 0.99, and the queue stores the most recent 16 minibatches.Source: §A.2 · p. 14
  • The MMD objective matches projected generated and real transition marginals but supplies no target counterfactual trajectory for an individual recipient.It uses a mixture of RBF kernels.Source: §A.2 · p. 14
  • Nuisance invariance trains the online tokenizer on a temporally consistent perturbed view and the EMA tokenizer on the corresponding clean clip.The perturbation uses a shared-per-clip crop, brightness scaling, and contrast scaling; effect-based objectives use unaugmented transitions.Source: §A.2 · p. 14
  • Training uses 65k self-reconstruction updates followed by 15k EAT-refinement updates while retaining self-reconstruction.The reported refinement weights are λe = 1, λr = 0.5, λd = 1, and λi = 1.Source: §A.2 · p. 14

B EXPERIMENTAL PROTOCOL

The representation experiments compare TERRA with a LAPA-style discrete baseline trained in the same frozen feature space and under a matched training budget. UniVLA is retrained at TERRA's latent-action pretraining data scale using its official tokenizer recipe.

B EXPERIMENTAL PROTOCOL

  • The LAPA-style discrete VQ baseline uses the same frozen DINO feature space and linear-reader protocol, with an 8-entry codebook and four codes per transition.It is trained for 65k updates on the same data, matching TERRA's self-reconstruction stage; it is not the released LAPA tokenizer.Source: §B.1 · p. 14
  • UniVLA is retrained at the same latent-action pretraining data scale as TERRA, following the official tokenizer recipe and its released Stage-1 task-irrelevant prior.Source: §B.1 · p. 15

B.2 DOWNSTREAM VLA PROTOCOL AND PROVENANCE

The downstream protocol compares complete latent-action systems at matched pretraining scale and specifies their policy interfaces, inference inputs, and baseline provenance. The action-reader and nuisance protocols use fixed evaluation procedures to assess representations and visual robustness.

B.2 DOWNSTREAM VLA PROTOCOL AND PROVENANCE

  • TERRA and retrained UniVLA start from OpenVLA-7B and use the same 10% pretraining-mixture subset before LIBERO fine-tuning.Closed-loop numbers use one fine-tuning seed, 10 tasks per suite, and 50 rollouts per task.Source: §B.2 · p. 15
  • Table 1 compares complete systems rather than a fixed policy interface: TERRA uses continuous latent readouts with a flow-matching expert, while UniVLA uses native discrete tokens.At inference, TERRA uses only the current observation and language instruction.Source: §B.2 · p. 15
  • The 91.8% retrained UniVLA row is a matched-scale baseline, not the published UniVLA model or a strict same-provenance-from-random comparison.It uses a reduced 10% pretraining mixture and inherits the released Stage-1 task-irrelevant prior.Source: §B.2 · p. 15
  • Perturbed closed-loop rollouts are qualitative comparisons only; Table 1 success rates are measured without perturbations.The perturbations combine LIBERO-Plus level L3 with a moving distractor.Source: §B.2 · p. 15

B.3 ACTION READERS AND CONTROLLED NUISANCE

  • Separate ridge readers are fit within each of 29 OXE sources and their source-level normalized mean-squared errors are macro-averaged.Held-out transitions are split by episode, with two thirds for fitting and one third for scoring.Source: §B.3 · p. 15
  • Nuisance robustness holds robot trajectories and action labels fixed while scaling the same moving visual distractor across four severity levels.The distractor magnitude ranges from α = 0 for clean frames to α = 1 for the fully pasted patch.Source: §B.3 · p. 15

B.4 TRANSPORTED-ACTION EVALUATION AND CONTEXT SENSITIVITY

This section evaluates transport with a frozen linear reader that scores transported transitions against donor actions. It also defines recipient distance and reports cross-context diagnostics for interpreting reuse.

B.4 TRANSPORTED-ACTION EVALUATION AND CONTEXT SENSITIVITY

  • Table 3 gives 95% bootstrap intervals for Table 2 readouts, reflecting variation across 29 sources rather than training seeds.The intervals use 2,000 source resamples; tokenizer training uses one seed.Source: Table 3 · p. 16
  • Table 4 reports cross-context transport diagnostics, including context sensitivity and latent effective rank.The caption identifies context sensitivity as CtxSens.Source: Table 4 · p. 16
  • The frozen transported-action reader maps feature-increment sequences to 12-step action chunks and scores them against recorded donor actions.Because the reader is fit on real held-out pairs and never sees generated transitions, low NMSE indicates the transported transition carries the donor action in a form recognizable from real transitions.Source: §B.4 · pp. 15–16
  • Recipient distance is cosine distance between spatially averaged frozen DINO initial-state features, ranked into equal-count quartiles within each data source.Q1 contains the nearest 25% and Q4 the farthest 25% of donor–recipient pairs.Source: §B.4 · p. 16

C ADDITIONAL RESULTS

Additional results qualify the same-budget comparison, examine cross-context transport, and test spatial evidence resolution under matched compression and readout conditions.

C.1 BOOTSTRAP INTERVALS FOR TABLE 2

  • The same-budget control separates EAT refinement from additional reconstruction and isolates nuisance invariance, but does not isolate Leffect, Lrel, and Ldist individually.Source: §C.1 · p. 16

C.2 TRANSPORT DIAGNOSTICS

  • On the farthest recipient quartiles, mean transported-action NMSE was .806 with full EAT versus .893 with reconstruction only; context sensitivity .487 argues against state-independent donor replay as the sole explanation.Full EAT was lower on 22 of 29 sources.Source: §C.2 · p. 16, Table 5 · p. 17

C.3 SPATIAL EVIDENCE RESOLUTION

  • Under matched 256-D compression and readout, robustness improves as resolution falls to 4 × 4, then largely plateaus; this does not establish that learned full-patch tokenizers are intrinsically worse.The authors attribute the pattern to added spatial capacity admitting nuisance faster than it improves linear action readout under a fixed budget.Source: §C.3 · p. 17, Table 6 · p. 17
Loading 2610.09509v1…