PaperlayerView exact revision on arXiv

Source-linked AI summary

POD-RoPE: Rotary Position Encoding over Physical Observation Distributions for Embodied AI

Xiaodi Huang, Zilin Wang, Jinpeng Liu, Pengchuan Xiao, Zhenlei Shao, Zhang Zhang

arXiv:2610.33745v2cs.ROcs.AI

TL;DR

Embodied perception needs positional representations that capture physical geometry and the spatial and temporal extent of observations, beyond discrete indices. POD-RoPE represents token footprints as distributions over physical observation addresses and reuses expected-rotation statistics for matching and content aggregation. Full Fast-WAM integration improves reported mean success on RoboTwin 2.0 and Sharpa North manipulation tasks.

  • Problem

    Standard positional encodings based on sequence indices or spatiotemporal grids do not represent viewing geometry or actual elapsed time, and a single center address cannot fully describe spatial and temporal observation footprints.

    Source: §1 · p. 1

  • Method

    POD-RoPE represents token footprints as distributions over camera centers, viewing directions, and sampling times, using expected rotations for relative Q/K interactions and coherence-dependent V/O transformations.

    Source: §1 · pp. 1–2

  • Results

    Full Fast-WAM integration raises mean success from 91.83% to 92.88% across all 50 RoboTwin 2.0 tasks and from 32.5% to 52.5% on Sharpa North.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Results support physical observation distributions as a basis for geometry-aware attention across static and temporal observations, with low additional model-block latency relative to Standard RoPE.

    Source: §5 · p. 9

Abstract

from arXiv · show

Embodied perception requires positional representations that capture both physical geometry and the spatial and temporal extent of observations. We introduce POD-RoPE (Physical Observation Distribution RoPE), which represents token observation footprints as distributions over camera centers, viewing directions, and sampling times. Under independent sampling, expected rotations encode relative relationships between distributions; when each distribution collapses to a single address, the construction recovers pointwise RoPE. The same rotation statistics support Q/K matching and coherence-dependent V/O transformations, allowing observation geometry to guide both information selection and content aggregation. Per-token pre-aggregation avoids increasing sequence length or processing pairs of observation nodes. We evaluate POD-RoPE across four robot policy architectures, simulation and real-world manipulation, and complementary reconstruction, depth-estimation, and novel-view-synthesis tasks. Integrating POD-RoPE into Fast-WAM improves mean success from 91.83% to 92.88% across all 50 RoboTwin 2.0 tasks, and from 32.5% to 52.5% on two real-robot tasks on Sharpa North. Controlled reconstruction ablations support the contributions of distribution encoding and V/O transformations. In H20 model-block benchmarks, POD-RoPE achieves median forward-plus-backward speedups of $1.91\times$ over P-RoPE and $5.34\times$ over RayRoPE, with a median latency overhead of 17.4% over Standard RoPE. These results support physical observation distributions as an effective and computationally efficient basis for geometry-aware attention.

1 INTRODUCTION

Embodied systems need positional representations that reflect observation geometry and actual spatial and temporal scales, rather than only discrete index differences. POD-RoPE encodes observation footprints as physical-address distributions and reuses rotation statistics for matching and content aggregation; evaluations report gains in manipulation success and lower latency than geometric baselines.

1 INTRODUCTION

  • Standard RoPE typically uses sequence indices or grid coordinates, which do not distinguish identical pixel offsets with different viewing angles or identical frame offsets with different elapsed times.The paper motivates positional encoding that represents observation geometry and actual spatial and temporal scales.Source: §1 · p. 1
  • Fast-WAM mean success rises from 91.83% to 92.88% across all 50 RoboTwin 2.0 tasks and from 32.5% to 52.5% on Sharpa North.The Sharpa North result covers two real-robot tasks.Source: §1 · p. 2
  • POD-RoPE represents token observation footprints as distributions over camera centers, viewing directions, and sampling times, recovering pointwise RoPE when a distribution collapses to one address.Expected rotations encode relative relationships between independently sampled addresses.Source: §1 · pp. 1–2
  • Shared per-token rotation statistics support Q/K matching and coherence-dependent V/O transformations without extra tokens or pairwise processing of observation nodes.This lets observation geometry guide both information selection and content aggregation.Source: §1 · p. 2
  • Evaluations span four robot policy architectures and three visual-geometry tasks, with task-dependent geometry gains, ablations supporting both components, and lower model-block latency than P-RoPE and RayRoPE.In H20 benchmarks, the full method has median forward-plus-backward speedups of 1.91× over P-RoPE and 5.34× over RayRoPE, with 17.4% median overhead over Standard RoPE.Source: §1 · p. 2

2 RELATED WORK

Prior work applies positional rotations and camera geometry to attention, while regional encodings and compressed visual tokens motivate representations that retain observation extent. POD-RoPE differs by representing token footprints as physical distributions and reusing their rotation statistics across attention interactions.

2 RELATED WORK

  • RoPE encodes relative displacement through query/key rotations, while GTA and PRoPE extend geometry-aware attention with camera transformations and projective view-frustum relationships (Miyato et al., 2024; Li et al., 2025).RayRoPE uses expected RoPE along projective rays and applies transformations to Q/K/V/O (Wu et al., 2026).Source: §2 · pp. 2–3
  • POD-RoPE targets tokenization footprints by jointly encoding camera centers, viewing directions, and actual time, then pre-aggregating rotation statistics per token.The full method uses rotation coherence to modulate V/O transformations with an identity branch.Source: §2 · p. 3
  • Mip-NeRF averages sinusoidal features over Gaussian approximations of conical frustums, while RayRoPE applies expected rotations to predicted-depth uncertainty intervals (Barron et al., 2021; Wu et al., 2026).DiT and Wan use visual tokens that represent multiple pixels and potentially multiple source frames (Peebles & Xie, 2023; Team Wan et al., 2025).Source: §2 · p. 3
  • The paper studies observation geometry across ACT, RDT, π0, and Fast-WAM policy architectures (Zhao et al., 2023; Liu et al., 2025; Black et al., 2025; Yuan et al., 2026).Evaluations include RoboTwin 2.0 and Sharpa North manipulation, reconstruction datasets, UniMatch depth estimation, and LVSM novel-view synthesis.Source: §2 · p. 3

3 METHOD

POD-RoPE maps physical observation geometry and time to addresses, averages address-dependent rotations over each token’s footprint, and uses the resulting statistics for relative attention and content aggregation.

3.1 FROM DISCRETE INDICES TO PHYSICAL OBSERVATION ADDRESSES

  • Camera centers, unit viewing directions, and actual time serve as positional inputs because discrete token indices do not express camera baselines, viewing directions, or elapsed-time intervals.The address construction uses camera calibration and maps observations to a shared reference frame.Source: §3.1 · pp. 3–4

3.2 FROM A SINGLE ADDRESS TO AN OBSERVATION DISTRIBUTION

  • Each token’s observation footprint is represented by a weighted address distribution, and averaging its address-dependent rotations yields a positional operator that recovers pointwise RoPE for a singleton address.The distribution describes the token’s observation composition, not uncertainty about an unknown scene position.Source: §3.2 · p. 4

3.3 PRESERVING RELATIVE RELATIONSHIPS BETWEEN DISTRIBUTIONS

  • For independently sampled addresses, the interaction between token operators equals the expected rotation over relative addresses, preserving relative relationships between observation sets in Q/K attention.Relative addresses include mapped time differences, camera-center displacement, and viewing-direction difference.Source: §3.3 · p. 4

3.4 EFFICIENT COMPUTATION AND DISTRIBUTION-GUIDED CONTENT AGGREGATION

  • Per-token rotation statistics can be reused across attention interactions without processing pairs of observation nodes, retaining computation over 2D feature blocks.Mean cosine and sine suffice to construct the positional operator for each token and frequency.Source: §3.4 · p. 5
  • Coherence-dependent transformations mix identity with the mean-direction rotation for content features, then transform values before weighted aggregation and apply an output transformation.The identity branch preserves content when rotations cancel; the full method fixes g = 1 without learnable gating parameters.Source: §3.4 · p. 5

4 EXPERIMENTS

Experiments evaluate POD-RoPE in robot policies, visual-geometry tasks, ablations, and model-block benchmarks. Results include higher mean manipulation success on RoboTwin and Sharpa North, dataset- and architecture-dependent geometry outcomes, and lower model-block latency than P-RoPE and RayRoPE.

4.1 END-TO-END EMBODIED MANIPULATION

  • 91.83% to 92.88% mean success across 50 RoboTwin 2.0 tasks with POD-RoPE integrated into Fast-WAM, under identical training and evaluation settings.Gains were 1.24 percentage points under Clean conditions and 0.86 under Randomized conditions.Source: §4.1 · p. 5, Table 1 · p. 6
  • All three single-task policies improve average success after POD-RoPE integration, but RDT and ACT improve primarily under Clean conditions and ACT reaches 3.07% under Randomized conditions.The gains vary substantially across architectures and evaluation conditions.Source: §4.1 · p. 6, Table 2 · p. 6
  • 32.5% to 52.5% mean success across two Sharpa North real-robot tasks with the complete POD-RoPE integration.Both tasks improved; evaluation used 20 trials per task at the 20,000-step checkpoint.Source: §4.1 · pp. 6–7, Table 3 · p. 7

4.2 FURTHER EVALUATION ON VISUAL GEOMETRY TASKS

  • Cross-view reconstruction improves all four metrics over P-RoPE on Sharpa North and RoboTwin, but on AGIBot improves PSNR while slightly worsening LPIPS.The dataset-dependent results use synchronized camera observations to reconstruct a third view.Source: §4.2 · p. 7, Table 4 · p. 7
  • At h16, POD-RoPE improves future-target PSNR over P-RoPE on all three datasets when given the target camera geometry and time.The current reference observations are fixed under matched settings.Source: Table 4 · p. 7, §4.2 · p. 8
  • Depth estimation results are mixed: full POD-RoPE improves 12 of 14 metrics over Q/K-only and 10 over P-RoPE, while RayRoPE performs better on four RGBD metrics.POD-RoPE leads on five Scenes11 metrics and ties RayRoPE on δ1.Source: §4.2 · p. 8, Table 5 · p. 8

4.3 ABLATION STUDIES

  • Distribution Q/K improves PSNR/SSIM/LPIPS over point Q/K by +0.534/+0.0259/−0.0373; V/O improves all metrics with both encodings, while direct-average V/O underperforms distribution Q/K-only by 0.373 dB.With V/O enabled, distributions improve over points, including 0.744 dB in PSNR; the full method gains 0.923 dB over direct averaging, supporting the identity branch as coherence vanishes.Source: §4.3 · p. 8, Table 6 · p. 9

4.4 COMPUTATIONAL EFFICIENCY

  • Full POD-RoPE achieves 1.91× speedup over P-RoPE and 5.34× over RayRoPE in median forward-plus-backward model-block benchmarks, with 17.4% median latency overhead over Standard RoPE.The benchmarks use separate model-block configurations; their absolute latencies are not pooled.Source: §4.4 · p. 8, Figure 3 · p. 9

5 CONCLUSION

POD-RoPE treats token footprints as distributions over physical observation addresses, using shared rotation statistics to guide matching and content aggregation. The construction preserves relative Q/K interactions while accounting for footprint coherence and physical spatiotemporal conventions.

5 CONCLUSION

  • POD-RoPE uses physical observation distributions for positional encoding and reuses their rotation statistics for Q/K matching and coherence-dependent V/O aggregation.The method handles static and temporal observations without increasing sequence length; the conclusion reports improved aggregate manipulation success and task-dependent visual-geometry gains.Source: §5 · p. 9

A.1 OBSERVATION DISTRIBUTIONS AND THE RELATIVE-POSITION PROPERTY

  • Independent sampling makes expected rotations encode relative relationships between observation distributions, and singleton distributions recover pointwise RoPE.A common address translation leaves the pairwise Q/K positional kernel unchanged.Source: §A.1 · p. 11

A.2 OBSERVATION FOOTPRINTS AND ROTATION MAGNITUDE

  • Distribution encoding retains footprint information that center-only encoding misses, distinguishing equal-center footprints through average-rotation magnitude.For ωa = π/6 and π/2, the magnitudes are approximately 0.955 and 0.637 despite the same center and mean direction; magnitude describes rotation coherence at a frequency.Source: §A.2 · p. 12

A.3 PROPERTIES OF DISTRIBUTION-GUIDED CONTENT AGGREGATION

  • The V/O transformation is a convex combination of identity and rotation, has norm at most one, and returns to identity when rotation coherence is zero.Unlike the Q/K positional kernel, distribution-guided content aggregation generally also uses absolute address information.Source: §A.3 · p. 12

A.4 PHYSICAL ADDRESSES AND PREPROCESSING CONVENTIONS

  • Physical addresses combine time, camera centers, and viewing directions, with units and preprocessing conventions governing their interpretation.Time uses seconds, camera centers retain input coordinate units, static tasks omit time, and image preprocessing updates intrinsics consistently after crop and resize.Source: §A.4 · pp. 12–13

A.5 NUMERICAL IMPLEMENTATION AND COMPLEXITY

The implementation estimates per-token rotation statistics from weighted observation nodes and reuses them across attention interactions. Its complexity and measured latency are reported alongside per-case timing details.

A.5 NUMERICAL IMPLEMENTATION AND COMPLEXITY

  • Per-token statistics are computed from weighted observation nodes and reused across attention interactions, with O(NM) storage and O(Ndh) costs for Q/K encoding and V/O transformations.Computing the statistics costs O(M Σ_i Ki), while the dominant attention term remains O(N^2dh).Source: §A.5 · p. 13
  • Visual tokens use 3 × 3 spatial nodes, and temporal bins cover at most four source frames, yielding at most 36 nodes per token.The temporal bin specifies observation nodes rather than the full receptive field of the causal VAE.Source: §A.5 · p. 13

B.1 PER-CASE LATENCY

  • Table 7 reports median per-case forward and forward-plus-backward latency in milliseconds, with 10 warm-up runs and 30 timed measurements per case.RayRoPE entries are missing for T3 and T5 because its reference implementation does not accept those multiframe shapes.Source: §B.1 · p. 13, Table 7 · p. 14

B.2 COMPLETE RE10K METRICS

The RE10K section reports LVSM novel-view synthesis metrics and relates a matched-protocol speed axis to a separate reconstruction-block microbenchmark.

B.2 COMPLETE RE10K METRICS

  • Figure 3(b) uses forward-plus-backward latency from the matched RE10K/LVSM timing protocol as its speed axis.The corresponding quality metrics are reported under the same 80k training protocol.Source: §B.2 · p. 14
  • The RE10K/LVSM timing results are reported separately from reconstruction-block microbenchmarks.The passage does not state the numerical metric outcomes.Source: §B.2 · p. 14
  • Table 8 reports LVSM novel-view synthesis on RE10K after 80k training steps, with n = 6474.The table includes PSNR, SSIM, and LPIPS; bold denotes the best result and pale blue identifies POD-RoPE variants.Source: Table 8 · p. 14

C POLICY RESULTS ON ALL ROBOTWIN 2.0 TASKS

The policy-results section reports Fast-WAM success rates across all 50 RoboTwin 2.0 tasks under shared training and evaluation settings. Results use a fixed checkpoint and equal task weighting.

C POLICY RESULTS ON ALL ROBOTWIN 2.0 TASKS

D RECONSTRUCTION UNDER INPUT CHANGES

The input-change evaluation tests reconstruction when reference inputs are altered, with effective intrinsics and actual times updated consistently. Results vary by transformation: image-size changes generally favor POD-RoPE, while temporal resampling alone can favor P-RoPE.

D RECONSTRUCTION UNDER INPUT CHANGES

  • The protocol changes reference inputs, unlike conditional temporal extrapolation, which holds references fixed and moves the future target.Each transformation updates effective intrinsics and actual times consistently.Source: D RECONSTRUCTION UNDER INPUT CHANGES · pp. 15–16, Table 10 · p. 16
  • On original RGB, POD-RoPE’s PSNR differences over P-RoPE are +4.160 dB on Sharpa North, +3.321 dB on RoboTwin, and +2.473 dB on AGIBot.Under res336, PSNR, SSIM, and LPIPS also improve over P-RoPE on all three datasets.Source: D RECONSTRUCTION UNDER INPUT CHANGES · p. 16, Table 11 · p. 16
  • For cropping and mixed-FOV conditions, all PSNR-difference intervals are above zero, although LPIPS is slightly worse on RoboTwin and does not improve under AGIBot’s native 70-degree FOV.Table 11 reports POD-RoPE − P-RoPE differences with trajectory-level paired 95% confidence intervals.Source: D RECONSTRUCTION UNDER INPUT CHANGES · p. 16, Table 11 · p. 16
  • When only the sampling interval changes, POD-RoPE has lower PSNR than P-RoPE under stride5/8 and irregular sampling on both Sharpa North and RoboTwin.When sampling-interval changes are combined with size changes, the PSNR differences become positive.Source: D RECONSTRUCTION UNDER INPUT CHANGES · p. 16

E.1 METHOD INTEGRATION AND TRAINING BUDGETS

The integration section specifies how POD-RoPE fits existing policy attention layers and configures observation support, scales, and transformations. The reconstruction evaluation uses fixed checkpoints and tests synchronized view reconstruction and conditional extrapolation across three datasets.

E.1 METHOD INTEGRATION AND TRAINING BUDGETS

  • POD-RoPE uses existing visual attention without extra adapters or trainable parameters in π0 and ACT; Fast-WAM simulation and RDT add residual attention modules with 3.154M and 0.887M parameters.Those additions equal approximately 0.052% and 0.072% of the respective baselines’ trainable parameter counts, excluding frozen encoders.Source: §E.1 · p. 17
  • Reconstruction uses nine Gaussian spatial nodes per source frame and equally weighted temporal support frames, with joint temporal-spatial weights normalized over each token’s support.Temporal coordinates use t0 as the minimum timestamp across reference and target support nodes and a fixed st = 0.125 s.Source: §E.1 · pp. 17–18
  • Spatial scales are calibrated from RE10K training-view baseline lengths and within-token ray-angle variation, while attention applies the positional operator after Q/K projection.The full method also aggregates content through Gjvj and applies G⊤.Source: §E.1 · p. 18

E.2 EMBODIED RECONSTRUCTION AND CONDITIONAL TEMPORAL EXTRAPOLATION

  • The three reconstruction models are trained independently per dataset, then evaluated on third-view reconstruction from two references using shared splits and context-consistent VAE-decoded targets.Metrics are task-averaged within trajectories and then trajectory-averaged; extrapolation fixes the references and evaluates actual future targets with known target geometry.Source: Figure 4 · p. 18, §E.2 · p. 19
  • On RoboTwin, POD-RoPE is 0.581 dB below P-RoPE at h0 but leads all four metrics at h16; LPIPS does not improve on AGIBot.The temporal-extrapolation figure covers h = 0, 1, 2, 4, 8, 16, with different actual delays for each dataset.Source: §E.2 · p. 19, Figure 5 · p. 19

E.3 SINGLE-TASK EVALUATION ACROSS POLICY ARCHITECTURES

The single-task section evaluates policies on 15 RoboTwin 2.0 tasks under seen and unseen instructions. The following all-task evaluation trains Fast-WAM jointly on 50 tasks and averages results equally across tasks and evaluation conditions.

E.3 SINGLE-TASK EVALUATION ACROSS POLICY ARCHITECTURES

  • For the 15-task study, each model is independently trained per task using its corresponding baseline recipe; each task uses 50 Clean demonstrations.ACT allocates 40 demonstrations for training and 10 for validation; Clean/Randomized correspond to Easy/Hard.Source: §E.3 · pp. 19–20
  • Each single-task evaluation uses 100 scenes per task and environment, native execution and success criteria, and equal averaging across the 15 tasks.Clean uses seen instructions; Randomized uses unseen instructions.Source: §E.3 · p. 20

E.4 FAST-WAM SIMULATION EVALUATION ACROSS ALL TASKS

  • The 50-task joint Fast-WAM study trains on 27,500 trajectories and evaluates 100 episodes per task in each of Clean and Randomized conditions.Results are averaged equally over tasks and then equally over the two conditions; both groups use identical training steps and episode counts.Source: §E.4 · p. 20

E.5 VISUAL GEOMETRY AND EFFICIENCY EVALUATION

The visual-geometry and efficiency experiments specify matched evaluation protocols for depth, novel-view synthesis, and model-block timing. Component ablations vary point versus distribution encoding and enable V/O transformations; the real-robot evaluation uses two Sharpa North tasks with shared preprocessing and training conditions.

E.5 VISUAL GEOMETRY AND EFFICIENCY EVALUATION

  • The H20 microbenchmark measures a six-layer reconstruction block in BF16, while the separate quality–efficiency plot uses a matched RE10K/LVSM timing protocol.Neither timing setup includes the full data-loading, decoding, optimizer, or simulation pipeline.Source: §E.5 · p. 20

E.7 REAL-ROBOT EVALUATION ON SHARPA NORTH

  • For real-robot evaluation, head-camera inputs are fixed while the robot reconstructs a wrist view, and each method receives 20 trials per task.The tasks are bottle opening and straw insertion; success requires completing all task steps.Source: §E.6 · p. 20, §E.7 · p. 21

E.6 COMPONENT ABLATION

  • The ablation compares five physical-encoding configurations by varying point versus distribution Q/K encoding and enabling V/O transformations.All five share frequency bands, scale, temporal mapping, and Q/K normalization; the full method changes the V/O gate from g = 0 to g = 1.Source: §E.6 · p. 21

E.7 REAL-ROBOT EVALUATION ON SHARPA NORTH

  • Robot camera links are mapped to the robot root through URDF forward kinematics, not an independently and precisely calibrated world-coordinate system.The policy reads only current observation geometry, and wrist intrinsics use an approximation based on a historical fisheye model.Source: §E.7 · p. 21
Loading 2610.33745v2…