PaperlayerView exact revision on arXiv

Source-linked AI summary

MinkowskiPE: Minkowski Positional Encoding for Spatiotemporal Perception

Yuhao Li, Louie Hong Yao, Tianyi Shi, Hanqun Cao, Hongxia Hao, Zhen Zhao, Shengchao Liu

arXiv:2609.33804v1cs.LGcs.AIcs.CVphysics.comp-ph

TL;DR

Spatiotemporal models face a trade-off between physics-motivated structure and learning-based flexibility, with coordinate relations often left implicit. MinkowskiPE uses joint time–space coordinates to parameterize Lorentz transformations of query and key features. It achieves the best results on all nine multi-trajectory molecular evaluations and reduces KTH video-prediction MSE by 9.9% relative to the best baseline while using roughly one-tenth as many parameters.

  • Problem

    Physics-motivated approaches may constrain flexibility, while learning-based approaches often leave the relation between temporal and spatial coordinates implicit.

    Source: §1 · p. 2

  • Method

    MinkowskiPE uses joint temporal and spatial event coordinates to parameterize Lorentz transformations of query and key features, so attention depends on position through relative spacetime displacement.

    Source: §1 · p. 2

  • Results

    MinkowskiPE achieves the best performance on all nine multi-trajectory molecular evaluations and reduces KTH video-prediction MSE by 9.9% relative to the best baseline while using roughly one-tenth as many parameters.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The same positional construction applies across molecular dynamics and video prediction despite differences in physical scale, token semantics, spatial structure, and prediction architecture.

    Source: §6 · p. 10

    MinkowskiPE uses a fixed geometric bias rather than learning geometry from data, and the experiments focus on molecular and visual forecasting.

    Source: §6 · p. 10

Abstract

from arXiv · show

Modeling spatiotemporal coupling is a key challenge in building physical intelligence across scales, from microscopic to macroscopic. Existing models capture such structure broadly through physics-motivated dynamical formulations or learning-motivated architectures. The former provide stronger priors but may constrain flexibility, whereas the latter are more flexible but leave the spatiotemporal coupling largely implicit. We therefore seek an approach that combines flexible learning with an explicit geometric bias for jointly modeling time and space. To this end, we propose Minkowski Positional Encoding (MinkowskiPE), which uses joint temporal and spatial coordinates to parameterize Lorentz transformations applied to query and key features. With MinkowskiPE, the query-key attention score depends on position only through the relative spacetime displacement between the two tokens and is therefore invariant to global translation of the coordinates. This paradigm retains the standard dot-product attention interface and remains compatible with efficient attention implementations. We evaluate MinkowskiPE on microscopic molecular dynamics and macroscopic video prediction tasks, achieving the best results on all nine multi-trajectory molecular evaluations and reducing KTH video-prediction MSE by 9.9% relative to the best baseline while using roughly one-tenth as many parameters.

1 Introduction

MinkowskiPE addresses the gap between physics-motivated models that may limit flexibility and learning-based models whose coordinate relations often remain implicit. It combines a geometric prior with data-driven learning and performs strongly on molecular dynamics and video prediction.

1 Introduction

  • Molecular motion and video can both be viewed as observations associated with spatiotemporal coordinates, despite their differences in scale, modality, and data structure.Molecular states pair atoms with time steps and 3D positions, while video patches pair frames with spatial locations.Source: §1 · p. 1
  • The paper asks whether spatiotemporal coupling can combine data-driven flexibility with a strong physical prior, using positional encoding to model temporal and spatial coordinates while preserving their distinct roles.The motivation is that physics-motivated methods may constrain flexibility, while learning-based methods often leave the relation between temporal and spatial coordinates implicit.Source: §1 · pp. 1–2
  • MinkowskiPE uses joint temporal and spatial coordinates to parameterize Lorentz transformations of query and key features, making pairwise attention depend on position only through relative spacetime displacement.It preserves the standard dot-product attention interface and is compatible with efficient implementations such as FlashAttention.Source: §1 · p. 2
  • MinkowskiPE achieves the best performance on all nine multi-trajectory molecular evaluations and reduces KTH video-prediction MSE by 9.9% relative to the best baseline while using roughly one-tenth as many parameters.It ranks among the top two in 26 of 30 single-trajectory molecular evaluations and achieves the best performance on four of five KTH metrics.Source: §1 · p. 2

2 Related Work

Prior spatiotemporal models either organize temporal structure through propagation or dynamical mechanisms, or learn dependencies through attention. MinkowskiPE instead places its inductive bias in the positional representation.

2 Related Work

  • Recurrent, convolutional, continuous-time, and physically structured approaches encode temporal evolution through prescribed propagation or dynamical mechanisms, potentially limiting flexibility in learning dependencies from data.Examples include differential-equation constraints, Hamiltonian dynamics, and graph-based simulation.Source: §2 · p. 3
  • Transformer methods learn spatiotemporal dependencies through attention organizations including factorized spatial–temporal attention, multiscale representations, local windows, and trajectory-aware attention.MinkowskiPE differs by introducing inductive bias at the positional-representation level rather than prescribing temporal evolution or a specific attention organization.Source: §2 · p. 3
  • Positional representations range from relative embeddings and attention biases [23] to multidimensional and group-based methods [29] [33] [35], including recent spatiotemporal video encodings.MinkowskiPE uses joint temporal and spatial event coordinates to parameterize Lorentz transformations within a single relative positional mechanism.Source: §2 · p. 3

3 Preliminaries

Relative positional encoding makes pairwise interaction depend on coordinates through their displacement, yielding invariance to a common translation. Minkowski geometry supplies an indefinite metric that distinguishes temporal from spatial directions and is used here as a representation structure.

3.1 Relative Positional Encoding

  • A relative positional encoding makes pairwise interaction depend on positions only through the displacement Δxij = xj − xi, so a common coordinate translation leaves the score unchanged.The score is formed from query and key feature transformations before attention scaling and softmax.Source: §3.1 · p. 3

3.2 Minkowski Geometry and Lorentz Transformations

  • A spacetime event has temporal coordinate t and spatial coordinate r, and Minkowski geometry assigns them different roles through an indefinite metric.The metric’s signature makes temporal and spatial directions enter its bilinear form with opposite signs; transformations preserving that form are Lorentz transformations.Source: §3.2 · pp. 3–4
  • MinkowskiPE uses coordinates to parameterize Lorentz transformations of query–key features rather than transforming the event coordinates or imposing Lorentz symmetry on the data.The Lorentz-preserving identity is the algebraic ingredient used to construct a relative positional interaction.Source: §3.2 · p. 4

4 Method

The method treats spatiotemporal tokens as events and uses their coordinates to parameterize Lorentzian query–key transformations whose positional interaction depends only on relative displacement. A practical multi-head form assigns learned projections to feature blocks.

4 Method

  • Tokens are represented as events (hi, xi), where xi = (ti, ri), and their relative displacement is Δxij = xj − xi.A video token may be an image patch with ri in 2D; a molecular-dynamics token may be an atom with ri in 3D.Source: §4 · p. 4

4.1 Minkowski Positional Encoding

  • MinkowskiPE parameterizes Lorentz transformations with event coordinates so the pairwise query–key interaction depends on coordinates only through relative displacement.A scalar projection of each coordinate determines the one-parameter Lorentz transformation; a one-parameter subgroup provides a tractable construction given the commuting-transformation requirement for multidimensional coordinates.Source: §4.1 · pp. 4–5

4.2 Practical Parameterization

  • For a full attention head, query and key features are partitioned into two-dimensional blocks, each with a learned coordinate projection shared across Transformer layers and attention heads.The block-wise Lorentz transformations can also be written in element-wise form, with the key transformation including the Minkowski metric factor.Source: §4.2 · p. 5

4 Method

  • MinkowskiPE is evaluated in a task-agnostic scheme that pairs task-specific features with spatiotemporal coordinates, processes them with a Transformer, and maps them to future states using task-specific decoders.The overview applies this scheme to molecular dynamics and video prediction at different physical scales.Source: Figure 2 · p. 6

5 Experiments

The experiments test MinkowskiPE on molecular dynamics and video prediction using a shared spatiotemporal formulation across different spatial structures. Results include best performance on all nine multi-trajectory molecular evaluations and improved KTH video-prediction metrics under the protocol-matched comparison.

5 Experiments

  • Molecular dynamics and video prediction differ in physical scale and data structure but fit the same task-agnostic spatiotemporal framework.The former models atomic motion in 3D; the latter models visual evolution on a 2D image plane.Source: §5 · p. 6

5.1 Molecular Dynamics Prediction

  • For molecular dynamics, ligand-atom event tokens combine time and 3D position; attention uses MinkowskiPE, and an MLP predicts future displacements.The semi-flexible task predicts multiple future positions with the protein held fixed; single-trajectory models use 10 observed and 20 target frames, while multi-trajectory models use 80 observed and 20 target frames.Source: §5.1 · pp. 6–7
  • MinkowskiPE achieves the best result in 17 of 30 single-trajectory evaluations and ranks in the top two in 26 of 30.It leads all three metrics on 4K6W, 1XP6, 4G3E, and 3B9S; Figure 3 compares final predicted ligand configurations with references.Source: Table 1 · p. 7, §5.1 · p. 8, Figure 3 · p. 8
  • MinkowskiPE achieves the best performance on all nine multi-trajectory evaluations across three dataset scales.The advantage persists as the number and diversity of training trajectories increase and spans coordinate and structural metrics.Source: Table 2 · p. 7, §5.1 · p. 8

5.2 Video Prediction

  • For video prediction, an eight-layer global Transformer processes latent-grid tokens with MinkowskiPE to forecast future frames.The KTH setup uses 10 grayscale frames at 128 × 128 resolution to predict 20 frames, with MSE, MAE, SSIM, PSNR, and LPIPS evaluation.Source: §5.2 · p. 8
  • MinkowskiPE reduces KTH MSE, MAE, and LPIPS by 9.9%, 5.7%, and 1.7%, respectively, versus the strongest baseline for each metric.It also achieves the best performance on PSNR while remaining competitive on SSIM, using 2.5M parameters.Source: Table 3 · p. 9, §5.2 · p. 9

5.3 Effect of MinkowskiPE

  • Controlled ablations improve every evaluated metric in both domains when MinkowskiPE is added while the other model and training settings remain unchanged.On MISATO-1000, MAE and Matching fall by 2.7% and 3.9%, while Stability rises by 1.2%; on KTH, MSE and MAE fall by 5.7% and 3.6%, and LPIPS improves by 4.5%.Source: Table 4 · p. 9, §5.3 · p. 9

6 Discussion

MinkowskiPE uses Minkowski geometry as an attention representation bias, not as an assumption about the system’s dynamics or symmetry. The discussion highlights cross-domain reuse alongside limits from its fixed geometry, parameterization, and current experimental scope.

6 Discussion

  • MinkowskiPE does not require relativistic dynamics or Lorentz symmetry; it uses Minkowski geometry as a representation-level inductive bias for attention.Its relative-position interaction follows from Lorentz transformations in query–key feature space and linear coordinate projections.Source: §6 · p. 10
  • The same positional construction applies to molecular dynamics and video prediction despite their different scales, token semantics, spatial structures, and prediction architectures.Both domains use an event-based view of time and space without changing the underlying formulation.Source: §6 · p. 10
  • The method fixes its geometric bias rather than learning geometry from data, and different applications may benefit from other geometric biases.For one particular class of finite-dimensional unitary token-wise constructions, exact Galilean invariance prevents retaining the Galilean-invariant spatial component; this does not rule out other Galilean encodings.Source: §6 · p. 10
  • The current construction uses learned scalar projections and one-parameter Lorentz transformations; richer multi-parameter versions require mutually commuting generators.Experiments focus on molecular and visual forecasting, while embodied dynamics, weather forecasting, and generative models are future directions.Source: §6 · p. 10

7 Conclusion

The conclusion presents MinkowskiPE as a relative positional encoding that couples time and space through Lorentz transformations in query–key feature space. It reports strong performance across molecular dynamics and video prediction, while the AI-use statement describes writing, literature, and experimental support tools.

7 Conclusion

  • MinkowskiPE achieves strong performance across distinct spatiotemporal settings, including molecular dynamics and video prediction.The conclusion identifies positional geometry as a design axis for spatiotemporal Transformers.Source: §7 · p. 10
  • Its attention scores depend on relative spacetime displacement, and the construction applies unchanged across different spatiotemporal tokenizations.The method uses a lightweight modification to query–key transformations.Source: §7 · p. 10

AI use statement

  • ChatGPT assisted with writing and literature discovery, while CodeX supported limited code review, debugging, utility scripts, and experimental organization; authors reviewed the AI-assisted material.The authors state they take responsibility for the final manuscript, code, and reported results.Source: AI use statement · p. 10

A Additional Derivations of MinkowskiPE

The appendix derives relative-position attention from a homomorphic Lorentz representation of event coordinates. Its key construction uses a metric factor so that query–key interaction reduces to a transformation of relative displacement.

A Additional Derivations of MinkowskiPE

  • The derivation formalizes how a homomorphic Lorentzian representation yields an exact relative-position interaction and characterizes smooth homomorphisms from additive spacetime coordinates into a Lorentz group.For the two-dimensional feature blocks used by MinkowskiPE, the construction reduces to a linear scalar projection followed by a one-parameter Lorentz boost.Source: A Additional Derivations of MinkowskiPE · p. 13

A.1 Relative Lorentzian Construction

  • The positional interaction depends on event coordinates only through their relative displacement when the coordinate-to-Lorentz mapping is a homomorphism.The homomorphism maps coordinate addition to composition of Lorentz transformations.Source: §A.1 · p. 14
  • The metric factor in the key transformation lets the query–key interaction reduce to a transformation of the relative coordinate.Without that factor, the interaction contains Γ(x_i)ᵀΓ(x_j), which generally does not reduce because Lorentz transformations are not orthogonal under the ordinary Euclidean inner product.Source: §A.1 · p. 14

A.2 Group-Theoretic Characterization

The appendix characterizes smooth homomorphisms from additive event coordinates into Lorentz groups. It shows why MinkowskiPE’s two-dimensional feature blocks yield a scalar-projection, one-parameter form, while higher-dimensional constructions face commuting-generator constraints.

Higher-dimensional Lorentz Representations

  • For higher-dimensional Lorentz representations, each coordinate direction generates a one-parameter subgroup, and an exact multidimensional homomorphism requires mutually commuting Lorentz generators.The event-coordinate group is additive and Abelian, so the images of its coordinate directions commute.Source: §A.2 · p. 14

The (1+1)-dimensional Case

  • Within a (1+1)-dimensional Lorentzian feature block, every smooth homomorphism from additive event coordinates into SO+(1, 1) has the scalar-projection, one-parameter boost form used by MinkowskiPE.The scalar projection is therefore the general smooth homomorphic parameterization in this feature-block class, including the trivial zero-projection case.Source: §A.2 · p. 15

Higher-dimensional Lorentz Representations

  • Independent boosts along different positive-signature directions do not commute, so they cannot directly serve as independent generators of an additive-coordinate homomorphism.For noncommuting generators, composition introduces commutator terms and the simple additive relation generally fails.Source: §A.2 · pp. 15–16
  • Higher-dimensional commuting constructions remain possible, but richer multi-parameter encodings require choices of commuting subalgebras and are left for future work.The (1+1)-dimensional blocks provide a minimal feature-space realization that mixes the two metric signatures while admitting an exact one-parameter homomorphic form.Source: §A.2 · p. 16

B Additional Experimental Details

The implementation normalizes event coordinates differently for molecular dynamics and video, then computes the Lorentzian coordinate projection and hyperbolic functions in float32. Attention uses PyTorch scaled dot-product attention.

B.1 Implementation Details of MinkowskiPE

  • Molecular-dynamics spatial coordinates are centered by the last observed ligand frame’s mass-weighted center of mass, while observed-frame times are mapped to [−1, 1].The spatial normalization uses ligand positions and cropped-protein Cα positions; the passage also describes a task-dependent spatial range.Source: §B.1 · p. 16
  • Video temporal and spatial coordinates are constructed on a regular grid in [−1, 1]3.The normalization keeps inputs in a numerically well-behaved regime for the hyperbolic functions.Source: §B.1 · p. 16
  • Coordinate projection and cosh/sinh evaluation use float32; transformed queries and keys are cast back to the attention dtype before scaled dot-product attention.On the experiments’ NVIDIA H200 GPUs, PyTorch dispatches the operation to FlashAttention.Source: §B.1 · p. 16

B.2 Molecular Dynamics Prediction

The appendix details molecular-dynamics and video-prediction data, architectures, and training procedures. The molecular model combines ligand–protein cross-attention with ligand-trajectory self-attention, while the video model processes observed and future latent maps jointly.

B.2 Molecular Dynamics Prediction

  • Molecular-dynamics experiments use MISATO trajectories, with single-trajectory models trained on 10 observed and 20 target frames and multi-trajectory windows using 80 observed and 20 target frames.The multi-trajectory datasets contain 80/10/10, 800/100/100, and 13,066/1,357/1,357 training/validation/test complexes for MISATO-100, MISATO-1000, and MISATO-Full, respectively.Source: §B.2 · pp. 16–17
  • Each molecular block performs frame-wise ligand-to-protein cross-attention followed by global self-attention over ligand event tokens; the model predicts future displacements from last-frame ligand representations.All future frames are predicted in parallel, and training combines coordinate prediction error with pairwise structural consistency.Source: §B.2 · pp. 17–18, Figure S1 · p. 17
  • Molecular-dynamics evaluation reports coordinate error, Matching, and Stability; multi-trajectory baselines are retrained under the in80/out20 protocol.Matching is an RMSE over valid ordered atom pairs, and Stability is the fraction of pairwise distances with absolute error at most 0.5 Å.Source: §B.2 · p. 18

B.3 Video Prediction

  • KTH video experiments use 10 observed grayscale frames to predict 20 future frames, with 5,200 training clips and 3,167 test clips.An eight-layer global Transformer processes observed and future latent maps with MinkowskiPE; the decoder generates all 20 future frames in parallel.Source: §B.3 · pp. 18–19
  • The video model is trained with frame-level mean squared error and evaluated using MSE, MAE, SSIM, PSNR, and LPIPS.The model contains 2,482,086 parameters, reported as 2.5M in Table 3.Source: §B.3 · p. 19

C Constraints on Galilean-Invariant Positional Encoding

This appendix studies a specific route to Galilean-invariant positional encoding: finite-dimensional unitary token-wise transformations. Under exact common-transformation invariance, that construction loses relative spatial information, but the result does not rule out other approaches.

C Constraints on Galilean-Invariant Positional Encoding

  • Exact invariance under common Galilean transformations forces finite-dimensional unitary token-wise pairwise interactions to discard the relative spatial component.The proposition concerns a particular construction and does not establish that Galilean positional encoding is impossible in general.Source: C Constraints on Galilean-Invariant… · pp. 19–20
  • For spacetime coordinates alone, relative spatial displacement changes under a common Galilean boost; adding token velocity yields an invariant spatial quantity r_ij.The relative quantity is r_ij = x_j − x_i − u_i(t_j − t_i), alongside relative time and velocity.Source: C Constraints on Galilean-Invariant… · pp. 19–20
  • The obstruction is specific to exact, finite-dimensional, unitary, token-wise factorizations; such encodings may still represent relative time and relative velocity.The example pairwise interaction contains no dependence on the Galilean-invariant spatial quantity r_ij.Source: C Constraints on Galilean-Invariant… · p. 21
  • Explicitly pairwise Galilean-invariant constructions and non-unitary or infinite-dimensional representations fall outside the proposition’s assumptions and remain open possibilities.Projective representations and central extensions are also identified as directions requiring separate analysis.Source: C Constraints on Galilean-Invariant… · p. 21
Loading 2609.33804v1…