PaperlayerView exact revision on arXiv

Source-linked AI summary

Local Temporal Contrasts on Human Motion Data Induce a Latent Geometry

Lourenço A. Rodrigues, Felix Putze

arXiv:2610.21630v1cs.LG

TL;DR

Time-series research still faces the challenge of preserving how latent states evolve in representations for downstream use. SemTAC uses temporal-neighborhood contrasts to induce structure in representation space without forecasting. The authors report improved performance on partial sequences and more structured representation spaces.

  • Problem

    Time-series models still face challenges identifying the right analysis timescale and preserving latent-state evolution for downstream use.

    Source: §1 · p. 1

  • Method

    SemTAC uses contrastive learning on temporally informed positive pairs to connect local temporal interactions with global structure in representation space.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    SemTAC improves robustness to partial sequences over contrastive and classification baselines, and its local-temporal contrastive approach yields better-structured representation spaces.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The results support modeling time-series representations as structures that reflect global temporal structure rather than only static embeddings.

    Source: §1 · p. 2, §5 · p. 9

    Inducing branching structure trades off some motion-type cluster isolation, while the self-similarity transformation also shows sensitivity in single-motion classification results.

    Source: §5 · p. 9

Abstract

from arXiv · show

Time series such as recordings of human motion data often encode structure across multiple temporal scales, and when a latent state determining this sequence evolves, this structure needs to survive in the representation space for downstream use. Yet the currently prevalent attention-based encoders paired with sequence-level pooling have made single-vector, pointwise comparison the default mode of inference, discarding the trajectory a sequence traces through the representation space. We propose recovering this geometric structure directly, without relying on forecasting-based objectives, by training an encoder model (SemTAC) with a temporal-neighborhood contrastive loss: representations that satisfy local consistency everywhere must give rise to a coherent global structure, letting inference happen at temporal scales beyond those seen during training. We evaluate SemTAC against four baselines on a five-step motion sequence discrimination task: the same architecture trained with a classification or a standard contrastive objective, and two established self-supervised time-series methods, TS-TCC and TS2Vec. At the full sequence length all five are broadly competitive, but on incomplete, truncated sequences SemTAC holds a large advantage: a median F1 of 0.77 one step before the sequence completes, against 0.27-0.70 for the four baselines. A hyperparameter-matched ablation attributes this effect specifically to the temporal-contrastive objective. SemTAC's representation trajectories are also the only ones among these five to exhibit interpretable geometric structures, exceeding a no-structure chance baseline on a displacement-smoothness metric which we introduce. This is evidence that its objective, not classification accuracy alone, produces genuinely temporally-ordered trajectories.

1 INTRODUCTION

The introduction identifies two challenges in retaining temporal structure: choosing a scale and preserving how latent states evolve. It proposes inducing trajectory geometry with temporal contrasts, then comparing sequence geometries at inference.

1 INTRODUCTION

  • Time-series data can span multiple temporal scales, while existing approaches often fix one resolution or aggregate across time.The introduction also identifies preserving latent-state evolution after choosing a scale as a challenge.Source: §1 · p. 1
  • Attention-based encoders paired with pooling reduce inference to point comparisons, discarding how representations evolve through a process.Such point representations can capture state similarity but not where a state lies in a process or what may follow it.Source: §1 · pp. 1–2
  • Forecasting-based approaches assume a signal’s dynamics can predict what comes next, an assumption that may fail when states change in stages.The introduction gives the example of a health condition advancing through a qualitatively new symptom rather than gradual symptom evolution.Source: §1 · p. 2
  • The approach uses temporal contrasts to induce a global latent structure from local temporal interactions, without relying on forecasting.The introduction argues that capturing local interactions along a trajectory requires a coherent structure beyond points seen during training.Source: §1 · p. 2
  • This global structure supports composing encoded states into longer trajectories and comparing them geometrically without retraining at a coarser scale.The paper also identifies this geometry as potentially useful for early prediction and monitoring by revealing which states precede others.Source: §1 · p. 2
  • The paper frames time-series representation learning as modeling latent trajectories rather than static embeddings, with trajectory portions informative for instantaneous classification.Its contributions also include geometry-level inference and empirical evaluation of partial-sequence robustness and representation structure.Source: §1 · pp. 2–3

2 RELATED WORK

Related work spans contrastive learning, sequential architectures, and trajectory-aware evaluation. The paper positions its contribution as using temporal structure to induce and use geometry across representation trajectories.

2 RELATED WORK

  • Contrastive learning, introduced as Contrastive Predictive Coding by Oord et al. (2019), has been applied broadly, including in computer vision.The passage cites Chen et al. (2020), He et al. (2020), and Radford et al. (2021) as examples.Source: §2 · p. 3
  • Contrastive learning work characterizes objectives through alignment and uniformity, but commonly relies on augmentations and does not explicitly model temporal dependencies.The related work describes this limitation for approaches operating primarily on static data.Source: §2 · p. 3
  • Recurrent and Transformer architectures model sequences, but temporal aggregation such as pooling can compress dynamics into a fixed-dimensional vector.The paper notes that this compression may discard information about how representations evolve over time.Source: §2 · p. 3
  • Time-series methods use local temporal consistency and transformation invariance; Tonekaboni et al. (2021) uses a similar temporal contrast but does not define pairs across motion stages.Other examples include Yue et al. (2022), Foumani et al. (2024), and Eldele et al. (2021).Source: §2 · p. 3
  • Existing time-series inference commonly compares point embeddings or clusters them, without explicitly using temporal-consistency structure across trajectories.Although multi-scale representations have been explored, the cited work does not leverage that structure at trajectory level.Source: §2 · p. 3
  • Trajectory clustering metrics exist in other domains, but the authors identify a gap in directly characterizing branch-like or temporal-order structure in learned representation trajectories.They cite longitudinal clinical clustering by Luong et al. (2019) and deep trajectory phenotyping by Wu et al. (2024).Source: §2 · p. 3

3 METHODS

The methods construct synthetic motion sequences with controlled temporal branching, train SemTAC using local temporal positive pairs, and assess sequence geometry, classification, and representation organization.

3.1 DATASET PREPARATION

  • The dataset uses generated motion prompts and arbitrary motion sequences to test whether models represent imposed temporal structure rather than physical signal progression.Two of the ten motion types were excluded from sequence creation after filtering.Source: §3.1 · p. 4
  • Sequences share their first two steps before gradually diverging, with opposite-side instances mixed in to prevent the first non-shared step from trivially identifying sequence type.Opposite-side relabeling is 40% for step motions and 20% for wave motions.Source: §3.1 · p. 4

3.2 TEMPORAL CONTRASTS

  • SemTAC combines a transformer-based Semantic Encoder with a TNC-like objective that treats adjacent motion instances as temporal positives.Training also uses partially overlapping segments from the same motion instance as positive pairs and computes InfoNCE loss.Source: §3.3 · p. 5, §3.2 · p. 5

3.4 EVALUATION

  • The evaluation compares full-sequence and step-4 sequence-type classification using DTW and Fréchet distances to left and right prototypes.Prototypes are sequences of step centroids computed from validation data.Source: §3.4 · pp. 5–6
  • Representation evaluation measures temporal encoding through displacement smoothness and order recovery, and sequence branching through silhouette and KNN consistency.Additional analyses test eight-class motion classification and visualize representations with PHATE plots.Source: §3.4 · p. 6

3.3 MODEL ARCHITECTURE

  • The comparisons include TS-TCC and TS2Vec, plus same-architecture ablations trained with self-similarity-only or supervised classification objectives.TS-TCC and TS2Vec timestamp representations are aggregated to match the experiment's five-point sequences.Source: §3.4 · pp. 6–7

4 RESULTS

SemTAC performs best on truncated sequences and learns temporal organization in its representation space, while remaining broadly competitive at full sequence length. Its sequence-branching performance remains below the classification ablation, and all models except Self-similarity perform significantly above chance on single-motion classification.

Performance at full sequence

  • At full sequence length, all models achieve average F1-scores above 0.8 with at least one of the two distance methods.Table 1 reports mean and standard deviation by approach and latent sequence distance.Source: Table 1 · p. 7, §4 · p. 7

Performance at truncated sequence

  • DTW yields higher F1-scores for four approaches, with the largest gains for TS2Vec (0.70 vs. 0.40), Self-similarity (0.64 vs. 0.37), and Classification (0.63 vs. 0.37).TS-TCC scores higher with Fréchet (0.51 vs. 0.47), while SemTAC’s DTW gain is 0.04 (0.72 vs. 0.68).Source: §4 · p. 7, Table 2 · p. 8
  • SemTAC’s median F1-scores on truncated sequences are 0.77 with Fréchet and 0.78 with DTW, significantly exceeding the other models except Classification with DTW.The Classification-versus-SemTAC difference with DTW is not statistically significant.Source: §4 · p. 7, Table 2 · p. 8

Space structure metrics

  • SemTAC is significantly better than the other models on Order Recoverability and smoothness, except for a nonsignificant Order Recoverability comparison with Classification.Its explained variability is 0.30 ± 0.23 versus 0.24 ± 0.18 for Classification, and it alone exceeds the -0.5 smoothness value associated with an unstructured sequence.Source: Table 3 · p. 8, §4 · p. 8

Sequence branching

  • SemTAC best approaches Classification’s sequence-branching separability, but a significant difference remains, indicating a trade-off between global structure learning and motion classification.The branching metrics assess separation of the two sequence types in representation space.Source: Table 4 · p. 8, §4 · p. 8

Single motion classification

  • All models except Self-similarity perform significantly above the 0.125 chance level on single-motion classification, supporting retention of information across latent structural resolutions.Table 5 reports mean and standard deviation for single-motion classification F1-scores.Source: §4 · p. 8, Table 5 · p. 9

Representation visualization

  • The selected PHATE visualization shows a divergent structure for SemTAC that is absent from the other model types, and the authors report that this pattern is representative across runs.Points are test data; colored poly-lines are validation-derived prototypes, with numbers indicating sequence steps.Source: Figure 3 · p. 9, §4 · p. 9

5 DISCUSSION

The discussion attributes the learned temporal geometry to adjacency-based contrasts, while noting a motion-cluster separation trade-off and that validation on real datasets remains future work.

5 DISCUSSION

  • Adjacency-based contrasts produce representation spaces organized like the dataset’s branching temporal structure, with segment adjacency driving the behavior.The authors identify this as the contribution, rather than a new architecture or objective.Source: §5 · p. 9
  • SemTAC’s branching structure comes at the cost of some motion-type cluster isolation, and the Classification ablation performs better on metrics rewarding motion separation.The discussion links this trade-off more to the self-similarity transformation than to branching alone.Source: §5 · p. 9
  • DTW is the better sequence-comparison choice in this inference approach, while the Fréchet distance may be sensitive to mislabeled samples at steps 3 and 4.The authors say confirming the proposed mislabeling explanation requires experiments with varying mislabeling rates.Source: §5 · p. 9
  • The structure-inducing effect was explored on synthesized data because its temporal structure was fully known; applying the methodology to real datasets remains future work.The authors identify real-data application as necessary to generalize the results into a validated approach for evolving time-series modeling.Source: §5 · pp. 9–10
  • The code and data are available, with environment and data-synthesis reproduction details provided in the paper and appendices.The authors also report full statistical comparison results in Appendix A.6.Source: REPRODUCIBILITY STATEMENT · p. 10

A.1 MOTION DATA GENERATION DETAILS

The motion dataset was generated with MDM using specified prompts, a checkpoint, and a fixed seed; two motion types were excluded after filtering, and lateralized labels received additional checks.

A.1 MOTION DATA GENERATION DETAILS

  • Twenty rephrasings were generated for each of the ten base prompts using ChatGPT (o3-mini), with instructions to preserve the original motion instruction.Source: §A.1 · p. 13
  • Motion instances were synthesized with MDM’s sample.generate command, random seed 42, and the humanml_enc_512_50steps checkpoint; other parameters remained at defaults.Source: §A.1 · p. 13
  • Two motion types—shaking an object and dancing the Hula-hoop—were excluded from sequence creation after low quality was observed during filtering.Source: §A.1 · p. 13
  • For lateralized motions, step labels were checked by net translation direction and wave labels by elbow-angle variability.Source: §A.1 · p. 13

A.2 MOTION PROMPT GENERATION

The prompt-generation procedure asked for varied phrasings of ten initial motion instructions while keeping each phrasing tied to the same motion.

A.2 MOTION PROMPT GENERATION

  • The prompt requested 20 different phrasings for each of ten initial motion instructions, from jumping and push-ups to lateral steps and waves.Source: §A.2 · p. 13
  • The rephrasings were intended to refer to the same motion while varying its wording to promote variability during generation.Source: §A.2 · p. 13

A.4 COMPUTATIONAL RESOURCES AND ENVIRONMENT

The paper reports the computing environment used for training and evaluation and identifies a separate table of post-filtering instance counts.

A.4 COMPUTATIONAL RESOURCES AND ENVIRONMENT

  • Training and evaluation ran on an Ubuntu 22.04 server; each model was trained on one NVIDIA RTX 6000 GPU with CUDA 12.4, using Python 3.10 and PyTorch 2.6.0.Source: §A.4 · p. 13
  • Table 6 reports accepted-instance counts after dataset filtering with the method from Rodrigues et al. (2026).Source: Table 6 · p. 14

A.5 NOTES ON SMOOTHNESS BASELINE SCORE

The smoothness score averages cosine similarity between consecutive displacement vectors, but unstructured embeddings have a chance-level expectation of −0.5 rather than 0. The appendix also describes statistical comparisons of sequence classification and temporal-structure metrics using corrected Mann–Whitney U tests.

A.5 NOTES ON SMOOTHNESS BASELINE SCORE

  • The smoothness score is the mean cosine similarity between consecutive displacement vectors in an embedding trajectory.The displacements are defined as the difference between successive embeddings.Source: §A.5 · p. 14
  • −0.5, not 0, is the chance-level baseline for smoothness on unstructured embeddings.Values approaching +1 indicate a smooth, locally linear trajectory.Source: §A.5 · p. 14

A.6 ABSOLUTE VALUES

  • Tables 7 and 8 compare SemTAC’s sequence-type classification with other model families at full length and length 4, respectively, using corrected Mann–Whitney U tests.Comparisons are reported separately for each inference distance method; the supplied captions do not state the test outcomes.Source: Table 7 · p. 15, Table 8 · p. 15
  • Tables 9 and 10 use corrected Mann–Whitney U tests to compare SemTAC’s sequence-branching and temporal-encoding metrics with other model families.The supplied captions identify the metrics and test, but do not provide comparison outcomes.Source: Table 9 · p. 15, Table 10 · p. 15

Related papers

Loading 2610.21630v1…