PaperlayerView exact revision on arXiv

Source-linked AI summary

Detecting Temporal Concepts in Video Transformers via a No-False-Positives Test

Jianghuai Li, Andrew Ren, Greg Shakhnarovich

arXiv:2610.33353v1cs.CVcs.LG

TL;DR

Video models may represent temporal concepts, but motion and appearance changes are coupled, making it difficult to tell whether a feature encodes motion or correlated static positions. The paper introduces a No-False-Positives test using controlled stimuli and finds temporal features in three video models, while interventions on those features have mixed results.

  • Problem

    Because motion and appearance changes are coupled, a feature that appears motion-sensitive may instead reflect static position concepts correlated with motion.

    Source: §1 · p. 1

  • Method

    The NFP test uses controlled synthetic stimuli so static features have constant expected activation over time, making significant covariance with a temporal concept evidence of non-static content.

    Source: §1 · p. 2, §8 · p. 10

  • Results

    The test identifies temporal features in VideoMAE, V-JEPA2, and TimeSformer, none in DINOv2, and yields mixed intervention results across video models.

    Source: §8 · p. 10

  • Takeaways & Limitations

    The relationship between detectable temporal features and task-relevant temporal reasoning varies across models.

    Source: §8 · p. 10

    The test detects non-static features covarying with the chosen temporal concept, not necessarily features encoding that concept specifically, and may miss features with zero covariance.

    Source: §3.5 · p. 6

Abstract

from arXiv · show

Video models have the architecture to support encoding temporal aspects of the input, i.e. concepts that require more than one frame to define, like object movement. However, because motion and appearance change are entangled, a feature that appears to encode temporal concepts like object movement may instead encode non-temporal concepts like spatial presence at different positions whose activations merely correlate with motion. To resolve this, we introduce the No-False-Positives (NFP) test, a statistical test with a formal guarantee of statistical specificity for identifying features encoding temporal concepts. Using specially designed stimuli, we apply this test to representations produced by VideoMAE, V-JEPA2, and TimeSformer, and observe that temporal features exist in each model. We validate the test first by showing it yields no temporal features for the DINOv2 image-encoder. Then, using a synthetic representation space, we show that detection rate increases with a direction's projection onto the temporal subspace, and that Sparse Autoencoder features test-flagged as temporal project much more of their norm onto the temporal subspace than unflagged features. Interventions on temporal SAE features yield mixed results across models, suggesting that intervening on temporal features may not outperform a non-temporal feature baseline.

1 INTRODUCTION AND MOTIVATION

The paper asks whether video-model features genuinely encode temporal concepts, since motion and appearance changes can make static position features appear motion-sensitive. It introduces the NFP test and applies it to video encoders, with additional validation and intervention analyses.

1 INTRODUCTION AND MOTIVATION

  • Motion and appearance changes are coupled, so static features for successive object positions can mimic activations attributed to motion.This makes observed behavior alone insufficient to distinguish temporal encoding from static concepts that correlate with motion.Source: §1 · p. 1
  • The NFP test is designed to ensure that a feature flagged as temporal is provably non-static, and is applied to VideoMAE, V-JEPA2, and TimeSformer.The paper frames the coupling problem as a false-positive problem: static features may match temporal activations or downstream effects.Source: §1 · p. 2
  • Synthetic moving-ball videos control frame-to-frame variation, while a constant distribution of ball positions at each time makes static features’ expected activation constant across time.Consequently, nonzero expected covariance with a temporal concept implies that a feature is non-static.Source: §1 · p. 2
  • The paper also presents a generalized NFP framework and an optimization procedure for applying it to arbitrary directions.It reports mixed results when erasing temporal SAE feature spans, suggesting such interventions do not generally outperform non-temporal baselines on temporally demanding tasks.Source: §1 · p. 2

2 EXISTING WORK ON TEMPORAL UNDERSTANDING IN MODELS

Prior work studies whether models use temporal information and whether temporal concepts are decodable, but neither resolves whether individual features encode temporal concepts rather than correlated static patterns. This paper instead asks that question at the feature level.

2 EXISTING WORK ON TEMPORAL UNDERSTANDING IN MODELS

  • Model-level benchmarks, probes, and action-direction or ordering tasks show whether models can use temporal information, but do not address its internal organization.The cited benchmark, probing, and action tasks include Goyal et al. (2017), Dave et al. (2023), and Shao et al. (2020).Source: §2 · p. 2
  • Linear decodability of motion direction in intermediate representations does not establish that a feature encodes motion rather than position patterns correlated with it.Joseph et al. (2026) report layers where motion direction is linearly decodable in V-JEPA2 and VideoMAE-v2.Source: §2 · pp. 2–3
  • Unlike representation-level probing, this work examines whether individual features are temporal and uses SAEs as an application use case.Whether temporal concepts have nonlinear encodings in representation space is explicitly outside this paper’s scope.Source: §2 · p. 3

3 THEORETICAL FRAMEWORK

The framework measures a representation-space direction’s activation at the token tracking a moving ball and compares its time series with temporal concepts. Controlled stimulus sampling gives static features constant expected activation over time, supporting a no-false-positives test based on covariance.

3 THEORETICAL FRAMEWORK

  • Synthetic videos depict a ball moving on a plain canvas, with motion profiles specifying its movement and position determining each frame’s content.The setup uses velocity profiles and assumes one-frame tubelets for the main framework.Source: §3 · p. 3

3.1 DEFINITIONS

  • A feature is static when its ball-tracking activation can be reduced to a function of the current ball position; temporal concept profiles describe per-frame properties such as velocity or direction.The measured activation series applies a function F to the direction’s activation at the token containing the ball’s center.Source: §3 · p. 3, §3.1 · p. 3

3.2 STIMULUS SET

  • The stimulus set independently varies starting position and motion profile, with coverage and off-screen zeroing arranged to equalize static features’ expected activation across frames.Different motion profiles are needed to associate a direction with a specific temporal concept; off-screen zeroing may reduce power but does not affect the NFP guarantee.Source: §3.2 · p. 4

3.3 TEST STATISTIC

  • The test statistic is within-video covariance between ball-tracking activation and a temporal concept; Pearson correlation does not preserve the NFP property.The resulting time series is compared with the selected temporal concept profile.Source: §3 · p. 3, §3.3 · p. 4

3.4 PROVING THE NFP PROPERTY

  • Under the stimulus conditions, static features have zero expected covariance with an independent temporal concept, so a significantly nonzero sample mean covariance implies the direction is non-static.The law of total expectation extends the argument across videos with differing motion profiles; a matching temporal feature can have positive covariance, giving the test detection power.Source: §3.4 · p. 5

3.5 THEORETICAL LIMITATIONS OF THE NFP TEST

  • The test detects non-static features associated with the chosen concept, not necessarily features encoding that concept specifically; it can also miss temporal features that do not covary with it.Detectability depends on the stimulus set, which supports ball-motion concepts but not unrelated concepts such as increasing brightness.Source: §3.5 · p. 6

4 EMPIRICAL EVALUATION: GENERAL SETUP

The empirical setup applies the NFP test to video-transformer representations and a DINOv2 image-encoder control, using rendered ball-motion videos and multiple temporal concepts. It also tests SAE features and evaluates significance with Bonferroni correction.

4 EMPIRICAL EVALUATION: GENERAL SETUP

  • The evaluation includes VideoMAE, V-JEPA2, and TimeSformer video transformers, with DINOv2 as an image-encoder negative control.VideoMAE is tested both pretrained and finetuned; TimeSformer is finetuned, and activations come from the final transformer layer.Source: §4 · p. 6
  • The stimulus set contains 3,000 rendered 16-frame ball videos and five pairwise-decorrelated profiles: speed, vx, vy, |a|, and direction.Starting positions are sampled uniformly under the coverage condition, and the pairwise covariance among concepts is at most 0.0022.Source: §4 · p. 6
  • A direction is flagged for a temporal concept when a one-sample t-test of its per-video covariance rejects zero at p < 0.05/D after Bonferroni correction.Undefined t-statistics are classified as non-valid rather than static or temporal.Source: §4 · p. 6
  • The study trains L1-penalized ReLU SAEs on final-layer activations to examine temporal SAE features and interventions.SAEs use an 8× expansion factor and are trained on 4,000 randomly selected SSv2 training clips per video model.Source: §4 · pp. 6–7
  • Table 1 reports flagged counts for representation axes and SAE features, with per-profile counts that may overlap and an “Any” count for directions flagged on at least one profile.The number of tested directions D is the representation dimension for axes or dictionary size for SAE features.Source: Table 1 · p. 7

5 IDENTIFYING TEMPORAL FEATURES

The NFP test flags temporal directions across the video models, with counts varying by model and motion profile. DINOv2 yields no flagged axes, while SAE findings and unresolved limits qualify how the counts should be read.

5 IDENTIFYING TEMPORAL FEATURES

  • 28.7%–92.1% of dimension axes are flagged as temporal across models, with |a| accounting for most flags except in VideoMAE finetuned.Zero direction axes are flagged for any tested concept in DINOv2, consistent with the test’s specificity on an image encoder.Source: §5 · p. 7, Table 2 · p. 8
  • The flagged-count patterns are suggestive rather than definitive: flags can include polysemantic features, and zero direction flags do not rule out concepts outside the test’s definition or sensitivity.The authors note that finetuning may change encoded temporal concepts, but present this as a possibility rather than a demonstrated cause.Source: §5 · p. 7
  • For some models, SAE-feature flag counts resemble dimension-axis counts, but VideoMAE finetuned has more SAE flags for speed, vx, and vy than for |a|.SAE and axis percentages are not directly comparable because some SAE features may not activate on the stimulus.Source: §5 · p. 7
  • Whether temporal concepts occupy a meaningful dedicated low-dimensional subspace remains unanswered by Table 1.The authors note that their activation-and-controlled-stimulus framework may make this structure difficult to study.Source: §5 · p. 8

6 SYNTHETIC VALIDATION OF THE NFP TEST

The synthetic validation controls temporal content in a representation space and tests whether NFP responses track that content. It finds no flags at zero temporal content, increasing detection as temporal content grows, and strong temporal alignment among SAE-flagged features.

6 SYNTHETIC VALIDATION OF THE NFP TEST

  • The synthetic representation controls temporal, static, and position-dependent subspaces; each test direction’s temporal fraction is set by β².Directions combine a vector in the planted temporal subspace with one from its orthogonal complement.Source: §6 · p. 8
  • No direction with zero temporal content is flagged; detection reaches 20/20 at β² = 6.3 × 10⁻⁵ in the synthetic representation.The authors caution that real representations may behave differently.Source: §6 · p. 8
  • Acceleration directions have lower t-scores than speed and vy directions at each tested β², with saturation near |t| ≈13 rather than ≈40.In this synthetic setup, saturation is tied to the ratio of within-video to across-video variance for each temporal concept.Source: §6 · p. 8
  • The NFP test flags 8 of 6144 synthetic SAE features, whose temporal-subspace norm fraction is 0.582 versus 0.003 for unflagged features.Flagged features have 198× the unflagged temporal-subspace fraction; none orthogonal to the planted temporal subspace is flagged.Source: §6 · p. 9, Table 3 · p. 9

7 INTERVENTION ANALYSIS

Span erasure of NFP-identified temporal SAE features has different effects across models and does not establish that temporal features are more important than non-temporal features for temporally demanding tasks. The intervention removes the flagged decoder span directly from layer activations.

7 INTERVENTION ANALYSIS

  • Span erasure subtracts the projection onto flagged decoder directions from each token’s layer activation, without encoding or decoding an SAE reconstruction.The SAE supplies the erased directions; evaluation uses 10 videos per SSv2 class, for n = 1740.Source: §7 · p. 9
  • Erasure effects differ by model: VideoMAE finetuned is most affected by temporal-feature erasure, V-JEPA2 by low-static erasure, and TimeSformer similarly by both.Random-feature erasure has a similar effect to low-static erasure for VideoMAE finetuned, but little effect for TimeSformer.Source: Table 4 · p. 9, §7 · p. 9
  • Further experiments do not characterize temporal features as more important than non-temporal features for temporally demanding tasks, and intervention may not outperform a non-temporal baseline.The authors note that static features can be composed across frames in classification heads to represent temporal concepts.Source: §7 · p. 10

8 CONCLUSION

The NFP test gives a formal guarantee that flagged features are non-static, and the authors find temporal features in three video models but none in DINOv2. Synthetic validation supports the test’s intended behavior, while interventions are mixed; the conclusion also identifies open questions, and reproducibility materials are provided.

8 CONCLUSION

  • The test guarantees that any flagged feature is provably non-static by controlling frame variation and equalizing the confounder’s distribution across time.The authors also describe a generalized NFP framework beyond the moving-ball setup and an optimization application for arbitrary directions.Source: §8 · p. 10
  • The test finds temporal features in VideoMAE, V-JEPA2, and TimeSformer, none in DINOv2, and synthetic validation supports its intended behavior.Interventions on temporal SAE features are mixed across video models, suggesting the relationship between detectable temporal features and task-relevant temporal reasoning varies across models.Source: §8 · p. 10
  • The conclusion calls for broader stimulus sets and further investigation of whether static features composed by classification heads explain temporal understanding, and why pretraining alone seemingly fails to produce impactful temporal features.Source: §8 · p. 10

REPRODUCIBILITY

  • The paper provides supplementary code sufficient to reproduce the main experiments and points to sections and appendices for dataset, model, and SAE-training details.Source: REPRODUCIBILITY · p. 11

LLM USAGE

The appendix generalizes the NFP problem to detecting dependence on a target concept rather than a co-varying confounder. It formalizes the setup and assumptions for decoupling them, while the LLM-usage statement describes assistance and author checking.

LLM USAGE

  • Claude Opus 4.6, Claude Opus 4.8, and Claude Fable 5 assisted with ideation, mathematical verification, SAE-toolkit adaptation, interventions, tables, code, and writing; authors checked the work and wrote and edited the final text and main mathematical results.Source: LLM USAGE · p. 14

A A GENERAL NFP FRAMEWORK

  • The general framework treats NFP as testing whether a feature depends on a target concept or only on a confounder that co-varies with it.Here the target is a temporal property such as speed, and the confounder is ball position.Source: A A GENERAL NFP FRAMEWORK · p. 14

A.1 SETUP

  • The observational setup comprises units, indexed observations, a representation, a target-concept profile, and a confounding variable that co-varies with the target within each unit.The difficulty is that a feature tracking the target may instead respond to the confounder.Source: §A.1 · p. 14

A.2 ASSUMPTIONS

  • Three assumptions decouple the confounder from the target: the confounder is generated by a per-unit random seed, that seed is independent of the target, and the confounder’s distribution is the same at every observation index.Equal distributions make a confounder-only feature’s expected value constant across conditions; bounded neural-network features satisfy the stated integrability condition.Source: §A.2 · pp. 14–15
  • The main-paper design achieves equal confounder distributions through uniform starting positions, off-screen zeroing, and coverage; periodic boundary conditions are another possible mechanism.Source: §A.2 · p. 15

A.3 THEOREM AND PROOF

This section establishes the NFP theorem and its assumptions, showing how within-unit covariance can rule out confounder-only features and, under additive linear encoding and pairwise decorrelation, align with a temporal concept’s encoding direction. It also clarifies the framework’s limits, including multiple confounders, tubelet-spanning tokens, and the distinction between covariance and correlation.

A.3 THEOREM AND PROOF

  • Under Assumptions 1–3, significant nonzero expected within-unit covariance means a feature is not confound-dependent and must depend on something beyond the confounder.This is the contrapositive of the General NFP theorem.Source: §A.3 · p. 15

A.4 THE PAPER’S SETUP AS AN INSTANCE OF THE GENERAL FRAMEWORK

  • In the ball-video setup, independently randomized uniform starting positions, coverage, and off-screen zeroing make expected activations of position-only features time-independent, satisfying Assumption 3.Thus the paper’s setup instantiates the general theorem for the effective confounder: ball position when visible and ⊥ when off-screen.Source: §A.4 · p. 16

A.5 MULTI-CONFOUNDER EXTENSION

  • For multiple confounders, the NFP property holds for jointly confound-dependent features when their seeds are mutually independent, independent of the target, and each confounder’s distribution is time-invariant.Shared seeds can break the property despite time-invariant marginals: relational features such as two balls being close can vary with time and be falsely flagged as temporal.Source: §A.5 · pp. 16–17

A.6 EXAMPLES IN OTHER DOMAINS AND REMARK ABOUT STOCHASTIC CONFOUNDERS

  • NFP can apply to neuron velocity attribution, manufacturing speed versus temperature, and financial momentum versus price level when randomization keeps the confounder distribution constant over time.The appendix notes, without proof, that extension to stochastic within-unit confounder dynamics may require modifications and remains outside the paper’s scope.Source: §A.6 · pp. 17–18

B.1 THEORETICAL IMPLICATIONS / B.2 EXPERIMENTAL IMPLICATIONS

  • For tubelets spanning multiple frames, confounders retaining the starting position can satisfy the coverage argument, but within-tubelet displacement cancels it and can be correctly flagged as temporal.For covariance, using every frame or averaging concept values within tubelets is mathematically equivalent; the paper averages and uses 8 effective time steps instead of 16 frames.Source: §B.1 · p. 18, §B.2 · p. 19

C.1 MORE ON COMPACT SUPPORT AND COVERAGE CONDITION

  • The coverage proof generalizes to any bounded activation-support region, and off-support activations need only be constant rather than zero.Compact support is the special case where that constant is zero.Source: §C.1 · p. 19

C.2 EMPIRICAL PROCEDURE FOR IMPLEMENTATION OF (S1) - (S5) AND THE NFP TEST

  • The empirical procedure simulates covered ball trajectories with independent uniform starting positions, extracts activations with off-screen values zeroed, computes within-video covariance, and tests its mean against zero.Directions significant at level α are flagged as non-static.Source: §C.2 · pp. 19–20

D COUNTEREXAMPLE AGAINST CORRELATION

  • A correlation-based test can falsely flag a static feature: position-dependent activation variance rescales covariance unequally, yielding nonzero expected Pearson correlation even when expected covariance is zero.The counterexample fixes the velocity profile across videos and shows that denominator variation breaks the covariance symmetry.Source: D COUNTEREXAMPLE AGAINST CORRELATION · p. 20

E.1 THE MEAN COVARIANCE VECTOR ¯cτ

  • If the representation is static, its mean covariance vector is zero; therefore a nonzero vector implies temporal information, and its normalized direction maximizes covariance with the concept.For any direction w, the covariance is zero under a static position-only representation; the unit direction ¯cτ/∥¯cτ∥ maximizes wᵀ¯cτ.Source: §E.1 · p. 21

E.2 STRUCTURED DECORRELATION

  • Under additive linear encoding and a complete concept list, pairwise decorrelation makes each mean covariance vector proportional to its concept’s true encoding direction.Without decorrelation, cross-covariances between distinct concepts can deflect the velocity vector away from its true direction; recovery depends on the stated encoding and concept-list assumptions.Source: §E.2 · p. 22

E.3 OBSTRUCTING CONCEPTS

  • Obstructing concepts covary with velocity with the same strict sign across profiles, preventing decorrelation; velocity is obstructing, while acceleration can be decorrelated as a distinct concept.Strictly monotone functions of velocity are obstructing; the practical implication is to treat obstructing concepts as part of the true velocity subspace rather than decorrelate against them.Source: §E.3 · pp. 22–23

E.4 ACHIEVING DECORRELATION IN PRACTICE

  • With 5 concepts, 4,000 candidate profiles, and 3,000 videos, the LP-weighted stimulus set achieved pairwise covariance ≤0.0022 for every concept pair.Feasibility requires covariance values of both signs for each pair; the stimulus design combines the weighted profile distribution for decorrelation with randomized starting positions for the NFP test.Source: §E.4 · pp. 23–24

E.5 LIMITATIONS OF DECORRELATED RECOVERY

Structured decorrelation can recover an encoding direction only relative to a chosen concept list and under perfect decorrelation, conditions that cannot be verified in practice. The section shows that no stimulus distribution can eliminate contamination from every non-obstructing concept and that recovery can fail sharply.

E.5 LIMITATIONS OF DECORRELATED RECOVERY

  • In practice, the true concept list is unknown and incomplete decorrelation can make the max-covariance direction orthogonal to the true encoding direction.The section notes that a direction maximizing covariance with velocity can be orthogonal to the velocity subspace.Source: §E.5 · pp. 24–26
  • For T≥3, no distribution with a positive velocity signal achieves zero worst-case contamination across all non-obstructing concepts.The proof exhibits a family of such concepts, at least one of which has large contamination regardless of the distribution.Source: §E.5 · p. 24
  • For T=8, even the best stimulus design leaves at least one non-obstructing concept with contamination of at least 6/7 (≈0.857).This lower bound restricts the method to structured decorrelation using a finite concept list chosen in advance.Source: §E.5 · p. 25
  • Recovery succeeds only when the representation has a specified concept decomposition and decorrelation is perfect; the resulting direction depends on that concept list.Different valid decompositions can assign different velocity directions to the same representation.Source: §E.5 · p. 25

F MODEL SPECIFICATIONS

  • Table 5 specifies model token grids, representation dimensions, and extracted layers; DINOv2 processes frames independently, while TimeSformer uses eight of sixteen frames.Table 6 lists the HuggingFace checkpoint identifiers.Source: Table 5 · p. 26, Table 6 · p. 26

G SAE TRAINING DETAILS

The paper uses standard ℓ1 sparse autoencoders with largely fixed training settings, while varying dictionary size, corpus, and training steps. Because the synthetic ball stimulus is out of distribution for SAEs trained on natural video, only a small fraction of their features are valid for the test, which limits what the test evaluates.

G SAE TRAINING DETAILS

  • All SAEs use a standard ℓ1 recipe with an 8× expansion factor; dictionary size, corpus, and gradient steps vary, with no early stopping or checkpoint selection.Training directly in float16 diverged to NaN; stored activations were cast to float32 at load.Source: G SAE TRAINING DETAILS · pp. 26–27, Table 7 · p. 27
  • Each model-specific SAE uses 4,000 SSv2 training clips, while the synthetic SAE uses an 80/20 split of 3,000 generated videos.DINOv2 processes each frame independently as 256 spatial patch tokens.Source: Table 8 · p. 27
  • Table 9 reports held-out FVE and L0, and its DINOv2 and synthetic FVE values are approximate because they were recomputed from checkpoints.The reported recomputation is about 0.04 above the logged metric for VideoMAE finetuned (0.63 vs. 0.59).Source: Table 9 · p. 27

H VALID FEATURES PER DICTIONARY

  • Only a small fraction of SAE features are valid on the ball stimulus, and this valid set—not the full dictionary—is the population the test evaluates.The Bonferroni denominator remains the full dictionary size D, although most SAE features are alive on held-out SSv2.Source: H VALID FEATURES PER DICTIONARY · pp. 27–28, Table 10 · p. 28
  • A feature that is non-valid on the stimulus may be dead there or may indicate a test artifact; it may also encode temporal concepts absent from the ball stimulus.Examples of omitted concepts include a hand moving left or increasing brightness.Source: H VALID FEATURES PER DICTIONARY · p. 28

I IDENTIFYING TEMPORAL FEATURES: RESULTS ACROSS LAYERS

Across layers, VideoMAE finetuned retains temporal-axis feature counts through its final layer, whereas V-JEPA2’s count drops there. The synthetic control plants temporal, static, and position-dependent components, with random signs designed to cancel expected covariance between position content and temporal concepts.

I IDENTIFYING TEMPORAL FEATURES: RESULTS ACROSS LAYERS

  • VideoMAE finetuned’s temporal-axis feature counts persist through the final layer, while V-JEPA2’s count decreases significantly there.The authors speculate that this may reflect more structured temporal concepts in later V-JEPA2 layers and conclude the models appear to learn temporal information differently.Source: I IDENTIFYING TEMPORAL FEATURES:… · p. 29

J.1 FULL CONSTRUCTION DETAILS

  • The synthetic 768-dimensional representation uses orthogonal temporal, static, and position-dependent subspaces, plus noise to prevent an exactly low-rank SAE input.Its five z-scored temporal profiles are speed, vx, vy, |a|, and direction; static content is constant within each video, while position channels depend on ball location.Source: §J.1 · p. 29

J.2 RANDOM-SIGN MECHANISM FOR POSITION-DEPENDENT CONTENT

  • Random signs make each position channel’s expected within-video covariance with any temporal concept zero, satisfying the test’s null hypothesis.The cancellation holds for each video individually, regardless of trajectory, starting position, or bump center.Source: §J.2 · pp. 29–30

J.3 FULL β2 SWEEP RESULTS

The appendix examines synthetic-control behavior and intervention results for NFP-flagged temporal features. It finds that detection responds to temporal-subspace content, while interventions do not consistently outperform non-temporal controls.

J.3 FULL β2 SWEEP RESULTS

  • The complete sweep tests 850 directions across 26 β² levels and three β² = 0 null sets, using a Bonferroni-adjusted threshold of |t| > 4.0.Table 12 reports the full sweep; the main text presents a representative subset.Source: §J.3 · p. 30, Table 12 · p. 30

J.4 INDIVIDUAL FLAGGED SAE FEATURES

  • Each of the eight flagged synthetic-SAE features placed 0.41–0.71 of its norm in the planted temporal subspace, despite imperfect alignment with individual concepts.Their maximum absolute cosine with a single planted concept was 0.57–0.76; SAE reconstruction is non-exact, and features can mix concepts.Source: §J.4 · pp. 30–31, Table 13 · p. 31

K.1 GENERAL CASE

  • The general-case derivation treats expected within-video covariance from the temporal component as signal, while static, randomized position-dependent, and independent-noise components have zero expected covariance.The t-statistic’s denominator also reflects across-video variance contributions from temporal and non-temporal components.Source: §K.1 · pp. 31–32

K.2 SINGLE-CONCEPT CASE

  • In the single-concept synthetic case, the t-statistic grows linearly with β at low values and saturates near β = 1 at a ceiling determined by the stimulus set.At low β, the denominator is approximately independent of β; near one, β and the temporal variance term cancel.Source: §K.2 · p. 33

K.3 IMPLICATIONS FOR INTERPRETING REAL-MODEL RESULTS

  • The synthetic β sweep suggests acceleration features’ high flagging in real models is unlikely to be a test artifact, though the proposed training-related explanation remains uncertain.The authors note that acceleration had lower single-concept t-scores across β > 0 in the synthetic control, which they interpret as the stimulus set discouraging its flagging.Source: §K.3 · p. 33

L SYNTHETIC CONTROL WITH NON-ORTHONORMAL TEMPORAL DIRECTIONS

  • With correlated planted temporal directions, the test flagged 17 SAE features at c = 0.25 and 20 at c = 0.5, with substantial temporal-subspace projection and no false positives.Individual features aligned less cleanly with single planted directions as correlation increased.Source: L SYNTHETIC CONTROL WITH… · p. 33, Table 14 · p. 33

M.1 FEATURE AMPLIFICATION

  • Amplifying temporal features modestly improved finetuned VideoMAE performance, harmed V-JEPA2 performance, and had no apparent TimeSformer effect; changes were likely within noise.The reported α = 3 pattern was representative of α values from 2 to 10, with improvements usually peaking between 3 and 5.Source: §M.1 · p. 34, Table 15 · p. 34

M.2 SPAN ERASURE EFFECT ON BINARY CLASSIFICATION

  • Erasing temporal-feature spans did not meaningfully change binary accuracy on kinematic pairs or outperform erasing an equivalent number of non-temporal features.The authors interpret this as evidence that these SSv2 classifications may not require temporal feature encoding in representation space.Source: §M.2 · p. 35

M.3 FLIPPER EXPERIMENTS

The appendix finds that some temporal SAE features can flip classifications, but comparable non-temporal features can as well, and intervention effects vary by model. It also demonstrates that stimulus entanglement can create misleading overlap in features flagged for different concepts.

M.3 FLIPPER EXPERIMENTS

  • Some temporal SAE features flipped predictions for opposing class pairs, but low-static features also acted as flippers, and model behavior varied.TimeSformer had no identified flippers on the randomly drawn pairs, with a maximum top-1 flip rate of 0.46.Source: §M.3 · p. 36, Table 18 · p. 36, Table 19 · p. 36

M.4 WHAT DOES ALL THIS MEAN?

  • Across interventions, temporal SAE features produced greater performance effects than non-temporal baselines for some models but not others.The authors suggest static features may combine across frames through the classification head, while noting this account and other explanations remain unresolved.Source: §M.4 · pp. 36–37

M.5 LISTS OF CLASS PAIRS USED

  • The hand-picked class pairs include direction, speed, and motion-outcome contrasts; depth pairs serve as controls for non-motion concepts.The randomly drawn pairs also include control pairs to check for flips on non-motion-related contrasts.Source: Table 20 · p. 37, Table 21 · p. 37

N THE STRONGLY ENTANGLED STIMULUS SET

  • When speed and acceleration were strongly correlated in the stimulus set, the flagged-feature subsets overlapped, illustrating how entanglement can mislead interpretation.The rendered speed–|a| correlation was 0.88, while every other pair had covariance ≤0.003; decorrelating stimuli avoids this ambiguity.Source: N THE STRONGLY ENTANGLED STIMULUS SET · p. 38, Table 22 · p. 38

O EXAMPLES OF NFP TEST VIDEOS

  • The example NFP videos vary ball movement patterns and starting positions across four timestamps while holding ball and background colors and sizes constant.The videos isolate motion-pattern and starting-position variation in the illustrated examples.Source: Figure 1 · p. 39

Related papers

Loading 2610.33353v1…