Your Probabilistic JEPA Is Secretly a Hidden Markov Model: A State-Space Interpretation of Joint-Embedding Predictive Learning

HMM correspondence

• Full PIB-VJEPA exposes an HMM-like encode–transition–emit structure: context encoding infers latent states, prediction defines transitions, and decoders or inverse encoders provide emissions.The correspondence is structural and does not alone imply identical sequence distributions or HMM maximum-likelihood training.Source: §1 · pp. 4, 6
• Exact sequence-level HMM equivalence requires Markov, emission, marginal-consistency, and filtering-consistency conditions, while model-and-objective equivalence additionally requires sequence-level probabilistic training.Without an explicit observation map, an implicit emission may exist but need not be tractable for generation or likelihood evaluation.Source: §1 · pp. 4, 6

Transition models

• MCJEPA replaces the latent predictor with a learned transition matrix whose powers guarantee exact Chapman–Kolmogorov consistency for direct and composed multi-step predictions.The transition construction extends to conditioned discrete transitions, continuous-state Markov kernels, and continuous-time dynamics, with deterministic temporal JEPA as a Dirac-kernel case.Source: §1 · p. 6

Predictive state construction

• Predictive information-bottleneck learning is framed as Markov-state construction, with predictive compression promoting minimality and residual predictability testing sufficiency.The paper separates JEPA latent prediction, hybrid JEPA–HMM learning, and HMM-style sequence learning by their probabilistic semantics.Source: §1 · p. 6

Experiments

• Four controlled experiments examine finite-state transition recovery and path consistency, filtering by the context encoder, predictive compression and Markovization, and JEPA–HMM objective distinctions.These experiments target transition composition, the filtering interpretation, predictive Markovization, and differing sequence-learning semantics.Source: §1 · p. 6

2.1 Latent Markov dynamics

• The framework covers discrete or continuous states and discrete or continuous time, while focusing first on discrete-time, discrete-state dynamics because they yield the clearest HMM matrix correspondence.Continuous-state versions use Markov kernels, continuous-time discrete-state versions use rate generators, and continuous-time continuous-state versions use stochastic processes such as SDEs.Source: §2.1 · p. 7

2.2 The HMM–PIB-VJEPA analogy at a glance

• Full PIB-VJEPA mirrors an HMM’s encode–transition–emit pipeline, but amortizes inference with an encoder and trains prediction primarily in representation space rather than by observation likelihood.A decoder, inverse target encoder, or induced conditional can provide the emission direction; inverse encoding requires strong invertibility and density conditions.Source: §2.2 · pp. 7–8, Table 2 · p. 7

2.3 Temporal JEPA

• Temporal JEPA supplies an online encoder, EMA target encoder, and latent predictor that forecast future target representations, with side information such as time, actions, positions, or covariates.The HMM filtering interpretation becomes exact only when the history-dependent context encoder satisfies the required filtering-consistency conditions.Source: §2.3 · p. 8, §2.2 · p. 8

2.4 MCJEPA: replace the predictor by a Markov chain

• MCJEPA replaces a general latent predictor with a categorical state Zt ∈ {1, . . . , K}, soft encoder assignments, and a constant row-stochastic transition matrix.Training matches matrix-propagated online distributions to EMA target-encoder distributions across selected horizons, optionally with discounted horizon weights.Source: §2.4 · pp. 8–9

2.5 Exact multi-horizon path consistency

• Exact matrix powers make MCJEPA predictions path-consistent: composing horizons h1 and h2 equals predicting their total horizon directly under a fixed time-homogeneous chain.This is the Chapman–Kolmogorov property and supports hierarchical planning without path-dependent discrepancies.Source: §2.5 · pp. 9–10

2.6 Avoiding discrete-state collapse

• MCJEPA can collapse through single-state or uniform assignments, so complementary occupancy and entropy regularizers encourage diverse state usage and confident per-observation assignments.Their weights require care: excessive occupancy pressure distorts imbalanced state frequencies, while excessive entropy pressure causes prematurely hard assignments.Source: §2.6 · pp. 10–11

3 From a Transition Matrix to a Neural Markov Kernel

• MCJEPA starts with discrete-time, discrete-state Markov dynamics, then progressively generalizes the transition matrix while preserving dependence on history only through the predictive state and side information.The time-homogeneous case reuses one transition matrix at every step.Source: §3 · p. 12

3.1 Neural discrete-state transitions

• A neural row-stochastic matrix conditioned on actions, elapsed time, goals, regimes, controls, or covariates supports noncommuting ordered transitions and exact path composition.Including elapsed time handles irregular intervals while retaining a discrete, interpretable latent state space.Source: §3.1 · p. 12

3.2 Continuous-state stochastic transitions

• Continuous-state dynamics replace matrix multiplication with Markov-kernel composition that marginalizes intermediate states, using sampling, Monte Carlo, moment propagation, or learned approximations when closed forms are unavailable.Gaussian kernels model mean evolution and uncertainty, but multimodal futures require mixtures (Huang, 2026a), flows, diffusion, or other expressive distributions; Gaussianity is not essential.Source: §3.2 · pp. 12–13

3.3 Continuous-time extensions

• Continuous-time extensions handle irregular horizons through interval-conditioned discrete transitions, generator-matrix exponentials under piecewise-constant conditioning, time-varying generator composition, or stochastic differential equations.A fixed generator spans intervals of different lengths; continuously varying conditioning requires appropriate generator composition.Source: §3.3 · pp. 13–14

3.4 Deterministic temporal JEPA as a degenerate kernel

• Deterministic temporal JEPA is the zero-uncertainty limit of this framework: deterministic encoders are point masses and deterministic predictors are Dirac transition kernels.MCJEPA, conditioned discrete-state JEPA, continuous probabilistic VJEPA, and deterministic JEPA differ mainly in state representation and transition family.Source: §3.4 · p. 14

3.5 When a recurrent predictor is Markov

• A recurrent predictor is first-order Markov only in the augmented state comprising its latent state and recurrent memory; PIB-VJEPA therefore seeks a predictive state sufficient for future prediction.If information remains outside Z_t, the model must enlarge the state, add memory, use higher-order dynamics, or accept non-Markovianity in the chosen representation.Source: §3.5 · pp. 14–15

4.1 The three distributions in PIB-VJEPA

• The online encoder infers a distribution over the current latent state, the predictor propagates it conditionally, and the target encoder supplies the future latent training target.The predictor may condition on actions, elapsed time, or exogenous covariates, while source and target regularization trade predictive accuracy against compression and regularity.Source: §4.1 · pp. 15–16

4.2 The encode–transition–emit correspondence

• The HMM encode–transition–emit analogy maps observations, predictive states, latent transitions, and context-based inference, but the encoder is not itself an emission distribution.Emission can come from an explicit decoder, an invertible target encoder, or an implicit construction; inverse encoding requires additional density modeling for normalized likelihoods.Source: §4.2 · pp. 16–17

4.3 When no decoder is present: an implicit emission

• A local stochastic encoder plus the data marginal induces a normalized emission whose posterior exactly equals the encoder, but this static construction does not ensure sequence-level HMM consistency.Sequence-level validity additionally requires latent distributions to evolve consistently with the transition and the history encoder to equal the induced Bayesian filter.Source: §4.3 · p. 17

4.4 Four levels of correspondence

• The hierarchy separates computational roles, emission-complete latent representations, sequence-level HMM equivalence, and model-and-objective equivalence.Standard probabilistic JEPA predicts target representations and regularizes information rather than generally optimizing observation-sequence likelihood; hybrid objectives retain both JEPA and HMM-style supervision.Source: §4.4 · pp. 17–18
• PIB-VJEPA reaches exact sequence-level HMM equivalence only when Markov dynamics, a valid emission, transition-consistent marginals, and filtering-consistent encoders all hold.These sufficient conditions are not guaranteed by standard JEPA training; implicit emissions additionally require observation-locality and Bayes consistency.Source: §4.4 · p. 18

Information-bottleneck objective

• Predictive information bottleneck learning compresses the past while retaining predictive information about the future, with λ controlling the compression–prediction trade-off and PIB-VJEPA implementing it through latent prediction and variational regularization.The underlying objective penalizes information retained from observation history while rewarding dependence between current and future latent states.Source: §5 · p. 18

Predictive sufficiency

• Minimizing the bottleneck objective alone does not guarantee that the learned representation is predictively sufficient for the future.The stronger target requires the learned state to retain all history information relevant to future observations.Source: §5 · p. 18
• Predictive sufficiency requires that, given Z_t, the remaining observation history adds no information about the future, matching the predictive-state representation perspective.This condition is closely related to predictive-state representations.Source: §5 · p. 19

Markovization result

• Predictive sufficiency implies one-step Markov sufficiency: conditioning on the current predictive state screens off older history from the next latent state.If transition-relevant side information ξ_t exists, the corresponding diagnostic also conditions on ξ_t; the implication follows from the conditional data-processing inequality.Source: §5 · p. 19

6 Residual Predictability as a Diagnostic of Markov Sufficiency

• Residual predictability from older observations or latent states indicates that the current state–predictor pair omitted transition-relevant information, but nondetection does not prove full Markov sufficiency.The diagnostics primarily test conditional-mean consequences of sufficiency rather than the complete conditional-independence condition.Source: §6 · p. 19

6.1 Categorical probability innovations

• For categorical MCJEPA, probability innovations have zero conditional mean under a correctly fitted transition, while Markov sufficiency additionally requires no predictable component from the full history.The transition is conditionally mean-correct with respect to its own inputs under the forward-KL objective when the model family represents and optimizes the population optimum.Source: §6.1 · p. 20

6.2 Testing incremental predictability from older history

• A history-augmented auxiliary predictor that improves held-out residual prediction beyond current inputs provides evidence against conditional-mean sufficiency of the current state–predictor pair.Covariance diagnostics detect only linear dependence; direct prediction of the target is equivalent to residual correction when both use a common baseline.Source: §6.1 · p. 20, §6.2 · p. 21

6.3 Distinguishing state insufficiency from predictor misspecification

• Residual predictability may reflect either state insufficiency or transition misspecification, so attribution to the representation requires expressive, well-fitted, capacity-matched, regularized, out-of-sample diagnostics.Current-input residual prediction tests transition misspecification, whereas additional gains from older latent history test information omitted by the current representation.Source: §6.3 · p. 21, §6.4 · p. 21

6.4 Continuous-state diagnostics

• For continuous predictors, standardized innovations should have conditional mean zero and identity covariance under a correct Gaussian model, but zero autocorrelation or calibration cannot establish Markov sufficiency.Diagnostics may include lagged residual, covariance, squared-residual, auxiliary-prediction, score-based, probability-integral-transform, and multivariate-calibration tests; singular covariances require regularization or pseudoinverses.Source: §6.4 · p. 22

7 Experiments

• The four experiments use known latent processes and five-seed evaluations to isolate transition recovery, filtering, predictive compression, and HMM-objective correspondence rather than maximize forecasting performance.Systems are intentionally small, with means and standard deviations reported across five random seeds.Source: §7 · pp. 22, 24

7.1 Experiment 1: finite-HMM recovery and Markov composition

• MCJEPA has exact direct-versus-composed path consistency, whereas independent horizon predictors violate composition; occupancy and entropy regularization are complementary for diverse, confident state assignments without collapse.A shared transition matrix makes path disagreement identically zero, while the ablation shows occupancy prevents state under-use and entropy prevents diffuse assignments.Source: §7 · p. 23, §7.1 · pp. 25–26
• In finite HMMs, MCJEPA recovers the latent process close to a correctly specified HMM when emissions are separated, while ambiguous emissions expose a trade-off between predictive flexibility and faithful shared transition structure.The HMM is strongest under ambiguity; horizon-specific predictors can improve state recovery or predictive NLL at some horizons, but independently learned transitions are less faithful to one coherent Markov law.Source: §7.1 · p. 24

7.2 Experiment 2: filtering resolves emission ambiguity

• As emissions overlap, oracle filtering becomes more informative than local observation evidence because it integrates propagated beliefs with current observations, while its advantage diminishes when emissions are locally separable.Filtering remains stable through ambiguous measurements and changes when accumulated temporal evidence supports a transition; the comparison uses exact oracle posteriors, not separately trained models.Source: §7.2 · p. 27, Figure 4 · p. 27

7.3 Experiment 3: predictive compression and Markovization

• In the second-order synthetic process, predictive compression removes the redundant older bit while retaining the recent pair, yielding a first-order predictive state with the same next-observation information and prediction NLL.The experiment contrasts overcomplete, minimal sufficient, and insufficient states; the four-state predictive pair compresses the three-bit history without sacrificing prediction.Source: §7.3 · p. 28
• The controlled compression boundary is eight states sufficient but redundant, four states sufficient and minimal, and two states insufficient, with further compression sharply reducing predictive information.Predictive information falls from 0.2807 to 0.0192 and prediction NLL rises from 0.3978 to 0.6593 when compressing to Z_t = X_t.Source: §7.3 · pp. 28, 31
• Across all deterministic compressions of eight histories, the globally optimal representation for positive compression weights is exactly the known four-state predictive state.The exhaustive enumeration provides a global deterministic reference, and compression selects the minimal sufficient Markov state when it removes redundancy without sacrificing prediction.Source: §7.3 · p. 29
• Residual-history gain complements compression by detecting insufficiency: adding Z_t−1 substantially helps the insufficient current-observation representation but provides essentially no benefit for minimal sufficient or overcomplete states.Thus, predictive compression tests minimality, whereas residual predictability tests whether discarded history still contains transition-relevant information.Source: §7.3 · pp. 30–31

7.4 Experiment 4: HMM-style training of PIB-VJEPA

• The hybrid HMM-plus-latent objective improves over JEPA-only training across ARI, NMI, sequence NLL, transition recovery, filtering KL, and true-state prediction NLL at every evaluated horizon and seed.It reduces transition-matrix error by approximately 46%, demonstrating that the gain is not driven by a single favorable run.Source: §7.4 · pp. 34–35
• The hybrid nearly closes the observation-sequence likelihood gap to HMM-style training while retaining the EMA-target MCJEPA latent-prediction loss.JEPA-only latent prediction does not directly optimize observation-sequence likelihood or jointly train an emission model with the representation.Source: §7.4 · pp. 34–35
• HMM+filter achieves the strongest state recovery and sequence likelihood, despite omitting the JEPA latent-prediction objective.At every evaluated horizon h ∈{1, 2, 4, 8}, HMM-style and hybrid training achieve lower true-state prediction NLL than JEPA-only training.Source: §7.4 · pp. 35, 37
• 98.6% of the JEPA-only sequence-NLL gap is closed by hybrid training, which also matches HMM-style transition recovery while retaining genuine JEPA latent prediction.Figure 6 reports the objective-level comparison: hybrid and HMM-style training closely match on transition recovery, while both outperform JEPA-only training.Source: §7.4 · pp. 35, 37, Figure 6 · p. 36
• Filtering diagnostics show that HMM+filter explicitly learns an amortized filter, while hybrid remains substantially filtering-aligned and JEPA-only has no such requirement.For HMM+filter and hybrid, filtering KL is training-aligned; for JEPA-only, it measures agreement with a post-hoc filter induced by the fitted observation model.Source: §7.4 · p. 36

7.5 Summary of Experiments

• Experiment 1 establishes that MCJEPA’s shared transition matrix composes exactly across horizons, while Gaussian HMMs remain strongest under ambiguous emissions and regularizers stabilize discrete-state learning.Independent horizon-specific predictors can be more flexible but yield less faithful, non-compositional transition laws.Source: §7.5 · p. 37
• Experiments 2–4 support filtering context encoders, predictive compression as compact state construction, and an objective continuum from JEPA-only through hybrid to HMM-style learning.History-based filtering outperforms local evidence under overlapping emissions; compression identifies a minimal sufficient state, while residual-history tests distinguish sufficiency from minimality.Source: §7.5 · pp. 37–38

• Full, time-indexed PIB-VJEPA matches HMM inference, latent-state propagation, and state-to-observation mapping, with exact sequence equivalence requiring Markov, marginal-consistency, and filtering-consistency conditions.The correspondence has four levels, and model-and-objective equivalence additionally requires HMM-compatible sequence-level probabilistic training.Source: §8 · p. 38
• MCJEPA makes the broader latent-Markov interpretation explicit through a finite-state transition matrix, but general nonlinear neural predictors are also Markov when conditioned only on the current latent state.If prediction uses recurrent memory or higher-order history, augmenting the state with transition-relevant information can restore a first-order interpretation.Source: §8 · pp. 38–39
• A first-order predictor does not guarantee a sufficient Markov representation: residual history may remain predictive after conditioning on Zt, requiring state augmentation, higher-order dynamics, or unrestricted history dependence.The representation should retain future-relevant information while discarding redundant history; compression promotes minimality, while residual-predictability tests sufficiency.Source: §8 · p. 39
• Architecture and objective are separate: hybrid HMM supervision improves MCJEPA’s latent-state recovery toward HMM-level likelihood and transition recovery, while the same architecture can be trained without LMC.Thus, the boundary between temporal JEPA and classical state-space modeling depends on the probabilistic semantics enforced by training, not model components alone.Source: §8 · p. 40
• Observation reconstruction is optional for predictive representation learning, but decoders support forecasting and likelihood, invertible encoders constrain compression, and implicit emissions may be intractable for generation or likelihood evaluation.JEPA can avoid reconstructing high-entropy observation details irrelevant to prediction while still admitting a probabilistic completion.Source: §8 · p. 40
• The correspondence is a representation-design target rather than a generic property of embeddings, and controlled synthetic experiments leave validation in video, control, and real-world partially observed systems for future work.Finite categorical states improve interpretability but do not ensure semantically meaningful labels, which require evidence from statistics, transitions, interventions, or downstream tasks.Source: §8 · p. 40

Markov-Chain JEPA

• MCJEPA replaces the latent predictor with a learned row-stochastic transition matrix, yielding multi-step predictions with exact Chapman–Kolmogorov consistency.Neural conditioned matrices, continuous-state Markov kernels, and continuous-time transitions extend the construction beyond finite homogeneous chains.Source: §9 · p. 40

HMM correspondence

• The HMM interpretation assigns filtering to the stochastic context encoder, latent dynamics to the probabilistic predictor, and emissions to a decoder, inverse target encoder, or induced implicit conditional.The paper distinguishes computational correspondence, emission-complete latent-state representation, sequence-level HMM equivalence, and model-and-objective equivalence.Source: §9 · pp. 40–41

Deterministic limit

• Deterministic temporal JEPA is the degenerate boundary of the same framework, with deterministic predictors represented as Dirac Markov kernels and deterministic latent states as point-mass distributions.Probabilistic formulations expose the state-space structure explicitly, while stronger HMM equivalence still requires emission and consistency conditions.Source: §9 · pp. 40–41

HMM correspondence

• Theorem 1 gives sufficient conditions for exact sequence-level HMM equivalence, requiring emission and consistency conditions beyond computational or latent-Markov correspondence.These stronger conditions include emission completeness, transition and marginal consistency, filtering consistency, and HMM-style sequence training.Source: §9 · p. 41

Predictive representation learning

• Predictive information bottleneck learning seeks a compact predictive state that approximately Markovizes observations at the chosen prediction scale: compression promotes minimality, while residual predictability tests sufficiency.Objective-level experiments span JEPA latent prediction, hybrid JEPA–HMM learning, and HMM-style sequence and filtering training.Source: §9 · p. 41

A Training Objectives and Minimal Algorithm

• The shared-A MCJEPA objective combines multi-horizon latent prediction with a discrete-state regularizer, and matrix powers require no additional Chapman–Kolmogorov penalty.Exact path consistency is automatic because every horizon uses the same transition matrix A.Source: A Training Objectives and Minimal… · p. 42
• A horizon-dependent transition variant loses guaranteed path consistency and therefore adds a weighted Chapman–Kolmogorov penalty, unlike the shared-A model used in the main experiments.The penalty weight satisfies λCK ≥0 and applies only to the separate-matrix variant.Source: A Training Objectives and Minimal… · p. 42
• The minimal training step samples time, horizon, history, target, and side information, computes online and EMA state distributions, propagates with A, evaluates the latent loss, and updates online and target parameters.The target uses sg(¯q_t+h), while the target encoder is updated by exponential moving average.Source: A Training Objectives and Minimal… · p. 42
• Conditioned discrete-state MCJEPA replaces shared-A propagation with ordered transition composition, while the remainder of training remains unchanged.This modification is confined to Step 4 of the minimal algorithm.Source: A Training Objectives and Minimal… · p. 43
• Residual-history gain is a held-out diagnostic of omitted state information, evaluated after fitting rather than included in the default MCJEPA objective.This prevents the representation from being trained directly to satisfy the diagnostic.Source: A Training Objectives and Minimal… · p. 43

B.1 Proof of Proposition 1

• Associativity of matrix multiplication gives A^h1A^h2 = A^(h1+h2), and left-multiplication by q_t yields the two-stage predictive-distribution identity.This establishes Eq. (9) for nonnegative horizons h1 and h2.Source: §B.1 · p. 43
• For any partition h = h1 + ··· + hm, repeated application of the same identity makes every decomposition produce the same predictive distribution.Thus shared matrix powers prove full multi-step Chapman–Kolmogorov consistency.Source: §B.1 · p. 43

B.2 Proof of Proposition 2; B.3 Proof of Proposition 3

• Proposition 2 gives qθ(z | x) an exact posterior interpretation under prior qθ(z) and implicit emission, but sequence-level HMM equivalence additionally requires transition, marginal, and filtering consistency.The construction reproduces pdata(x) as the observation marginal and defines a valid static latent-variable model.Source: §B.2 · p. 44
• Under predictive sufficiency and target-encoder randomness independence, Zt screens off older observations from Zt+1; deterministic target encoders need no auxiliary randomness.The result follows from conditional data processing and the stated conditional-independence assumption.Source: §B.3 · p. 44

C Three Realizations of the Emission Direction

• Emission-complete latent-state representations can use an explicit decoder, an inverse target encoder, or an induced implicit conditional, but none alone guarantees sequence-level HMM equivalence.Exact equivalence also requires transition, marginal-consistency, and filtering-consistency conditions.Source: C Three Realizations of the Emission… · p. 45

C.1 Explicit decoder

• An explicit decoder directly supplies the HMM state-to-observation direction, with categorical emission matrices or conditional observation models for high-dimensional data.Post-hoc decoders act as observation probes, whereas jointly trained decoders become part of the generative latent-state model and can affect observation-sequence likelihood.Source: §C.1 · p. 45

C.2 Invertible target encoder

• A bijective target encoder supplies a deterministic inverse emission, but exact invertibility restricts dimensionality reduction and may preserve details that compressed JEPA representations discard.Noise augmentation can produce a non-degenerate conditional emission, while standard compressed encoders need not support the invertible construction.Source: §C.2 · p. 45

C.3 Implicit emission

• The implicit conditional exactly completes the one-time observation–state relationship, yet may be intractable and cannot reinterpret arbitrary history-dependent encoders as emissions without filtering and transition consistency.Its dependence on the generally unknown pdata(x) can obstruct direct sampling and likelihood evaluation.Source: §C.3 · p. 46

D HMM-Style Training Objectives for PIB-VJEPA; D.1 Sequence likelihood; D.2 Exact likelihood and filtering for categorical MCJEPA

• HMM-style training learns transition and emission models from observation-sequence likelihood while aligning the context encoder with Bayesian filtering, unlike standard JEPA’s primarily latent-space objective.For categorical MCJEPA, forward recursion gives exact sequence likelihood and filtering posteriors; encoder distillation separately trains one-pass amortized filtering.Source: D HMM-Style Training Objectives for… · p. 46, §D.1 · p. 46, §D.2 · p. 47

D.3 Continuous-state extension

• For continuous or nonlinear latent states, approximate sequence posteriors yield a variational lower bound, with causal filtering or full-sequence smoothing depending on training access.The paper uses exact finite-state inference in its experiments, while the extension generalizes the model-and-objective interpretation beyond categorical states.Source: §D.3 · p. 47

D.4 Relation to JEPA and hybrid training

• JEPA latent prediction and HMM-style sequence learning remain distinct objectives, while hybrid training combines latent prediction, HMM sequence, and filtering losses through shared latent dynamics.The exact HMM filtering posterior supervises Lfilter but does not replace the EMA future target used by LMC.Source: §D.4 · p. 48
• Adding an emission model alone is insufficient for HMM-equivalent training; observation-sequence evidence and sequence-inference semantics must also participate in the objective.The transition matrix is shared by the HMM and MCJEPA objectives, constraining latent dynamics from observation-space and representation-space perspectives.Source: §D.4 · p. 48

E.1 Marginal and filtering consistency

• Dynamic consistency requires one-time latent marginals to be generated by the same transition kernel used by the temporal model.Filtering proceeds by propagating the prior through the transition and applying Bayes’ rule to the current observation.Source: §E.1 · pp. 48–49
• The history-dependent PIB-VJEPA encoder is filtering-consistent only when it equals the filtering distribution generated by the specified transition and emission model.An arbitrary history encoder need not satisfy this equality.Source: §E.1 · p. 49

E.2 Explicit and invertible emissions

• An explicit decoder or invertible target encoder can supply the emission, and with consistent marginals and filtering the resulting temporal JEPA has the theorem’s sequence-level HMM interpretation.An invertible target encoder is interpreted through its induced deterministic kernel.Source: §E.2 · p. 49

E Exact HMM Representation Conditions

• Exact sequence-level HMM equivalence requires consistent latent marginals, a valid transition and emission model, and a history encoder matching the resulting filtering distribution.The construction applies to discrete states and extends to general Markov kernels; side information is treated as observed.Source: E Exact HMM Representation Conditions · p. 48, §E.2 · p. 49

E.3 Implicit-emission case

• In the implicit-emission case, a local evidence encoder depending only on the current observation induces a normalized state-to-observation conditional.The locality assumption separates current-observation evidence from the history-dependent context encoder, which represents the filtering belief.Source: §E.3 · p. 49
• Exact implicit-emission equivalence additionally requires the induced marginals to be dynamically consistent with the transition and the history encoder to coincide with the resulting filtering distribution.The induced emission can be substituted into the filtering recursion to express the history-dependent belief in equivalent form.Source: §E.3 · p. 50

E.4 Completion of the proof for Theorem.1

• Theorem 1 follows because At supplies first-order latent Markov dynamics, valid emissions exist, latent marginals follow the same kernel, and the context encoder equals the Bayesian filter.These conditions yield the joint sequence distribution in Eq. (40) and exact sequence-level HMM equivalence, beyond architectural correspondence alone.Source: §E.4 · p. 50

F Experimental Details

• Experiments are deliberately small and synthetic, isolating structural claims under known latent dynamics rather than benchmarking large-scale forecasting performance.Unless otherwise stated, sampled-data and learned-model results use five seeds and report mean ± one standard deviation; deterministic finite calculations omit seed variability.Source: F Experimental Details · p. 50
• The finite-HMM protocol uses 400 training and 160 test sequences of length 80, horizons H = {1, 2, 4, 8}, paired data within seeds, and no validation split.Experiment 4 regenerates separated-emission data independently with the same process and seeds, without reusing fitted Experiment 1 models.Source: F Experimental Details · pp. 50–51

F.1 Evaluation metrics

• Representation recovery reports permutation-invariant ARI and NMI between learned and ground-truth state assignments; transition matrices and predicted categorical probabilities are aligned using a Hungarian assignment computed on training-set hard state assignments.ARI equals 1 for identical partitions and near 0 for chance-level agreement, while NMI lies in [0, 1] and larger values indicate greater shared information.Source: §F.1 · p. 51
• Transition recovery uses normalized Frobenius error, true-state prediction NLL evaluates future latent-state probability, and observation-sequence NLL evaluates marginalized observed-sequence density.Observation likelihood is computed exactly with the HMM forward algorithm rather than enumerating latent trajectories, and its training role differs across Experiment 4 regimes.Source: §F.1 · pp. 51–52
• Path disagreement tests direct-versus-composed multi-step predictions, with one shared MCJEPA transition matrix guaranteeing agreement up to numerical precision unlike independently learned horizon-specific matrices.Filtering KL measures amortized context beliefs against exact filtering distributions; in Experiment 4 it is training-aligned for HMM+filter and hybrid regimes, so it is a role diagnostic.Source: §F.1 · p. 52
• Experiment 2 additionally reports accuracy, multiclass Brier score, and posterior entropy, distinguishing hard correctness and probabilistic confidence from descriptive uncertainty.Entropy is not a performance metric by itself.Source: §F.1 · p. 53

F.2 Experiment 1: finite-HMM recovery and Markov composition

• Experiment 1 uses stationary four-state Gaussian-HMM sequences with separated (σ = 0.35) and ambiguous (σ = 0.80) emissions, training on 400 and testing on 160 sequences per regime.Sequences have length 80 and observations are two-dimensional.Source: §F.2 · p. 53
• MCJEPA combines GRU context and target encoders with one learned 4 × 4 transition matrix, whereas the baseline learns independent matrices A1, A2, A4, A8.The shared model predicts every horizon using powers of one row-normalized transition matrix.Source: §F.2 · pp. 53–54
• For Experiment 1, after Hungarian alignment, evaluation uses ARI and NMI for hidden-state partition recovery, transition error for recovering A⋆, and true-state NLL across h ∈ {1, 2, 4, 8}.Effective state count measures diversity of state usage, while assignment entropy measures confidence of individual assignments; occupancy regularization promotes broad global usage and entropy regularization promotes confident assignments.Source: §F.2 · p. 55, Figure 9 · p. 57
• Across horizons, all models remain close under separated emissions; under ambiguity, the correctly specified HMM is strongest, while both JEPA variants show similar predictive NLL despite differing consistency.Figure 7 reports complete multi-horizon state-prediction curves with mean ± one standard deviation over five seeds.Source: §F.2 · p. 55, Figure 7 · p. 56
• MCJEPA has zero direct-versus-composed path disagreement by construction, while independently trained horizon-specific matrices exhibit nonzero disagreement in separated emissions.The same structural distinction holds in the ambiguous regime; Fig. 8 visualizes the separated-emission result.Source: §F.2 · p. 55, Figure 8 · p. 56

F.3 Experiment 2: filtering under emission ambiguity

• Experiment 2 compares exact local evidence p(S_t | X_t) with exact filtering that combines propagated previous beliefs and current emissions under a persistent two-state HMM.Both are oracle calculations under the known generating process, not learned HMM or MCJEPA models, isolating temporal history’s informational value.Source: §F.3 · p. 57
• Filtering’s state-NLL advantage grows as µ/σ decreases, matching the main-text accuracy trend as Gaussian emissions become more ambiguous.Accuracy, state NLL, Brier score, and posterior entropy provide complementary evaluation measures; Fig. 10 reports state NLL across separations.Source: §F.3 · pp. 57–58, Figure 10 · p. 58

Exact deterministic compression

• Of 4140 deterministic partitions, 36 are nondominated on the information–prediction Pareto frontier, with the four-state predictive pair removing redundant history while preserving one-step predictive information.The frontier evaluates exact I(H; Z), I(Z; Y), and H(Y | Z); Table 9 provides structural controls and Figure 11 shows the frontier.Source: §F.4 · p. 59, Table 9 · p. 59, Figure 11 · p. 62

Exact compression sweep

• For every tested positive β through 0.014, the four-state predictive pair is optimal; at β = 0.015 it switches to two states, and at β = 1 to one state.At β = 0, tie-breaking reports the lower-information four-state sufficient representation rather than a uniquely preferred prediction-only solution.Source: §F.4 · p. 60, Table 10 · p. 60

Learned stochastic continuation

• The learned stochastic continuation is a warm-started gradient path from an overcomplete eight-state initialization, so it need not coincide with the exact deterministic optimum.Five perturbation seeds are averaged, and the continuation differs because stochastic parameterization and optimization path dependence can prevent global-optimum recovery.Source: §F.4 · pp. 60–62

Residual-predictability diagnostic

• Positive residual-history gain indicates that earlier representation states contain omitted predictive information, whereas values near zero indicate one-step sufficiency.The insufficient current-state representation benefits from adding its previous state, while the minimal sufficient and overcomplete controls show effectively zero gain; tiny negative values are finite-sample variation.Source: §F.4 · p. 61

F.5 Experiment 4: HMM-style training of PIB-VJEPA

• Experiment 4 controls the latent-state family, data realization, GRU encoder family, state-use regularization, and evaluation protocol across independently trained regimes.Each seed uses 400 training sequences and 160 test sequences of length 80, with K = 4 latent states and hidden dimension 48.Source: §F.5 · p. 63, §F.6 · p. 68
• The three regimes separate JEPA latent prediction from HMM sequence learning: JEPA-only fits an observation model post hoc, HMM+filter distills detached exact filtering, and hybrid shares the transition while retaining the original EMA-target JEPA loss.In the hybrid, the HMM filtering posterior updates only Lfilter, while the future JEPA target remains separate from that posterior.Source: §F.5 · p. 64
• Hybrid training retains only a very small sequence-evidence gap relative to HMM-style training, whereas JEPA-only remains clearly separated.The paired comparison uses identical test realizations within each seed and is descriptive rather than a formal hypothesis test.Source: §F.5 · p. 66
• HMM-style and hybrid training improve true-state prediction NLL over JEPA-only at every evaluated horizon, with the largest differences at shorter horizons and narrower gaps by h = 8.Multi-horizon evaluation uses powers of each learned transition matrix and reports results in Table 13.Source: §F.5 · p. 66, Table 13 · p. 66
• Across all five paired seeds, hybrid training improves over JEPA-only in ARI, NMI, observation-sequence NLL, transition error, filtering KL, and true-state prediction NLL at every evaluated horizon.The pattern is reported descriptively, without inferring formal statistical significance from five seeds.Source: §F.5 · p. 67

F.6 Reproducibility and computation

• The implementation is reproducible through released Python code, fixed seeds across Python, NumPy, PyTorch, and CUDA, and explicit experiment-specific seed offsets.Single-precision PyTorch is used, with CUDA selected when available and CPU otherwise.Source: §F.6 · pp. 67–68
• Experiment 4 is an objective-behavior diagnostic rather than a compute-matched efficiency benchmark because JEPA-only uses sequence mini-batches while HMM-based regimes use full-data updates.The scripts do not guarantee bitwise reproducibility across CUDA libraries or hardware, and they omit pinned package versions, accelerator models, and wall-clock timings.Source: §F.6 · p. 68

Disclaimer

• The disclaimer states that the author originated the probabilistic-JEPA/HMM correspondence, ChatGPT assisted with development and execution, and the author manually reviewed and takes responsibility for the final work.The reported contribution split is approximately 60% author and 40% ChatGPT, with readers advised to interpret claims cautiously because errors may remain.Source: Disclaimer · p. 69