PaperlayerView exact revision on arXiv

Source-linked AI summary

JEPA-Anything: Learning Predictive Models across Different Worlds

Taoyong Cui, Zhongyao Wang, Xinyue Xu, Weiyang Liu, Zhaochen Yu, Yuying Zhang, Qiang Gao, Mengyue Yang, Wanli Ouyang, Pheng Ann Heng, Yingcheng Wu, Zhenfei Yin, Ling Yang

arXiv:2609.20800v1cs.CL

TL;DR

World models need to represent predictive states across systems with different structures, while conventional JEPA formulations use a monolithic target. JEPA-Anything factorizes targets into complementary components within a shared predictive core, and reported results improve across diverse tasks and domains.

  • Problem

    Conventional JEPA formulations use one target embedding and prediction pathway, which can let dominant structure crowd out weaker predictive structure when a state contains multiple entities or scales.

    Source: §1 · p. 3

  • Method

    JEPA-Anything uses domain adapters for observation geometry and orthogonal predictive factorization to organize latent targets into complementary coordinates within a shared interface.

    Source: §1 · pp. 4–5, §2.1 · p. 5

  • Results

    Reported metrics improve across the ten-task matched dynamics benchmark and four molecular systems, with stronger readouts, intervention and OOD prediction, and scientific diagnostics across the evaluated domains.

    Source: §1 · p. 4, §5 · p. 24

  • Takeaways & Limitations

    The results support OPF as a capacity-organizing core for a reusable latent world-state interface.

    Source: §5 · p. 24

    Orthogonal factors alone do not establish causal mechanisms, and a shared predictive architecture does not guarantee aligned representations across domains.

    Source: §6 · p. 25

Abstract

from arXiv · show

World modeling enables intelligence to anticipate consequences, guide interventions, and learn from interaction. Yet predictive models remain domain-specific: can a common learning principle support world modeling across radically different systems? We introduce JEPA-Anything, a domain-agnostic framework based on orthogonal predictive factorization (OPF). Extending joint-embedding predictive architectures, OPF decomposes latent targets into complementary factors, learns them through dedicated pathways, and recombines them within a shared predictive design. We evaluate JEPA-Anything across seven domains: vision, biology, clinical trajectories, control, molecular dynamics, physical fields, and weather. Experiments span representation learning, intervention prediction, out-of-distribution generalization, and long-horizon dynamics, including 10 matched dynamics tasks, forecasting of over 1,000 clinical events, and 100-step molecular rollouts across four systems. Against matched JEPA baselines, JEPA-Anything improves reported metrics on all 10 dynamics tasks and reduces single-intervention prediction error on Interventional Pong by 34.8%. It achieves the lowest one-step and 100-step molecular errors among compared methods in all four systems. Beyond prediction, a factor-nominated biological intervention receives experimental support in cell co-cultures, patient-derived organoids, tumor fragments, and mice; latent orbital modes recover the Keplerian scaling exponent with a fitted slope of -1.4991. These results support a common factorized predictive principle across heterogeneous worlds, connecting world modeling with intervention and experimentally grounded scientific discovery. Code: https://github.com/Gen-Verse/JEPA-Anything

1. Introduction

The introduction asks whether one predictive interface can support world models across differently structured systems. JEPA-Anything addresses this with orthogonal predictive factors and a shared core that is evaluated across domains and task types.

1. Introduction

  • The paper uses latent world model for a model that constructs a state from context and predicts another state of the same underlying system for reuse across queries or transitions.JEPA provides the context encoder, target encoder, and predictor that map between representations.Source: §1 · p. 3
  • A monolithic JEPA target can let easily predicted or high-variance structure dominate and create conflicting gradients, motivating capacity allocation across complementary latent components.The introduction frames the question as whether multiple factors can organize predictive states across context–target systems.Source: §1 · pp. 3–4
  • JEPA-Anything uses learned basis matrices and dedicated prediction branches to predict target components, with orthogonality and activity objectives plus online encoder variance regularization.Predicted components can be synthesized into a complete latent state; factor identities emerge from predictability rather than predefined semantics.Source: §1 · pp. 4–5
  • Domain adapters specify observation geometry and context–target views, while a shared factorized predictive core operates once context and target states are available.Temporal conditioning, masking, and partial observation instantiate different structured targets.Source: §1 · p. 4
  • Reported results improve readout quality across vision, cells, and disease forecasting, intervention prediction on observed and unseen combinations, and metrics across ten matched dynamics tasks and four molecular systems.Biological and orbital analyses also connect learned coordinates to wet-lab evidence and a known physical scaling law.Source: §1 · p. 4

2. JEPA-Anything

JEPA-Anything uses a shared context–target interface and orthogonal predictive factorization to distribute prediction across complementary latent factors. Activity regularization supports training, while the learned representations serve reusable readouts, latent transitions, and factor-level analysis.

2.1. One Interface, One Predictive Core

  • Domain adapters and view samplers expose context, target, and optional descriptors through a shared interface, while the predictive core remains common across instantiations.The interface requires a meaningful predictive relation between states of the same underlying system.Source: §2.1 · pp. 5–6

2.2. Orthogonal Predictive Factorization

  • Orthogonal predictive factorization divides the stop-gradient target into learned subspaces, with dedicated predictors for each factor and synthesis of their predictions into a complete latent state.Within-factor and cross-factor constraints discourage repeated directions; under exact orthogonality, the factors preserve all target information and support stable synthesis.Source: §2.2 · pp. 6–8

2.3. Maintaining Factor and Encoder Activity

  • Factor-activity regularization keeps projected target coordinates active, while a separate online-encoder activity term supplies a direct anti-collapse gradient.These terms are added to each domain’s existing base objective.Source: §2.3 · p. 8

2.4. Shared Predictive Pretraining and Task-Specific Readout

  • The shared pretraining supports reusable encoder features for task-specific readouts and explicit latent transitions for prediction, planning, or simulation.Learned projectors also provide a diagnostic interface for analyzing predictive modes at the factor level.Source: §2.4 · p. 9

3. Experiments

This section evaluates JEPA-Anything across visual and cellular representations, clinical forecasting, dynamics, planning, and scientific analyses using domain-specific observations and a shared factorized predictive-state design. It examines factor coordinates in intervention tests and scientific analyses, alongside molecular forecasting across four systems.

3.1. Common Evaluation Logic

  • Direct standard-JEPA comparisons hold the adapter, encoder, sampler, optimization budget, data split, and downstream readout fixed, while using domain-specific metrics rather than averaging incompatible scales.Continuous-control results are summarized over five training seeds and matched environment resets.Source: §3.1 · p. 10

3.2.1. Controlled Visual Binding

  • JEPA-Anything increases INJ and grid recovery while reducing collapse for both DINOv3 and SigLIP2 under the same learned-grid readout.The evaluation uses frozen encoders, image-disjoint splits, leave-one-cell-out readout, and ten-seed averages.Source: §3.2.1 · p. 12, Table 3 · p. 12

3.2.2. Single-Cell State Representation and Perturbation Prediction

  • JEPA-Anything attains the highest value on both PBMC clustering metrics and both perturbation-response datasets.Evaluation includes finetuned and zero-shot PBMC-10K clustering and Adamson and Norman perturbation prediction; Pearson correlation measures absolute post-perturbation expression.Source: §3.2.2 · p. 13

3.2.3. Longitudinal Disease-State Forecasting

  • JEPA-Anything yields higher mean PRAUC than matched monolithic JEPA across the full vocabulary of more than 1,000 future clinical events.Mean PRAUC is the primary metric, and methods use five seeds and the same fixed patient-level cohort split.Source: §3.2.3 · p. 14, Figure 3 · p. 14

3.3.1. Intervention-Conditioned Compositional Prediction

  • Versus dense standard JEPA, JEPA-Anything reduced single-intervention one-step MSE by 34.83% (0.009541 to 0.006218), combined-intervention one-step MSE by 12.90% (0.009441 to 0.008223), and six-step free-rollout MSE by 8.58% (0.009478 to 0.008665).Source: §3.3.1 · p. 15

3.3.2. Out-of-Distribution and Long-Horizon Dynamics

  • Burgers errors decrease approximately 49.5% on the held-out late segment and 44.7% over six rollout steps, while KS late-state prediction improves approximately 13.2%.Each improvement occurs in every seed; the separate capacity-matched Burgers evaluation reports gains on both in-distribution and high-frequency OOD trajectories at 20 and 50 steps, with a narrower gain at H50.Source: Table 7 · p. 17, §3.3.2 · p. 18, Table 8 · p. 18

3.3.3. Continuous-Control Planning

  • JEPA-Anything attains higher mean CEM return on Walker2d and HalfCheetah, while Hopper favors standard JEPA.The comparison is capacity-matched; parameter differences are at most 0.246% and prediction-head FLOP differences at most 1.449% across calibrated branch widths.Source: §3.3.3 · pp. 18–19, Figure 6 · p. 19

3.3.4. Factor Activity and Functional Interventions in Locomotion

  • All four locomotion factors remain active, with factor-wise standard deviations of 0.692–0.715 on Hopper, 0.587–0.601 on Walker2d, and 0.673–0.678 on HalfCheetah.Single-factor probes show differentiated but not exclusive motion preferences; replacing a factor by its training-set mean tests its effect on 20-step rollout error and real-environment CEM return.Source: §3.3.4 · pp. 19–20, Figure 7 · p. 20
  • Masking any factor increased rollout error in all three locomotion environments, while masking F1 or F3 reduced HalfCheetah return in all five seeds.Masking F2 or F4 reduced HalfCheetah return in four of five seeds; factor activity, single-factor readout, and masking jointly support active, readable, individually intervenable state channels.Source: §3.3.4 · p. 20

3.3.5. Force-Free Molecular Forecasting

  • JEPA-Anything achieves the lowest one-step displacement MAE and 100-step RMSD on all four molecular systems.Metrics are reported in Å, averaged over five trained seeds, with lower values better; 100-step RMSD is the primary rollout-stability measure.Source: Table 9 · p. 21, §3.3.5 · p. 21

3.4. Group III: Scientific Analysis of Learned World States

  • The scientific-analysis experiments test factor coordinates as an interface for nominating experimentally testable interventions and evaluating latent modes against established scaling laws.Both analyses connect factor-level analysis of u_k(x) to external validation.Source: §3.4 · p. 21

3.4.1. Wet-Lab Evaluation of a Factor-Nominated Intervention

  • Factor-based analysis nominated IL-18 plus NT5E/CD73 blockade, and wet-lab studies supported enhanced antitumor activity across co-cultures, organoids, tumor fragments, and mice.The studies included Huh7–PBMC co-cultures, patient-derived organoids and tumor fragments, and immunocompetent mice.Source: §3.4.1 · p. 21

3.4.2. Orbital Scaling as a Physical-Law Diagnostic

  • A JEPA-Anything orbital-mode analysis recovered the Keplerian relation f∝a^-3/2 with fitted slope −1.4991 and R^2 = 0.9999999.The modes were derived by pairing spectral frequencies with semimajor axes from simulated position–velocity trajectories without physical labels.Source: §3.4.2 · p. 22, Figure 9 · p. 23

4. Related Work

Prior work develops latent world models for visual prediction and control, joint-embedding prediction, and structured or factorized representations. OPF differs by organizing predictive target capacity without presuming semantic factor identities, while retaining synthesis into a complete state.

Latent world models and joint-embedding prediction

  • Latent world models primarily target visual prediction or control, while JEPA work extends representation prediction across images, video, physical understanding, planning, and single-cell data.Prior work also studies context–target-dependent invariances in JEPA objectives.Source: §4 · p. 23

Structured and factorized representations

  • Structured representation methods use independent-variable, entity-slot, object-relation, or intervention-based decompositions, while unsupervised semantic disentanglement is not identifiable from observations alone.The cited approaches include β-VAE, Slot Attention [23], structured world models [17], and CITRIS [21]; the identifiability caveat is [22].Source: §4 · p. 23

Redundancy reduction and collapse prevention

  • Barlow Twins reduces redundancy through cross-correlation matching [36], as stated in the cited passage, which identifies that matching approach as the method for reducing redundancy.Source: §4 · p. 23
  • OPF factorizes predictive target capacity without assuming object, causal, or semantic factor identities, using orthogonality and activity constraints with an explicit synthesis map.This links redundancy reduction to a complete latent world state.Source: §4 · p. 24

5. Summary

JEPA-Anything applies a shared OPF core across diverse systems and uses its latent interface for readout, recursive prediction and planning, or factor analysis. Experiments span seven domains and support several predictive uses, while showing that continuous-control gains do not hold in every environment.

5. Summary

  • JEPA-Anything separates domain-specific state construction from a shared OPF core that assigns target capacity to complementary subspaces, keeps them active, and synthesizes their predictions.The resulting interface supports terminal encoder readout, recursive transition and planning, or factor-level analysis.Source: §5 · p. 24
  • Experiments across visual, biological, clinical, control, molecular, physical-field, and weather systems support readout, intervention composition, OOD prediction, repeated rollout, and external analysis of learned coordinates.Continuous-control means improve on two environments but not Hopper; factor interventions show all locomotion channels contribute to rollout.Source: §5 · p. 24
  • Geometry analysis shows non-overlapping, unit-conditioned coordinates and near-exact synthesis, while biological and orbital cases illustrate external analysis of learned coordinates.These findings are reported alongside stronger visual, cellular, and disease-state readout metrics.Source: §5 · p. 24

6. Future Work

The paper proposes connecting predictive models to systems that formulate and investigate scientific questions, while identifying reliability, causal interpretation, and representation alignment as unresolved challenges. Future evaluation should test experiment selection, externally validated hypothesis revision, and transfer to later investigations.

6. Future Work

  • JEPA-Anything could provide a predictive substrate for comparing candidate interventions and identifying discrepancies between expected and observed outcomes.The proposed connection to Discovery Foundation Models frames discovery as problem formulation, representation construction, hypothesis proposal, and revision through external evidence [35].Source: §6 · p. 24
  • ScienceIDE and ScienceBuddy could connect learned predictions with executable analyses, simulations, verifiable feedback, and researcher-directed candidate experiments.The proposed workflow would compare predictions with computational or physical evidence and use the resulting experience to update models, policies, and research procedures.Source: §6 · pp. 24–25
  • Orthogonal factors do not by themselves establish causal mechanisms, and a shared predictive architecture does not guarantee aligned representations across domains.The paper also identifies uncertainty calibration under distribution shift and deciding when a model is reliable enough to guide an experiment as open questions.Source: §6 · p. 25
  • Future evaluation should measure experiment-selection quality and efficiency, externally validated hypothesis revision, and transfer to subsequent investigations.These evaluations address progress needed beyond predictive accuracy.Source: §6 · p. 25

A. Approximate Pseudoinverse-Synthesis Stability

With an approximately orthogonal learned analysis matrix and pseudoinverse synthesis, the appendix establishes unbiased reconstruction for exact factor coordinates and bounds amplification of factor-prediction error by the orthogonality residual.

A. Approximate Pseudoinverse-Synthesis Stability

  • The stability result extends exact orthogonal decomposition to an approximately orthogonal learned analysis matrix when synthesis uses the pseudoinverse.The passage identifies this as the continuous counterpart of Proposition 1 and Equation (5).Source: A. Approximate Pseudoinverse-Synthesis… · p. 29
  • For exact coordinates u = P^T z and predicted coordinates û = u + e, synthesis uses the pseudoinverse of P^T.The passage states P is invertible and introduces the coordinate relation used in the result.Source: A. Approximate Pseudoinverse-Synthesis… · p. 29
  • Pseudoinverse synthesis has no reconstruction bias for exact factor coordinates, while measured orthogonality residual controls amplification of factor-prediction error.The proof uses invertibility of P under the stated eigenvalue bound.Source: A. Approximate Pseudoinverse-Synthesis… · p. 29

B.1. Implementation Details

Implementation settings specify separate configurations for the APEBench physical-field models and the Norman single-cell model, including an observed-minus-control target for Norman.

B.1. Implementation Details

  • APEBench Burgers and Kuramoto–Sivashinsky use an MLP with first-layer weights of shape [256, 64] and OPF dimensions d=128, K=4, and r=32.Source: §B.1 · p. 29
  • The Norman single-cell model uses OPF dimensions d=512, K=4, and r=128.Source: §B.1 · p. 29
  • Norman’s prediction target is z_response = z_observed − z_control.Source: §B.1 · p. 29

B.2. OPF Coefficients

The appendix reports OPF coefficient documentation, AI-use practices, and the governance conditions for the clinical-data analysis.

B.2. OPF Coefficients

  • Table 10 reports experiment-specific OPF dimensions and three additive regularizer coefficients, while domain-specific terms remain part of L(δ).Source: §B.2 · p. 29, Table 10 · p. 30

B.3. AI Use Statement

  • The authors used OpenAI Codex for language editing, translation, LaTeX restructuring, checks, figure layout, and feedback, then reviewed proposed changes and retained responsibility for the manuscript.Source: §B.3 · p. 30

B.4. Ethics Statement

  • Clinical analyses use de-identified UK Biobank data accessed through controlled governance, within an approved application and applicable ethics and data-use requirements; participant-level data are not redistributed.Source: §B.4 · p. 30

B.5. Reproducibility Statement

The reproducibility statement identifies the experiment documentation, matched settings, and reporting conventions used to support reproducibility.

C. Reproducibility Statement

  • Section 2 specifies the additive OPF loss, its combination with the implementation loss, activity statistics, and synthesis map; Section 3 identifies data, matched-baseline controls, and metrics.Source: §B.5 · p. 30
  • Table 2 records context–target construction, factor dimensions, evaluation splits, and seed counts, while Appendix Table 10 reports OPF regularization coefficients.Source: §B.5 · p. 30
  • Experiments use cited-backbone default architecture and optimization settings, with non-OPF settings shared within matched comparisons; tables report absolute metrics where comparable and figures state cross-task aggregation.Source: §B.5 · p. 30

C. Author Organizations and Correspondence

The author-organization section lists four institutions.

C. Author Organizations and Correspondence

Loading 2609.20800v1…