PaperlayerView exact revision on arXiv

Source-linked AI summary

What Has a Foundation Model Found? Using Inductive Bias to Probe for World Models

Keyon Vafa, Peter G. Chang, Ashesh Rambachan, Sendhil Mullainathan

arXiv:2507.06952v4cs.LGcs.AI

TL;DR

It is difficult to tell whether sequence-prediction success reflects a foundation model’s grasp of a deeper world model. The paper probes this by testing how models adapt to new tasks generated from postulated world models, and finds that sequence models often lack the corresponding inductive biases and instead behave as if they use task-specific heuristics.

  • Problem

    Foundation models and world models operate in different spaces, making it challenging to determine whether prediction reflects recovery of a world model.

    Source: §1 · p. 1, §2.1 · p. 2

  • Method

    The inductive bias probe tests whether a foundation model adapts to new tasks in ways that reflect a specified world model.

    Source: §1 · p. 1, §6 · p. 10

  • Results

    Across domains, sequence models often show limited inductive bias toward genuine world models, including orbital models that produce poor force vectors and gravity laws unlike Newton’s law.

    Source: §1 · p. 1, §6 · p. 10

  • Takeaways & Limitations

    The models instead behave as if they use task-specific heuristics that fail to generalize, with different apparent laws across physics tasks.

    Source: §1 · p. 2

    The probe requires analysts to specify a world model, limiting its use for identifying the exact representation underlying a model’s behavior.

    Source: §6 · pp. 10–11

Abstract

from arXiv · show

Foundation models are premised on the idea that sequence prediction can uncover deeper domain understanding, much like how Kepler's predictions of planetary motion later led to the discovery of Newtonian mechanics. However, evaluating whether these models truly capture deeper structure remains a challenge. We develop a technique for evaluating foundation models that examines how they adapt to synthetic datasets generated from some postulated world model. Our technique measures whether the foundation model's inductive bias aligns with the world model, and so we refer to it as an inductive bias probe. Across multiple domains, we find that foundation models can excel at their training tasks yet fail to develop inductive biases towards the underlying world model when adapted to new tasks. We particularly find that foundation models trained on orbital trajectories consistently fail to apply Newtonian mechanics when adapted to new physics tasks. Further analysis reveals that these models behave as if they develop task-specific heuristics that fail to generalize.

1. Introduction

The paper asks whether sequence prediction can reveal the deeper mechanisms behind observed patterns, as it did in the path from Kepler’s planetary predictions to Newtonian mechanics. It introduces an inductive bias probe and finds that accurate prediction does not ensure transfer of the underlying world model.

1. Introduction

  • An inductive bias probe tests whether a foundation model learned a postulated world model by examining how it extrapolates from limited information.The paper frames this as analogous to scientists using world models to make inferences from small amounts of data.Source: §1 · p. 1
  • The orbital model predicts trajectories even for solar systems it has not seen, yet this predictive success does not establish a Newtonian inductive bias.The introduction places this contrast against the historical progression from Kepler’s predictions to Newton’s mechanics.Source: §1 · p. 1
  • Across lattice problems and Othello, neural sequence models show weak inductive biases toward the given world models.Models that perform better on probes also perform better when fine-tuned on new tasks relying on the world model.Source: §1 · pp. 1–2
  • Orbital models produce poor force vectors and, across many solar systems, generalize using gravity laws unlike Newton’s law.The figure contrasts true Newtonian forces with model predictions after force-vector fine-tuning.Source: §1 · p. 1, Figure 1 · p. 2
  • The models instead behave as if they develop task-specific heuristics that fail to generalize, including different nonsensical laws across physics tasks.In lattice and Othello, their bias favors legal next tokens rather than the world model itself.Source: §1 · p. 2

2. Framework

The framework tests whether a foundation model’s adaptation behavior reflects a postulated world model, despite the models describing different kinds of objects. It compares the model’s extrapolations on synthetic, world-model-consistent datasets with the functions that world model permits.

2. Framework

  • A foundation model is a learning algorithm that maps a dataset to a prediction function, while a world model specifies states and how outputs depend on them.The framework treats their different representational spaces as a challenge in defining whether the foundation model has recovered the world model.Source: §2 · p. 2, §2.1 · p. 2

2.1. Comparing foundation models to world models

  • The probe treats a model’s extrapolation from limited data as evidence of its inductive bias, testing whether it learns functions allowed by the postulated world model.It repeatedly adapts the model to synthetic datasets consistent with that world model rather than scoring each extrapolation against one uniquely correct function.Source: §2.1 · p. 3

2.2. Special case: finite state space and binary outputs.

  • For binary outputs and finite states, R-IB measures whether predictions agree for inputs in the same state, while D-IB measures whether they differ across states.Both are needed: predicting the same output everywhere can score well on R-IB but poorly on D-IB.Source: §2.2 · p. 4, Figure 3 · p. 4

2.3. Inductive bias probe

  • For general outputs and state spaces, the probe compares model and oracle extrapolative predictability across pairs of inputs with varying state similarity.The oracle has access to the true state space and admissible functions; a 45-degree calibration line indicates perfect inductive bias toward it. R-IB and D-IB are special cases under stated conditions.Source: §2.3 · p. 5

3. Orbital Mechanics

The orbital transformer predicts held-out trajectories accurately, but probes and force-law tests show it has not recovered Newtonian mechanics. Its implied gravitation law varies across galaxy samples rather than yielding one universal law.

3. Orbital Mechanics

  • A 109M-parameter transformer was trained on 10M simulated orbital sequences and 20B tokens, with planet interactions omitted.Sequences interleaved planet and sun coordinates over 1,000 observations, with either six-month or one-week intervals.Source: §3 · p. 6
  • Held-out trajectory R2 exceeded 0.9999, significantly outperforming most-recent-position and per-orbit-mean baselines.The model could also generate long orbits with high accuracy.Source: §3 · p. 6
  • The probe found poor bias toward Newtonian state: extrapolations disagreed across similar states and aligned across very different states.The test used 100 synthetic datasets with outputs that were linear functions of the sequence state.Source: §3 · p. 6
  • Fine-tuning on force vectors yielded poor predictions and a nonsensical gravitation law, unlike an oracle trained on true state.The force-vector dataset provided 1% of the true forces for training; symbolic regression of the transformer’s predictions did not recover the physical law.Source: §3 · p. 7
  • Across five galaxy samples, symbolic regression found a different implied gravitation law in each, while the true-state oracle recovered the same correct law.The transformer never recovered the true law.Source: §3 · p. 7, Table 1 · p. 7

4. Other Applications

Across lattice and Othello tasks, models show weak inductive biases toward the underlying state structure despite strong next-token performance. Stronger probe performance is associated with better transfer to new state-dependent tasks, while Othello results suggest models may rely on legal-next-token patterns rather than full board state.

4. Other Applications

  • The evaluations cover finite line lattices with 2–5 states and Othello games, using five sequence-model classes pretrained on next-token prediction.The lattice data contain 10M training tokens; the Othello data contain 20M training games.Source: §4 · pp. 7–8
  • As lattice state counts increase, inductive biases decline; Othello models also show poor state biases despite generating legal moves nearly 100% of the time during pretraining.The transformer consistently underperforms recurrent and state-space models on the lattice probe.Source: Figure 5 · p. 7, §4 · p. 8
  • Models with stronger inductive-bias metrics transfer better to new Othello tasks defined as deterministic functions of board state.The tasks include Majority Tiles, Board Balance, and Edge Balance.Source: §4 · p. 8
  • Othello models can predict an incorrect full board yet recover its legal next moves, consistent with an inductive bias toward next-token equivalence rather than the true board.The tested hypothesis is that sequences with distinct states but equivalent legal next tokens are grouped together.Source: §4 · pp. 8–9, Figure 6 · p. 9
  • Across lattice and Othello, distinct states sharing the same legal next tokens yield significantly more similar extrapolations than states with different legal next tokens.This pattern suggests models group states by next-token partitions rather than distinguishing the full state structure.Source: §4 · p. 9

5. Related Work

Related work studies fixed-model behavior, internal representations, and knowledge synthesis, while this paper tests the inductive bias of the learning algorithm as it adapts to new tasks. The paper also connects its findings to work on heuristic reasoning and scientific law discovery.

5. Related Work

  • Unlike studies that test whether a fixed model predicts legal moves, this paper asks whether the learning algorithm itself embodies a world model.The distinction is between properties of a fixed model’s outputs and the behavior of a foundation model when used as a learning algorithm.Source: §5 · p. 9
  • The probe sidesteps representation-probe reliability questions by studying how a model learns, complementing work on internal representations and behavioral knowledge synthesis.Prior representation approaches include probes and sparse autoencoders; behavioral work asks whether latent training knowledge transfers downstream.Source: §5 · p. 10
  • The method is motivated by model multiplicity and causal representation learning: similar task performance may arise from different mechanisms, while causal mechanisms are linked to robust generalization.The paper evaluates inductive bias across adaptations rather than generations on one task.Source: §5 · p. 10
  • Prior work finds heuristic-like computations in GPT models, while this paper reports that a transformer without Newtonian biases does not recover gravitational law; imposing domain-specific biases is presented as promising.The cited orbital-data study by Lemos et al. (2023) recovered Newton’s law using a graph neural network with Newtonian laws explicitly imposed.Source: §5 · p. 10

6. Conclusion

The paper evaluates whether sequence models acquire postulated world models by measuring inductive biases during transfer, finding that strong next-token performance often coexists with weak world-model biases. Its measures require a world model to be specified, limiting use when analysts seek to discover the model implicit in behavior.

6. Conclusion

  • Sequence models can excel at next-token prediction while showing limited inductive bias toward genuine world models, instead relying on coarsened or non-parsimonious representations.The paper evaluates these biases as models adapt to new tasks.Source: §6 · p. 10
  • The metrics require analysts to specify a world model, constraining their use for discovering the exact representation implicit in a foundation model’s behavior.The paper proposes testing candidate models, such as next-token partitions, and identifies automatic construction of the implicit model as future work.Source: §6 · pp. 10–11

A. Model and Training Details

The experiments use five sequence-model classes with specified architectures and shared optimization settings. Physics sequences are generated from sampled solar-system parameters and trajectories obtained by solving Kepler’s equation.

A. Model and Training Details

  • The RNN uses six layers for Othello and two for lattice tasks, while the LSTM follows the same specification; the transformer decoder has 12 layers, 12 heads, and 768-dimensional embeddings.The lattice RNN uses two layers because that setting optimizes to better in-sample and out-of-sample loss.Source: A. Model and Training Details · p. 14
  • Mamba uses 24 layers and Mamba-2 uses the same architecture with a Mamba-2 mixer; both use 768-dimensional embeddings, state expansion 16, block expansion 2, and convolutional width 4.The paper treats 24 Mamba layers as analogous to 12 transformer layers.Source: A. Model and Training Details · p. 14
  • All models use Adam with learning rate 6e-4, 2,000 warmup iterations, weight decay 0.1, and gradient clipping at 1.Next-token pretraining uses a prediction head tied to the initial embedding weights.Source: A. Model and Training Details · p. 14
  • Physics systems sample planet count, eccentricity, semi-major axis, and masses from specified distributions before trajectories are generated by solving Kepler’s equation.The sampled distributions are chosen to keep the solar systems within the model’s training distribution.Source: A. Model and Training Details · p. 14

B.1. Physics

The physics and lattice/Othello experiments detail how the inductive-bias probe is constructed and tested under modified physics settings. In the modified two-body and same-trajectory physics evaluations, the model still shows poor bias toward Newtonian mechanics.

B.1. Physics

  • For the physics probe, 100 datasets of 100 examples produce separately fine-tuned models whose predictions are evaluated on held-out inputs.The procedure selects a state projection by its Spearman correlation with pairwise distances, then computes metrics from sampled predictions and binned oracle-prediction distances.Source: §B.1 · p. 14
  • Pretraining on two-body systems and evaluating on partially observed trajectories yielded results nearly identical to the main-text findings.The two-body model was pretrained on 10 million systems; the alternative evaluation fine-tuned on two observed outputs per sequence and extrapolated over the rest.Source: §B.1 · p. 15
  • The modified two-body evaluation still shows poor Newtonian inductive bias, and symbolic regression of force-magnitude predictions fails to recover Newton’s law.The modified metric evaluates partially observed training sequences; the reported equation is F ∝m1 × 1 exp(r).Source: §B.1 · p. 15, Figure 7 · p. 15

B.2. Lattice and Othello

  • For lattice and Othello, the probe assigns consistent random binary outputs by state, fine-tunes separate models, and compares prediction losses for same-state versus different-state inputs.The metrics are rescaled so 0 is perfect accuracy and 1 is random guessing; larger values indicate stronger state-related inductive bias.Source: §B.2 · p. 15
  • The lattice inputs comprise valid move sequences, while Othello uses permutations of openings leading to a repeated board to prevent sequence-permutation heuristics from mimicking state sensitivity.The Othello set contains 210 openings of length 10, yielding 2,100 move subsequences.Source: §B.2 · p. 16

C. Force Prediction Implementation Details

The force-prediction implementation tests whether fine-tuned transformers recover gravitational force patterns from orbital data. Symbolic regression does not recover the true law from transformer predictions, while the oracle procedure does.

Force Prediction Implementation Details

  • The transformer is fine-tuned on solar-system trajectories with 1% of observations labeled for force vectors, then extrapolated across the remaining points.Force vectors are normalized per sequence; fine-tuning uses MSE for 10,000 steps, with the checkpoint selected by held-out loss.Source: C. Force Prediction Implementation… · p. 16
  • For force-magnitude tests, the model trains on 9,000 two-body problems and is evaluated on selected nearby test states, including partial observations from each test sequence.Symbolic regression searches a constrained operator space over predictions sampled from the test set; the oracle uses the same held-out points and regression procedure.Source: C. Force Prediction Implementation… · p. 16
  • Symbolic regression never recovered the true gravitational law from transformer predictions across five sampled galaxies, whereas the true-state oracle recovered it for all five.The oracle uses a k-nearest-neighbor model with k = 2 based on Euclidean distance to the true state.Source: C. Force Prediction Implementation… · pp. 16–17

D. LLM Physics Experiments

The LLM experiment uses in-context force-magnitude prediction as a small-scale alternative to fine-tuning and a full inductive-bias probe. Most predictions are poor, and symbolic regressions corroborate that outcome.

LLM Physics Experiments

  • The experiment evaluates o3, Claude 4 Sonnet, and Gemini 2.5 Pro in context because fine-tuning is infeasible and a full probe is expensive.Each model receives five solar systems with 450 observations each and force magnitudes for 10 observations per system, then predicts the remaining outputs.Source: D. LLM Physics Experiments · p. 17
  • Figure 9 compares predicted and true force magnitudes across timesteps for five randomly sampled solar systems.Each LLM receives the full trajectory and a random 2% sample of force magnitudes to impute the remaining outcomes.Source: Figure 9 · p. 18, D. LLM Physics Experiments · p. 18
  • Most LLM force-magnitude predictions are poor, as also indicated by symbolic regressions.The symbolic regressions are simpler than those for domain-specific foundation models, but the authors note that differences in experimental setup may explain this.Source: D. LLM Physics Experiments · p. 18

E. Inductive Bias Ablations

The Othello ablations vary fine-tuning iterations and example counts to examine the inductive-bias metrics.

Inductive Bias Ablations

F. Additional Transfer Results

Additional transfer tests relate inductive-bias metrics to transfer performance, while next-token prediction can look strong; extrapolation analyses find predictability aligned with legal-next-token partitions.

F. Additional Transfer Results

  • Transfer performance across new functions of state correlates with inductive-bias metrics: models with low inductive bias perform worse at transfer.Table 6 reports transfer performance across new functions of state and includes an unsigned IB-correlation measure.Source: Table 6 · p. 20, F. Additional Transfer Results · p. 20

H. What are models using to extrapolate?

  • For lattice and Othello, the analysis coarsens states into pseudo-states, then compares extrapolative predictability within and across those groups.Lattice states are mapped into three pseudo-states; Othello boards are grouped by their sets of legal next moves, which can be shared by multiple boards.Source: H. What are models using to extrapolate? · pp. 20–21

G. Next Token Performance

  • Pre-trained lattice and Othello models make good next-token predictions that appear to obey state.The next-token test measures the share of top model predictions that are true for the underlying state.Source: Table 7 · p. 20, G. Next Token Performance · p. 20

H. What are models using to extrapolate?

  • Across all models, D-IBq= is smaller than D-IBq̸=, indicating more predictable extrapolations for different-state sequences sharing legal next tokens.The table caption says low D-IBq= and high D-IBq̸= suggest failures to differentiate state are driven by a bias toward the legal next-token partition.Source: Table 9 · p. 21, H. What are models using to extrapolate? · p. 21
Loading 2507.06952v4…