PaperlayerView exact revision on arXiv

Source-linked AI summary

Understanding the Inner Workings of a State-Space Foundation Model for Limit Order Books

Valentin Mohl, Alexandre Bismuth, Aramis Marti-Shahandeh, George Nigmatulin, Kang Li, Stefan Zohren, Jakob Foerster, Mihai Cucuringu, Anisoara Calinescu

arXiv:2610.28753v1q-fin.TRcs.LG

TL;DR

This paper asks what autoregressive LOB foundation models learn internally, since black-box performance on measured distributions does not establish behavior under market shifts. It analyzes an autoregressively trained Mamba-3 model and finds market dynamics decodable across stocks and metrics, with state steering shifting next-message beliefs.

  • Problem

    LOB foundation models generate realistic message flow, but their internal representations are poorly understood, limiting evidence about behavior beyond measured distributions.

    Source: §1 · p. 2

  • Method

    The authors analyze an 8.2M-parameter Mamba-3 trained autoregressively on message streams from eight large-cap US equities, probing and steering its recurrent state.

    Source: §1 · p. 2, §4.3 · p. 3, §5.1.5 · p. 6

  • Results

    OFI decodes from the recurrent state at R^2≈0.59 versus ≈0 for a random-weights twin, with traded volume and realized volatility also represented, generalization to unseen stocks, and monotonic belief shifts under steering.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The findings support an internal representation of market dynamics that is decodable and steerable, rather than only a picture of the market.

    Source: §6 · p. 8

    A frozen-state readout predicts the next 25 messages’ mid-price move at only R^2≈0.04, on par with a readout of the visible book.

    Source: §1 · p. 2

Abstract

from arXiv · show

Autoregressive foundation models offer state-of-the-art capabilities to simulate the market by generating the message flow of limit order books (LOBs). Yet, without understanding their inner workings, research directions are hard to narrow and practical use in real-world trading environments remains uncertain. This study therefore asks what these models actually learn about market microstructure. We find that, similarly to games where sequence models build an internal representation of the board, an 8.2M-parameter Mamba-3 state-space model trained autoregressively on the tokenized message stream of eight large-cap US equities learns an internal representation of the market and its dynamics. In particular, order-flow imbalance, a central driver of short-horizon price moves, is decodable at $R^2 \approx 0.59$, far above controls and baselines. This representation extends to unseen stocks, suggesting that these models learn general representations rather than ticker-specific patterns. The model also generalizes across metrics beyond order-flow imbalance: results hold across traded volume and volatility. Additionally, the representation is causally steerable: pushing the state along a feature's axis shifts the model's next-message beliefs as that feature implies, monotonically in dose, consistent with the emergence of a world model.

1 Introduction

The paper asks whether a message-only LOB foundation model develops an internal representation of market dynamics, addressing a black-box gap that limits confidence beyond measured distributions. Its 8.2M-parameter Mamba-3 model represents flow across stocks and metrics, and its recurrent state can be steered to shift next-message beliefs.

1 Introduction

  • Because LOB foundation models are black boxes, backtested performance on measured distributions does not establish their behavior under market shifts or costly tail events.The authors argue that understanding model representations has both practical and scientific value.Source: §1 · p. 2
  • The study tests whether next-token training on message streams yields an internal market representation using an 8.2M-parameter Mamba-3 model trained on four years of data from eight large-cap US equities.The model is described as efficient and competitive on LOB-Bench.Source: §1 · p. 2
  • OFI decodes from the recurrent state at R^2≈0.59 versus ≈0 for a random-weights twin; traded volume and realized volatility also decode, including on unseen stocks.A frozen-state readout predicts the next 25 messages’ mid-price move at R^2≈0.04, on par with a visible-book readout.Source: §1 · p. 2
  • The study presents a first mechanistic interpretability analysis of a foundation model trained natively on LOB data, extending emergent-world-model analysis to real market data.The authors characterize the learned representation as market flow, not only a static picture, and report generalization to stocks unseen in training.Source: §1 · p. 2
  • Steering the recurrent state along the OFI direction shifts next-message beliefs monotonically with dose, unlike matched-norm null directions.The authors report this causal steerability as evidence that the representation affects model expectations.Source: §1 · p. 2

2 Background

The background describes LOB messages and the market statistics used to probe the model, especially order-flow imbalance (OFI), which tracks net buying or selling pressure over a message window.

2 Background

  • A LOB holds outstanding buy and sell limit orders at price levels, while submissions, cancellations, and executions arrive as messages; the model consumes each message as a 26-token block.The best bid and ask form the touch, with orders queuing under price-time priority.Source: §2 · p. 2
  • OFI measures net buy-side pressure over a lookback window of messages, using best-quote prices and aggregate queue sizes after each event.Positive OFI indicates net buying pressure, while negative OFI indicates net selling pressure.Source: §2 · p. 2
  • Depth-k OFI evaluates the same expressions at the k-th best bid and ask, with OFI_1 equal to the standard best-quote measure.The depth measure sums bid-side minus ask-side event contributions across the selected levels.Source: §2 · p. 2

3 Related Work

Prior work shows emergent representations in sequence models and a growing range of financial market models, but mechanistic analysis of native LOB foundation models remains largely unaddressed.

3 Related Work

  • Sequence models trained on Othello, chess, and program execution can develop representations of hidden states, while prior work also finds static real-world features in sequence models [27] [37] [22] [20] [16].Probe accuracy may overstate world-model coherence, and probed states can be causally inert [49] [50] [3].Source: §3 · p. 2
  • Financial market modeling spans supervised LOB predictors and GAN-, RNN-, and diffusion-based generators, alongside autoregressive message-stream foundation models [21] [40] [46] [51] [57] [2] [5] [9] [52] [23] [26] [28] [29] [53].Autoregressive approaches outperform GAN-based and parametric alternatives on distributional benchmarks [36].Source: §3 · pp. 2–3
  • Most prior market-model evaluation measures predictive accuracy or distributional realism rather than internal computation.Earlier interpretability work largely uses post-hoc input attribution, while mechanistic finance studies have mainly focused on language models or generic time-series interventions [30] [42] [13] [34] [47] [7] [54] [45].Source: §3 · p. 3
  • The authors identify no prior mechanistic interpretability study of a foundation model trained natively on LOB data or of recurrent-state probing and steering in an SSM outside language and synthetic-game domains.The closest cited concurrent work probes a Transformer’s latent volatility state on simulated returns [18].Source: §3 · p. 3

4 Method

The study trains a message-only Mamba-3 model on LOB streams, then probes its internal states, intervenes on them, and tests forward prediction and robustness. These methods assess what market information the state contains, how the model uses it, and whether the findings withstand alternative evaluation and controls.

4.1 Training pipeline

  • The model uses a message-only 8.2M-parameter Mamba-3 trained on eight equities, with probes evaluated on held-out GOOG data and generalization tests on other training and out-of-distribution stocks.Training covers 2022-01 to 2025-12; evaluation uses January 2026 windows.Source: §4.1 · p. 3

4.2 Probing

  • Probes read touch, OFI, traded volume, realized volatility, and book depth from per-layer residual streams and flattened recurrent states at a fixed message boundary.The recurrent state has 65,536 dimensions; message contributions are reconstructed from the recurrence’s equivalent attention form and verified against the model state and layer output.Source: §4.2 · p. 3, §4.1 · p. 3, Figure 1 · p. 4

4.3 Causal steering

  • Causal steering adds a feature-specific high–low mean-state direction across all 13 layers, then tests seven doses against shuffled-label and isotropic random directions.The primary readout compares next-message beliefs after teacher forcing the same real future against an unperturbed pass; engine replay is a secondary check.Source: §4.3 · p. 3

4.4 Forward predictions

  • Forward-prediction probes use the frozen state to regress future price moves and OFI or classify future mid-price direction from real messages after the observation boundary.Regression uses 700-message windows with at least 200 future messages; classification uses 1,500-message windows with at least 1,000 future messages and reports accuracy and macro-F1 against the majority-class baseline.Source: §4.4 · p. 4

4.5 Robustness

  • Robustness checks use day-grouped and forward-chaining cross-validation, day-block bootstrap intervals, random-weight controls, and book-snapshot baselines.The probes use dual-form kernel ridge regression; the passage reports that 64 PCA components recover full OFI R^2 performance.Source: §4.5 · p. 4

5.1 Encoding of market statistics

The model’s market statistics are most readable in its recurrent state, with OFI and related signals concentrated in particular layers and integrated over message history. Depth is an exception to the state’s advantage over book snapshots.

5.1.1 Residual stream vs. recurrent state.

  • At W=500, OFI, traded volume, and realized volatility decode at R² 0.50–0.61 from the recurrent state, versus at most 0.06 from the residual stream.Touch reaches R² 0.83 in the recurrent state and 0.26 in the residual stream.Source: §5.1.1 · p. 4, Figure 2 · p. 5

5.1.2 Order-flow imbalance in the recurrent state.

  • Training accounts for nearly all OFI and traded-volume signal, with random-twin R² values of 0.01 and 0.03, respectively.The model’s touch readout rises from 0.43 in the twin to 0.83; a book snapshot from about 500 messages earlier reaches only 0.49 for current touch.Source: §5.1.1 · p. 5, Table 1 · p. 5

5.1.3 Volume, volatility, and depth.

  • The recurrent state also decodes traded volume and realized volatility above controls and snapshot baselines, but stale book snapshots predict depth better: 0.64 versus 0.38.The authors attribute the depth exception to persistent resting liquidity and the state’s limited view of book activity within its 500-message history.Source: §5.1.3 · p. 5, Table 2 · p. 5

5.1.4 Per-layer encoding variations.

  • OFI L1 peaks at R² 0.59 in layer 3, falls near zero mid-stack, then recovers to 0.16 at the penultimate layer; touch follows a similar pattern.Linear, quadratic, RBF-kernel, and one-hidden-layer MLP probes follow the same curves, so the dip is not a nonlinear re-encoding.Source: §5.1.4 · p. 5, Figure 3 · p. 5

5.1.5 Relevance of specific messages.

  • The newest message contributes 58% of total state-write magnitude but only 7% of the decoded OFI signal, which draws on 56 messages for half its value.At the OFI probe layer, four of eight heads have 28–66-message half-lives; the random twin’s readout instead gets 51% from one message and 89% from five.Source: §5.1.5 · p. 6, Figure 4 · p. 6

5.2 Forward predictions

State interventions shift next-message beliefs in the encoded directions, but their effects only partly survive autoregressive generation. The recurrent state carries limited forward signal, with directional decoding strongest at short horizons and often no better than the visible book.

5.2.1 Causal steering.

  • 0.42 to 0.61 buy probability, about 1.4× P(type_t+1=exec) per unit dose, and nine-fold turbulence-driven forecast widening show feature-specific steering.Across the seen range, trained responses are monotonic and outside the full range of shuffled-label axes; large volatility pushes also widen forecasts under the null.Source: §5.2.1 · p. 6, Figure 5 · p. 7
  • Generated market control is strongest for traded volume (25 to 187 shares at α=+2.4; 25 to 1 at α=−2.1), weaker for sell-side flow, and absent for volatility.The volatility push widens the model’s forecast but does not increase realized volatility; the authors rank control as traded volume > order flow > volatility.Source: §5.2.1 · p. 6

5.2.2 Future-time decoding.

  • 4% of next-25-message mid-price-move variance and at most 5% of future top-5 OFI are explained, with no signal remaining beyond H≈100.For mid-price moves, the state is level with an L10 book snapshot; for future OFI, the book snapshot carries almost no signal.Source: §5.2.2 · p. 6

5.2.3 Directional accuracy.

  • At H=10/25/50, state macro-F1 is 0.33/0.36/0.42 versus 0.28/0.27/0.38 for the book-snapshot baseline; the book matches or exceeds the state from H=200.Short-horizon accuracy is nearly identical across approaches, including the majority class, because most windows have no mid-price move.Source: §5.2.3 · p. 7, Figure 6 · p. 7
  • On windows with a mid-price move, binary directional accuracy is 61.2% at H=25 and 60.0% at H=50, level with the book readout, then reaches chance by H=500.The book-snapshot readout is 61.8% and 60.9% at H=25 and H=50, respectively.Source: §5.2.3 · p. 7

5.3 Learning dynamics and cross-stock generalization

The recurrent state’s market-feature readouts develop at different rates, and stock-level OFI decoding varies with the prevalence of extreme bursts. Tail-robust metrics show OFI representation across both training and unseen stocks.

5.3.1 Pretraining behavior.

  • Touch decoding plateaus quickly, while OFI decoding improves throughout training even as cross-entropy and LOB-Bench plateau.Figure 7 places the LOB-Bench plateau after about a third of training; the authors suggest longer pretraining may reinforce OFI encoding.Source: §5.3.1 · p. 7, Figure 7 · p. 7

5.3.2 Generalization across stocks.

  • Touch decodes on all ten stocks, including the two OOD stocks; raw level-1 OFI R² is 0.26 to 0.60 except META and MSFT, where it is near zero.The boundary-book probe stays at R² ≤0.14 and the random-weights twin at R² ≤0.04 on every stock.Source: §5.3.2 · p. 7, Figure 8 · p. 8
  • Winsorized or rank-based decoding yields OFI L1 R² ≥0.4 and OFI top-5 R² ≥0.5 across in-training and OOD stocks, while tail prevalence explains much of raw-R² variation.Raw L1 OFI R² falls as the share of target variance in the top 1% of windows rises; the fitted relationship has R²=0.94.Source: §5.3.2 · p. 8, Figure 9 · p. 8

6 Conclusion

The paper concludes that a message-only foundation model’s recurrent state represents market dynamics, including flow, and can be read and steered. It also identifies potential uses for latency-sensitive readouts and scenario generation, alongside future interpretability work.

6 Conclusion

  • A message-only model’s recurrent state represents market dynamics: its features decode above baselines, remain decodable on unseen stocks, and steering shifts model expectations in the pushed direction.The conclusion characterizes this as learning not only a picture of the market, but its flow.Source: §6 · p. 8
  • Linear maps read market dynamics with one matrix product, potentially helping latency-sensitive applications such as high-frequency trading; steering may support scenario generation without conditioning inputs or retraining.The authors frame the state as a representation engine that can be read and steered, beyond its use in market simulation.Source: §6 · p. 8
  • Future sparse-autoencoder work could search for features beyond the concepts already probed and potentially uncover predictors not yet identified in market microstructure research.Source: §6 · p. 8
Loading 2610.28753v1…