PaperlayerView exact revision on arXiv

Source-linked AI summary

Do LLMs Understand Limit Order Book Dynamics?

Junxiao Chen, Paul Glasserman

arXiv:2608.23706v1cs.AI

TL;DR

The paper asks whether an LLM that generates valid LOB event sequences has learned the LOB’s state and dynamics well enough for forecasting. It trains an LLM on synthetic LOB data and evaluates it with tests adapted to stochastic dynamics. The model performs extremely well on validity-oriented sequence tasks but exhibits systematic world-model errors, biased forecasts, and spurious predictability.

  • Problem

    The paper asks whether generating valid LOB event sequences implies a correct world model, and whether that understanding matters for forecasting.

  • Method

    The authors train an LLM from scratch on simulated LOB events and evaluate its world model using stochastic validity and compression tests, total variation measures, and regression diagnostics.

  • Results

    The LLM generates valid sequences extremely well yet shows systematic world-model errors that produce biased forecasts and spurious predictability.

  • Takeaways & Limitations

    Valid-sequence performance is insufficient evidence of a correct LOB world model for downstream forecasting.

  • Takeaways & Limitations

    The experiments use small LOB settings to make training from scratch feasible, while larger LOBs would likely require vastly more training data.

Abstract

from arXiv · show

A large language model (LLM) trained on synthetic limit order book (LOB) data achieves near perfect scores in generating valid sequences of LOB events. However, the LLM's implicit world model fails to learn the state of the LOB. This deficiency leads to biased estimates and spurious predictability in using the LLM to forecast future LOB events. Our analysis uses novel tests of an LLM's world model, extending prior work from deterministic settings to the stochastic dynamics needed for the LOB.

1. Introduction

The paper asks whether generating valid LOB event sequences implies a correct world model, and whether that understanding matters for forecasting. Using simulated stochastic LOB data and new diagnostics, it finds strong sequence validity alongside systematic world-model errors, biased forecasts, and spurious predictability.

  • The paper investigates whether an LLM that recognizes and generates valid LOB events has developed a correct LOB world model.
  • The transformer-based LLM performs extremely well at generating valid sequences and finding sequences that move the LOB between states.These tests capture validity and multi-step reachability, including key matching constraints.
  • Despite strong validity performance, the LLM exhibits systematic errors in its implicit world model of LOB dynamics.
  • In forecasting, the LLM finds predictability where none exists, producing misleading forecasts for future LOB events.The evaluation uses simulated data whose LOB state is a Markov chain, so past history is irrelevant given the current state.
  • The paper adapts prior deterministic world-model tests to stochastic LOB dynamics and introduces kernel-level and history-level total variation measures plus a regression test.
  • Valid-sequence tests are insufficient for downstream LOB tasks; larger datasets or new training methods are needed to improve the LLM’s LOB world model.The experiments use small LOB settings to make training from scratch feasible, and the documented issues would likely be more severe with a larger state space.

2. Limit Order Book

The paper represents an LOB as a finite vector encoding best ask, best bid, and quantities across fixed price levels. State transitions arise from valid limit, cancellation, and market-order events, with probabilities defining a finite-state Markov chain.

  • 2.1. States: An LOB state is represented by (ℓa, hb, v1, . . . , vP), where P is the fixed number of possible price levels.ℓa and hb identify the lowest asking and highest bidding prices, while vi records quantities on the relevant side.
  • 2.1. States: Price levels between the best bid and ask form the spread region and must have zero volume.The state space is kept finite by capping total volume per price level and individual order size.
  • 2.2. Events and State Transitions: LOB events include limit arrivals, cancellations, and market orders, each represented by a token describing side, price, and size.Market sell orders consume volume from the bid side starting at the best bid.
  • 2.2. Events and State Transitions: Matching rules determine valid events and map each valid event deterministically from state s to a new state s′.Multiple events may produce the same state transition.
  • 2.2. Events and State Transitions: Assigning event probabilities μ(a|s) turns the deterministic event updates into a Markov chain over LOB states.The transition probability from s to s′ aggregates probabilities of events that map s to s′.

3. Synthetic Datasets Construction

The paper constructs synthetic LOB-event datasets from a stationary Markov chain and trains models to generate valid event sequences conditioned on start and goal states. It compares random walks, intended to resemble observations, with shortest paths that emphasize reachability.

  • Each training sequence contains a start-state ID, goal-state ID, valid event tokens, and an end token.The end token is valid only when the current state equals the target state.
  • The model is prompted with (sstart, sgoal) to generate valid events that move the LOB from the start state to the goal state.This requires learning validity plus multi-step reachability and may suggest planning over the transition structure.
  • The study constructs two sequence datasets: random walks and shortest paths.The shortest-path dataset is used for comparison and is expected to explore the state space less thoroughly.
  • Random-walk dataset: Random walks start from the stationary distribution, sample events from the state-dependent kernel, and stop geometrically with mean sequence length 1/pstop.The construction is intended to approximate observational data.
  • Shortest-path dataset: Shortest-path sequences are constrained to reach the goal state using the fewest possible transitions.
  • Goal conditioning is averaged over goals drawn from the same data-generating process in later prediction tests.

4. LLM Training

The paper trains decoder-only GPT-style transformers from scratch with next-token cross-entropy prediction on synthetic LOB sequences. It uses a base configuration, three training datasets, held-out evaluation, and larger LOB settings for scaling experiments.

  • Each model is a decoder-only GPT-style transformer trained with standard next-token prediction and cross-entropy loss.The base architecture has 12 layers, embedding dimension 768, 12 attention heads, and approximately 86.4 million parameters.
  • The larger transformer has 48 layers, embedding dimension 1600, 25 attention heads, and approximately 1.5 billion parameters, with similar qualitative results.
  • The base experiments use LOB settings (P, V1, V2) = (3, 5, 2) and train three models: SP, RW-S, and RW-L.
  • SP is trained on 500K shortest-path segments, while RW-S and RW-L each use 10M random-walk segments with different geometric mean lengths.The geometric mean lengths are 5 for SP, 20 for RW-S, and 50 for RW-L.
  • The data are split into 99% training and 1% held-out evaluation, with SP trained for 10 epochs and RW models trained for one epoch.Training takes 4–6 hours for SP and 1–2 days for RW models.
  • Scaling experiments increase either depth capacity to (P, V1, V2) = (3, 12, 3) or the number of price levels to (P, V1, V2) = (8, 2, 2).

5. Empirical-Kernel Baseline

The empirical-kernel baseline predicts next events from the current LOB state using observed transition frequencies, providing a comparison for separating data-driven errors from representation errors.

  • Baseline construction: The baseline uses empirical next-token frequencies conditioned only on the current LOB state.For state s and event a, it uses observed counts Ntrain(s, a) relative to total observed outgoing events.
  • Baseline construction: The baseline shares the RW-L model’s training dataset, enabling direct comparison under the same data coverage.More than 99.8% of evaluation states are observed in the training data, and baseline results are computed over covered states.
  • Interpretation: If the baseline avoids an error pattern found in the LLM, that pattern is attributed to the LLM’s learned representation rather than the training data.The baseline has no hidden state or attention mechanism and predicts only from the current state.

6. Model Diagnostics

The LLMs nearly always generate valid, often short goal-directed LOB trajectories, but compression tests reveal that their predicted feasible continuations depend incorrectly on history rather than only the current state.

  • 6.1. Valid Traversal Test: The valid traversal test checks whether generated events obey true LOB transition rules and whether termination occurs at the target state.The evaluation reports the fraction of valid sequences, alongside shortest-path diagnostics.
  • 6.1. Valid Traversal Test: All three LLMs nearly always generate valid sequences, while the baseline performs poorly because it lacks goal-directed generation.The test uses held-out start-goal pairs and greedy next-token selection.
  • 6.1. Valid Traversal Test: 90.6% of RW-S generated sequences reach the goal by a shortest path, versus 10.4% in its training set.RW-S generated paths exceed the shortest path by 0.1 steps on average, compared with 16.3 steps in its training data.
  • 6.2. Compression Test: The compression test evaluates whether feasible suffixes depend only on the current state, not on the history that led there.Two histories terminating in the same state are compared using suffixes generated from one history and accepted under the other.
  • 6.2. Compression Test: All three LLMs have low compression scores, whereas the baseline score is close to 100%.For RW models, more than half the states are judged history-dependent even though true LOB dynamics give identical valid suffix sets for histories ending in the same state.
  • Interpretation: The diagnostics therefore present a mixed picture: strong valid-sequence generation coexists with failure to learn a key feature of LOB dynamics.Valid-sequence success alone suggests rule understanding, but history dependence exposes a deeper state-modeling error.

7. Evaluating LLM Forecast Distributions

The paper evaluates whether LLM forecast distributions reflect stochastic LOB dynamics, rather than merely producing valid event sequences. Kernel-level TV measures distributional bias, history-level TV measures dependence on irrelevant past events, and regression tests assess spurious predictability.

  • Stochastic counterparts to validity and compression tests are needed because deterministic-style tests can ignore probabilities assigned to valid events.Under greedy generation, valid traversal depends only on whether the highest-probability event is valid; compression mainly checks common support.
  • Kernel-level TV compares the LLM’s goal-averaged H-step path distribution with the true H-step transition kernel, including probability assigned to invalid events.The measure lies in [0, 1], with smaller values indicating stronger agreement.
  • 7.2. Example 1: Kernel-Level TV: 0.665 is the one-step kernel-level TV in an example with 20 valid events, where the true kernel assigns 0.05 to each event.The model assigns nearly 50% probability to two cancellations at p1, although their actual probability is 10%, and overestimates limit sells at p1 or p2.
  • 7.3. History-Level TV Distance: History-level TV measures how much forecasts differ across histories ending in the same current state, violating the Markov property when past events influence predictions.The measure captures dispersion around the within-state mean model distribution.
  • 7.3. History-Level TV Distance: Both TV measures increase at longer forecasting horizons, while their small variances indicate consistent forecast errors and excessive dependence on past events across states.The kernel-level TV uses 1000 sampled state-prefix pairs, and the history-level TV uses 200 states with 10 prefixes each.
  • 7.4. Example 2: History-Level TV: 0.385 is the history-level TV for two histories reaching the same state; the model assigns 0.278 to a buy at p2 after one history and 0.313 to a sell at p3 after the other.These history-dependent probabilities constitute spurious predictability because the Markov-generated data contain no such predictive information.
  • 7.5. Regression Test: Regression results show state-dependent forecast miscalibration and prefix-history effects, including systematic overestimation of buys with larger depth imbalance and market orders with larger spreads.The RW-S results report prefix-history effects in eight of twelve regressions; the supplied passage truncates the remaining details.

8. Scaling to Larger LOB Settings

The study repeats its pipeline in larger LOB environments to test whether its findings are specific to small state spaces. The qualitative conclusions remain stable, while diagnostics indicate weaker state representation as complexity increases.

  • The base setting uses (P, V1, V2) = (3, 5, 2), producing a finite LOB space with 756 states.
  • Two larger settings expand the number of reachable LOB states to 8281 and 41553.
  • Qualitative conclusions remain stable across the larger environments, with high valid traversal rates and short generated sequences.
  • Compression scores decline and kernel-level and history-level TV distances increase as the LOB environment becomes larger, especially for RW models.
  • The scaling results suggest that the LLM’s ability to represent the correct Markov state abstraction weakens as the LOB becomes more complex.

9. Conclusion

The paper shows that LLMs trained on LOB event sequences can generate valid sequences yet fail to understand LOB dynamics. Using synthetic data and small settings, it links this deficiency to biased forecasts and spurious predictability while identifying larger training requirements as a practical boundary.

  • The paper shows that an LLM trained on LOB event sequences may generate valid sequences yet fail to understand LOB dynamics.
  • This deficiency leads to biased forecasts and spurious predictability in LOB events.
  • Synthetic data lets the study evaluate LLM forecasts in a setting without predictability, while actual LOB events may exhibit some predictability.
  • The study keeps LOB settings small to train an LLM from scratch, while larger LOBs would likely require vastly more training data.

A.1. Shortest Paths Datasets Construction

The shortest-path dataset construction uses reverse breadth-first search from a sampled goal state to build training sequences. The supplied passages also identify tables presenting diagnostic results and example histories.

  • A goal state sg is sampled from the stationary distribution π before constructing a shortest-path training sequence.
  • Reverse breadth-first search works backward from sg using state-transition distances to identify valid incoming events.
  • Table 6 presents diagnostic and multi-step forecast results for larger LOB settings.
  • Table 7 presents the two prefixes used in the history-level TV analysis.
Loading 2608.23706v1…