Source-linked AI summary
Entity tracking emerges in sub-billion parameter language models and exceeds human performance in naturalistic narratives
Karolina Drożdż, Micha Heilbron
TL;DR
Whether language models track entities across discourse as humans do remains unclear because prior evaluations use artificial tasks and lack human comparisons. This study evaluates humans and models on naturalistic narratives and finds human-level tracking at 410 million parameters, with performance improving with scale and eventually exceeding humans.
Problem
Whether language models track entities across discourse as humans do remains unclear, especially under naturalistic conditions and direct comparison with human comprehenders.
Method
The study evaluates humans and two open model families across scales using controlled, naturalistic narratives with explicit and implicit entity-tracking measures.
Results
Human-level entity tracking emerges at 410 million parameters, improves with scale, and contemporary models far exceed human performance.
Takeaways & Limitations
Entity tracking emerges at model scales far below those previously associated with the ability and can surpass human performance at sufficient scale.
Takeaways & Limitations
Because evaluated pretrained models contain code in their training data, the study cannot isolate code’s contribution to entity-tracking capacity.
Abstract
from arXiv · showhide
Understanding language requires tracking entities across discourse - i.e., knowing where things are and how they change, even when not explicitly stated. Whether language models perform such tracking in a human-like fashion remains unclear, in part because existing evaluations rely on artificial tasks, far removed from natural language comprehension, and lack comparisons to humans. Here, we evaluate entity tracking in both language models and humans (N = 48) using naturalistic narratives at multiple levels of complexity. In humans, we find that entity tracking degrades specifically with narrative complexity, not narrative length. In language models, we find that human-level entity tracking is already present at 410 million parameters - well below the multi-billion parameter, code-specialised models identified by prior work - and improves with scale, with contemporary models far exceeding human performance. Together, these results demonstrate that entity tracking, a core component of language understanding, emerges at model scales far smaller than previously thought.
1 Introduction
This section motivates naturalistic, human-compared evaluation of entity tracking as a test of language understanding. It introduces evidence that tracking emerges in small language models, improves with scale, and can exceed human performance.
- Motivation: Entity tracking tests whether language models maintain and update representations of entities’ underlying states and locations across discourse, as human comprehenders do.The question bears on whether coherent text reflects genuine language understanding or sophisticated pattern matching.
- Limitations: Prior evaluations lacked human comparisons and used formulaic, artificial tasks resembling reasoning puzzles or program execution rather than natural-language comprehension.Reported parameter and data requirements may therefore reflect task demands rather than limits on entity tracking itself.
- Approach: The study uses readable short narratives spanning five situational-complexity levels, assesses explicit and implicit tracking, and compares models with humans (N = 48).It evaluates Pythia (70M–12B), OLMo 2 (1B–32B), Llama 3.3, and Qwen 2.5, including pseudoword and semantically improbable-object narratives.
- Findings: Human entity tracking degrades specifically with situational complexity, revealing a cognitive complexity cost to maintaining structured situation models.The evaluation distinguishes complexity effects from the broader question of tracking entities across naturalistic narratives.
- Findings: 410 million parameters marks the emergence of robust, human-level entity tracking, which improves predictably with scale as complexity effects diminish and disappear in contemporary 70B models.The study concludes that entity tracking emerges below previously reported scales and that sufficiently large models can far exceed human performance.
- Contribution: The shared evaluation paradigm enables naturalistic entity-tracking research in small, fully open base models without task-specific fine-tuning.This extends the contribution beyond the reported empirical findings.
2 Related Work
Prior work has investigated structured world or situation models in artificial domains and natural language, but evidence has been mixed and direct entity-tracking assessments have relied on artificial tasks. These evaluations have not established whether the findings generalize to natural narrative comprehension or compare models with humans.
- Structured representations: Evidence for structured world models is inconsistent across artificial domains, although Othello-trained models develop board-state representations supporting predictions on unreachable states.These findings come from studies of domains with fully specified ground truth, including games and planetary motion.
- Natural-language tracking: Representational and mechanistic studies have identified entity-tracking circuits in base language models and found that fine-tuning enhances these pathways.The studied task was a simplified entity-tracking version without state changes.
- Prior entity-tracking evaluations: Prior direct assessment found that pure-text models up to 175 billion parameters failed entirely on the boxes-and-objects task, with tracking first appearing in GPT-3.5.GPT-3.5 was described as a closed, code-specialised model.
- Open limitations: Because these studies used an artificial, programmatic task and lacked human comparisons, they left open whether their findings generalize to situation-model construction in natural narrative comprehension.The present work addresses both limitations.
3 Methods
The study introduced a naturalistic entity-tracking task requiring participants and language models to infer an object's final location after movements across narratives of varying complexity. It combined explicit and implicit evaluations, controlled recency and template-based strategies, and compared 48 human participants with openly accessible model families across scales.
- Entity-tracking task: Participants and models tracked moving objects across locations in short narratives, then answered a question about the target object's final location.The task was designed as a novel entity-tracking evaluation using narratives with varying complexity.
- Narrative complexity: Five complexity levels varied the number of objects, locations, and movements, from C1 with one object and location to C5 with four objects across four locations.Humans were tested on C1–C4 because pilot data indicated that C4 already imposed substantial cognitive load; language models were evaluated at higher complexity.
- Evaluation formats: Explicit evaluations required free generation or open-ended responses, whereas implicit evaluations used probability read-outs for models and forced-choice selection for humans.Both formats were included because explicit question-answering can systematically underestimate model capabilities.
- Recency control: Target-object positions were varied within longer narratives to distinguish recency-based recall from the demands of tracking multiple entities.Higher-complexity narratives were also longer, creating a potential confound between complexity and recency.
- Human procedure: N = 48 native English speakers completed 46 trials, including 36 main trials evenly split between explicit and implicit tasks and 10 filler trials.Reading time was limited to 40 seconds and response time to 15 seconds; participants were recruited via Prolific.
- Model evaluation: The study evaluated Pythia models from 70M to 12B parameters and OLMo 2 models from 1B to 32B, using 50 trials per model condition and matched human–model comparisons.Models were compared descriptively across systematic variation in model size, instruction tuning, and template type.
4 Results
Humans tracked entities above chance but were specifically impaired by increasing situational complexity, whereas contemporary 70B-class models maintained near-ceiling performance. Open models showed entity tracking below 1B parameters, improving with scale and remaining robust to lexical manipulations.
- Human performance: Human accuracy was 75.3% on the explicit task and 79.3% on the implicit task, but declined as narrative complexity increased.Explicit-task accuracy fell from 87.2% at C1 to 65.0% at C4; implicit-task accuracy fell from 89.0% to 76.0%.
- Human performance: Controlling for target-object position, complexity remained significant while recency was not a significant predictor.This suggests human performance was limited primarily by the number of entities and relations in the situation model rather than surface-level recency.
- Model comparisons: Both Llama 3.3 and Qwen 2.5 maintained near-ceiling accuracy across complexity levels, unlike humans.The 70B-class models showed no trace of the complexity cost observed in human comprehenders.
- Model scaling: Pythia accuracy increased monotonically from 53.5% at 70M to 89.6% at 12B, while OLMo 2 showed higher-baseline performance with smaller complexity costs.OLMo 2’s complexity costs were 18 points at 1B and 7 points at 13B; OLMo 2 32B underperformed the 13B model as an outlier.
- Instruction tuning: Instruction-tuned OLMo 2 explicit-task accuracy improved from 55.3% at 1B to 93.5% at 32B, whereas instruction tuning did not improve implicit performance.For OLMo 2 1B, implicit accuracy decreased from 88.5% to 78.0% after instruction tuning.
- Lexical controls: Model accuracy stayed within 3 percentage points across standard, pseudoword, and improbable-object templates despite large surprisal differences.Pseudoword narratives were 1017 to 1027 times less likely than standard narratives, and improbable-object narratives were 108 to 1016 times less likely.
5 Discussion
The discussion introduces a controlled, naturalistic paradigm for evaluating entity tracking and argues that it revises influential prior conclusions. In this paradigm, language models reach human-level tracking at much smaller scales than previously reported.
- The study proposes a controlled entity-tracking evaluation using naturalistic narratives to challenge and refine prior conclusions.The paradigm targets entity tracking as it occurs during natural language comprehension.
- The boxes paradigm differs fundamentally from natural-language tracking because it requires explicit step-by-step movement reasoning through abstract locations.Humans may need extended time and external aids such as pen and paper to solve it.
- 410M parameters marks the scale at which human-level entity tracking appears on the implicit task.The implicit task is described as a more sensitive measure of underlying capacities than explicit generation, especially for smaller models.
- Prior claims placed entity tracking at 13B parameters with 500B additional code specialised tokens or 7B parameters with 2T code tokens.The discussion contrasts those requirements with human-level performance at 410M parameters in the naturalistic paradigm.
- Code pretraining may improve non-code abilities, but reported gains typically concern formal domains such as reasoning and semantic parsing.These domains differ substantially from naturalistic entity tracking.
Limitations
The study has three main limitations: it cannot isolate code’s contribution, identify the algorithms underlying model behavior, or fully assess contemporary models and extended reasoning. Its evaluation prioritizes human–model comparability, limiting the complexity and model scales examined.
- Limitations: Pretrained models included code in their data, so the study cannot isolate code’s contribution to observed entity-tracking capacity.Controlled text-only versus code-augmented pretraining ablations are needed.
- Limitations: Output-based evaluation cannot identify the underlying algorithms models use for discourse-based entity tracking.Mechanistic interpretability could test whether structured-task circuits generalize or whether models use different or cheaper heuristics.
- Limitations: Human–model comparability constrained testing to complexity levels where humans performed above chance and smaller, fully open model families.This design supported tracing scaling effects and instruction tuning but limited the evaluation range.
- Limitations: Contemporary Llama 3.3 and Qwen 2.5 models saturated the task at every tested level, preventing evaluation to their breaking point across scales.Future work should also characterize performance when models and humans use more extended reasoning.
A Performance of OLMo 2 Instruct Models on the Explicit Task
OLMo 2 Instruct models show strong scaling on the explicit generation task, with accuracy rising substantially as model size increases. The 32B model remains robust at the highest complexity, whereas smaller models decline sharply.
- Overall performance: Accuracy improves from 55% for the 1B model to 94% for the 32B model overall.These results indicate substantial scaling effects among instruction-tuned models.
- Complexity effects: The 32B model maintains high accuracy even at C5 complexity.
- Complexity effects: Smaller instruction-tuned models show steep performance declines as narrative complexity increases.
B Object Words by Template Condition
The study used five object-word template conditions spanning common, pseudoword, rare, and physically or logically impossible objects. Conditions varied in familiarity, plausibility, and violation type, with sample sizes of N = 12 or N = 4.
- Template conditions: N = 12 conditions used common and possible objects in the standard template or pseudowords derived from those standard objects.These were the two larger template conditions.
- Template conditions: N = 4 conditions used rare but possible objects or rare and physically impossible objects involving a size violation.The rare-object conditions separated plausibility from physical impossibility.
- Template conditions: N = 4 conditions used rare and logically impossible objects involving an abstract violation.This condition tested logical rather than physical impossibility.
C Evaluation Prompt
The explicit evaluation used a common instruction prompt across all models. Models had to identify an object’s final location and answer with only the corresponding single location word.
- Evaluation prompt: The same instruction prompt was used across all models for the explicit evaluation.The prompt introduced instructions for reading each story and question.
- Evaluation prompt: Models had to carefully read the story and question to identify the object’s final location.The required answer used the location’s single word, such as “box” for “the red box” or “bag” for “the bag.”
- Evaluation prompt: Responses had to contain only the single location word, without reasoning, explanations, introductory phrases, sentences, or punctuation.The response format ended with the label “Location:” followed by an empty quoted field.
D Example Narratives · Explicit Task
The explicit task uses naturalistic room-tidying narratives in which objects move among containers, and questions require tracking each queried object's final location. Narratives increase in complexity from one to five while varying the number of tracked objects and intervening events.
- Explicit Task: Complexity 1 asks readers to track a book moved once, with the expected answer being basket.Dora picks up the book and puts it into the basket before the question.
- Explicit Task: Complexity 2 requires tracking a bookmark across two placements, yielding drawer as the expected final location.The bookmark moves from the bag to the drawer before an unrelated window glance.
- Explicit Task: Complexity 3 introduces two objects and distractor state changes while querying either the key in box or the letter in bag.The key moves from the blue basket to the modern box, while the letter moves from the vintage drawer to the bag.
- Explicit Task: Complexity 4 presents three objects and multiple intermediate events, with queried objects ending in box, basket, or basket.The postcard is placed in the box, the coin moves to the colorful basket, and the watch is put into the basket.
- Explicit Task: The examples combine explicit object-location statements with anaphoric and sequential updates, requiring the queried answer to reflect the latest relevant movement.For example, objects are moved between containers, while other entities are introduced or relocated before the final question.
- Explicit Task: Complexity 5 uses four tracked objects and longer chains, querying a pen in bag, a letter in bag, a key in box, or a postcard in box.These narratives include additional object introductions, transfers, and unrelated pauses or phone checks before the question.
- Explicit Task: Focus objects vary from 0 to 3 at Complexity 5, expanding the task beyond single-object tracking.The Complexity 5 examples query focus objects 0, 1, 2, and 3 across separate narratives.
Implicit Task
The implicit task tests entity tracking by distinguishing narrative-consistent from inconsistent object locations across increasing levels of complexity. Complexity varies through additional objects, containers, intervening actions, and distractors while the target object's final location is manipulated.
- Complexity 1: Complexity 1 contrasts a consistent prompt_good with an inconsistent prompt_bad by changing the photo's earlier location from the bag to the drawer.In the consistent version, Dora retrieves the photo from the bag; in the inconsistent version, she retrieves it from the drawer.
- Complexity 2: Complexity 2 introduces an intermediate wallet transfer between the bag and box before the final retrieval.The consistent prompt retrieves the wallet from the box, whereas the inconsistent prompt retrieves it from the bag and places it into the basket.
- Complexity 5: Complexity 5 further increases tracking demands with longer chains of object movements and distractor entities before the target object's final-location test.The prompts track targets such as a scarf, wallet, or watch across several containers, with prompt_bad changing the object's prior location at retrieval.