PaperlayerView exact revision on arXiv

Source-linked AI summary

"Act Like a 5th Grader" is Not Enough: Bounding Knowledge in LLM-Based User Simulators

Krisztian Balog, Arild Michel Bakken

arXiv:2608.30033v1cs.CLcs.AI

TL;DR

LLM simulators often exhibit superhuman bias, failing to reproduce the natural variability and cognitive constraints of young readers. The paper introduces CBUS, an episodic working-memory bottleneck with two test-taking strategies, and finds that explicit cognitive bounds narrow the simulation gap across LLM backbones. The study is scoped to binary recognition and retrieval tasks among Norwegian students in grades 4–6.

  • Problem

    Standard persona prompting produces near-perfect simulations that fail to capture the natural variance and cognitive limitations of developing readers.

  • Method

    CBUS forces LLMs to operate through limited-capacity episodic memory bottlenecks and models two distinct test-taking strategies.

  • Results

    Explicitly modeling cognitive boundaries narrows the simulation gap across a diverse set of LLM backbones, improving student-centric and item-level alignment.

  • Takeaways & Limitations

    High-fidelity reading simulation benefits from architectural cognitive constraints rather than superficial persona assignment alone.

  • Takeaways & Limitations

    The evidence primarily concerns binary recognition and retrieval tasks among Norwegian 4th- to 6th-grade students, so universality across languages, systems, stages, and generative comprehension remains unvalidated.

Abstract

from arXiv · show

Large language models (LLMs) are increasingly used to simulate human behavior but frequently fail to exhibit realistic cognitive constraints, suffering from a "superhuman bias." Using a dataset of over 71,000 reading comprehension responses from 2,359 primary-school students (grades 4--6), we demonstrate that standard persona prompting yields near-perfect, deterministic performance, failing to capture the natural variance of developing readers. To address this, we introduce the Cognitively Bounded User Simulator (CBUS), an architectural framework that explicitly models the restricted working memory of young readers through an episodic bottleneck. Within this framework, we formalize two distinct test-taking strategies to emulate different reading behaviors. Our evaluation shows that explicitly modeling cognitive bounds significantly narrows the simulation gap across multiple LLM backbones, demonstrating that enforcing architectural constraints is more effective for high-fidelity simulation than simply scaling raw model capabilities.

1 Introduction

LLM user simulators can display a “superhuman bias,” producing unrealistic performance despite persona prompts. The paper proposes CBUS, which imposes cognitive bottlenecks and evaluates simulation with specialized metrics against real student responses.

  • Reading comprehension provides a controlled simulation testbed because each student’s knowledge and cognitive load are bounded by a source text and developmental stage.
  • Over 71,000 responses from 2,359 students in grades 4–6 show that prompting models to act like fifth-graders overestimates student capabilities and misses developing readers’ challenges.
  • Standard persona prompting can produce a “superhuman bias,” with LLMs displaying knowledge and reasoning beyond their intended student personas.
  • CBUS architecturally models restricted working memory through a parameterized, two-stage episodic bottleneck and two test-taking strategies.
  • The paper introduces evaluation metrics that assess both student-centric and item-centric alignment rather than aggregate accuracy alone.
  • Explicit cognitive bounds reduce the calibration gap between simulated and real student responses across diverse LLM backbones.

2 Related work

Related work increasingly treats LLM simulators as scalable proxies for people but finds that simple role prompting does not adequately constrain model behavior. This paper differs by targeting population-level reading distributions with a working-memory bottleneck and no interaction data or misconception taxonomy.

  • LLM simulators support evaluation, training, and social-science applications, but merely prompting them to act human produces poor adherence to knowledge constraints and instructions.
  • Recent approaches add explicit personas, internal states, behavioral traits, evolving knowledge, information needs, and non-collaborative behavior.
  • Educational simulation seeks high-fidelity predictions of how different student profiles interact with content, partly to reduce reliance on costly human calibration.
  • Prior learner simulators model fine-grained behavior or generate tutoring data, whereas this work focuses on population-level score and per-item difficulty distributions.
  • Unlike related work, the proposed mechanism requires no interaction data, knowledge-component tagging, or misconception taxonomy and models young learners’ working-memory limits.

3 Reading Comprehension Dataset

The dataset contains Norwegian reading-comprehension materials and responses from primary-school students, standardized into binary outcomes for controlled evaluation. Classroom-level splitting creates separate ground-truth and held-out distributions for measuring simulation fidelity.

  • Texts and Questions: The corpus contains Norwegian texts, comprehension questions, and responses from 2,359 primary-school students in grades 4–6.
  • Texts and Questions: The 156 texts use multiple-choice, true-or-false, checkbox, and free-text questions, with free-text responses excluded to yield 750 objectively gradable questions.
  • Texts and Questions: Questions are categorized as Locate, Interpret & Connect, or Reflect & Evaluate using an LLM-classifier ensemble followed by expert review.
  • Responses: Responses are binarized as correct or incorrect; checkbox items receive credit only when the selected subset exactly matches the gold standard.
  • Dataset Statistics & Difficulty: The dataset includes 71,789 responses across 80 classes, with checkbox questions more difficult and Interpret & Connect questions slightly less successful overall.
  • Data Splitting: Classroom-level splitting prevents leakage from shared educational environments or teacher effects and defines ground-truth and held-out evaluation distributions.

4 Pilot Study: Evaluating Baseline Simulators

The pilot evaluates zero-shot persona prompting for simulating fifth-grade reading comprehension and finds a stable superhuman bias: near-perfect performance that misses real students’ variance.

  • Pilot Experimental Setup: Baseline zero-shot persona simulators are evaluated across flagship commercial and open-weight LLMs using a standardized persona template.The protocol uses Norwegian student personas, English structural instructions, temperature T = 0.7, and n = 3 runs.
  • Pilot Experimental Setup: Mean Accuracy measures correct-answer rates, while JSD compares simulated and real per-student score distributions.A held-out human split provides an empirical upper bound for simulation fidelity.
  • Results and Key Findings: 0.687 human mean accuracy contrasts with baseline simulators’ deterministic near-perfect performance, whose JSD values are far worse than the human reference of 0.006.The baselines fail to reproduce the natural spread of student scores.
  • Results and Key Findings: 0.378 mean accuracy and 0.518 JSD make random guessing distributionally closer to humans than flagship LLMs, whose JSD ranges from 0.634–0.773.This comparison motivates explicit modeling of cognitive limitations rather than merely reducing accuracy.
  • Results and Key Findings: Increasing model capability pushes persona-prompted mean accuracy toward 0.98 and increases divergence from the human baseline.Norwegian-language prompting also remains superhuman, ruling out instruction language as the explanation.

5 A Cognitively Bounded User Simulator

CBUS replaces superficial role-play with a capacity-limited episodic bottleneck that separates information encoding from answer execution. It uses SPR and TS to model distinct reading strategies while forcing the central executive to reason only over retained propositions.

  • Motivation: Standard persona prompting produces a superhuman bias because the LLM retains unbounded global text access and performs global synthesis.CBUS instead treats the LLM as a central executive mediated by parameterized memory modules.
  • Cognitive Model: The architecture draws on the episodic buffer and central executive: one stores integrated propositions, while the other reasons over them.The buffer supports cross-sentence inference by keeping earlier propositions available during reading.
  • Cognitive Model: CBUS simplifies working memory to a capacity-limited store, where overload forces shallow, literal interpretation when required propositions exceed capacity.The approach follows Cowan’s account of a small number of chunks, approximately four in adults.
  • Capacity Parameter: C defines the maximum number of distinct text propositions the simulated student can actively maintain at once.The parameter operationalizes the working-memory bottleneck computationally.
  • Encoding Strategies: SPR extracts the C most salient propositions from the source text once, retaining a fixed memory trace for all later questions.This models holistic single-pass reading.
  • Encoding Strategies: TS extracts at most C propositions relevant to each specific question, modeling a scan-and-search strategy.The paper sets C = 4 for SPR and C = 2 for TS from independent cognitive principles.
  • Computational Implementation: Stage 1 fills the buffer without answering, then Stage 2 purges the source text and answers using only the retained propositions.With C = 2, an inference requiring three facts must be attempted from incomplete information.

6 Experimental Setup

The experiments compare CBUS with a zero-shot persona baseline across diverse backbones using metrics that assess both population-level and question-level fidelity. Simulated populations mirror the real students’ question exposure.

  • Experimental Design: CBUS is evaluated against the standard zero-shot persona simulator across diverse backbone LLMs.The comparison is designed to isolate explicit cognitive modeling from underlying model capability.
  • Metrics: Student-centric metrics assess population performance and variance, while item-centric metrics assess alignment on specific question difficulties.This separates macro-level simulation fidelity from question-level fidelity.
  • Metrics: Absolute Performance Gap, JSD, and ECE measure population-level alignment between human and simulated students.They respectively compare mean accuracy, score-distribution shape, and success rates across difficulty bins.
  • Metrics: Item-Level Difficulty Correlation uses Pearson (ρ) and Spearman’s rank (r_s) correlations between human and simulator question difficulties.Question difficulty is defined as mean success rate across all subjects.
  • Inference Protocol: Six open-weight and proprietary backbones are tested zero-shot at temperature T = 0.7, with results averaged across runs.The protocol aims to prevent in-context memorization and emulate population-level response variance.
  • Population Construction: Each simulated population mirrors 1,198 real subjects answering the same question sets, but student-centric metrics compare score distributions rather than paired individuals.This construction matches human and simulated populations on exposure by design.

7 Results and Analysis

CBUS consistently narrows the simulation gap across backbones, but its strategies trade off global population fidelity against question-level difficulty alignment. Architectural constraints matter more than raw backbone capability for realistic simulation.

  • Overall Results: C = 4 for SPR and C = 2 for TS are fixed from independent cognitive principles rather than tuned on human response data.Sensitivity analysis places both values at or near the empirical optimum.
  • Closing the Simulation Gap: Across all backbones and strategies, CBUS improves every student-centric metric over baseline and brings simulated score distributions closer to real behavior.The result directly mitigates the superhuman bias through explicit cognitive boundaries.
  • Trade-offs in Test-Taking Strategies: SPR outperforms TS on all student-centric metrics, producing the most realistic macro-level score distributions.The two strategies therefore differ in the type of fidelity they capture.
  • Trade-offs in Test-Taking Strategies: TS improves Pearson and Spearman item-difficulty correlations over baseline for every backbone and achieves the highest Pearson correlation across variants.SPR also improves item-centric correlations in most cases, making the strategies complementary.
  • Model Capability vs. Simulation Fidelity: Gemini-3.5-Flash-Lite and Llama-3.3-70B achieve SPR absolute gaps of 0.031 and 0.037, respectively, while TS leads item-centric correlations by backbone.The strongest results come from smaller or open-weight backbones rather than the most capable proprietary models.
  • Diagnostic Baselines and Ablation: Random dropout performs on par with CBUS-SPR across metrics, indicating that the capacity bound is more important than the salience heuristic.Prompting and answer-noising fail to match CBUS’s distributional and item-level fidelity even when aggregate accuracy is adjusted.
  • Impact Across Question Categories: CBUS strategy effectiveness varies by question category, with superhuman bias most pronounced on retrieval-oriented Locate questions.Baselines answer these questions almost uniformly correctly and therefore miss the natural spread of student scores.

8 Conclusion

The study finds that explicit cognitive bounds narrow the simulation gap for primary-school reading comprehension across diverse LLM backbones, while strategy choice and model scale create alignment trade-offs.

  • CBUS narrows the simulation gap across diverse LLM backbones, improving both student-centric and item-centric metrics.The framework evaluates explicit memory bottlenecks and two test-taking strategies against a large dataset of real student responses.
  • Different test-taking strategies produce distinct alignment trade-offs that require further investigation.
  • Smaller, less capable models achieve the highest simulation fidelity rather than the most capable frontier LLMs.This result suggests that realistic simulation depends more on explicit cognitive constraints than on raw model capability.

Limitations

The limitations concern incomplete cognitive modeling, idealized information retention, rigid reading strategies, and a restricted evaluation scope.

  • CBUS models only working-memory capacity, omitting noncognitive influences such as anxiety, fatigue, motivation, and attention drift.The simulator therefore lacks mechanisms for performance degradation caused by apathy or physical fatigue.
  • CBUS limits retained information but idealizes which propositions are selected and how accurately they are encoded.Real children may retain irrelevant details, miss the gist, or misremember, whereas CBUS stores accurate paraphrases selected by a capable LLM.
  • CBUS treats Single-Pass Reading and Targeted Scanning as rigid, mutually exclusive strategies rather than dynamic human behaviors.Future work should examine strategy switching and probabilistic capacity allocation.
  • The evaluation primarily covers recognition and retrieval tasks using binarized responses from Norwegian students in grades 4–6.Free-text generation was excluded, and the universality of the cognitive bounds across languages and educational populations remains outside the stated scope.
  • The dataset uses a text-based platform in which students choose between two randomly paired reading texts before answering comprehension questions.This design was intended to encourage engagement, but the binary choice does not guarantee genuine interest.
  • Questions requiring visual inspection were excluded to keep the text-based evaluation environment fair.The filtering protocol retained questions answerable from the reading material and associated image alt text alone.

G Full Pilot Results with Run Variance

Pilot baselines were evaluated across three runs with complete metric reporting, and their small run-to-run variance indicates that persona prompting’s superhuman bias is stable rather than sampling-driven.

  • Table 6 reports the complete pilot-baseline metric set as mean ± standard deviation over n = 3 runs.The main pilot table reports only Mean Accuracy and JSD for readability.
  • The study maps each evaluated LLM to its specific model snapshot tag queried through the OpenRouter API.
  • Small run-to-run variance across all backbones confirms that persona prompting’s superhuman bias is a stable property rather than a sampling artifact.

H Diagnostic Baselines and Ablation

Diagnostic comparisons show that matching human mean accuracy does not reproduce realistic item difficulty, while the capacity bound—not proposition selection—is the primary driver of CBUS-SPR’s fidelity gains. Results also vary by backbone and question category, with CBUS improving different alignment dimensions in different settings.

  • Diagnostic baselines: Abs. Gap 0.001 on both backbones does not prevent answer-noising’s item-difficulty correlation from nearly vanishing, with Spearman rs of 0.065 and 0.090.Uniformly random flips match the real-student mean but carry no information about which questions are difficult.
  • Diagnostic baselines: The Calibrated low-persona baseline is backbone-dependent: Llama-3.3-70B remains near baseline, whereas Gemini-3.5-Flash-Lite improves but still trails cognitively bounded simulators.Llama reports Abs. Gap 0.277 versus 0.279; Gemini complies with the 65–70% accuracy instruction.
  • Ablation: Random dropout performs on par with CBUS-SPR, indicating that the number of retained propositions matters more than which propositions are selected.The ablation holds the capacity bound fixed while replacing salience-based selection with a larger random proposition pool.
  • Category-specific results: Baseline JSD is 0.64–0.70 for Locate versus ∼0.28 for Interpret & Reflect, because superhuman accuracy erases the natural spread on easier Locate items.CBUS narrows both category gaps, with SPR reducing Locate JSD to ∼0.08.
  • Category-specific results: Pearson up to 0.556 and 0.594 occurs for Interpret & Reflect under TS, while the largest student-centric gains occur on Locate under SPR.CBUS therefore improves student-centric and item-centric alignment along different category-specific axes.

J Capacity Sensitivity Analysis

A capacity sweep shows a trade-off: lower capacity better matches student-level score distributions, while higher capacity improves item-level difficulty discrimination. Thus, no single capacity jointly optimizes both metric families.

  • Capacity sweep: C ∈ {2, 4, 6, 8, 10, 12} is swept for SPR and C ∈ {1, 2, 3, 4, 5} for TS against the Ground Truth split.Each capacity setting uses a single run because main experiments showed small run-to-run variance.
  • SPR: For SPR, student-centric error is lowest at low capacity and grows monotonically as C increases.The student-centric measures are absolute gap and Score JSD.
  • SPR: For SPR, item-centric difficulty correlation is flat-to-increasing with C, even as the score distribution drifts farther from real students.Retaining more propositions improves item-level discrimination while reducing student-level distributional fidelity.
  • Operating point: No single capacity is jointly optimal because student-centric and item-centric metrics respond to C in opposite directions.Figure 15 marks cognitively motivated operating points at SPR C = 4 and TS C = 2.
Loading 2608.30033v1…