PaperlayerView exact revision on arXiv

Source-linked AI summary

KnowSim: Evaluating Information Calibration in LLM Assistants with User Simulators that Learn

Yoonjoo Lee, Hyoungwook Jin, Tae Soo Kim, Shaoyang Zhang, Philippe Laban, Q. Vera Liao

arXiv:2608.17150v1cs.AIcs.CLcs.HC

TL;DR

Existing user simulators do not explicitly model users’ knowledge states or how understanding evolves, limiting calibration evaluation across knowledge levels. KNOWSIM addresses this with a knowledge-state-aware simulator and benchmark, finding 73–74% agreement with human rankings and that the best LLM shifts by user knowledge level.

  • Problem

    Current user simulators lack explicit knowledge-state modeling and knowledge-evolution tracking needed to evaluate information calibration across user knowledge levels.

  • Method

    KNOWSIM models user understanding as a prerequisite graph of Information Units whose states evolve during interaction, yielding Knowledge Gain, Delivery Calibration, and Cognitive Overload metrics.

  • Results

    73–74% sign agreement with human rankings was achieved across 705 conversations, while evaluation of 9 LLMs showed the best model shifted by user knowledge level.

  • Takeaways & Limitations

    KNOWSIM exposes knowledge-level-dependent assistant rankings and aptitude–treatment interactions that aggregate leaderboards do not reveal.

  • Takeaways & Limitations

    The simulator underspecifies motivational and affective dynamics, which manifests as knowledge-gain saturation for advanced simulated users.

Abstract

from arXiv · show

To effectively collaborate with users on knowledge-intensive tasks, Large Language Models (LLMs) must perform information calibration: matching content to a user's evolving understanding and cognitive capacity. Yet user simulators used to evaluate and train LLMs do not explicitly model user knowledge so they neither produce realistic interactions across knowledge levels nor reflect how interactions unfold as that knowledge evolves. To close this gap, we introduce KNOWSIM, an evaluation framework built around a user simulator that maintains explicit knowledge states, represented as a graph of Information Units with prerequisite relationships, that evolve under update rules grounded in learning theory. KNOWSIM computes three metrics (Knowledge Gain, Delivery Calibration, Cognitive Overload) directly from the knowledge state trajectory, reflecting key mechanistic aspects of information calibration. We validate KNOWSIM against 705 human-AI sessions across two domains, stratified by knowledge level: its rankings align significantly with human judgments (73-74% sign agreement), outperforming three baseline simulators. Applied to 9 LLMs, KNOWSIM reveals that the best model shifts by user knowledge level, revealing aptitude-treatment interactions invisible to standard evaluation.

1 Introduction

KNOWSIM evaluates information calibration by simulating users whose explicit knowledge states evolve as they interact with LLM assistants. Its mechanistic metrics and human validation show that assistant performance depends on users’ knowledge levels.

  • Motivation: Current LLM-based user simulators lack knowledge-state modeling, limiting evaluation to aggregate utility rather than calibration across users with different knowledge levels.They also do not track how user understanding evolves across interaction turns.
  • Framework: KNOWSIM represents user understanding as Information Units connected by prerequisite relationships and updates the knowledge state as the assistant explains concepts.Absorption is moderated by whether the user can understand each unit and by cognitive load.
  • Framework: Knowledge Gain, Delivery Calibration, and Cognitive Overload are computed directly from evolving knowledge states to explain why an assistant helps or hinders a user.These metrics reflect key aspects of information calibration and are grounded in learning theory.
  • Validation: 705 human–assistant sessions across math problem solving and expert-level question answering validated KNOWSIM across novice, intermediate, and advanced users.The benchmark varied information delivery strategy and base model, with subjective ratings and, for MathQA, pre/post knowledge tests.
  • Validation: 73–74% sign agreement with human rankings showed that KNOWSIM’s condition rankings aligned with human judgments and outperformed three baseline simulators.The alignment was statistically significant (p=.003) and strongest at the novice level.
  • Frontier model benchmarking: Applied to 9 LLMs, KNOWSIM found that the best model shifts by user level: DeepSeek V4 maximizes novice knowledge gain, while Gemini 3.1 Pro best serves advanced users.This reveals performance differences across user knowledge levels that standard evaluation can miss.

2 Knowledge-Grounded Simulator for Information Calibration Evaluation

KNOWSIM models information calibration through an evolving user knowledge state over a prerequisite graph of Information Units. It updates that state from knowledge-consistent interactions and learning-theoretic constraints, then computes interpretable calibration metrics from the resulting trajectory.

  • Knowledge representation: The simulator represents topic understanding as a directed acyclic graph of Information Units linked by prerequisite relationships, with each user state labeling units as unaware, struggling, partial, or knows_well.The initial state is parameterized by user knowledge level, and updates reflect provided information and the user’s ability to absorb or learn it.
  • State evolution: User messages are generated consistently with the current knowledge state, while each turn updates that state through extracted teaching and engagement signals followed by deterministic learning-theoretic rules.The simulator prevents spontaneous mention of unaware concepts, models realistic errors and correct application, and uses teaching quality plus cognitive constraints to determine transitions.
  • Cognitive constraints: Three constraints limit absorption: prerequisite mastery sets an attainable-state ceiling, cognitive overload weights less-known information and reasoning attempts more heavily, and progress-based conditions terminate stalled or overloaded conversations.Termination occurs when target Information Units are mastered or when persistent nonprogress or overload is observed.
  • Calibration metrics: The framework computes Knowledge Gain, Delivery Calibration, and Cognitive Overload deterministically from knowledge-state trajectories and per-turn classifications.Knowledge Gain sums ordinal advancement; Delivery Calibration is the harmonic mean of precision over explained IUs and recall over teachable IUs; Cognitive Overload is average per-turn load relative to capacity, saturating at one.

3 KNOWCHAT: A Benchmark of Knowledge-Stratified Conversations with Learning Experience Annotations

KNOWCHAT benchmarks knowledge-stratified human–AI conversations across tasks, delivery strategies, and assistant models, combining objective learning outcomes with subjective learning-experience annotations. Its protocol stratifies participants by domain knowledge and controls ordering and item effects across repeated sessions.

  • Dataset and study design: The dataset collects real user–AI conversations to validate the simulator, covering objective learning outcomes and subjective perceptions across knowledge levels and delivery strategies.The conversations involve assistants using different information-delivery strategies and models.
  • Dataset and study design: Four study arms cross comparison target—strategy versus model—with task, using math and ExpertQA and separate participant pools.Strategy arms fix the base model while model arms hold the strategy fixed and vary base models.
  • Tasks: 15 competition-mathematics problems at difficulty level 5 span five topic categories, while ExpertQA lets participants select an interested professional domain.Math problems require distinct Information Unit graphs within categories; ExpertQA prioritizes professional-domain coverage.
  • Recruitment and protocol: Each arm recruits ∼60 participants stratified as novice, intermediate, or advanced using a 10-item domain-specific prescreening test.This stratification enables comparisons between simulator and human outcomes within each knowledge level.
  • Recruitment and protocol: Sessions compare three strategies or three models per participant, rotate math condition–problem pairings, and annotate conversations with ratings for Delivery Calibration, Cognitive Overload, and Interaction Quality.Math additionally uses pre- and post-tests to measure Knowledge Gain, KG = (Post − Pre)/(Maxpost −Pre); ExpertQA omits these tests.

4 Experiment Setup

The experiment evaluates KNOWSIM by comparing its human-outcome alignment and baseline performance across knowledge levels and conditions. It uses shared configurations, cell-level aggregation, common IQ scoring, and KNOWSIM-specific state-trajectory metrics.

  • Baselines: KNOWSIM is compared with three increasingly sophisticated Gemini-3-Flash baselines: Zero-shot, ZS-CoT, and ZS-CoT-Prof.ZS-CoT adds per-turn chain-of-thought reasoning, while ZS-CoT-Prof additionally uses a synthetic profile and initial knowledge state without updating it.
  • Evaluation protocol: Each simulator runs on every (q, ℓ, c) KNOWCHAT configuration, with conversations capped at 15 turns.KNOWSIM samples its initial knowledge state from the question-specific IU graph by knowledge-level ratios; its ending turn follows its termination rule, whereas baselines use an LLM judge.
  • Evaluation protocol: Human and simulator outcomes are compared at the (ℓ, c) cell-mean level to test population-level condition effects within knowledge strata.Human means aggregate participants assigned to each cell, while simulator means aggregate simulated conversations at the same configuration.
  • Metrics: All four simulators receive Interaction Quality (IQ) scores from a shared LLM rater, enabling direct cross-method comparison.KNOWSIM additionally computes Knowledge Gain (KG), Delivery Calibration (DC), and Cognitive Overload (CO) deterministically from its knowledge-state trajectory; baselines lack these metrics.
  • Alignment analysis: Alignment is assessed by pairwise sign agreement on condition contrasts with human Cliff’s δ ≥0.15, tested against 50% chance using a one-sided binomial test.The analysis reports metric-aggregated and metric-specific agreement across arms, tasks, and knowledge levels; human DC, CO, and IQ are self-ratings, while KG uses pre/post tests for MathQA.

5 Results

KNOWSIM’s rankings significantly align with human judgments across tasks and knowledge levels, with especially strong novice-level agreement. It also outperforms baseline simulators on IQ alignment and better matches real users’ information-unit engagement.

  • Human alignment: 73% and 74% sign agreement on MathQA and ExpertQA, respectively, significantly aligned KNOWSIM’s rankings with human judgments across both domains.Pooled across Strategy and Model arms, MathQA reached 27/37 (p=.003) and ExpertQA 26/35 (p=.003).
  • Human alignment: 100% novice-level agreement on MathQA and 83% on ExpertQA made novices the strongest-aligned knowledge level.Intermediate and advanced levels were more variable across tasks.
  • Metric analysis: IQ and CO were the most consistent diagnostic metrics, reaching 75% agreement on MathQA and 80% and 79% on ExpertQA, respectively.DC was lowest in both tasks: 67% on MathQA and 64% on ExpertQA.
  • Baseline comparison: 77% IQ agreement made KNOWSIM the top simulator, ahead of ZS-CoT-Prof at 73%, ZS at 68%, and ZS-CoT at 64%.Pooling all four arms, KNOWSIM led or tied the strongest baseline in every arm.
  • Baseline comparison: On MathQA, KNOWSIM’s simulated users had greater full-turn Jaccard overlap with real users’ engaged information units than any baseline.This result extends beyond metric-level agreement to the information units users engage with during sessions.

6 Comparing Assistants with KNOWSIM

KNOWSIM evaluates nine LLM assistants across simulated users with three knowledge levels, showing that overall rankings conceal knowledge-level tradeoffs and distinct calibration failures. Per-level metrics and IU tracking reveal which assistants fit different populations and why interactions fail.

  • Evaluation setup: 9 LLMs were evaluated on 30 MathQA and 30 ExpertQA items with simulated users at three knowledge levels per item.The evaluation reports KNOWSIM’s internal metrics—KG, DC, and CO—alongside IQ.
  • Overall ranking: Claude Opus 4.7 leads overall (¯R=2.75), while Gemini 3.1 Pro ranks second (¯R=3.50) with the highest IQ (8.24) and lowest CO (0.833).DeepSeek V4 ranks first on KG but eighth on CO (0.890), indicating a knowledge-dumping pattern.
  • Overall ranking: Gemini 3.1 Flash ranks fourth overall with strong KG (7.1, rank 2) but the highest CO (0.895), reflecting an overload pattern among high-throughput models.Its throughput advantage is therefore accompanied by greater cognitive overload.
  • Aptitude–treatment patterns across knowledge levels: No single model serves all knowledge levels equally well: DeepSeek V4 maximizes novice KG (15.5), whereas Gemini 3.1 Pro minimizes novice CO (0.875) and has the highest novice IQ (8.39).For advanced users, the relative picture inverts, with Gemini 3.1 Pro identified as best-calibrated.
  • Aptitude–treatment patterns across knowledge levels: Gemini 3.1 Pro ranks sixth on novice KG while DeepSeek V4 overwhelms novices (CO=0.945), exposing tradeoffs hidden by level-agnostic leaderboards.KNOWSIM’s per-level decomposition supports selecting assistants for target user populations.
  • Failure mechanisms: 65% of explained novice IUs are prerequisite-blocked or overload-wasted, compared with 89% dominated by redundant re-explanation in advanced sessions.IU-level tracking decomposes these directional calibration failures across assistants.

7 Related Work

Prior work evaluates multi-turn assistants, builds LLM-based user simulators, and models students for educational AI. However, these approaches span fidelity-scalability tradeoffs and diverse simulation or knowledge-modeling strategies.

  • Multi-turn LLM evaluation: Multi-turn LLM evaluation faces challenges because each assistant response shapes subsequent user turns and affects evaluation outcomes.Static benchmarks cannot react to assistant replies, whereas real-user studies have limited scalability.
  • LLM-based user simulators: LLM-based user simulators augment generation through external knowledge, self-play, dialogue-data fine-tuning, reinforcement learning, implicit profiles, and search-oriented agent architectures.These simulators are often used as evaluation infrastructure.
  • Student simulation and knowledge state modeling: Student simulation research models cognitive levels, learning dynamics, student errors, and simulator quality for educational AI applications.Quality is audited with human teachers or automated metrics.

8 Conclusion

The paper presents KNOWSIM, a knowledge-state-aware user simulator, alongside KNOWCHAT, a benchmark of 705 knowledge-stratified human–AI conversations. KNOWSIM aligns with human judgment and reveals that the best model shifts by user knowledge level across 9 frontier LLMs.

  • Contributions: KNOWSIM models how understanding evolves across multi-turn conversations through an internal knowledge representation.Its design is explicitly knowledge-state-aware.
  • Contributions: 705 knowledge-stratified human–AI conversations across two task domains comprise the KNOWCHAT benchmark, with learning outcome annotations.KNOWCHAT accompanies KNOWSIM as an evaluation benchmark.
  • Findings: 73–74% sign agreement with human judgment and outperformance of baseline simulators establish KNOWSIM’s evaluation performance.The reported alignment is measured against human judgment.
  • Findings: Across 9 frontier LLMs, KNOWSIM reveals aptitude–treatment interactions invisible to aggregate leaderboards, with the best model shifting by user knowledge level.The model ranking depends on the user’s knowledge level.

Limitations

KNOWSIM captures information-processing constraints and human-aligned rankings but leaves motivational dynamics, individual knowledge-state estimation, broader domains, and training applications for future work. Its multi-call implementation also limits evaluation throughput, motivating distillation of LLM-based modules.

  • Simulator dynamics: The simulator models cognitive load and prerequisite-gated absorption but underspecifies frustration and boredom, leading to knowledge-gain saturation for advanced simulated users.It nonetheless captures the ATI crossover in Top-1 strategy ranking across novice, intermediate, and advanced learners.
  • Evaluation scope: Evaluation covers only two domains—math problem-solving and ExpertQA—leaving open-ended settings such as creative writing and exploratory data analysis for future work.The framework’s extension to domains with more open-ended success criteria is ongoing work.
  • User-state estimation: Although tuning the KS ratio expresses within-group variation, estimating an individual’s initial knowledge state from behavior remains challenging in open-ended conversations.Bridging group-level cognitive abstractions with individual state estimation is identified as a future research direction.
  • Implementation: Multiple LLM calls per turn constrain throughput when evaluating many assistant models, while deterministic structured components are inexpensive.Distilling only the LLM-based modules into smaller fine-tuned simulators could reduce cost without sacrificing alignment.
  • Future applications: The simulator has been validated as an evaluation proxy but not yet applied as a training signal, such as a reward model for RL fine-tuning.Using it for assistant training is left as a natural future extension.

Ethical Considerations … A.7 IU Graph Example

The ethical discussion cautions that KNOWSIM is an evaluation proxy whose rankings may disadvantage less-knowledgeable users, while the appendices specify simulator construction, prompting, state updates, and supporting examples. The framework uses explicit IU graphs, knowledge-conditioned interaction rules, and code-based state advancement to model learning trajectories.

  • Ethical Considerations: KNOWSIM should not substitute for human studies because simulation-derived rankings may deploy assistants that underserve users, especially where metric alignment is weaker.The released system is an evaluation proxy rather than a replacement for human evaluation.
  • Ethical Considerations: Aggregate scores may systematically disadvantage less-knowledgeable users when novice knowledge gain coincides with high cognitive overload or advanced-user calibration underserves novices.The paper identifies this as an equity dimension in model selection.
  • A User Simulator Details: The appendix documents IU graph construction, initial knowledge states, user-message generation, signal extraction, state updates, conversation termination, and a worked IU graph example.These subsections mirror the simulator description in §2.
  • A.1 IU Graph Construction: An IU graph is a directed acyclic representation of the understanding needed to comprehend an answer, with independently assessable units linked by prerequisite relationships.IUs are designed as self-contained explanations of 2-4 sentences.
  • A.2 Initial Knowledge State: Initial knowledge states are instantiated from per-level ratios, assigning knows_well labels first to shallow IUs, followed by partial_understanding and struggling in topological order.Matched validation comparisons initialize each simulated user at the corresponding participant’s knowledge level.
  • A.3 User Message Generation: User messages are conditioned on connectable unknowns and use articulation thresholds based on neighborhood density: d ≥0.6 is Explicit, 0.3 ≤d < 0.6 is Vague, and d < 0.3 is Deferential.Separate prompts generate initial requests and follow-up messages.
  • A.4 Signal Extraction: Signal extraction classifies each IU’s teaching quality and user engagement using the current state, latest exchange, and analyst-only reference answer.Teaching quality distinguishes well_explained, shallow, and not_mentioned, while user attempts are reasoning, articulation, or none.
  • A.5 State Update Rules: State updates enforce a prerequisite ceiling, compute cognitive overload from state-weighted load, and reset per-IU carry-forward understanding on state transitions.The ceiling depends on pre-turn prerequisite states processed in topological order, and articulation uses WA = 0.35.

B Details in Data Collection … D Analysis of KNOWCHAT

The paper operationalizes KNOWCHAT through tutor strategies, human-study materials and ratings, and baseline user simulators with varying degrees of reasoning and state tracking. These design choices support analysis of how assistants adapt explanations, while contrasting KNOWSIM with simpler simulators that lack evolving knowledge updates.

  • [Model Arms] Simple Strategy: The Simple Strategy frames tutoring as helping students understand concepts and reasoning rather than simply supplying answers, with explanations adapted to apparent understanding.Its guidelines also prohibit giving the complete answer initially and encourage clear language and concrete examples.
  • [Strategy Arm] Socratic Strategy: The Socratic Strategy uses one reasoning-level question, a short conceptual hint with follow-up, or minimal clarification per turn, while forbidding factual-recall prompts and tutorial-shaped dumps.Hard rules limit replies to one reasoning move and at most 2–3 short sentences, with full answers and enumerations disallowed.
  • [Strategy Arm] Adaptive Strategy: The Adaptive Strategy reassesses the user at every turn and scales each teaching unit to novice, intermediate, or advanced capacity.Novices receive one concept, intermediates one logical step, and advanced users one sticking-point resolution, with limited multi-unit responses allowed only for advanced users.
  • [Strategy Arm] Comprehensive Strategy: The Comprehensive Strategy provides expansive, organized answers covering prerequisites, connections, reasoning, and relevant background, while requiring new ground rather than repetition on later turns.Replies may contain at most 10 sentences and should confirm or correct the user before expanding to related concepts.
  • B.2 Study Material Creation: Study questions covered remember, understand, and apply levels of Bloom’s taxonomy and were piloted with 20 participants to capture varying domain knowledge.The questions targeted knowledge components associated with three domain-specific chatbot problems.
  • B.3 Subject Rating Questions: Human ratings measured Delivery Calibration, Perceived Overload, and Interaction Quality using 10-point response scales with anchored descriptions.Calibration ranged from completely mismatched to perfectly matched, overload from no to severe overload, and interaction quality from very poor to excellent.
  • C.1 Baseline Simulator Details: The three baselines and KNOWSIM all used gemini-3-flash-preview, but only KNOWSIM explicitly models evolving understanding; the baselines rely on implicit role-play, independent reasoning, or static profiles.ZS-CoT regenerates reasoning each turn without external state, while ZS-CoT-Prof specifies initial knowledge and behavior but does not update them.

D.1 Per-Level Dataset Statistics … E.3.2 Baseline IQ Spearman ρ

The analyses show that knowledge-level groups are well separated, subjective ratings can diverge from objective learning, and KNOWSIM generally best matches human interaction patterns and IQ judgments. Metric alignment varies by task, with MathQA favoring KG/IQ and ExpertQA favoring CO.

  • D.1 Per-Level Dataset Statistics: MathQA DC and IQ rise monotonically with knowledge level, while Strategy conversations shorten from 13.1 to 6.4 turns; KG remains low and slightly negative for advanced Model users.The KG pattern reflects a pre-test ceiling that leaves little room for measured gain.
  • D.2 Validating Knowledge-Level Stratification: All four arms show highly significant knowledge-level separation (p<.001), with large effect sizes (ε2 = 0.89–0.97).Groups are defined by objective prescreening scores ranging from 0–10.
  • D.3 Divergence of Subjective and Objective Measures: IQ negatively correlates with KG (ρ=−0.27, p=.003), whereas higher CO accompanies lower gain at the advanced level (ρ=−0.30, p<.05).The IQ pattern is already present among novice and intermediate users, with ρ=−0.21 and −0.36, respectively.
  • E.1 Basic Conversation Statistics: On MathQA, KNOWSIM produces 9.1-turn conversations versus 8.6 for humans, while ZS and ZS-CoT terminate earliest at 3.1 turns.On ExpertQA, ZS-CoT-Prof and KNOWSIM are longest at 7.4 and 7.3 turns, exceeding the human length of 5.4.
  • E.2 Per-Arm IQ Sign Agreement: KNOWSIM leads or ties ZS-CoT-Prof in every arm, with the clearest advantage on MathQA-Model (83% vs. 67%).The ExpertQA Strategy and Model arms are tied at 86% and 67%, respectively, but contain few signal pairs.
  • E.3.1 Per-Metric Spearman ρ: On MathQA, KG and IQ show the strongest alignment at ρ = 0.45, p=.061 and ρ = 0.44, p=.069, while ExpertQA is dominated by CO at ρ = 0.49, p=.039.DC is weakest on ExpertQA, with ρ = 0.08, compared with IQ at ρ = 0.32.
  • E.3.2 Baseline IQ Spearman ρ: On MathQA, KNOWSIM reaches ρ=0.44 (p=.069), ahead of ZS (0.30), ZS-CoT-Prof (0.25), and ZS-CoT (0.16).On ExpertQA, all four methods fall within 0.29–0.38, with none reaching significance.

E.4 Concept Coverage Analysis … F.2 Overall Ranking

KNOWSIM’s simulated users overlap with human users’ concepts most strongly when assistant turns are included, despite systematically covering more information units than humans. The benchmark evaluates nine LLMs with KNOWSIM’s calibration metrics and interaction quality under level-marginalized ranking.

  • E.4 Concept Coverage Analysis: The concept-coverage analysis tests whether simulated and human users engage with the same MathQA concepts at each turn.It matches 202 human conversations to simulated counterparts by problem, strategy, and knowledge level.
  • E.4 Concept Coverage Analysis: Human and simulated conversations are tagged with the same GPT-4.1-nano IU tagger, prompt, and per-problem IU graph.The analysis compares user-only IU overlap with full-turn overlap, including assistant messages.
  • E.4 Concept Coverage Analysis: 0.614 full-turn Jaccard is highest for KNOWSIM, significantly exceeding ZS-CoT and ZS-CoT-Prof, while user-only Jaccard is 0.479 versus 0.487 for ZS and 0.482 for ZS-CoT.All three user-only results significantly outperform ZS-CoT-Prof at 0.410 (Wilcoxon p<.001).
  • E.4 Concept Coverage Analysis: Simulated users engage with 71–78% of graph IUs versus 48–56% for humans, because prerequisites expose newly connectable concepts throughout sessions.This over-coverage enlarges |H ∪S| without proportionally increasing |H ∩S|, lowering absolute Jaccard values.
  • F.1 Models and Configuration: Nine LLMs are benchmarked using default decoding and non-thinking mode through hosted APIs, including GPT, Gemini, Claude, and open-weight models.The setup disables reasoning features to isolate conversational calibration from chain-of-thought depth.
  • F Assistant Benchmarking Details: The benchmark reports KG, DC, CO, and LLM-judged Interaction Quality under the same rater πr.These are the internal KNOWSIM metrics plus the external interaction-quality measure.
  • F.2 Overall Ranking: Each metric is averaged over three knowledge levels and two tasks with equal contribution from each level.Conversations are first averaged within levels, then across the three level means, preventing unequal cell counts from reweighting levels.
  • F.2 Overall Ranking: Mean Rank averages the four per-metric ranks, with lower values preferred for the overall ordering.CO is lower-is-better because it measures cognitive overload.

G Failure Mode Decomposition by Knowledge Level

KNOWSIM’s IU-level tracking decomposes calibration failures into five mutually exclusive categories and shows that failure patterns differ sharply by user knowledge level. Novices primarily experience over-reaching, advanced users over-repetition, and intermediate users a more balanced profile with under-absorption emerging as a bottleneck.

  • Failure Mode Taxonomy: KNOWSIM classifies each explained Information Unit into exactly one of five mutually exclusive categories using an ordered decision tree.The categories are REDUNDANT, PREREQ-BLOCKED, EFFECTIVE, OVERLOAD-WASTED, and UNDER-ABSORBED.
  • Per-level distribution: 65% of novice explained IUs are wasted through over-reaching: PREREQ-BLOCKED 37% plus OVERLOAD-WASTED 28%.Only 11% of novice IUs result in effective knowledge advancement.
  • Per-level distribution: 89% of advanced explained IUs are REDUNDANT, indicating over-repetition, while only 3% result in effective knowledge advancement.REDUNDANT means the user already knows the information well, so re-explanation adds no value.
  • Per-level distribution: At the intermediate level, REDUNDANT dominates at 68%, while UNDER-ABSORBED rises to 10%.This profile suggests that depth of teaching becomes the bottleneck when prerequisites are largely met.
Loading 2608.17150v1…