PaperlayerView exact revision on arXiv

Source-linked AI summary

Modeling Student Sensemaking with LLMs and Knowledge-Graph-Guided Inference

Özge Alacam, Zübeyde Demet Kirbulut Güneş, Funda Ekici, Nurcan Turan-Oluk, Dilay Dinçdemir, Hakkı Kadayıfçı, Sevinç Nihal Yeşiloğlu, Burcu Işık, Halil Tümay, Sinem Gencer

arXiv:2609.31046v1cs.CL

TL;DR

Expert analysis of collaborative sensemaking is labor-intensive and depends on interpreting dialogue beyond surface cues. The paper evaluates two mid-size instruction-tuned LLMs with varied prompting and knowledge-graph-derived diagnostics; reasoning and knowledge-state context improve selected outcomes, but no configuration leads across every dimension.

  • Problem

    Expert analysis of collaborative sensemaking is labor-intensive because labels depend on utterance function, explanation structure, and whether learners resolve a knowledge gap.

    Source: §1 · p. 1

  • Method

    The study evaluates two mid-size instruction-tuned LLMs across prompting conditions and augments dialogue with knowledge-graph-derived knowledge-state diagnostics.

    Source: §4 · p. 4

  • Results

    Reasoning-enabled prompting improved unsuccessful-sensemaking identification, and knowledge-state diagnostics with an appropriate reasoning configuration yielded the highest human-model alignment (κ = .60), while performance remained uneven across dimensions.

    Source: §6 · p. 8

  • Takeaways & Limitations

    The findings suggest that sensemaking prompting may need to be tailored to specific dimensions rather than rely on one universally best configuration.

    Source: §6 · p. 8

    Evidence is bounded to 23 episodes from one chemistry topic and two mid-sized open-weight models, and should not be treated as broad statistical generalization or representative of other model sizes or proprietary models.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Collaborative science learning requires nuanced interpretation of student dialogue to characterize how learners identify knowledge gaps, build explanations, and work toward resolution - a theory-driven analysis that is labor-intensive and difficult to scale. We investigate whether instruction-tuned large language models (LLMs) can support multidimensional analysis of collaborative sensemaking without task-specific training, and whether structured knowledge-state information improves model inference. We evaluate two mid-size LLMs on 23 richly annotated, expert-labeled episodes across prompting conditions that vary definitional scaffolding, reasoning mode, and turn structure. Without reasoning, models tend to overpredict successful sensemaking; reasoning-enabled prompting improves identification of unsuccessful cases. Knowledge-state diagnostics provide additional grounding, improving detection of unsuccessful sensemaking and increasing agreement with expert annotations. No single configuration performs best across all sensemaking dimensions, underscoring the multidimensional nature of the task.

1 Introduction

Analyzing collaborative sensemaking requires experts to interpret how dialogue functions in explanation building and whether learners resolve their knowledge gaps, making reliable automation difficult. This study evaluates instruction-tuned LLMs across prompting conditions and introduces a framework for examining both sensemaking outcomes and how learner knowledge is organized around a focal gap.

1 Introduction

  • Expert analysis must identify episode outcomes, knowledge gaps, epistemic emotions, and explanation building; these labels depend on utterance function and gap resolution, not surface cues.The required interpretation makes manual analysis labor-intensive and difficult to automate reliably.Source: §1 · p. 1
  • Whether instruction-tuned LLMs can reliably perform multidimensional, theory-driven sensemaking annotation—and how prompting choices affect performance—remains unclear.The motivating questions concern definitional scaffolding, reasoning-oriented prompting, and enriched representations of student knowledge.Source: §1 · p. 1
  • The study evaluates two mid-size instruction-tuned LLMs on expert-annotated Boyle’s law dialogues across conditions varying definitions, reasoning mode, turn structure, and KG-derived diagnostics.The framework tests whether knowledge-state information improves inference beyond dialogue alone.Source: §1 · p. 1
  • The framework examines both whether sensemaking succeeds and how learners’ expressed knowledge is organized relative to the focal gap driving the episode.The study also evaluates model configurations on a theory-driven annotation scheme, including the effect of chain-of-thought prompting.Source: §1 · pp. 1–2

2 Related Work

The paper frames sensemaking as iterative explanation building around a knowledge gap, with success determined by whether learners resolve that gap. It situates LLMs as promising dialogue-based tools while noting the challenges of modeling temporal, collaborative, and multidimensional learning processes.

2 Related Work

  • Sensemaking involves iteratively constructing or revising explanations to resolve knowledge gaps; its stages run from organizing prior knowledge to resolving the focal gap.Source: §2 · p. 2
  • Knowledge gaps can be conceptual, procedural, or epistemic, while explanation building can be represented through claim-evidence-rationale structures.Source: §2 · p. 2
  • A successful episode resolves the focal gap from learners’ perspective, whereas unsuccessful sensemaking engages with it without coherent resolution (Kirbulut Gunes et al., 2026).Scientific correctness is not required for success; computational analysis must assess whether ideas form a gap-resolving explanation.Source: §2 · p. 2
  • Epistemic emotions provide complementary signals about learners’ experience of uncertainty and progress, but are treated as indicators rather than defining criteria of sensemaking (Pekrun, 2021; D’Mello and Graesser, 2012; Kirbulut Gunes et al., 2026).Emotion changes across dialogue may accompany productive engagement or persistent difficulty.Source: §2 · p. 2
  • LLMs can infer learner knowledge from dialogue, but prompt-only approaches remain prone to hallucination and fine-tuning may not resolve temporal-coherence limits (Hooshyar et al., 2025).This work extends learner modeling toward collaborative processes by tracing knowledge, sensemaking features, and emotions as they co-evolve across dialogue.Source: §2 · pp. 2–3

3 Corpus

The corpus consists of expert-annotated collaborative dialogue from a first-year undergraduate chemistry activity. Its 23 lengthy episodes provide richly labeled material for evaluating models on a theoretically grounded, multidimensional construct.

3 Corpus

  • The researchers report using a corpus and describe the material in that corpus as collaborative dialogues.Source: §3 · p. 3
  • The 23 episodes are lengthy multi-turn discussions averaging 6,500 words and 45 turns, covering experimental design, observations, data interpretation, and explanation building.Source: §3 · p. 3
  • Domain experts annotated episodes for sensemaking variations and steps, focal gaps and their types, vexing questions, and claim-evidence-rationale structures.The annotation effort is substantially greater than for standard sequence-labeling or sentiment tasks.Source: §3 · p. 3

4 Sensemaking Modeling with LLMs

The modeling framework first analyzes raw dialogue, then augments inference with knowledge-state diagnostics derived from student knowledge graphs. It evaluates multiple dimensions and prompting configurations, with expert validation emphasizing whether knowledge becomes organized to resolve the focal gap.

4 Sensemaking Modeling with LLMs

  • The two-stage framework first analyzes dialogue directly, then adds KG-derived knowledge-state diagnostics to test whether explicit conceptual coverage and relational correctness improve inference.Source: §4 · pp. 3–4

4.1 Sensemaking Analysis From Raw Dialogue

  • Gemma-3-27B-IT and Qwen3-32B are evaluated using standard zero-shot or chain-of-thought inference and four prompt variants with incrementally added definitions.The variants scaffold epistemic emotions and knowledge-gap types while retaining the shared sensemaking framework.Source: §4.1 · p. 4
  • Models return episode-level sensemaking labels, focal knowledge gaps and types, claim-evidence-rationale components, and two prominent emotions from a predefined set.Outputs are compared with human annotations to assess scaffolding, reasoning, and cross-model effects.Source: §4.1 · p. 4

4.2 Integrating Knowledge-State Diagnostics into LLM Inference

  • For each episode, LLMs map dialogue onto a controlled domain vocabulary, extract expressed entities and relations, and flag or discard hallucinated entities and invalid predicates.Source: §4.2 · p. 5
  • Knowledge-state diagnostics compare student graphs with domain and focal-question graphs using node and edge recall, cosine similarity, missing concepts, contradictory relations, and structural similarity.The diagnostics and expert-curated interpretation guidelines are provided to the model as context.Source: §4.2 · p. 5
  • Expert validation found that successful episodes organized concepts into a relevant, gap-resolving structure, while unsuccessful episodes lacked such integration through fragmentation, contradiction, or irrelevance.The distinction concerns organization and gap resolution, not concept count or correctness alone.Source: §4.2 · p. 5
  • In representative episodes, SM5 resolved its procedural gap as relevant resources became organized, while USM4 cycled around an unresolved Boyle’s law inconsistency and ended in confusion and abandonment.Expert validation found the derived graph diagnostics consistent with episode outcomes and raw dialogue content.Source: §4.2 · pp. 5–6

5 Results

The results compare prompting configurations on raw-dialogue sensemaking classification, then test KG diagnostics and feature extraction. KG diagnostics helped most with single-turn thinking for unsuccessful-case detection, while no configuration performed best across all sensemaking features.

5.1 Base models with Raw Dialogs

  • Thinking mode benefited both models, especially on unsuccessful episodes, but definitional scaffolding had mixed effects and Qwen3’s emotion definitions hurt its thinking-prompt performance.The three hardest episodes were unsuccessful and contained partial explanations that models treated as resolved.Source: §5.1 · p. 6, Table 2 · p. 6
  • Knowledge-gap prediction was largely unsuccessful, with models defaulting to “conceptual”; thinking with gap definitions raised procedural-gap accuracy to approximately 30%, while Qwen3 achieved κ = 0.51 overall agreement with human annotations.Source: §5.1 · p. 6
  • Two-turn prompting improved identification of unsuccessful episodes but hurt successful-case classification, with no average gain in human-label alignment.Two-turn prompting uniquely solved two unsuccessful cases, while thinking uniquely solved another missed by the other configurations.Source: §5.1 · p. 6, Table 3 · p. 7

5.2 Effect of Knowledge Diagnostics

  • Single-turn thinking with KG diagnostics achieved the highest human-annotation agreement (κ = .60), substantially improving unsuccessful-sensemaking detection while sensemaking performance remained stable.Without thinking, KG diagnostics slightly degraded classification; with two-turn thinking, they hurt performance substantially.Source: §5.2 · p. 7, Table 3 · p. 7
  • No prompting configuration uniformly improved alignment across focal questions and claim-evidence-rationale components; rationale similarity was consistently lowest.No-thinking KG performed well for claim and rationale, two-turn for evidence, and thinking with KG for focal questions.Source: §5.2 · p. 7, Figure 2 · p. 7
  • Emotion predictions varied with prompting: two-turn outputs omitted or nulled emotion fields, while unsuccessful episodes were more associated with confusion and frustration than successful episodes.Confusion appeared in both outcomes; the distinction was more evident in its emotional context, including greater frustration and reduced contentment for unsuccessful episodes.Source: §5.2 · p. 8

6 Conclusion

Reasoning-enabled prompting improved identification of unsuccessful sensemaking, while KG-state diagnostics paired with suitable reasoning achieved the highest human-model alignment. Performance remained uneven across dimensions, supporting expert-in-the-loop use rather than standalone annotation.

6 Conclusion

  • KG-state diagnostics combined with an appropriate reasoning configuration yielded the highest human-model alignment, κ = .60.Reasoning-enabled prompting improved unsuccessful sensemaking identification, but no single configuration performed best across all dimensions.Source: §6 · p. 8
  • Performance remained uneven across knowledge gap types, CER components, and epistemic emotions, while two-turn prompting had mixed effects across dimensions.The authors suggest ensemble or adaptive prompting tailored to specific dimensions as a possible direction.Source: §6 · p. 8
  • LLM-based sensemaking analysis is not yet a standalone annotation tool; the authors frame it as support for expert interpretation of theoretically grounded features.The study also emphasizes representing how student concepts relate to the focal knowledge gap, not only which concepts students mention.Source: §6 · p. 8

Limitations

The study evaluates a small corpus from one chemistry topic and restricts experiments to two mid-sized, open-weight LLMs. Its findings are not intended as broad statistical generalizations or as representative of other model scales and types.

Limitations

  • The 23-episode corpus covers a single chemistry topic, so replication on larger, more topically diverse corpora remains a future need.The authors frame the corpus as suited to evaluating whether LLMs can operationalize a complex theoretical construct, not to establishing broad statistical generalization.Source: Limitations · p. 9
  • Experiments cover only two mid-sized, open-weight LLMs, so results should not be treated as representative of smaller, larger, or proprietary models.The study prioritizes examining interactions between prompting configurations, knowledge-state diagnostics, and multidimensional sensemaking components rather than conducting a broad model benchmark.Source: Limitations · p. 9

Ethical Section

The analyses use anonymized dialogue and expert annotations from an existing corpus collected with ethics approval and participant consent.

Ethical Section

  • The study uses anonymized student dialogue and expert annotations, collected with institutional ethics approval and informed participant consent.Data use follows the original project’s scope and applicable institutional ethical requirements; no personally identifiable information is included.Source: Ethical Section · p. 9

A.1 Sample Expert Annotation

Figure 3 presents excerpts from a successful episode to illustrate sensemaking features as they unfold over time.

A.1 Sample Expert Annotation

  • Figure 3 tracks a successful episode across Steps 1–3, showing the focal knowledge gap alongside recurring VQs, CER components, and complementary emotional cues.The figure illustrates how sensemaking features emerge and develop over time.Source: Figure 3 · p. 10, §A.1 · p. 10

A.2 Computational Environment and Runtime Details

The appendix specifies the models’ runtime and prompting procedures, the sensemaking classification framework, and the distribution of predicted epistemic emotions. It also describes prompt variants and tables for comparing sensemaking predictions.

A.2 Computational Environment and Runtime Details

  • Both models ran in 4-bit NF4 on one A100; Gemma’s reasoning was prompt-elicited, while Qwen3 used its native thinking flag.Qwen3 received separate system and user messages; Gemma’s content was concatenated into one user turn.Source: §A.2 · pp. 10–11

A.3 Two-turn Prompt Design

  • In two-turn prompting, the model first analyzes the episode in plain text, then converts its prior reasoning into the requested JSON schema.Separating analysis from formatting is intended to keep schema demands from collapsing inference into surface-level pattern matching.Source: §A.3 · p. 11

A.4 Epistemic Emotion Distributions

  • Curiosity, confusion, contentment, and frustration dominated Qwen3 predictions; enjoyment and anxiety were rare, while surprise and boredom were absent.The distributions cover successful and unsuccessful episodes across Qwen3 configurations without knowledge-diagnostic integration.Source: §A.4 · p. 11, Figure 4 · p. 12

A.6 Fine-grained Breakdown for Prompt variation and knowledge gap type classification

  • Sensemaking is successful when students identify a gap, attempt to explain it, and reach a shared resolution; unsuccessful cases lack effective explanation or resolution.Step 1 is identifying an inconsistency; Step 2 is actively evaluating explanations; Step 3 is reaching a shared explanation that resolves the gap.Source: §A.6 · p. 17
  • Explanation quality is assessed through a clear claim, relevant evidence, and a rationale connecting the evidence to the claim.The rationale may invoke relevant scientific principles, mechanisms, or theories.Source: §A.6 · p. 17
  • The prompt variants add definitions for epistemic emotions and knowledge-gap types to the base sensemaking framework, and the instructions classify gap type as none, conceptual, procedural, or epistemic.Table 4 reports baseline performance across prompt variations; Table 5 is titled as Qwen3-32B sensemaking-variation prediction.Source: §A.6 · pp. 12, 17, Table 4 · p. 12, Table 5 · p. 12

Related papers

Loading 2609.31046v1…