PaperlayerView exact revision on arXiv

Source-linked AI summary

IntrEx: A Dataset for Modeling Engagement in Educational Conversations

Xingwei Tan, Mahathi Parvatham, Chiara Gambi, Gabriele Pergola

arXiv:2509.06652v2cs.CL

TL;DR

Research has examined interestingness in educational texts, but how linguistic features shape engagement in educational conversations remains less understood. IntrEx adds sequence-level learner ratings of interestingness and expected interestingness to teacher–student dialogues and evaluates LLM prediction. Fine-tuned small models outperform GPT-4o on predicting human interest ratings.

  • Problem

    How linguistic properties shape engagement in educational conversations remains an open question, while whole-document assessments offer limited insight into interest across exchanges.

    Source: §1 · p. 1

  • Method

    IntrEx adds sequence-level interestingness and expected-interest annotations to teacher–student conversations using over 100 second-language learners and a comparison-based rating approach.

    Source: §1 · p. 2

  • Results

    Fine-tuned 7B–8B models outperform GPT-4o in predicting interestingness in educational dialogues.

    Source: §1 · p. 2

  • Takeaways & Limitations

    IntrEx’s labels offer a resource for reward-modeling research and efforts to improve conversational models for second-language learning.

    Source: §5 · p. 9

    Interestingness remains subjective, with agreement limited by annotators’ differing interests and background knowledge, so results should be interpreted accordingly.

    Source: Limitations · p. 10

Abstract

from arXiv · show

Engagement and motivation are crucial for second-language acquisition, yet maintaining learner interest in educational conversations remains a challenge. While prior research has explored what makes educational texts interesting, still little is known about the linguistic features that drive engagement in conversations. To address this gap, we introduce IntrEx, the first large dataset annotated for interestingness and expected interestingness in teacher-student interactions. Built upon the Teacher-Student Chatroom Corpus (TSCC), IntrEx extends prior work by incorporating sequence-level annotations, allowing for the study of engagement beyond isolated turns to capture how interest evolves over extended dialogues. We employ a rigorous annotation process with over 100 second-language learners, using a comparison-based rating approach inspired by reinforcement learning from human feedback (RLHF) to improve agreement. We investigate whether large language models (LLMs) can predict human interestingness judgments. We find that LLMs (7B/8B parameters) fine-tuned on interestingness ratings outperform larger proprietary models like GPT-4o, demonstrating the potential for specialised datasets to model engagement in educational settings. Finally, we analyze how linguistic and cognitive factors, such as concreteness, comprehensibility (readability), and uptake, influence engagement in educational dialogues.

1 Introduction

Research on educational texts has identified interesting features, but how linguistic properties shape engagement in conversations—and how interest evolves across exchanges—remains understudied. IntrEx adds sequence-level interestingness and expected-interest labels to teacher–student interactions and uses them to study learner judgments and LLM prediction.

1 Introduction

  • Whole-document assessments offer limited insight into how interest develops across exchanges, where thematic continuity, discourse structure, and interaction flow shape engagement.Existing work has examined interestingness in texts, including linguistic features such as concreteness.Source: §1 · p. 1
  • IntrEx introduces sequence-level annotations of interestingness and expected interestingness across extended interactions, based on annotations from over 100 second-language learners.Annotators compare original utterances with automatically rewritten “boring” versions rather than rating turns in isolation.Source: §1 · p. 2
  • Fine-tuned 7B–8B models outperform GPT-4o in predicting interestingness in educational dialogues.The experiments examine how IntrEx data can align LLMs with human interest judgments.Source: §1 · p. 2
  • IntrEx is presented as the first dataset annotated for interestingness and expected interestingness in educational conversations for second-language learners.The dataset includes individual annotator ratings, demographic information, and aggregated scores.Source: §1 · p. 2
  • The paper analyzes concreteness, comprehensibility, and uptake as factors associated with engagement in teacher–student dialogues.It uses a range of metrics and linear mixed-effect regressions to examine these factors.Source: §1 · p. 2

2 Related Work

Prior conversational corpora cover a range of settings, including educational interaction, while psychological and educational research relates interestingness to content, comprehensibility, and complexity. The reviewed corpora lack interestingness annotations, motivating IntrEx’s focus.

2.1 Conversational Corpora

  • The reviewed conversational corpora include tutoring and educational datasets, but none provides annotations for interestingness.TSCC includes annotations for sequence types, pedagogical focus, and grammatical-error correction.Source: §2.1 · pp. 2–3

2.2 Interestingness

  • Research relates interest to tangible, personalized, contextualized content and suggests comprehensibility may be most engaging within an optimal range.The Goldilocks effect and information-gap theory describe interest as lower for stimuli that are too simple or too complex.Source: §2.2 · p. 3

3 The IntrEx Dataset

IntrEx extends TSCC V2 with turn- and sequence-level ratings of interestingness and expected interestingness for teacher–student English-learning conversations. The annotation design prioritizes sequence-level judgments to capture engagement across multi-turn discourse and improve reliability.

3.1 Background: The TSSC V2 Corpus

  • TSCC V2 contains 260 one-to-one English-learning conversations involving 2 teachers and 12 students, with sequence labels marking shifts in conversational flow and teaching focus.Lessons took place in online private chatrooms and averaged 52 dialogue snippets per conversation, ranging from 26 to 90.Source: §3.1 · p. 3

3.2 Task Definition

  • Interestingness rates current attention or curiosity, while expected interestingness predicts the annotator’s interest after the next message, before that message is seen.Both scores are integers from 0 to 4.Source: §3.2 · p. 3
  • Sequence-level annotators rated teacher messages within TSCC-labeled sequences, reducing workload while representing engagement across multi-turn lesson segments.Turn-level annotators instead rated the other party’s responses; turn-level ratings were often unchanged across consecutive messages and were more fatiguing and variable.Source: §3.2 · p. 4, §3.3 · p. 5

3.3 Data Collection and Validation

  • Second-language learners were recruited as annotators, and comparison-based ratings paired each message with a GPT-4o-generated rewrite that removed engaging features while preserving content.Calibration conversations were included in annotation tasks, and the study used checks for repeated identical scores and a reward system tied to sequence-level agreement.Source: §3.3 · pp. 4–6
  • Sequence-level annotations achieved significantly higher agreement than turn-level annotations for INT (p = 3.19e−8), EXP INT (p = 6.82e−5), and combined scores (p = 5.12e−7).Table 1 reports dataset statistics and average agreement for both annotation levels.Source: Table 1 · p. 5, §3.3 · p. 6

3.4 Dataset Statistics and Demographics

  • Sequence-level annotations cover 259 conversations and 5,801 sequences, while turn-level annotations cover 64 conversations and contain 7,118 score pairs.Sequence-level ratings average 22.4 sequences per conversation, ranging from 8 to 41.Source: §3.4 · p. 6
  • Most annotators reported B2 or C1 English proficiency, and the authors note that this high-proficiency skew may bias ratings of instruction as overly simple.Annotators whose proficiency matched the original learner’s were associated with higher ratings; the authors also note Prolific’s Europe and North America user base favors moderate-to-advanced English proficiency.Source: §3.4 · p. 6

4 Experiments

The experiments assess whether LLMs align with human judgments of conversational interestingness and examine how linguistic features relate to engagement. They compare model predictions and analyze concreteness, comprehensibility, and uptake in teacher–student interactions.

4.1 Reward Modeling with IntrEx

  • After IntrEx fine-tuning, Llama3-8B-Instruct and Mistral-7B-Instruct surpassed GPT-4, GPT-4o, and Mixtral on human-interest prediction.The comparison is reported in Table 3.Source: Table 3 · p. 7, §4.1 · p. 7
  • Instruction-tuned models aligned better with human interestingness ratings, while base models scored below the random baseline even after fine-tuning.The authors suggest base models’ lack of pre-training exposure to explicit rating tasks may hinder instruction following in rating contexts.Source: §4.1 · p. 7

4.2 Linguistic Predictors of Human Interest

  • Concreteness negatively correlated with interestingness consistently at turn and sequence levels.Source: §4.2 · p. 9
  • For comprehensibility, predictors were retained in a linear mixed-effects model only when robust across models of interestingness and expected interestingness.Source: §4.2 · p. 9
  • Several readability metrics positively related to interest at both levels, but some sequence-level metrics reversed; turn-level length, GIS, and Gunning fog showed inverted-U relations.The authors interpret the turn-level pattern as suggesting that both very simple and very complex messages may lower interest relative to messages that challenge readers “just enough”.Source: §4.2 · p. 9
  • Uptake-interest correlations varied by measure and level: LCS and student-uptake-teacher were positive at turn level, propTinS at sequence level, while cosine similarity was negative at both.The authors suggest lower similarity may reflect the introduction of novel ideas, and that student uptake of the teacher may index different phenomena at turn and sequence levels.Source: §4.2 · p. 9

5 Conclusion

IntrEx adds interestingness and expected-interestingness annotations to teacher–student dialogues, including sequence-level labels that track engagement across lessons. Small instruction-tuned LLMs fine-tuned on the dataset surpassed GPT-4 and GPT-4o in predicting human interest ratings.

  • IntrEx is the first dataset to annotate both interestingness and expected interestingness for teacher–student dialogues, extending TSCC with sequence-level labels.The sequence labels track how engagement evolves across a lesson.Source: §5 · p. 9
  • Small instruction-tuned LLMs fine-tuned on IntrEx surpassed GPT-4 and GPT-4o in predicting human interest ratings.Source: §5 · p. 9
  • Future work can use IntrEx to train generation reward models and extend the approach to new learner groups and domains.Source: §5 · p. 9

Limitations

The paper’s findings are bounded by annotator proficiency, the subjectivity of interestingness, the dataset’s domain, and the scope of model evaluation.

Limitations

  • Most annotators had at least B2 English proficiency, so lower-proficiency learners’ engagement perspectives may be underrepresented.Capturing those perspectives would require task simplification and dedicated data collection.Source: Limitations · p. 10
  • Individual interests and background knowledge limit agreement, and the authors say subjectivity cannot—and probably should not—be eliminated.They recommend interpreting results accordingly.Source: Limitations · p. 10
  • IntrEx covers teacher–student interactions in English-as-a-second-language learning, so findings may not fully generalize to peer conversations, other subjects, or multilingual settings.The authors identify broader conversational contexts as future work.Source: Limitations · p. 10
  • LLM evaluation predicts ratings on human-authored conversations; it does not assess whether fine-tuned models generate more engaging conversations.Generation quality and direct preference optimization would require new human evaluation and fall outside this dataset release’s scope.Source: Limitations · p. 10

Ethics Statement

The paper anonymized annotations for public release and describes its model fine-tuning experiments, including unsuccessful and stronger training settings.

Ethics Statement

  • Annotations were anonymized before public release, and demographic data were not linked to individual participants.Participants had a withdrawal procedure and were compensated fairly; the study received institutional review board approval.Source: Ethics Statement · p. 10

A Fine-tuning LLMs on Human Interest Annotations

  • Regression training sometimes produced scores outside the 0–4 range, while adding rating instructions yielded inferior models compared with training without them.The instructions asked models to rate message interestingness using the whole-conversation context.Source: A Fine-tuning LLMs on Human Interest… · p. 14
  • Instruction-tuned Llama3 and Mistral models had higher agreement than their base-model counterparts.Both models were trained for 3 epochs with a learning rate of 5e−6.Source: A Fine-tuning LLMs on Human Interest… · p. 14

B Boring Alternative

The appendices show a teacher–student example and provide prompts for generating less engaging alternatives and rating conversational interest.

B Boring Alternative

  • The boring alternative replaces the teacher’s conversational feedback with a more formal request to elaborate and wait.The student’s response remains the same in the two examples.Source: B Boring Alternative · p. 14

C Prompts

  • The alternative-generation instruction asks for a straightforward, less engaging version that strips colorful language or extra context while preserving the main information..Source: C Prompts · p. 14
  • The instruction-tuned rating prompt asks for 0–4 ratings of interest in the teacher’s reply and expected interest in the next exchange, with a reason and prior context considered.Source: C Prompts · p. 14
  • The base-model rating prompt frames the task as assigning a 0–4 interestingness score to the student’s message.Source: C Prompts · p. 14

D Metric Implementation

Metric implementation uses separate tools for readability, GIS, and conversational-uptake estimates.

D Metric Implementation

E Dataset Information

The dataset is intended for research use and follows Prolific’s access conditions and ethics approval guidelines. It is anonymized and documented to support transparency and reproducibility.

E Dataset Information

  • The dataset is for research purposes and complies with Prolific’s original access conditions and ethics approval guidelines.Source: E Dataset Information · p. 14
  • All personally identifiable information was removed, and participants were informed about anonymization and data protection.Source: E Dataset Information · p. 14
  • Documentation covers domains, languages, linguistic phenomena, and demographics to support transparency and reproducibility.Source: E Dataset Information · p. 14

F Annotation Interface

The annotation interface section describes recruitment pages on Prolific and shows the sequence-level task interface. Figures 3 and 4 show recruitment pages; Figure 5 shows a sequence-level annotation page.

F Annotation Interface

Loading 2509.06652v2…