PaperlayerView exact revision on arXiv

Source-linked AI summary

From Granular Revision Operations to Meaningful Revision Units: Evaluating LLMs for Revision Boundary Detection

Yu Tian, Andrew Potter, Katerina Christhilf, Motahareh Darvishpour Ahandani, Jessica Early, Steve Graham, Danielle S. McNamara

arXiv:2609.33720v1cs.CLcs.AI

TL;DR

Granular edit operations can split a purposeful revision into multiple analytic units, so this study tests whether LLMs can identify meaningful revision boundaries beyond simple baselines. Fine-tuned Qwen3-32B exceeded the proximity heuristic, while prompted models did not; the study also notes that its comparisons omit learned non-LLM classifiers and assess boundaries rather than resulting units.

  • Problem

    Granular operations may divide one purposeful revision into multiple units, while grouping them appropriately requires determining which adjacent edits serve a shared purpose.

    Source: §1 · p. 2

  • Method

    The study frames MRU identification as boundary classification and compares prompted and fine-tuned Qwen3-32B and GPT-5.5 with majority and proximity baselines.

    Source: §1 · pp. 2–3

  • Results

    Macro-F1 was .859 for fine-tuned Qwen3-32B using the ±2 representation versus .825 for the proximity heuristic; prompted LLMs did not outperform the heuristic.

    Source: §5 · p. 14

  • Takeaways & Limitations

    Task-specific adaptation can add value beyond transparent structural heuristics, whereas general-purpose prompting alone may not outperform them.

    Source: §6 · p. 16

    The study did not compare LLMs with learned non-LLM classifiers, and boundary-level evaluation does not fully measure the quality of resulting multi-operation units.

    Source: §5.1 · p. 16

Abstract

from arXiv · show

Revision traces provide valuable evidence about students' writing processes, but their usefulness for learning analytics depends on how individual revisions are represented. Automated draft-comparison methods often produce granular edit operations that can fragment a single purposeful revision into multiple analytic units. This study evaluates whether LLMs can identify meaningful revision unit boundaries in structured revision operation data and whether they provide value beyond simple non-LLM baselines. Using 113 matched draft--revision pairs from undergraduate writing, expert annotation yielded 4,344 candidate boundaries. We compared zero- and few-shot GPT-5.5 and base Qwen3-32B, parameter-efficient fine-tuning of Qwen3-32B, and majority and proximity-based baselines. Despite receiving revision context and task instructions, no prompted LLM condition outperformed the proximity heuristic (macro-F1 = .825). In contrast, fine-tuned Qwen3-32B using the two context representation achieved the highest macro-F1 (.859), identifying more same-unit relationships while maintaining precision comparable to the heuristic. Deterministic post-processing substantially improved the prompted models but added little benefit to the strongest fine-tuned model. These findings suggest that LLMs can support revision boundary judgment when task-adapted, but general purpose prompting alone may not outperform transparent structural heuristics.

1 Introduction

Granular edit operations may fragment purposeful revisions, complicating learning-analytics measures; the study asks whether context-aware LLMs can group operations into meaningful revision units and outperform simpler baselines.

1 Introduction

  • Revision traces preserve decisions about ideas, reasoning, organization, wording, and form, making them potential evidence for studying writing behavior, feedback uptake, and development.The introduction situates this question within computational approaches that extract operations and classify their intentions, purposes, or quality.Source: §1 · p. 2
  • Fragmented operations can inflate revision counts, distort purpose distributions, and assign conflicting labels; fixed sentence- or paragraph-level units can also combine distinct purposes.The challenge is to group adjacent operations contextually rather than impose uniformly finer or coarser units.Source: §1 · p. 2
  • A meaningful revision unit (MRU) comprises adjacent operations that together serve a single revision purpose.The study frames identifying these units as boundary classification: retain boundaries between units and remove boundaries between operations in the same unit.Source: §1 · pp. 2–3
  • The evaluation compares majority and proximity baselines with prompted GPT-5.5 and Qwen3-32B and fine-tuned Qwen3-32B, varying context windows and post-processing.The study also examines zero- versus few-shot prompting and tests a deterministic rule involving preserve operations.Source: §1 · p. 3
  • The study contributes an MRU boundary task, a human-annotated benchmark, and a common comparison of structural baselines with LLM approaches.The comparison covers prompting, parameter-efficient fine-tuning, contextual information, and deterministic post-processing.Source: §1 · p. 3

2 Related Work

Prior edit-merging work offers proximity rules, supervised classification, and linguistically informed rules, chiefly for grammatical correction; student revision units extend beyond correcting errors to shared communicative or rhetorical purposes.

2 Related Work

  • Swanson and Yamangil (2012) found that merging consecutive nonmatching operations with no unchanged word between them outperformed treating edits independently or merging across larger distances.Their proximity heuristics targeted teacher corrections to ESL essays.Source: §2 · p. 4
  • Xue and Hwa (2014) treated correction merging as supervised pairwise classification, using edit and surrounding-context features to improve correction detection over heuristic merging.Pairs of consecutive edits were labeled according to whether they belonged to the same expert-annotated correction.Source: §2 · p. 4
  • Felice et al. (2016) used linguistically enhanced alignment and recursive rules for merging edits, outperforming simpler alternatives; their extraction approach was later incorporated into ERRANT.The rules covered patterns including punctuation, capitalization, possessives, whitespace, and multi-token changes.Source: §2 · pp. 4–5
  • Grammatical correction research groups atomic edits into complete correction units, while student revision may group operations serving a broader communicative or rhetorical purpose.The latter requires judging how operations function together in context, beyond proximity or predefined linguistic patterns.Source: §2 · p. 5
  • Keystroke analytics faces a related measurement challenge: granular actions must be aggregated into revision events that can be interpreted as learner activity.Conijn et al. (2024) reconstructed revision events, while Mouchel et al. (2023) examined revision behavior as an indicator of engagement and self-regulated writing.Source: §2 · p. 5
  • LLM studies of revision intention assume revisions are already delineated; this study instead tests whether adjacent operations should be grouped into the same MRU.The target is an earlier stage of revision analysis than assigning a purpose to a predefined revision.Source: §2 · p. 5

3 Methods

The methods use matched undergraduate draft–revision pairs, TRACE operations, expert MRU boundary annotation, and adjudication to create data for model development and evaluation.

3.1 Data Source

  • The data came from 113 undergraduates at a U.S. public university who completed a narrative essay and revised it about a month later after AI-generated feedback.Of 135 initial participants, 113 completed the revision task and yielded matched draft–revision pairs.Source: §3.1 · p. 5

3.2 TRACE Revision Analysis

  • TRACE represents five operation types—preserve, insert, delete, replace, and move—in source-to-revision sequence order.Across the 113 pairs, TRACE produced 4,457 operation rows; Table 1 presents their distribution.Source: §3.2 · p. 6, Table 1 · p. 6

3.3 Meaningful Revision Unit Annotation

  • MRUs could range from local edits to discourse-level reorganization, and annotators judged how adjacent operations functioned together in context.A preserve operation could occur inside a unit when linking related revisions but could not independently begin or end one; examples included lexical, semantic, and sentence-structuring changes.Source: §3.3 · pp. 6–7, Figure 1 · p. 7
  • Four trained education experts annotated boundaries in TRACE operation sequences, yielding 4,344 candidate decisions from 4,457 operation rows across 113 draft–revision pairs.Each file was independently annotated by two annotators, and an experienced expert adjudicated disagreements to establish gold-standard labels.Source: §3.3 · pp. 6–7
  • Paired annotators agreed on 94.28% of boundaries, with Gwet’s AC1 = .91 and Cohen’s κ = .85 before adjudication.Gwet’s AC1 was selected as the primary chance-corrected statistic because of class imbalance and kappa’s prevalence sensitivity.Source: §3.3 · p. 7

3.4 Boundary Instance Construction and Data Partitioning

Each classification instance centers on a boundary between adjacent TRACE operations, includes sentence context, and varies the neighboring-operation window; draft pairs are partitioned to keep samples separate across sets.

3.4 Boundary Instance Construction and Data Partitioning

  • Each instance predicts whether two consecutive TRACE operations belong to the same MRU or should remain separated by a unit boundary.The focal pair is accompanied by source- and target-sentence contexts.Source: §3.4 · p. 8
  • The ±1 context condition adds up to one preceding and following operation, while ±2 includes up to two on each side of the focal pair.The windows differ in how much neighboring revision-operation context is available to the model.Source: §3.4 · p. 8, Figure 2 · p. 8
  • Draft pairs, rather than individual boundary instances, were assigned to approximately 80/10/10 training, validation, and test splits to prevent sample overlap.The resulting sets contained 3,532 training, 371 validation, and 441 test instances; the test set was reserved for final evaluation.Source: §3.4 · p. 8, Table 2 · p. 9
  • Few-shot demonstrations for GPT-5.5 and base Qwen3-32B came exclusively from the training set.Source: §3.4 · p. 9

3.5 Non-LLM Baselines

The study uses majority and proximity-based baselines to assess how much MRU boundary structure can be recovered without linguistic or semantic modeling. The proximity heuristic relies on operation type and immediate adjacency.

3.5 Non-LLM Baselines

  • The majority baseline predicts same_unit = False for every boundary because False is the majority class.It provides a reference for interpreting accuracy and macro-F1 under substantial class imbalance.Source: §3.5 · p. 9
  • Together, the baselines contextualize LLM performance by contrasting a majority-class prediction with a structural heuristic that uses only operation type and adjacency.The comparison tests how strongly TRACE operation regularities predict MRU boundaries without linguistic or semantic modeling.Source: §3.5 · p. 9
  • The proximity heuristic predicts same-unit when both focal operations are non-preserve revisions, and different-unit when either is preserve.It uses operation type and immediate adjacency, without text, wider context, labeled training data, or model-based reasoning.Source: §3.5 · p. 9

3.6 LLM-Based Boundary Judgement

The study evaluates prompted GPT-5.5 and base Qwen3-32B alongside task-adapted Qwen3-32B for revision boundary classification. Models are evaluated with separate ±1 and ±2 boundary-instance formats.

3.6 LLM-Based Boundary Judgement

  • Three configurations were evaluated: prompted GPT-5.5, base Qwen3-32B without task-specific fine-tuning, and Qwen3-32B adapted through parameter-efficient fine-tuning.Each configuration was assessed with the ±1 and ±2 boundary-instance formats.Source: §3.6 · p. 9

3.6.1 GPT-5.5 Prompting

  • GPT-5.5 received the MRU definition and decision criteria in zero-shot and few-shot prompts; few-shot added four fixed training examples, two per outcome class.The examples included straightforward and more difficult decisions, while the zero-shot prompt had no labeled demonstrations.Source: §3.6.1 · p. 10

3.6.2 Base Qwen3-32B

  • Base Qwen3-32B was evaluated with zero-shot and few-shot prompts using the same task information and four demonstrations as GPT-5.5.Holding the prompting materials and context formats constant enabled a more direct comparison of the model families and a reference for assessing fine-tuning.Source: §3.6.2 · p. 10

3.6.3 Qwen3-32B Fine-Tuning

  • Qwen3-32B was fine-tuned with separate LoRA adapters for the ±1 and ±2 formats, using adjudicated same_unit labels as supervision.Each adapter was evaluated with its matching context format; the design compared fine-tuned models with corresponding zero-shot base conditions.Source: §3.6.3 · p. 11

3.7 Post-Processing Procedure

A deterministic procedure adjusts existing LLM predictions using a local preserve-operation constraint. It makes only single-pass corrections within each draft pair.

3.7 Post-Processing Procedure

  • Post-processing conditionally modifies LLM predictions based on surrounding boundary structure and a preserve-operation constraint considered in human annotation.Unlike the proximity heuristic, it operates on existing model predictions rather than predicting directly from the focal operation types.Source: §3.7 · p. 11
  • For an original False prediction, the immediately adjacent boundary is changed from True to False when the shared operation is preserve.Predictions already labeled False remain unchanged, and the procedure never changes False to True.Source: §3.7 · p. 11
  • Corrections are single-pass, do not trigger further corrections, and do not cross draft-pair boundaries.Raw and post-processed predictions were retained and evaluated separately to distinguish the rule's contribution from the LLM's.Source: §3.7 · p. 11

3.8 Evaluation Measures

The study evaluates predictions against adjudicated gold-standard annotations using classification measures. Macro-F1 is the primary aggregate measure because the boundary data are imbalanced toward the negative class.

3.8 Evaluation Measures

  • Macro-F1 is the unweighted mean of positive- and negative-class F1, while positive-class F1 is the harmonic mean of precision and recall.Same_unit = True is treated as the positive class.Source: §3.8 · p. 12
  • Accuracy measures the proportion of all boundaries classified correctly, while precision and recall measure correctness among predicted same-unit boundaries and coverage of gold same-unit instances.Predictions were compared with adjudicated gold-standard annotations using accuracy, precision, recall, F1, and macro-F1.Source: §3.8 · p. 12
  • Macro-F1 is the primary aggregate measure because boundary labels are imbalanced toward the negative class; accuracy is reported as a complement.The majority and proximity baselines were evaluated once on the held-out test set, while LLM measures were computed for raw and post-processed predictions.Source: §3.8 · p. 12

3.9 AI Use Disclosure

The authors disclose that generative AI systems were used as research objects and analytic tools.

3.9 AI Use Disclosure

  • The authors take full responsibility for the manuscript’s accuracy, integrity, and final content.Source: §3.9 · p. 12

4 Results

The results compare LLMs and non-LLM baselines on a held-out test set, using macro-F1 as the primary aggregate measure because the test set is imbalanced toward the negative class.

4.1 Baseline and Raw Model Performance

  • .859 macro-F1 for fine-tuned Qwen3-32B with ±2 context exceeded the proximity heuristic’s .825, with higher recall (.688 vs. .597) and nearly identical precision (.855 vs. .852).The strongest configuration’s advantage was primarily associated with identifying additional positive same-unit boundaries rather than reducing false positives.Source: §4.1 · pp. 12–13, Table 3 · p. 13
  • .633 macro-F1 was the strongest raw prompted result, from few-shot base Qwen3-32B with ±2 context, but remained below the proximity heuristic.The majority baseline had .825 accuracy but identified no positive same-unit boundaries and achieved .452 macro-F1; the heuristic achieved .912 accuracy and .825 macro-F1.Source: §4.1 · pp. 12–13
  • Context-window effects varied across LLM conditions, while the fine-tuned ±2 model exceeded its ±1 counterpart on accuracy, recall, F1, and macro-F1.Source: §4.1 · p. 13

4.2 Performance After Post-Processing

  • Post-processing generally increased precision and reduced recall because it could change positive predictions to negative but not the reverse.Source: Table 4 · p. 13, §4.2 · p. 13
  • .845 macro-F1 was the post-processed fine-tuned ±2 model’s best result, below its raw .859; the strongest prompted condition reached .808, still below the heuristic’s .825.Only the fine-tuned ±2 configuration exceeded the heuristic before and after post-processing, and its strongest performance was without post-processing.Source: Table 4 · p. 13, §4.2 · p. 14

5 Discussion

Task-specific fine-tuning let Qwen3-32B outperform the proximity heuristic, while prompted models did not. The discussion also qualifies these findings by the study’s data, baselines, and boundary-level evaluation.

5 Discussion

  • Prompted GPT-5.5 and base Qwen3-32B did not outperform the heuristic, despite receiving revision context and, in some conditions, labeled examples.The prompted models generally had high recall but low precision, tending to merge adjacent operations too often.Source: §5 · p. 14
  • Macro-F1 was .859 for fine-tuned Qwen3-32B with ±2 context, versus .825 for the proximity heuristic.Their precision was .855 and .852, respectively, while recall was .688 and .597.Source: §5 · p. 14
  • Fine-tuning was the clearest source of improvement; prompting and context-window effects were more limited and less consistent.Few-shot prompting generally brought modest accuracy and macro-F1 gains over zero-shot prompting, often with reduced recall; ±2 context helped the fine-tuned model most.Source: §5 · p. 15
  • Deterministic post-processing improved non-fine-tuned LLM predictions but slightly reduced macro-F1 for the strongest fine-tuned configuration.For prompted models, the rule increased precision by reducing false-positive same-unit predictions.Source: §5 · p. 15
  • Transparent heuristics and task-adapted models can play complementary roles when operation type alone misses meaningful relationships.The discussion frames MRU identification as both a modeling and representation problem for learning analytics, with similar aggregation challenges in other writing-process data.Source: §5 · p. 15

5.1 Limitations and Future Work

  • The evaluation used undergraduate narrative essays from one institution, and the heuristic’s strong performance may depend partly on TRACE’s structure.The authors call for testing whether similar regularities arise with other revision representations and datasets.Source: §5.1 · pp. 15–16
  • Non-LLM comparisons were limited to simple baselines; the study did not establish an advantage over conventional supervised language models.The authors recommend comparing task-adapted LLMs with smaller learned models.Source: §5.1 · p. 16
  • Few-shot results may depend on the single fixed set of four demonstrations, and boundary-level metrics do not fully assess resulting unit quality.The authors recommend multiple demonstration sets and unit- or sequence-level evaluation of segmentation errors and downstream effects.Source: §5.1 · p. 16

6 Conclusion

Prompted general-purpose LLMs did not outperform a proximity heuristic, but task-specific fine-tuning did. The findings support choosing between transparent rules and adapted models according to the structure of revision-boundary decisions.

6 Conclusion

  • Prompted general-purpose LLMs did not outperform a proximity heuristic based only on TRACE operation types, despite task instructions, examples, and local revision context.The comparison underscores the value of transparent baselines that reflect data structure.Source: §6 · p. 16
  • Macro-F1 was .859 for fine-tuned Qwen3-32B with the ±2 contextual representation, compared with .825 for the proximity heuristic.The improvement was driven primarily by higher recall while maintaining comparable precision.Source: §6 · p. 16
  • Transparent rules may suffice when MRU boundaries align with structural regularities, whereas task-specific adaptation may help when decisions require contextual interpretation.Deterministic post-processing also substantially improved non-fine-tuned LLMs.Source: §6 · p. 16
Loading 2609.33720v1…