PaperlayerView exact revision on arXiv

Source-linked AI summary

A Unified Framework to Elicit Structured Feedback for Interpretable Multi-Trait Essay Scoring

Shihang Yang, Sanwoo Lee, Ningning Zhao, Yunfang Wu

arXiv:2608.28407v1cs.CL

TL;DR

Multi-trait AES needs rubric-grounded reasoning that remains consistent with interdependent trait scores, but prior methods often separate feedback from scoring. HiFTS jointly generates hierarchical feedback and scores with supervised and GRPO alignment, while CFMS-34 adds a Chinese 34-trait benchmark. Across CFMS-34 and ASAP++, HiFTS improves scoring and produces more coherent rubric-aligned feedback.

  • Problem

    Existing multi-trait AES methods often decouple feedback from scoring or generate traits independently, weakening unified rubric-grounded reasoning.

  • Method

    HiFTS trains a unified autoregressive model to generate hierarchical global-to-local feedback before holistic and trait scores, using teacher CoT supervision, GRPO, and inference-time global prior guidance.

  • Results

    Experiments on CFMS-34 and ASAP++ show improved holistic and average trait QWK, reduced CFMS-34 MSE, and more coherent rubric-aligned feedback.

  • Takeaways & Limitations

    Feedback-to-score reasoning is supported as a promising direction for interpretable multi-trait essay scoring.

  • Takeaways & Limitations

    HiFTS uses a predefined global-to-local generation order and has been evaluated only on CFMS-34 and ASAP++, leaving broader genres, populations, and classroom interactions for future study.

Abstract

from arXiv · show

Multi-trait Automated Essay Scoring (AES) requires rubric-grounded reasoning across interdependent traits, rather than isolated score prediction. Existing feedback-enhanced methods often decouple feedback from scoring or assess traits independently, weakening score--feedback consistency and rubric alignment. We propose HiFTS, a unified autoregressive framework that generates hierarchical CoT feedback before predicting trait-level and holistic scores. HiFTS distills rubric-grounded hierarchical CoT feedback from a teacher LLM and trains student models to jointly generate feedback and scores. HiFTS further applies Group Relative Policy Optimization with a composite reward balancing score agreement, calibration, feedback quality, and structural validity. At inference, a lightweight global prior provides holistic guidance to reduce drift during long-form reasoning. We also introduce CFMS-34, a Chinese multi-trait AES dataset with 951 essays annotated with holistic scores and 34 rubric-based traits. Experiments on CFMS-34 and ASAP++ show that HiFTS achieves strong holistic and trait-level scoring while producing coherent, rubric-aligned feedback.

1 Introduction

HiFTS addresses the disconnect between feedback and scoring in multi-trait AES with a unified autoregressive framework that generates hierarchical rubric-grounded feedback before scores. It combines hierarchical supervision, GRPO alignment, prior-guided inference, and CFMS-34 evaluation across Chinese and English benchmarks.

  • Motivation: Existing AES systems often prioritize score agreement while leaving decisions opaque, motivating feedback grounded in multiple rubric dimensions.Human-crafted features and post-hoc explanations provide only partial interpretability.
  • Motivation: Current feedback-enhanced methods separate feedback from scoring, generate traits independently, or use separate regression heads, weakening inter-trait and score–rationale consistency.These limitations prevent a unified autoregressive treatment of rubric-based reasoning.
  • HiFTS: HiFTS jointly generates global-to-local feedback and rubric-aligned holistic and trait scores in one autoregressive process.Its hierarchical input template is designed to capture inter-trait relations.
  • HiFTS: HiFTS uses teacher-prompted CoT traces for warm-up SFT, then applies GRPO to align feedback with ground-truth trait scores as verifiable rewards.The framework further conditions inference on a lightweight BERT-predicted prior to improve scoring stability and accuracy.
  • Contributions: CFMS-34 provides 34 expert-annotated Chinese traits for cross-lingual evaluation, while experiments on CFMS-34 and ASAP++ show improved scoring accuracy and feedback quality.Ablations attribute gains to hierarchical CoT supervision, GRPO alignment, and essay-specific prior guidance.

2 Related Work

Multi-trait AES extends holistic scoring with rubric-defined dimensions and increasingly models trait interactions. Feedback-oriented methods add rationales or joint training, but often retain separations between reasoning, scoring, and trait feedback.

  • Multi-trait essay scoring: Multi-trait AES predicts rubric-defined dimensions to provide finer-grained assessment than holistic scoring.Recent approaches use cross-prompt modeling, trait-aware representations, rubric-assisted features, mixture-of-experts, and graph-based interactions.
  • Feedback-based scoring and alignment: LLM-based AES research explores rubric-aware scoring and feedback through prompting, supervised training, and distillation.These methods seek more human-like evaluation and natural-language rationales.
  • Feedback-based scoring and alignment: Feedback-based systems often separate reasoning from scoring through external feedback generation, separate score heads, or independent trait-level feedback.Post-training methods such as RLHF, preference optimization, and GRPO are used to improve alignment.
  • Feedback-based scoring and alignment: HiFTS instead aligns structured multi-trait feedback with calibrated trait and holistic scores in a single autoregressive process.This directly addresses the separation identified in prior feedback-based scoring methods.

3 CFMS-34 Dataset Construction

CFMS-34 is an expert-annotated Chinese multi-trait essay-scoring dataset designed for fine-grained rubric evaluation and cross-lingual benchmarking. Its annotations show substantial human agreement, while frontier LLMs achieve only moderate agreement and calibration remains challenging.

  • Data collection: CFMS-34 contains 951 primary-school essays written in textbook-based classroom prompts under a 50-minute limit.The collection was designed to preserve consistent writing conditions and authentic classroom settings.
  • Annotation: Two Chinese-language education experts independently annotate 34 sub-traits across Content, Structure, Expression, and Conventions, plus an overall score.The dataset uses a fine-grained rubric tailored to Chinese essay writing.
  • Annotation: 97.4% of essays have overall-score differences within one point, with human–human overall QWK of 0.57.Across sub-traits, average agreement is 0.48 exact match, 0.90 within one point, and 0.41 QWK.
  • Dataset split: The dataset is split 8:1:1, with perfectly agreed overall-score samples reserved for the test set to reduce label noise.This test-set design is intended to improve evaluation reliability rather than simplify the task.
  • Benchmark role: CFMS-34 offers a fine-grained Chinese benchmark for more precise multidimensional essay-quality evaluation than datasets with less detailed rubrics.Its expert annotations, dual-rater scoring, and 34 rubric-based traits support cross-dataset comparison.
  • Benchmark difficulty: Current frontier LLMs reach best QWK below 0.44 on CFMS-34, while their mean scores reveal underprediction or overprediction calibration issues.The reported model QWKs are computed against resolved human scores and are not strictly comparable to the human QWK of 0.57.

4 Method

HiFTS unifies hierarchical rubric-grounded feedback generation and multi-trait scoring in one autoregressive process, using teacher CoT supervision, GRPO alignment, and global-prior-guided inference. Its rewards jointly target scoring agreement, numerical accuracy, feedback alignment, and structural compliance.

  • HiFTS generates rubric-aware hierarchical feedback before predicting trait-level and holistic scores in a unified autoregressive process.
  • Warmup SFT: Teacher-generated top-down CoT feedback moves from global essay understanding to dimensions and sub-traits, preparing structured reasoning for student training.The feedback is combined with labeled trait-level and overall scores into a single feedback-to-score target sequence.
  • Warmup SFT: Warmup SFT anchors the student’s reasoning trace to its eventual scoring decisions before reinforcement learning.The student is fine-tuned with token-level cross-entropy on concatenated feedback and labeled scores.
  • Reinforcement Learning: GRPO uses a composite reward balancing score agreement, numerical error, semantic feedback alignment, and structural compliance.The balancing coefficient α controls score-oriented versus feedback-oriented alignment, while δ is a rule-based structural bonus or penalty.
  • Global Prior Guidance: A lightweight BERT regressor supplies a coarse holistic prior that guides decoding without replacing the LLM’s final autoregressive scores.The prior acts as a soft semantic anchor and adds no decoding-time hyperparameters.
  • Global Prior Guidance: Prior-guided decoding reduces inconsistencies between local trait analyses and final scoring decisions by mitigating semantic drift in long-form reasoning.

5 Experimental Setup

HiFTS is evaluated on Chinese and English multi-trait essay scoring using common trait targets, agreement and calibration metrics, and feedback preference comparisons. The setup uses two-stage SFT-to-GRPO training with BERT-based holistic-prior regressors and structured decoding.

  • Datasets: Experiments cover CFMS-34 and ASAP++, spanning Chinese and English multi-trait essay scoring.CFMS-34 evaluation uses 20 core rubric traits scored on a 0–10 scale from two raters’ 0–5 annotations.
  • Evaluation Metrics: Scoring agreement is measured with QWK for holistic and trait-level performance, while CFMS-34 calibration is measured with MSE.WinRate from DeepSeek-V3.2 pairwise judgments compares each SFT model with its GRPO-aligned counterpart.
  • Training Details: Qwen2.5-7B and Qwen3-4B follow two-stage training: hierarchical-CoT SFT followed by GRPO-based alignment.BERT regressors initialized from Chinese or English base checkpoints estimate holistic essay quality for global-prior guidance.
  • Training Details: GRPO samples G = 4 responses per input, uses β = 0.02 and a learning rate of 1 × 10^-6, and computes holistic QWK over h = 64 recent generations.
  • Inference: Inference produces structured [Analysis], [Trait Scores], and [Overall Score] fields with temperature 0.6 and top-p = 0.9.The global score prior is enabled when applicable.
  • Baselines: Baselines include traditional, multi-trait, and rationale-enhanced essay scoring models, with published ASAP/ASAP++ results used when settings are comparable.

6 Results and Analysis

HiFTS performs consistently across CFMS-34 and ASAP++, while reward alignment improves scoring, feedback grounding, and hierarchical reasoning quality. Ablations show that hierarchical generation, meaningful prior guidance, and reward trade-off choices each affect performance and stability.

  • 6.1 Scoring Performance: 0.677 Overall QWK and 0.741 MSE are achieved by the Qwen3-based model with GRPO on the reported cross-dataset evaluation.HiFTS remains consistent across CFMS-34 and ASAP++, while GRPO improves several scoring metrics.
  • 6.2 Fine-grained Trait Analysis: GRPO-aligned models generally outperform SFT counterparts on CFMS-34 trait-level QWK, with Qwen3 achieving the highest average.The gains extend across most evaluated trait dimensions.
  • 6.3 Feedback Assessment: GRPO improves feedback grounding over SFT, and full HiFTS obtains the best rule-based grounding score.The metric combines coverage, trait-level reasoning, and reasoning density.
  • 6.3 Feedback Assessment: HiFTS hierarchical CoT moves from global understanding to rubric dimensions and sub-traits, producing more cohesive and inspectable feedback than flat reasoning.Connective expressions link different dimensions and support rubric alignment.
  • 6.4 Ablation Study: The BERT prior stabilizes inference but is insufficient alone; meaningful essay-specific guidance improves Overall QWK and WinRate, whereas random or constant priors degrade performance.GRPO also outperforms PPO in WinRate, and adding the meaningful prior yields the best ablation results.
  • 6.5 Hyperparameter Analysis: α = 0.8 provides the best tested trade-off between feedback alignment and score-oriented optimization, while h = 64 is selected despite h = 128 performing slightly best.The window-size analysis attributes instability with smaller windows to QWK being a set-level metric.

7 Conclusion

The paper contributes CFMS-34 and HiFTS for interpretable multi-trait essay scoring. Experiments show improved scoring metrics and more coherent rubric-aligned feedback, with ablations supporting reward alignment and prior-guided decoding.

  • 7 Conclusion: CFMS-34 provides an expert-annotated Chinese benchmark with dual ratings, holistic scores, and 34 fine-grained rubric-based traits.It extends interpretable writing assessment beyond English settings.
  • 7 Conclusion: HiFTS generates global-to-local rubric-grounded feedback before deriving holistic and trait-level scores in a unified autoregressive framework.Training uses teacher-generated hierarchical feedback and GRPO, while a lightweight BERT prior anchors inference.
  • 7 Conclusion: Experiments on CFMS-34 and ASAP++ show improved holistic and average trait QWK, reduced CFMS-34 MSE, and more coherent rubric-aligned feedback.Ablations verify benefits from GRPO alignment and prior-guided decoding.

Limitations

The paper’s reasoning structure and evaluation scope constrain its conclusions. It uses a predefined global-to-local order and evaluates only Chinese and English multi-trait AES settings.

  • Limitations: HiFTS uses a predefined global-to-local autoregressive reasoning order, leaving alternative generation orders and more flexible feedback organizations for future study.The current structure is effective in the reported experiments and aligns with rubric-based assessment.
  • Limitations: Evaluation covers CFMS-34 and ASAP++, so scalability and educational impact remain to be assessed across more genres, learner populations, and classroom interactions.The authors identify direct student and educator engagement with feedback as a future evaluation setting.

Ethics Statement

The paper reports research-use and data-governance conditions for its benchmarks and models. CFMS-34 was collected with institutional authorization and consent procedures, and the system is not intended for unsupervised high-stakes decisions.

  • Ethics Statement: ASAP++ is used as a publicly available benchmark, while pretrained models and API-based LLMs are used under their respective licenses or service terms.The creators of datasets, models, and baselines are cited in the corresponding sections.
  • Ethics Statement: CFMS-34 essays were collected from routine educational assessments with institutional authorization and applicable consent or guardian-consent procedures.The dataset is constructed for research on multi-trait essay scoring and rubric-grounded feedback.
  • Ethics Statement: The dataset and system are intended for research and educational support, not as the sole basis for high-stakes educational decisions without human oversight.
  • Ethics Statement: AI assistants were used only to improve writing clarity and coherence; the paper’s ideas and experiments are presented as original to the authors.

A Selection of 20 Core Traits

The study selects 20 core traits from CFMS-34 using their correlation with holistic scores. This reduces noise and reasoning complexity while retaining traits most associated with overall performance.

  • Motivation: The selection is motivated by holistic scores being dominated by key traits, while marginal traits may add noise.This reflects real-world grading conditions described by the authors.
  • Motivation: Using fewer traits shortens the reasoning chain and reduces the complexity of structural constraints.The authors link this design to greater stability in rubric adherence.
  • Selection criterion: The model selects the 20 traits with the highest Pearson correlations with overall scores on the training set.These traits form the core reasoning subset for modeling and evaluation.

B Per-trait Score Statistics and Inter-rater Agreement

Table 11 summarizes score distributions and dual-rater agreement for all 34 traits across 951 CFMS-34 essays. The 20 traits used in the main experiments are explicitly marked for distinction.

  • Table scope: Table 11 reports per-trait score statistics and dual-rater agreement for all 951 CFMS-34 essays.Mean and standard deviation use pooled 0–5 ratings from both experts.
  • Agreement measures: Exact measures identical ratings, Within-1 measures agreement within one point, and QWK measures quadratic weighted kappa.These metrics characterize agreement between the two expert raters.
  • Experimental subset: The 20 traits used in the main experiments are marked with † in Table 11.This marker distinguishes the modeling and evaluation subset from the full 34-trait annotation set.
Loading 2608.28407v1…