PaperlayerView exact revision on arXiv

Source-linked AI summary

ArguLens: An Open-Source System for Automated Essay Scoring and Label-Aware Feedback Generation

Weiran Wang, Hongxiang Shi, Huitao Tang, Wenjuan Qin

arXiv:2608.17356v1cs.CL

TL;DR

Automated essay scoring often lacks interpretable evidence, local deployment, and label-aware feedback. ArguLens addresses these gaps with an open-source modular pipeline, whose components achieve strong reported classifier and scorer benchmarks, including 0.813 mean QWK. However, the evaluation remains component-level rather than end-to-end.

  • Problem

    Automated essay scoring often provides only holistic scores, while closed APIs limit interpretability, local deployment, privacy, and cost-effective label-aware feedback.

  • Method

    ArguLens uses a decoupled open-source pipeline for discourse classification, feature-based scoring, and label-aware feedback generation.

  • Results

    0.813 mean QWK was achieved by the full 31-feature LightGBM scorer, while the logit-probe classifier reached 82.58% accuracy and 0.727 macro-F1.

  • Takeaways & Limitations

    The modular design allows discourse classification, scoring, and feedback components to be developed, tested, and replaced independently.

  • Takeaways & Limitations

    The reported scorer uses gold discourse annotations, so the +0.055 QWK result is an oracle component diagnostic rather than an end-to-end classifier-to-scorer evaluation.

Abstract

from arXiv · show

Most automated essay scoring (AES) systems output a single holistic score without interpretable evidence and rely on closed APIs that introduce data privacy and cost barriers. We present ArguLens, an opensource, locally deployable system that decomposes AES into three decoupled components: a discourse-move classifier (Qwen2.5-7B-Instruct fine-tuned with LoRA on PERSUADE 2.0), a grade-independent LightGBM scorer over 31 linguistic and discourse features, and a label-aware feedback generator served through vLLM with a Qwen2.5-14BInstruct backbone. A Gradio web UI exposes pluggable inference backends and supports single-essay and batch scoring with downloadable per-essay breakdowns. On an essaydisjoint PERSUADE 2.0 test split, the logitprobe classifier achieves 82.6% accuracy and 0.727 macro-F1; under prompt-grouped 5-fold cross-validation the scorer reaches a mean QWK of 0.813 under an oracle discoursefeature protocol, and an ablation shows that adding gold discourse annotations yields an increment of +0.055 QWK over the lexical+syntactic configuration (paired t-test, p = 0.010). This is a component-level diagnostic rather than an end-to-end classifier-to-scorer result. The feedback generator ships with a structured evaluation protocol; its human-rater study is left to future work. The system is released under Apache 2.0 at https://github.com/wwrwbs/AI_AWE.

1 Introduction

ArguLens addresses persistent AES trade-offs between interpretability, performance, local deployment, and privacy by decomposing scoring and feedback into modular components. The paper evaluates its classifier and scorer on PERSUADE 2.0, while limiting feedback evaluation to its documented workflow and repository contract.

  • Motivation: ArguLens targets AES systems that provide opaque holistic scores, depend on commercial APIs, and lack fine-grained, label-aware revision guidance.These gaps hinder instructor understanding, increase cost, latency, and privacy barriers, and limit assistive deployment in resource-constrained educational settings.
  • System contribution: The Apache 2.0-licensed system decomposes scoring into discourse-move classification, feature extraction, LightGBM scoring, and label-aware feedback.This modular design is presented as the paper’s response to the identified interpretability and deployment gaps.
  • System contribution: ArguLens supports local deployment through vLLM with tensor parallelism and a 4-bit HuggingFace fallback, while retaining an optional API backend.The implementation preserves deployment flexibility without making local execution dependent on commercial APIs.
  • System contribution: The system ships reproducible artifacts and a config-driven, offline-tested stack with zero hardcoded paths and environment-variable precedence.Artifacts include the scorer model, scaler, LoRA adapter configuration, and versioned Release weights with checksums.
  • Evaluation scope: Evaluation uses PERSUADE 2.0 middle-school argumentative essays graded 1 to 6, with classifier metrics, scorer QWK-based cross-validation, and workflow-level feedback assessment.Human-rater results for feedback are explicitly outside the scope of this release.

2 Related Work · 3 System Architecture

The paper situates ArguLens within interpretable, argument-aware, and responsibly evaluated automated writing assessment, then describes a modular architecture with reproducible configuration and verifiable model assets.

  • 2 Related Work: Classical feature engineering exposes linguistic evidence, whereas neural AES models learn richer representations but reduce transparency.Both paradigms reflect the field’s tension among predictive accuracy, interpretability, and instructional use.
  • 2 Related Work: Argument-mining research models essay structure and quality, while recent work combines argument-segment and cohesion features for feedback-related scoring.EssayJudge further evaluates AES across lexical, sentence, and discourse levels, highlighting granularity gaps.
  • 2 Related Work: Feedback evaluation is multidimensional: ArguLens defines relevance, actionability, and tone but leaves human ratings to future work.Prior work emphasizes explicit dimensions, calibration against human judgments, and risks of specific yet misprioritized or inappropriate criticism.
  • 2 Related Work: Aggregate AES accuracy does not establish fairness, so subgroup analysis across demographic and learner characteristics remains a required future study.This limitation is treated as an open research requirement rather than an implied property of the released system.
  • 2 Related Work: The release documents models, training context, limitations, and asset locations, distributing small artifacts in the repository and larger adapters as versioned Release assets with checksums.This is explicitly a release-design choice rather than an experimental finding.
  • 3 System Architecture: The Gradio frontend orchestrates three independent services: sentence-level move classification, 31-feature extraction, scoring, and grounded feedback generation.The classifier supports HF or vLLM backends; features combine TAALED lexical indices, QuanSyn dependency metrics, and discourse counts, while the scorer emits a score, confidence, and flags.
  • 3.1 Configuration and Reproducibility: Runtime configuration follows precedence from environment variables to config.yaml to default HuggingFace model IDs, with paths resolved against the repository root.A checksums.sha256 manifest supports sha256sum -c verification of scorer, scaler, adapter configuration, tokenizer, and LoRA-weight assets.

4 Discourse-Move Classifier

ArguLens classifies each essay sentence into one of four discourse-move labels using local context and a LoRA-adapted Qwen2.5-7B-Instruct model. A deterministic logit probe supports inference through interchangeable Hugging Face and vLLM backends, with an API backend reserved for feedback generation.

  • Classifier design: Each sentence receives one of four labels—claim, data, counterclaim, or rebuttal—using the preceding sentence and essay opening as local context.The classifier operates at sentence level over an essay of N sentences.
  • Model and training: Qwen2.5-7B-Instruct is adapted with a LoRA module using rank r = 32, α = 64, and dropout 0.05 across linear projection modules.Training uses PERSUADE 2.0 with 4-GPU DDP, BF16 precision, a peak learning rate of 1 × 10−4, a 2,048-token cutoff, two epochs, and effective batch size 32.
  • Data split: 22,605 training, 2,825 validation, and 2,826 test essays form an essay-level seed-42 split with zero essay overlap.Reported test results use a natural-distribution test set of 12,000 sentences from 2,730 essays.
  • Inference: The discriminative logit probe extracts four code-token logits at the final non-padding position, applies softmax, and selects the argmax label.This avoids free-text generation and provides deterministic inference.
  • Deployment: Three backends are available: in-process hf, persistent vllm, and an api backend reserved for the feedback generator.The frontend falls back to hf when the vLLM service is unavailable.

5 Grade-Independent LightGBM Scorer

The grade-independent scorer combines lexical, syntactic, and discourse features in a six-class LightGBM model trained with prompt-grouped cross-validation. Reported evaluation uses gold discourse annotations, while production inference substitutes classifier-predicted move labels.

  • Feature design: 16 lexical features from TAALED capture lexical diversity, word count, average word length, function-word counts, and lexical density.The features span academic-word, content-word, and function-word classes.
  • Feature design: 8 dependency-based features from QuanSyn comprise mdd, ndd, mhd, mtdl, vk, mtw, hi, and mrd.
  • Feature design: 7 discourse features are derived from classifier outputs for deployment, but reported scorer evaluation uses corresponding PERSUADE gold discourse annotations.The deployment features include sentence, claim, data, percentage, counterclaim, rebuttal, and a binary counterclaim-and-rebuttal indicator.
  • Model and training: The LightGBM scorer uses a multiclass objective over 6 ordinal classes and holds out prompt groups during cross-validation.Training uses the PERSUADE 2.0 grade-independent feature table, seed 42, and fold-indexed model seeds.
  • Inference: At inference, z-score-normalized features produce a six-class probability vector whose expected value is rounded to an integer from 1–6.The same expected-value decoding rule is used for cross-validation, ablation, and production inference.
  • Inference: The system flags low-confidence essays when maximum class probability is below 0.5 and marks borderline cases when adjacent runner-up probability exceeds 0.30.Both thresholds are fixed heuristics rather than tuned values.

6 Label-Aware Feedback Generation · 7 Frontend, Batch Processing, and Export · 8 Experiments and Evaluation

ArguLens combines label-aware feedback generation, pluggable inference and batch export, and component-specific evaluation protocols. Experiments report strong classifier and scorer results, while feedback quality and full-pipeline latency remain unevaluated.

  • 6.1 Prompt Design: Feedback prompts combine essay context, predicted score, confidence, flags, and 14 feature values paired with Low, Medium, or High band labels.The 14 features cover lexical, syntactic, discourse, and length properties.
  • 6.2 Backends: Three inference backends support a persistent vLLM server, single-GPU 4-bit Hugging Face inference, or any OpenAI-compatible API endpoint.API keys are read at runtime from the UI or environment and never stored.
  • 7 Frontend, Batch Processing, and Export: The Gradio 6.x interface supports single-essay analysis, batch upload with summary tables, and ZIP export containing per-essay results and root-level summary files.Per-essay archives include essay text, scores, linguistic and discourse features, feedback, results, and sentence labels.
  • 8 Experiments and Evaluation: Classifier evaluation uses an essay-disjoint sentence-level test split, whereas scorer evaluation uses prompt-grouped cross-validation that withholds complete prompts.The protocols are deliberately not combined into one end-to-end score.
  • 8.3 Scorer Evaluation: +0.055 QWK was gained from gold discourse features over lexical+syntactic features alone, rising from 0.759 to 0.813 in an oracle diagnostic.The paired t-test across prompt-aligned folds gives p = 0.010; this does not measure downstream classifier gain.
  • 8.4 Feedback Generation and Evaluation Protocol: Human-annotator results for feedback relevance, actionability, and tone are left for future work, and reported component latency is not full-pipeline latency.The Qwen2.5-14B feedback backend was not benchmarked; classifier inference peaks at 16.7 GB allocated GPU memory for one 21-sentence essay.
  • 8.2 Classifier Evaluation: 82.58% accuracy and 0.727 macro-F1 were achieved by the Qwen2.5-7B + LoRA logit-probe classifier on 12,000 PERSUADE 2.0 test sentences.Claim–data confusion accounts for 73% of classifier errors.
  • 8.3 Scorer Evaluation: 0.813 mean QWK was achieved by the full 31-feature LightGBM scorer under prompt-grouped 5-fold cross-validation.Fold-level QWK ranges from 0.754 to 0.872, indicating heterogeneous difficulty across withheld prompts.

9 Open-Source Release Strategy · 10 Testing and Quality Assurance

The release includes an offline test suite covering eight system aspects, with all 22 tests passing without GPU or network access. Tests verify batch outputs, classifier prompting, feedback bands, frozen assets, Gradio I/O, model configuration, scorer behavior, and the vLLM API contract.

  • 10 Testing and Quality Assurance: Eight system aspects are validated by an intentionally offline tests/ suite.The suite avoids requiring a live model service by using a fake engine for the vLLM API contract test.
  • 9 Open-Source Release Strategy: Table 6 specifies the asset distribution policy.The supplied passage provides the table title but no further policy details.
  • 10 Testing and Quality Assurance: 3/3 batch-output tests cover ZIP structure, manifests, and CSV files.The classifier-prompt tests cover the prompt template, label parsing, and shared-context contract.
  • 10 Testing and Quality Assurance: 4/4 feedback-band tests verify band thresholds, while frozen-asset tests check adapter configuration, tokenizers, and SHA-256 hashes.The frozen-assets result is reported as 1/1.
  • 10 Testing and Quality Assurance: Gradio contract tests cover single- and batch-I/O schemas and branding.The cited passage does not report a pass-rate value for this module.
  • 10 Testing and Quality Assurance: 2/2 model-configuration tests cover precedence and path resolution, while scorer-contract tests verify feature order, scaling, flags, and score decoding.The vLLM API contract records a 1/1 pass rate using a fake engine.
  • 10 Testing and Quality Assurance: 22 tests pass without a GPU or network connection.The vLLM API contract test uses a fake engine instead of a live model service.

11 Conclusion

ArguLens concludes by presenting a modular, open-source pipeline that independently combines discourse classification, feature-based scoring, and label-aware feedback generation. Experiments report strong component results, while future work targets multilingual extension and larger-scale human evaluation.

  • Contributions: ArguLens separates discourse classification, feature-based scoring, and feedback generation into independently developable, testable, and replaceable components.The system is presented as an open-source modular architecture for automated essay scoring and label-aware feedback.
  • Results: 82.58% accuracy and 0.727 macro-F1 were achieved by the logit-probe classifier on the PERSUADE 2.0 test set.These results evaluate the discourse classifier component.
  • Results: 0.813 QWK was obtained by the LightGBM scorer with all 31 features over 5-fold cross-validation.In the oracle component evaluation, adding gold discourse features contributed +0.055 QWK over lexical and syntactic features alone, with paired t-test p = 0.010; this is not the gain from predicted classifier features.
  • Future Work: Future work includes extending the pipeline beyond English argumentative essays and one educational context, alongside larger-scale human evaluation of feedback actionability.The proposed extensions include XLM-RoBERTa for discourse classification, multilingual LLMs for feedback, and evaluation across diverse learner populations.
  • Release: The code, release metadata, evaluation protocol, and model artifacts are released under Apache 2.0 to facilitate community-driven extensions.Third-party models and data retain their upstream licenses.

12 Responsible Use and Limitations

ArguLens has important limitations in domain generalization, subgroup reporting, formative-use validity, and end-to-end evaluation. Its computational requirements also vary by component, with local 14B feedback deployment costs not measured.

  • Domain and fairness limitations: Training exclusively on US middle-school argumentative essays limits expected generalization across genres, age groups, and languages without domain-specific fine-tuning.The corpus reflects specific demographic and curricular contexts, and scores may carry demographic biases.
  • Domain and fairness limitations: Subgroup metrics for gender, race/ethnicity, and other groups are not reported, leaving support and uncertainty for future study.The passage recommends using corpus metadata in a future study.
  • Evaluation limitations: The classifier is evaluated on completed essays, not partial drafts, while the scorer uses a single holistic 1–6 target without rater-level variation.These limitations matter for formative use and borderline-case interpretation.
  • Evaluation limitations: +0.055 QWK is an oracle component diagnostic, not an end-to-end classifier-to-scorer result, because scorer features use PERSUADE gold discourse annotations.The classifier uses an essay-disjoint sentence-level split, whereas scorer cross-validation uses a feature table with gold discourse fields.
  • Deployment limitations: 16.7 GB allocated and 18.5 GB reserved VRAM were used for the classifier benchmark on one RTX 5000 Ada; local 14B feedback deployment requirements were not measured.The scorer runs on CPU, and the feedback backend can call an OpenAI-compatible service.

A Reproducibility Checklist

ArguLens provides a reproducibility checklist covering released code, model and data resources, pinned environments, integrity checks, tests, launch scripts, training configuration, and split statistics.

  • Released Resources: The code is released at https://github.com/wwrwbs/AI_AWE under Apache 2.0, with LoRA weights available separately.LoRA weights are provided at the v0.1.0 release tag.
  • Released Resources: The checklist identifies Qwen2.5-7B-Instruct and Qwen2.5-14B-Instruct models from HuggingFace Hub, plus PERSUADE 2.0 data under its upstream non-commercial license.The models are Apache 2.0 licensed.
  • Execution and Verification: Reproducibility support includes three version-pinned requirement files, model_checksums.sha256, and offline pytest -q tests.The requirement files cover the frontend, vLLM, and training environments.
  • Execution and Verification: The run all services.sh script autodetects virtual environments, GPUs, and ports, while the LoRA training configuration is explicitly provided.The configuration path is qwen_move_classifier/configs/qwen25_7b_move_lora_sft.yaml.
  • Data Splits: Split statistics are recorded in dataset_summary.json for an essay-level split with seed 42 and zero overlap between splits.The file is located under qwen_move_classifier/data.

B Full Feature List

The scorer uses 31 features grouped by family, with lexical-diversity variants computed over all, content, and function words.

  • Feature organization: 31 features comprise the scorer’s complete feature set, organized into feature families.Table 8 enumerates all 31 scorer features.
  • Lexical diversity: Lexical-diversity variants are computed over all words, content words, and function words.These variants are denoted aw, cw, and fw, respectively.
Loading 2608.17356v1…