ArguLens: An Open-Source System for Automated Essay Scoring and Label-Aware Feedback Generation

1 Introduction

• AES faces a tension between transparent feature-based scoring and accuracy, while neural-model opacity and closed-API barriers constrain interpretability, local deployment, and privacy.Existing systems also rarely provide fine-grained, label-aware revision suggestions beyond a holistic score.Source: §1 · p. 1
• ArguLens decomposes scoring into discourse-move classification, feature extraction, LightGBM scoring, and label-aware feedback.It is presented as a locally deployable system licensed under Apache 2.0.Source: §1 · p. 1
• The system supports local inference through vLLM with tensor parallelism or a 4-bit HuggingFace fallback, while retaining an optional API backend.Its repository includes reproducible scorer and LoRA artifacts, versioned weights with checksums, config-driven paths, environment-variable precedence, and an offline test suite.Source: §1 · p. 1
• Evaluation uses PERSUADE 2.0 US middle-school argumentative essays graded 1 to 6, assessing classifier accuracy and macro-F1 and scorer QWK, accuracy, and macro-F1 under prompt-grouped cross-validation.Feedback is evaluated only through its documented generation workflow and repository contract; human-rater results are outside the release’s scope.Source: §1 · pp. 1–2

2 Related Work

• Classic feature engineering exposes linguistic evidence, while neural AES learns richer representations less transparently; ArguLens pairs an explicit-feature LightGBM scorer with a rhetorical-structure classifier.The related work also notes holistic scoring with general-purpose AI and questions about rubric alignment and validity.Source: §2 · p. 2
• ArguLens uses four PERSUADE 2.0 labels—claim, data, counterclaim, and rebuttal—and a Qwen2.5-7B LoRA classifier without full-parameter updates.Prior work covers argument structure and essay quality (Stab and Gurevych, 2017), feature combinations for feedback-related scoring, and discourse-level AES gaps.Source: §2 · p. 2
• The release defines relevance, actionability, and tone for feedback evaluation but reports no human ratings, and treats subgroup analysis as future work rather than claiming fairness from aggregate metrics.The cited work highlights multidimensional evaluation and risks of specific comments missing key problems or misjudging criticism (Hashemi et al., 2024; Rashkin et al., 2025); subgroup performance can vary with demographic and learner characteristics.Source: §2 · p. 2
• The release documents its model, training context, limitations, and asset locations; small artifacts remain in the repository, while the larger adapter is a versioned Release asset with a checksum.The paper presents this as a release design choice, not an experimental finding (Mitchell et al., 2019).Source: §2 · p. 2

3 System Architecture

• The Gradio frontend coordinates a move classifier, a 31-feature extractor, a scorer, and a feedback generator across pluggable inference backends.The scorer returns a score, confidence, and flags; feedback is served through vLLM or an OpenAI-compatible API.Source: §3 · p. 2, Table 1 · p. 2, Figure 1 · p. 3

3.1 Configuration and Reproducibility

• Runtime settings prioritize environment variables over config.yaml and defaults, resolve relative paths from the repository root, and use a SHA-256 manifest to verify model and adapter assets.The manifest covers the scorer, scaler, adapter config, tokenizer, and LoRA weights.Source: §3.1 · p. 3

4.1 Task Formulation

• The classifier assigns each sentence one of four labels—claim, data, counterclaim, or rebuttal—using the preceding sentence and essay opening as local context.The task is formulated at sentence level for an essay of N sentences.Source: §4.1 · p. 3

4.2 Base Model and LoRA

• The model uses Qwen/Qwen2.5-7B-Instruct with a rank-32 LoRA adapter and is trained on PERSUADE 2.0 using LLaMA-Factory.LoRA uses α = 64 and dropout 0.05; training uses 4-GPU DDP, BF16, two epochs, and an effective batch size of 32. The 309 MB adapter weights are released as v0.1.0.Source: §4.2 · p. 3

4.3 Training Data and Model Selection

• For model evaluation, the cited description identifies a natural-distribution test set of exactly 12,000 sentences, drawn from 2,730 essays, as the test data.Source: §4.3 · p. 4

4.4 Logit-Based Classification; 4.5 Inference Backends

• For deterministic classification, the logit probe extracts logits for four single-token label codes, applies softmax, and selects the highest-probability label.The maximum softmax probability is reported as sentence confidence. HF chunks prompts, while vLLM sends bounded requests to a persistent service with one-token decoding and only the four label tokens allowed. The HF backend loads the model and adapter in-process; the frontend falls back to HF if vLLM is unavailable.Source: §4.4 · p. 4, §4.5 · p. 4

5.1 Feature Set

• All features exclude learner grade information to prevent direct target leakage, though this does not establish fairness because linguistic features may correlate with demographic or curricular variables.A single shared spaCy dependency parse supplies lexical and syntactic metrics.Source: §5.1 · p. 4
• The 31-feature set combines 16 lexical features, eight dependency-based syntactic features, and seven discourse features; cross-validation uses gold discourse annotations rather than predicted moves.Deployment derives discourse features from Qwen2.5-7B LoRA classifier outputs, so the reported scorer results are an oracle component evaluation, not an end-to-end classifier benchmark.Source: §5.1 · p. 4

5.2 Model Training

• The scorer uses LightGBM with a six-class ordinal objective, trained on grade-independent PERSUADE 2.0 features with prompt groups held out during cross-validation.Training uses 1,000 estimators, 63 leaves, and a learning rate of 0.03; the model and feature scaler are committed to the repository.Source: §5.2 · p. 4
• At inference, z-score-normalized features produce six class probabilities, whose expected value is rounded to an integer score from 1 to 6.Low confidence is flagged when the maximum class probability is below 0.5; an essay is borderline when the runner-up exceeds 0.30 and is adjacent to the predicted class. These are fixed heuristics, not tuned thresholds.Source: §5.2 · pp. 4–5

6.1 Prompt Design

• The feedback prompt uses 14 of 31 features across lexical, syntactic, discourse, and length dimensions, each paired with a corpus-tertile band label.Bands guide feedback prompting only and never enter the scorer, preventing score-information leakage.Source: §6.1 · p. 5
• The prompt supplies essay text, prompt name, predicted score, confidence, and flags, and requires human review recommendations for low-confidence or borderline cases.It also forbids invented metrics, requires quotations from the essay, and enforces constructive English-only output.Source: §6.1 · p. 5
• Feedback follows five fixed sections: overall assessment, lexical, syntactic, discourse/argumentation, and one or two actionable improvements.Source: §6.1 · p. 5

6.2 Backends

• The vllm backend is the default and serves the 14B model through a persistent OpenAI-compatible server; the hf backend uses 4-bit BitsAndBytesConfig with SDPA on one GPU.The vllm server uses port 8000 and tensor parallelism of 2.Source: §6.2 · p. 5
• The api backend accepts any OpenAI-compatible endpoint, and its API key is read from the UI or environment at runtime and never stored.Source: §6.2 · p. 5

7 Frontend, Batch Processing, and Export

• The Gradio 6.x interface supports single-essay analysis, batch uploads with a summary table, and ZIP export.Source: §7 · p. 5
• Each exported essay folder contains text, score and feature files, feedback, results, and sentence labels; the archive root contains summary.csv and manifest.json.Source: §7 · p. 5

8.1 Experimental Setup

• Evaluation used an essay-disjoint sentence-level classifier test split and prompt-grouped scorer folds that withheld complete prompts, making scorer results an oracle component evaluation, not an end-to-end estimate.PERSUADE 2.0 contains more than 25,000 US student essays across 15 prompts, grades 6–12, each scored on a 1–6 rubric.Source: §8 · p. 5, §8.1 · p. 5

8.2 Classifier Evaluation

• Data had the highest class F1 (0.862), followed by claim (0.805), while claim–data confusions accounted for 1,528 of 2,091 errors (73%).Counterclaim and rebuttal had F1 scores of 0.604 and 0.638, with low precision but comparatively high recall, consistent with class-balanced resampling during training.Source: §8.2 · p. 6, Table 3 · p. 6

8.6 Summary of Experimental Findings

• The classifier achieved 82.58% accuracy and 0.727 macro-F1; the full scorer achieved mean QWK 0.813 over five folds, ranging from 0.754 to 0.872.The classifier was evaluated on 12,000 test sentences; scorer results use prompt-grouped cross-validation on the grade-independent feature table with gold discourse annotations.Source: Table 4 · p. 6, Table 2 · p. 6, §8.3 · p. 6

8.3 Scorer Evaluation

• Adding gold discourse features raised QWK by +0.055 over lexical-plus-syntactic features (0.813 vs. 0.759; paired t-test p = 0.010) in the oracle component evaluation.All five prompt-aligned fold differences were positive; the result is descriptive, not evidence of the downstream classifier’s deployment gain.Source: §8.3 · p. 6, §8.1 · p. 6

8.4 Feedback Generation and Evaluation Protocol

• The feedback generator uses predicted discourse labels and feature bands to produce label-aware revision suggestions, but human-annotator results are deferred.The repository defines an evaluation protocol for relevance, actionability, and tone.Source: §8.1 · pp. 5–6, §8.4 · p. 6

8.5 Latency Benchmarks

• Component latency was measured over 100 runs on one RTX 5000 Ada, but it is not full-pipeline latency; the classifier used 16.7 GB peak allocated GPU memory for a 21-sentence essay.The Qwen2.5-14B feedback backend was not benchmarked; the summary reports scorer-only CPU latency below 10 ms and classifier latency of approximately 1.3 seconds.Source: §8.5 · pp. 6–7, Table 5 · p. 7, §8.6 · p. 7

8.6 Summary of Experimental Findings

• The release’s offline test suite passed 22/22 tests.The classifier and scorer used separate evaluation protocols; the reported scorer result does not combine predicted classifier outputs with scorer performance.Source: §8 · p. 5, §8.6 · p. 7

10 Testing and Quality Assurance

• The offline test suite covers eight system aspects, uses a fake engine for the vLLM API contract test, and passes all 22 tests without a GPU or network connection.Table 7 summarizes test coverage.Source: §10 · p. 7, Table 7 · p. 7

11 Conclusion

• ArguLens decouples discourse classification, feature-based scoring, and feedback generation so each component can be developed, tested, and replaced independently.The system is presented as an open-source modular system for automated essay scoring and label-aware feedback.Source: §11 · p. 7
• 82.58% accuracy and a macro-F1 of 0.727 were achieved by the logit-probe classifier on PERSUADE 2.0, while the 31-feature LightGBM scorer obtained a QWK of 0.813 over 5-fold CV.In an oracle component evaluation, gold discourse features added +0.055 QWK over lexical and syntactic features alone (paired t-test, p = 0.010); this is not the gain from predicted classifier features.Source: §11 · p. 7
• Classification took approximately 1.3 seconds and scoring 6–9 ms in the reported component benchmarks, but feedback-backend and full-pipeline latency were not benchmarked.The feedback generator has a defined evaluation protocol, while a human-annotator study of feedback actionability remains future work.Source: §11 · p. 7
• Extending the pipeline to multilingual argumentative essays and conducting larger-scale human evaluation of feedback actionability across diverse learner populations are identified as next steps.The proposed multilingual direction includes XLM-RoBERTa for discourse classification and multilingual LLMs for feedback.Source: §11 · pp. 7–8
• Code and release metadata are provided under Apache 2.0, while third-party models and data retain their upstream licenses.The evaluation protocol and model artifacts are released to facilitate community-driven extensions.Source: §11 · p. 8

• Training exclusively on US middle-school PERSUADE 2.0 essays limits expected generalization across genres, age groups, and languages, while demographic and curricular contexts may bias scores.The release reports no gender, race/ethnicity, or other subgroup metrics; future study should report subgroup support and uncertainty and evaluate score accuracy and feedback quality before educational deployment. Until then, use is intended for research and assistive classrooms, not high-stakes decisions.Source: §12 · p. 8
• Evaluation covers completed essays, not partial drafts, and the scorer’s single holistic 1–6 target omits rater-level variation, limiting interpretation of borderline cases.Argument-mining models may be less robust with incomplete context even when they perform well on completed essays.Source: §12 · p. 8
• +0.055 QWK is an oracle component diagnostic, not an end-to-end result, because scorer cross-validation uses gold discourse annotations while the classifier is evaluated separately.The classifier uses an essay-disjoint sentence-level split; an end-to-end prompt-held-out rerun requires licensed raw essay text. Comparisons with ordinal logistic regression, prompting baselines, and fine-tuned encoder classifiers remain future work.Source: §12 · p. 8
• The classifier benchmark allocated 16.7 GB and reserved 18.5 GB VRAM on one RTX 5000 Ada; the scorer runs on CPU, and local 14B feedback deployment requirements were not measured.The feedback backend can call an OpenAI-compatible service, and LoRA weights must be downloaded separately and verified against the SHA-256 checksum before first use.Source: §12 · p. 8

• The code is released under Apache 2.0, and LoRA weights are available in the v0.1.0 release.Source: A Reproducibility Checklist · p. 9
• The checklist identifies Qwen2.5-7B-Instruct and Qwen2.5-14B-Instruct models under Apache 2.0, and PERSUADE 2.0 data under its upstream non-commercial license.Source: A Reproducibility Checklist · p. 9
• Reproduction materials include version-pinned frontend, vLLM, and training requirements, model checksums, and offline tests run with pytest -q.Source: A Reproducibility Checklist · p. 9
• The launch script autodetects virtual environments, GPUs, and ports, and the checklist provides the LoRA training configuration path.Source: A Reproducibility Checklist · p. 9
• Split statistics document an essay-level split using seed 42, with zero overlap between splits.Source: A Reproducibility Checklist · p. 10

B Full Feature List

• The 31 scorer features are grouped by family, and lexical-diversity variants are computed over all words, content words, and function words.Table 8 enumerates the full feature list.Source: B Full Feature List · p. 10, Table 8 · p. 10