PaperlayerView exact revision on arXiv

Source-linked AI summary

Rubric Spans are Label Representations: Joint LLM Encoding for Short Answer Scoring

Zhifan Sun, Sebastian Gombert, Fabian Zehner, Leon Camus, Longwei Cong, Hendrik Drachsler

arXiv:2610.09660v1cs.CL

TL;DR

ASAS must score answers against question-specific criteria despite variation in score schemes and the need for efficient, transferable models. RUSPAN jointly encodes rubric levels with the answer, and RIM restricts rubric-span attention; RIM with position reindexing yields substantial transfer gains on PT-ASAG under combined language and rubric-structure shift.

  • Problem

    ASAS scoring schemes vary across questions and benchmarks, while joint reasoning over questions, sample solutions, student answers, and rubric criteria remains underexplored.

    Source: §1 · p. 1, §2.2 · p. 3

  • Method

    RUSPAN scores rubric-span representations alongside a whole-sequence representation in one forward pass, while RIM prevents rubric spans from attending to one another.

    Source: §1 · p. 2, §3.2 · p. 4, §6 · p. 9

  • Results

    RIM with position reindexing yields consistent and substantial gains on PT-ASAG under combined language and rubric-structure shift.

    Source: §6 · p. 9

  • Takeaways & Limitations

    Across six benchmarks, RUSPAN offers competitive mono-benchmark scoring while avoiding pairwise answer-rubric expansion and autoregressive score generation.

    Source: §6 · p. 9

    Public benchmarks do not fully capture classroom assessment complexity, and preventing rubric-span interactions may remove useful information when score levels are meaningfully related.

    Source: §7 · p. 10

Abstract

from arXiv · show

Automatic Short Answer Scoring (ASAS) requires models that can score student responses against question-specific criteria while remaining efficient and transferable across rubric sets. We propose RUSPAN, a rubric-conditioned ASAS framework that treats rubric descriptions as semantic label representations. RUSPAN serialises the question context, student answer, and all candidate rubric levels into a single sequence, then scores the levels listwise from the rubric-span and whole-sequence representations produced in a single LM pass. We further introduce RUSPAN-RIM, in which a Rubric-Independent Mask prevents rubric spans from attending to one another, making rubric representations depend only on the answer and question context and preventing overfitting to rubric patterns during training for zero-shot transfer. On six ASAS benchmarks spanning English, German, and Portuguese, RUSPAN improves mono-benchmark scoring over discriminative and generative baselines, while RIM with position reindexing delivers consistent and substantial gains on PT-ASAG, the held-out benchmark with the strongest combined language and rubric-structure shift.

1 Introduction

ASAS models must handle scoring schemes that vary across questions and benchmarks, while fixed-label classifiers and rubric-based alternatives face transfer or efficiency constraints. RUSPAN scores rubric levels jointly in one pass, and RIM improves transfer under combined language and rubric-structure shift.

1 Introduction

  • Fixed-head classifiers perform well within a benchmark but do not directly transfer when a new question or dataset uses a different scoring scheme.Their output dimensions correspond to one dataset’s score levels.Source: §1 · p. 1
  • Rubric retrieval uses score-level descriptions as semantic labels, but expands each answer into multiple answer-rubric pairs, increasing training and evaluation cost.RUSPAN builds on the insight that rubric spans can serve as label representations; its whole-sequence representation is more effective than answer-rubric alignment.Source: §1 · pp. 1–2
  • RUSPAN represents each score level with its rubric span and scores variable numbers of levels in a single forward pass.Its sequence-rubric alignment variants pair rubric spans with sequence representations.Source: §1 · p. 2
  • RIM blocks rubric spans from attending to one another, making their representations depend on the answer and task context while retaining single-pass efficiency.The design addresses standard causal attention conditioning rubric representations on preceding rubrics and approaches rubric-set invariance of retrieval.Source: §1 · p. 2
  • RUSPAN with RIM delivers consistent, substantial gains under combined language and rubric-structure shift, while retrieval remains competitive on English-only cross-benchmark targets.The evaluation covers six public ASAS benchmarks and compares against sequence classification, rubric retrieval, and LLM generation baselines.Source: §1 · p. 2

2 Related Work

Prior ASAS methods use fixed labels, rubric or reference-text comparisons, and generative LLM scoring, each with distinct flexibility and efficiency trade-offs. Decoder-only LLM encoders may support ASAS, but joint reasoning across its heterogeneous inputs remains underexplored.

2.1 Automatic Short Answer Scoring

  • Fixed-label transformer classifiers work when scoring schemes are stable, but often fail to generalize to new questions or datasets as schemes vary.Rubric descriptions offer semantic grounding, while rubric and reference-answer comparisons trade flexibility for the cost of expanding each answer into multiple pairs.Source: §2.1 · pp. 2–3

2.2 LLMs as Encoders

  • Decoder-only LLMs can encode text for classification without generation, but their ability to reason jointly over questions, solutions, answers, and rubric criteria remains underexplored.Prior encoder adaptation has largely focused on single-sequence and sentence-pair tasks.Source: §2.2 · p. 3

2.3 Label Semantics for NLU

  • Label verbalization and NLI-style classification express labels as text; rubric retrieval applies a related text-label formulation by testing answers against rubric criteria.These approaches replace purely index-based classification, which has been effective in zero- and few-shot settings.Source: §2.3 · p. 3, §2.1 · p. 3

3 RUSPAN Architecture and Benchmark Augmentation

RUSPAN serializes task context, student answers, and candidate rubrics for single-pass, rubric-conditioned scoring. Its architecture extracts and aligns rubric-level representations, while RIM blocks cross-rubric attention and the benchmarks are augmented where needed.

3.1 Input Formulation

  • RUSPAN forms each input by serializing the question, sample solution, student answer, concatenated rubric text, and sampled instruction into one sequence.An instruction is randomly sampled from ten templates during training and testing; multilingual instructions use the dataset language.Source: §3.1 · p. 3

3.2 Rubric Span Extraction and Sequence Representation

  • Each rubric level is represented by its span’s final-token state, while the final input token supplies a whole-sequence representation.The rubric span boundaries are tracked in the formatted input before language-model encoding.Source: §3.2 · pp. 3–4

3.3 Label-to-Sequence Alignment Functions

  • Four alignment functions combine rubric and sequence representations in three cases, while NO-ALIGN uses the rubric representation directly; a shared head scores each level.The shared scoring head projects each resulting representation to a scalar score.Source: §3.3 · p. 4

3.4 Rubric-Independent Mask (RIM)

  • Under standard causal attention, each rubric representation depends on preceding rubrics, a conditioning that degrades transfer to unseen benchmarks with different rubric patterns.RIM blocks attention between rubric spans so each attends only to the context and itself; position reindexing also makes rubric-token position embeddings independent of other rubrics.Source: §3.4 · p. 4, Figure 2 · p. 5

3.5 Training Objective

  • Training uses cross-entropy over gold labels and masks padded rubric-level slots so they do not affect normalization.The validity mask supports benchmarks with different label cardinalities and batches containing different numbers of rubric levels.Source: §3.5 · p. 5

3.6.2 Benchmark Augmentation

  • LLMs synthesize per-level rubric descriptions for datasets without native rubric annotations, preserving their original labels.For ASAP-SAS, which lacks sample solutions, LLMs also generate sample solutions.Source: §3.6 · p. 5, §3.6.2 · p. 5

3.6.1 Source Benchmarks

  • The evaluation uses six public ASAS benchmarks, with Table 1 summarizing their languages, available context, dataset sizes, and test splits.The supplied text specifies that the original three-way SCIENTSBANK and BEETLE labels are recast as correct, partially correct, and incorrect, mapping contradictory, irrelevant, and non_domain to incorrect.Source: §3.6.1 · p. 5

4 Experimental Setup

The experimental setup compares RUSPAN with classification, rubric-based, and generative baselines, using both per-benchmark training and joint training with cross-benchmark evaluation. The protocol distinguishes same-language transfer from evaluation under stronger language or rubric shifts.

4.1 Baselines

  • Baselines include fixed-label sequence classification, rubric retrieval, answer-rubric alignment, fine-tuned score generation, and zero-shot score generation.Sequence classification excludes ASAP-SAS because its number of score levels varies by question.Source: §4.1 · p. 5

4 Experimental Setup

  • The setup reports six ASAS benchmarks’ languages, available context, dataset sizes, and test splits; detailed hyperparameters and model choices are in Appendix E.Source: §4 · p. 5, Table 1 · p. 6

4.2 Experimental Protocol

  • Mono-benchmark experiments train and evaluate separately on each benchmark, while joint training uses English benchmarks except ISTUDIO and tests transfer on ISTUDIO, PT-ASAG, and ALICE-LP.ISTUDIO tests same-language transfer; PT-ASAG and ALICE-LP test transfer under stronger language or rubric shifts.Source: §4.2 · p. 6

5 Results and Discussion

Mono-benchmark results show RUSPAN is competitive across settings, though no alignment variant or method wins uniformly. In cross-benchmark transfer, RIM with position reindexing improves results most consistently under stronger shifts, while per-level analysis shows that label frequency and rubric difficulty shape performance.

5.1 Mono-Benchmark Results

  • At least one RUSPAN alignment variant is strongest on ASAP-SAS across all four backbones, but sequence classification and rubric retrieval remain competitive and win in some settings.No single alignment function is universally optimal; CONDIFF performs generally well.Source: §5.1 · p. 6
  • RUSPAN generally outperforms Trained Generation in mono-benchmark scoring; with the 1B backbone on ALICE-LP, it scores 66.5 macro-F1 versus 46.5.RUSPAN’s whole-sequence conditioning is more effective than answer-rubric alignment in most reported settings.Source: §5.1 · p. 6

5.2 Joint Training and Cross-Benchmark Results

  • Standard RUSPAN transfers unevenly, with PT-ASAG especially difficult under English training data and mostly three-level scales; it falls behind baselines there.It remains broadly competitive with cross-benchmark baselines on ALICE-LP, matching or exceeding Trained Generation for the 3B and 8B backbones.Source: §5.2 · pp. 6–7
  • RIM matches or exceeds the strongest baseline in 7 of 12 combinations and improves over standard RUSPAN in 11 of 12; PT-ASAG gains are 6.9 to 12.5 macro-F1 points.With the 8B backbone, reindexed RIM is strongest on all three transfer targets.Source: Table 3 · p. 7, §5.2 · pp. 7–8
  • RUSPAN is modestly slower than fixed-head classifiers but much faster than retrieval and generation methods, while retaining competitive mono-benchmark accuracy.Figure 3 reports relative evaluation throughput on ISTUDIO, normalized by RUSPAN within each model size.Source: §5.2 · p. 8, Figure 3 · p. 8

5.3 Per-Level Analysis

  • Majority or near-majority levels consistently achieve higher F1, while rare levels remain harder; PT-ASAG level 3 has roughly 13% of training examples.In roughly balanced BEETLE, SCIENTSBANK, and ALICE-LP, intermediate levels are consistently hardest across methods.Source: §5.3 · p. 8
  • Trained Generation degrades more sharply on higher score levels; on BEETLE, its level 1 F1 is 0.27 versus 0.53 for RUSPAN.Because rubrics are presented in ordinal order, the highest score level appears last in the input sequence.Source: §5.3 · p. 8
  • RUSPAN achieves the highest or near-highest F1 on the hardest levels in SCIENTSBANK, BEETLE, ALICE-LP, and PT-ASAG.On PT-ASAG, reindexed RIM improves F1 on all four levels, with the largest relative gains on the harder higher-score levels.Source: §5.3 · p. 8

6 Conclusion

RUSPAN represents score levels as rubric spans jointly encoded with student answers, scoring all candidate levels in one forward pass. Across six benchmarks it offers competitive mono-benchmark performance, while RIM with position reindexing improves transfer to PT-ASAG under combined language and rubric-structure shift.

6 Conclusion

  • Across six ASAS benchmarks, RUSPAN delivers competitive mono-benchmark performance without pairwise answer-rubric expansion or autoregressive score generation.It scores variable score spaces that fixed-head classifiers cannot accommodate.Source: §6 · p. 9
  • RUSPAN represents score levels as rubric spans jointly encoded with the student answer and scores all candidates in a single forward pass.This offers a flexible and efficient alternative within the tradeoff among flexibility, accuracy, and efficiency in prior ASAS approaches.Source: §6 · p. 9
  • RIM with position reindexing yields consistent and substantial gains on PT-ASAG, the held-out benchmark with the strongest combined language and rubric-structure shift.The approach mitigates inter-rubric attention dependence during cross-benchmark transfer.Source: §6 · p. 9
  • Future work should explore richer feedback generation, multi-trait scoring, and robustness to noisy or underspecified rubrics.Source: §6 · p. 9

7 Limitations

The study’s claims concern how models use natural-language score-level descriptions, not whether generated rubrics are pedagogically optimal. Its scope is further bounded by limited context in public datasets and the controlled tasks in public ASAS benchmarks.

7 Limitations

  • Rubric annotation quality is a central limitation, and rubric generation is not the object of study.The claims concern using natural-language descriptions as score-level representations, not the pedagogical optimality of generated rubrics.Source: §7 · p. 9
  • Public ASAS datasets usually lack the curriculum goals, instructional materials, grading policies, and broader pedagogical context available to trained educators writing rubrics.That missing context can yield descriptions that are underspecified, overly generic, or misaligned with intended criteria.Source: §7 · p. 10
  • Public ASAS benchmarks are controlled short-answer tasks and do not fully capture classroom assessment involving multi-trait scoring, richer feedback, or longer and more diverse student work.Source: §7 · p. 10
  • Cross-benchmark transfer remains uneven under language and rubric-structure shifts, and preventing interactions among rubric spans may remove useful information when score levels are meaningfully related.Source: §7 · p. 10

A Benchmark Augmentation

The appendix describes benchmark augmentation, model input formatting, multilingual instructions, zero-shot prompts, and experiment design choices. For benchmarks without native materials, it uses generated reference answers or rubric descriptions to support rubric-conditioned evaluation.

A Benchmark Augmentation

  • Benchmarks lacking reference answers or rubrics are augmented with gpt-4o-mini-generated materials.For ASAP-SAS, a reference answer is generated from each question; rubrics are 2–3 sentences per score level and use question text, reference answers, up to 10 training answers per level, and target labels.Source: A Benchmark Augmentation · p. 12

B Input Examples

  • The ASAP-SAS input example includes the question, sample solution, student answer, candidate rubric labels, and a scoring instruction; the final > token pools each rubric representation.Source: Figure 8 · p. 14

C Multilingual Input Format and Instruction Pool

  • For multilingual benchmarks, field tags and scoring instructions are translated into the dataset language, while multibenchmark training uses English formatting.The rubric tag remains in English, and one of 10 scoring instruction templates per language is randomly sampled for each instance at training and test time.Source: C Multilingual Input Format and… · p. 14, Table 5 · p. 14, Table 6 · p. 15

D Zero-shot Prompt Templates

E Hyperparameters and Model Choices

E.1 Mono-Benchmark Training

Mono-benchmark experiments fine-tune the listed backbones with LoRA and select checkpoints by development macro-F1. Training duration and learning rates vary by model, and retrieval evaluation is omitted for two larger backbones because pair expansion is prohibitively expensive.

E.1 Mono-Benchmark Training

  • The experiment backbones are Llama-3.2-1B-Instruct, Llama-3.2-3B-Instruct, Mistral-7B-v0.1, and Llama-3.1-8B-Instruct.Source: §E.1 · p. 16
  • RUSPAN, Standard Classification, and Rubric Retrieval use LoRA with rank 64, α = 64, dropout 0.1, adapters on all linear modules, maximum sequence length 2048, and effective batch size 16.Optimization uses a cosine learning-rate schedule with warmup over the first 1% of training steps.Source: §E.1 · p. 16
  • Checkpoints are selected by highest development macro-F1; ALICE-LP is trained for 3 epochs and other models for 5.For benchmarks without an official development split, a held-out subset is sampled from training for checkpoint selection.Source: §E.1 · p. 16
  • Mistral-7B-v0.1 and Llama-3.1-8B-Instruct are not evaluated with Rubric Retrieval because expanding instances into answer-rubric pairs is prohibitively expensive at this scale.Source: §E.1 · p. 16
  • Generated labels are decoded greedily at temperature 0; Llama-3.2-1B-Instruct and Llama-3.2-3B-Instruct train for 3 epochs with learning rate 1 × 10^-4.Other hyperparameters follow the preceding settings.Source: §E.1 · p. 16

E.2 Joint Training and Cross-Benchmark Evaluation

The joint-training setup uses two epochs, and the rubric-quality experiments show that generated rubrics, appended rubric text, and detailed rubric descriptions do not help uniformly across settings. The results point to sensitivity to rubric quality, semantic encoding, and whether questions were seen during training.

E.2 Joint Training and Cross-Benchmark Evaluation

  • Jointly trained models use two epochs, selected to maximize performance on unseen benchmarks; other hyperparameters follow the mono-benchmark setup.Source: §E.2 · p. 16

F.1 Rubric Quality and Use

  • The experiments test appending rubrics to a fixed-head classifier, replacing detailed rubrics with label names, and replacing human-written rubrics with generated ones.PT-ASAG is excluded from the label-name comparison because it has numerical scores rather than concrete level names.Source: §F.1 · pp. 16–17
  • Appending rubric text to Sequence Classification is insufficient: removing it often leaves performance unchanged or improves it, including on SCIENTSBANK 1B and ISTUDIO 3B.The authors argue rubric text is useful when explicitly bound to candidate score levels rather than simply appended to a fixed-head classifier input.Source: §F.1 · p. 17
  • Across alignment functions, DIFF underperforms with detailed generated rubrics, indicating limited semantic representation capacity in those rubrics.Source: §F.1 · p. 17
  • On ISTUDIO-UA, detailed generated rubrics appear less beneficial when test questions were seen during training, where pattern recognition can substitute for semantic rubric understanding.The authors present this explanation as a hypothesis.Source: §F.1 · p. 17
  • Generated rubrics have mixed effects on ASAP-SAS and underperform human-written rubrics in most ALICE-LP UQ settings.The authors note that rubrics generated from public-dataset information may omit task-specific distinctions, scoring conventions, and question expectations.Source: §F.1 · p. 17

F.2 Sample-Solution Ablation

The sample-solution ablation finds little average change when the generated solution is removed, though effects vary by model and alignment function. The ablation is limited to ASAP-SAS, and the solution is retained for input-format consistency.

F.2 Sample-Solution Ablation

  • The sample-solution ablation applies only to ASAP-SAS, the sole benchmark with an LLM-generated sample solution.Source: §F.2 · p. 17, Table 10 · p. 17
  • −0.1 macro-F1 on average when the generated sample solution is removed from the full-input setting.The ablation reports changes relative to the full-input setting and applies only to ASAP-SAS.Source: §F.2 · p. 17, Table 10 · p. 17
  • The authors retain the sample solution to keep ASAP-SAS input formatting consistent with the other ASAS benchmarks.Source: §F.2 · p. 17
  • Tables 7 and 8 compare human-written and LLM-generated rubrics qualitatively for one ASAP-SAS item and one ALICE-LP item.Table 8 shows the original German text alongside an English translation.Source: Table 7 · p. 18, Table 8 · p. 18
  • Table 9 reports macro-F1 ablations for detailed-versus-short generated rubrics, human-versus-generated rubrics, and Sequence Classification without appended rubric text.The caption describes the compared conditions but does not provide their result values.Source: Table 9 · p. 18

H Non-ASAS Datasets

The non-ASAS evaluation tests rubric-based label representations across seven classification benchmarks, including conditions with unseen targets and cross-language transfer. RUSPAN variants improve over fixed-head classification in the unseen-target stance setting and remain competitive when rubric descriptions add little signal.

H Non-ASAS Datasets

  • The benchmark suite spans commonsense reasoning, sentiment, news topics, stance detection, figurative-language interpretation, and scientific edit intent.Examples include PIQA, IMDB, AG News, SemEval-2016 Task 6, FigQA, and EIC.Source: H Non-ASAS Datasets · p. 19
  • The comparison includes Sequence Classification with and without rubric text and four RUSPAN variants; PIQA and FigQA cannot be tested without candidate-solution rubric text.Source: H Non-ASAS Datasets · p. 19
  • xStance tests transfer from German and French training data to Italian test data, while SemEval-2016 Task 6 includes both seen and unseen targets.Source: H Non-ASAS Datasets · p. 19
  • IMDB and AG NEWS achieve near-identical macro-F1 across methods, at 94 to 97%, regardless of whether rubric text is present or how it is encoded.The authors describe these as saturation tasks whose short label names are recoverable from training data.Source: H Non-ASAS Datasets · pp. 19–20
  • On the unseen Donald Trump target, 3B Sequence Classification without rubrics scores 42.2, versus 70.2 for RUSPAN-CONDIFF and 69.6 for RUSPAN-NO-ALIGN; Sequence Classification with rubrics scores 49.6.Table 11 reports macro-F1 results for the non-ASAS benchmarks.Source: Table 11 · p. 20, H Non-ASAS Datasets · p. 20
  • Where Sequence Classification without rubrics is competitive, RUSPAN variants match or come within 1 point, with no systematic penalty when rubrics add no discriminative signal.The cited examples include EIC, IMDB, and AG News.Source: H Non-ASAS Datasets · p. 20
Loading 2610.09660v1…