Source-linked AI summary
Alice: A Large-Scale German Benchmark for Rubric-Based Multi-Dimensional Automatic Short Answer Scoring
Zhifan Sun, Sebastian Gombert, Jannik Lossjew, Tobias Wyrwich, Berrit Katharina Czinczel, David Bednorz, Marcus Kubsch, Knut Neumann, Hendrik Drachsler
TL;DR
Existing ASAS benchmarks mostly evaluate direct answers, with limited coverage of concept mastery and epistemic skills. ALICE adds three rubric-based German scoring dimensions and benchmarks rubric retrieval and other approaches; results favor rubric retrieval and show that rubric information is particularly useful for KE and SK.
Problem
Public ASAS benchmarks are scarce, and existing datasets largely assess direct answers rather than mastery of concepts and epistemic activities.
Source: §1 · pp. 1–2Method
Results
LLM encoders achieve the best rubric-retrieval results across subtasks, while rubric text more consistently helps KE and SK than LP.
Source: §5.2 · pp. 8–9Takeaways & Limitations
Supervised rubric retrieval remains the most reliable configuration, and rubric descriptions are especially useful for fine-grained KE and SK scoring.
Source: §5.2 · p. 9ALICE focuses on German-language scientific education, so results may not directly transfer to other languages or subject domains.
Source: §7 · p. 9
Abstract
from arXiv · showhide
Automatic Short Answer Scoring (ASAS) is central to NLP for Education. However, openly available benchmarks remain scarce, and existing datasets largely address how well students answer a question directly rather than how well they master underlying concepts (knowledge elements) such as thermal energy or epistemic activities (skills) such as reasoning or claim. To address this gap, we introduce Alice, a large-scale, rubric-based German ASAS dataset that is pedagogically aligned and comprises three subtasks: (i) learning performance (Alice-LP), (ii) knowledge elements (Alice-KE), and (iii) skills (Alice-SK). We further formulate rubric-based ASAS as a rubric-retrieval task and benchmark the dataset with a range of language models, from encoder-only models to lightweight LLMs. We also benchmark the dataset with zero-shot prompting via LLMs and a standard classification baseline. The experiments show that LLMs, in particular, struggle to score knowledge elements and skills in the zero-shot setting. They also indicate that rubric text is often useful, especially for Alice-KE and Alice-SK, while on Alice-LP gains over sample-solution-focused inputs are more modest and vary by model and input format.
1 Introduction
Existing ASAS benchmarks emphasize direct answer correctness, leaving question-specific rubrics and assessment of concept mastery and epistemic skills underrepresented. ALICE expands a German benchmark across three dimensions and evaluates rubric retrieval alongside classification and prompting approaches.
1 Introduction
- Existing benchmarks often assume uniform score levels across questions, despite teachers using question-specific rubrics shaped by pedagogical needs.They also commonly focus on direct responses rather than mastery of scientific concepts and epistemic activities.Source: §1 · p. 2
- ALICE expands ALICE-LP 1.1 with additional responses and annotations for learning performance, knowledge elements, and skills.LP measures how well students answer the question; KE captures target-concept mastery; SK evaluates epistemic activities.Source: §1 · p. 2
- ALICE instances pair each question and sample solution with per-level rubrics for learning performance, knowledge elements, and skills.The rubric structure is intended to align with how teachers evaluate responses.Source: Figure 1 · p. 1, §1 · p. 2
- The benchmark frames scoring as retrieval over question-specific rubric candidates and compares it with sequence classification and zero-shot prompting.The retrieval formulation accommodates rubrics whose levels vary across questions.Source: §1 · p. 2
- MLMs more often degrade with added question and solution context, whereas LLM encoders more often benefit, suggesting they use richer rubric context more reliably.The paper analyzes the effects of rubric information and optional context across the expanded setting.Source: §1 · p. 2
2 Related Work
Prior ASAS benchmarks are scarce, especially publicly available non-English resources, and largely score direct question responses rather than underlying concepts or skills. Modeling has moved toward pretrained language models, with rubric use and LLM prompting emerging alongside solution-focused approaches.
2.1 Short Answer Scoring: Benchmarks
- Public ASAS benchmarks, particularly non-English ones, remain scarce, and most assess how well responses address questions rather than concept or skill mastery.The section identifies SCIENTSBANK and ASAP-SAS2 as widely used datasets and notes that privacy leads some work to use proprietary data.Source: §2.1 · p. 2
- Solution-based scoring compares an answer with the reference answer, while rubrics describe answers at each performance level and can characterize partially correct responses.Rubrics are defined as natural-language descriptions of student answers at each performance level.Source: §2.1 · p. 3
- Among public rubric-based benchmarks, ASAP-SAS and ALICE use exclusive rubric matching, whereas RICECHEM and SAS-BENCH sum scores from matched rubrics.The latter datasets assume additive relations among rubrics.Source: §2.1 · p. 3
2.2 Short Answer Scoring: Modelling
- Pretrained language models shifted ASAS toward jointly encoding answers and sample solutions for similarity or entailment, while some work also incorporates rubrics.Examples of rubric use include Wang et al. (2019) and Sonkar et al. (2024).Source: §2.2 · p. 3
- LLM prompting avoids task-specific fine-tuning but typically relies on large proprietary models with high inference costs and latency.The section argues that ASAS requires classification or ranking rather than freeform generation, for which encoder-based models can produce alignment scores in one forward pass.Source: §2.2 · p. 3
3 The ALICE Dataset
ALICE extends German learning-performance scoring with expert-designed knowledge-element and skill rubrics, and evaluates answers across three complementary dimensions. The dataset uses a question-held-out split; its rubric levels often overlap semantically, while LP correlates with but remains distinct from KE and SK.
3.1 Collection and Definition
- ALICE extends ALICE-LP 1.1 with additional responses and fine-grained annotations for learning performance, knowledge elements, and skills.LP scores direct response to the prompt; KE assesses targeted concept use; SK captures specified cognitive or reasoning behaviors.Source: §3.1 · pp. 3–4
3.2 Annotation
- Subject specialists designed KE/SK dimensions, and expert annotators scored responses through four subject-specific phases with pilot feedback refining guidelines and rubrics.Agreement is reported by subject and scoring dimension using quadratic weighted kappa.Source: §3.2 · p. 4, Table 2 · p. 4
3.3 Semantic Overlap between Rubric Levels
- Adjacent rubric levels across the three subtasks usually do not contradict one another but often overlap semantically.The analysis applies NLI with higher-level rubrics as premises and lower-level rubrics as hypotheses.Source: §3.3 · pp. 4–5, Figure 2 · p. 5
3.4 Dataset Statistics
- The corpus contains 16,572 answers and uses Test-UQ for 22 held-out questions and Test-UA for unseen answers from the remaining 90-question pool.The split differs from ALICE-LP 1.1, so results are not directly comparable with published baselines for that release.Source: §3.4 · p. 5
- LP–KE correlation is r = 0.785, p < 0.001, and LP–SK correlation is r = 0.631, p < 0.001, yet the dimensions capture distinct aspects of student understanding.High or low KE/SK scores do not necessarily correspond to high or low LP scores.Source: §3.4 · p. 5
4 Modelling Approach
The modelling approach treats rubric-based scoring as retrieval over question-specific rubric candidates, using joint answer–rubric encoding and comparing it with sequence-classification and zero-shot prompting baselines.
4 Modelling Approach
- The encoder experiments test whether lightweight LLMs can use rubric context and generalize to unseen questions as discriminative encoders.This setup avoids autoregressive decoding costs.Source: §4 · p. 5
- Rubric retrieval jointly encodes each student response with a rubric item, optionally adding the question and sample solution as context.The benchmark also tests lightweight decoder-only LLMs as encoders, without autoregressive decoding.Source: §4 · pp. 5, 7
- Cross-encoders jointly represent answer–rubric pairs, enabling direct cross-attention; their extra encoding cost is negligible for the small candidate sets.The authors choose this over bi-encoders, which encode answers and rubrics separately, because joint encoding supports richer alignment features and variable rubric sets.Source: §4 · p. 5
- Softmax cross-entropy contrasts the correct rubric against alternatives because rubric items can overlap and are not additive.Additional alignment scores are masked when training batches contain different numbers of rubric levels.Source: §4 · p. 6
- The sequence-classification baseline concatenates the student answer and sample solution and predicts with a classification head.For KE and SK, it uses globally assigned level indices and omits levels undefined for a given question.Source: §4 · p. 7
- Zero-shot prompting tests open-weight and API-hosted LLMs with full context plus rubrics, no context plus rubrics, or label names alone.The broader input-format comparison contrasts base and full-context configurations across modelling approaches.Source: §4 · p. 7
5 Benchmarking on ALICE
The benchmark compares rubric-retrieval, sequence classification, and zero-shot prompting across three ALICE subtasks and unseen-answer/question settings. Results favor rubric-retrieval most clearly on unseen questions for knowledge elements and skills, while context benefits vary by task and approach.
5.1 Experimental Setup
- Evaluation spans ALICE-LP, ALICE-KE, and ALICE-SK using unseen-answer and unseen-question tests, with the latter testing generalisation to entirely unseen questions.The comparison includes two MLMs and four lightweight LLM encoders, with and without question and sample-solution context.Source: §5.1 · p. 7
5.2 Results and Discussion
- On unseen questions, rubric-retrieval gains are consistent across subtasks and larger for ALICE-KE and ALICE-SK; on unseen answers, sequence classification remains competitive on ALICE-LP.The main Macro-F1 table covers fine-tuned unseen-answer and unseen-question results, plus zero-shot unseen-question prompting.Source: §5.2 · pp. 7–8, Table 4 · p. 8
- Lightweight LLM encoders produce the best fine-tuned rubric-retrieval results across subtasks; proprietary zero-shot models are competitive, but supervised rubric-retrieval remains most reliable.Llama-3.2-3B-Instruct is strongest on ALICE-LP UQ, while Llama-3.2-3B leads on ALICE-KE and ALICE-SK UQ.Source: §5.2 · pp. 8–9
- Additional context benefits ALICE-KE and ALICE-SK more consistently than ALICE-LP in fine-tuned approaches, while full question context can hurt smaller open-weight models in zero-shot scoring.Rubric descriptions often improve zero-shot results, especially for stronger proprietary models.Source: §5.2 · p. 9
6 Conclusion
ALICE combines a three-dimension German benchmark with rubric-retrieval scoring. Experiments find clearer retrieval advantages on knowledge elements and skills than on learning performance, with rubric text especially useful for those dimensions.
6 Conclusion
- ALICE covers learning performance, knowledge elements, and skills, and its rubric-retrieval formulation accommodates question-specific rubric structures.The formulation is presented as a practical alternative to fixed-label sequence classification.Source: §6 · p. 9
- Rubric-retrieval and sequence classification are both competitive on ALICE-LP, while retrieval shows clearer advantages on ALICE-KE and ALICE-SK.For fine-tuned models, rubric information is particularly important for KE and SK; for LP, its benefit over sample-solution-focused scoring is smaller, while zero-shot prompting benefits from per-level rubric text across all subtasks.Source: §6 · p. 9
7 Limitations
The study is limited to German-language scientific education and discriminative models, while rubric-retrieval also has computational and data requirements that constrain its practical scope.
7 Limitations
- ALICE focuses on German-language scientific education, so results may not directly transfer to other languages or subject domains.The dataset design is described as general, but the reported results have this scope boundary.Source: §7 · p. 9
- The study evaluates encoder architectures but does not explore generative scoring or feedback generation.The authors identify feedback generation and pedagogical capabilities of LLMs as future-work directions.Source: §7 · p. 9
- Rubric-retrieval expands each answer into multiple answer–rubric pairs, increasing training and inference costs and limiting scalability to larger models.Rubric-based scoring also depends on high-quality rubrics, which may not be accessible in real-world settings.Source: §7 · p. 9
8 Ethical Statement
The dataset was collected from routine classroom assessment items with privacy protections and monitored annotation quality; ALICE is intended for research, not as a replacement for expert judgment.
8 Ethical Statement
- Student responses came from routine classroom assessment items, and identifiers were removed before annotation to safeguard privacy.The data were obtained under educational governance and privacy protections, and responses were anonymised.Source: §8 · pp. 9–10
- Annotators were compensated, and annotation quality was monitored through pilot rounds, rubric revisions, and adjudication.Source: §8 · p. 10
- ALICE is intended for research, and benchmark models should support instructional practice rather than replace expert human judgment.The authors urge attention to bias, fairness, and interpretability and caution against uses that reinforce inequitable treatment.Source: §8 · p. 10
A Dataset Example
The appendix materials illustrate ALICE’s question-specific, multidimensional rubrics and describe input formats and rubric-entailment decisions. They show how example responses are assessed across learning performance, knowledge elements, and skills.
A Dataset Example
- A photovoltaic-cell example pairs a prompt, sample solution, and student answer with separate learning-performance, knowledge-element, and skill rubric levels.The learning-performance levels distinguish no statement, one statement, and two statements covering both angle and illuminated area; the example also omits level 1 from the KE rubric for this item and from the skill rubric.Source: A Dataset Example · p. 13
B Input Formats of LLMs
- Structured LLM input marks sequence boundaries with XML-style tags; the authors report better results than natural-language boundaries and use the structured format.Source: B Input Formats of LLMs · p. 14
Sequence Classification
- The example combines <Frage>, <Loesung>, <Antwort>, and <Rubrik> tags; its rubric rates no je-desto statements about angle or illuminated area as incorrect, one as partly correct, and two as correct.Source: Sequence Classification · p. 14
Rubric-Retrieval
- The rubric-retrieval examples pair the student answer with candidate rubric levels, including correct and partially correct criteria.Source: Rubric-Retrieval · p. 14
C Full KE and SK Items List
- The KE and SK tables list, by subject, domain concepts and inquiry competencies that are scored independently.Source: Table 5 · p. 15, Table 6 · p. 15
D Rubric Entailment Experiments
- Educational rubric entailment is behavioral: satisfying a higher-level criterion means satisfying weaker lower-level conditions, which standard NLI models struggled to identify.The authors used LLM prompting with a task-specific definition; its results were consistent with manual inspection of labelled pairs.Source: D Rubric Entailment Experiments · p. 16
G Additional QWK Results
The appendix adds per-subject Macro-F1 views for both test sets and complementary QWK results, alongside details of model configuration and validation.
G Additional QWK Results
- Figures 7 and 8 report per-subject Macro-F1 for each subtask, averaged across input formats within each approach.Figure 7 covers Test-UA for fine-tuned models only; zero-shot prompting was not evaluated there, while Figure 8 covers Test-UQ.Source: Figure 7 · p. 17, Figure 8 · p. 17
F Model Configuration and Hyperparameters
- Checkpoints are selected by validation accuracy on a 10% instance-sampled training subset, which is question-disjoint from Test-UQ but instance-sampled relative to Test-UA.Both the validation set and Test-UA are drawn from the same pool of 90 questions.Source: F Model Configuration and… · p. 17
- The fine-tuned comparison uses XLM-R-Longformer-base-4096, mmBERT-base, and Llama 3.2 models, while zero-shot prompting uses five instruction-tuned models with greedy decoding.MLMs are fine-tuned for four epochs; LLMs use 4-bit QLoRA and are trained for three epochs.Source: F Model Configuration and… · p. 17
- Table 7 presents complementary QWK results, separating fine-tuned UA and UQ panels from zero-shot UQ prompting.It marks best results, cells where sequence classification beats rubric retrieval, and full-context inputs that underperform base formats; dashes indicate unavailable QWK values.Source: Table 7 · p. 18
H Ablation Studies
This section presents ablation results for unseen answers and questions, alongside task-specific LLM prompt details. It also shows a user-prompt template that places rubric, question, and sample-solution placeholders before the student-answer placeholder.
H Ablation Studies
- The ablations compare contrastive-loss training with a bi-encoder that separately embeds answers and rubric elements, combining their embeddings by element-wise multiplication.The bi-encoder uses the same language-model encoder for both inputs and SCE loss; answer encoding can include the question and/or sample solution.Source: H Ablation Studies · p. 18
- On Test-UA, both contrastive-loss and separate-encoding variants are competitive, but they generally underperform rubric retrieval on Test-UQ, suggesting the proposed approach generalizes better to new questions.Several settings match or exceed the main LP configuration on Test-UA, while a few model-specific gains occur on Test-UQ.Source: Table 8 · p. 18, H Ablation Studies · p. 19, Table 9 · p. 19
I.1 System Prompts
- System prompts tailor scoring to overall task performance for LP, mastery of each knowledge element for KE, and whether each skill is present or absent for SK.Common rules direct scoring by rubric criteria, treating sample answers as optional aids and ignoring style unless the rubric requires it.Source: §I.1 · p. 20
I.2 User Prompt
- The user prompt assembles the element section and rubric set with optional question and sample-solution sections, followed by the student answer.Source: §I.2 · p. 21
- The user prompt includes element, rubric, question, and sample-solution placeholders, followed by a student-answer placeholder.Source: §I.2 · p. 21
J Benchmarking on ASAP-SAS
The paper benchmarks joint rubric-retrieval models on ASAP-SAS against prior methods that generally train per prompt. LLM encoders are competitive and outperform MLM encoders on average, while the joint approach trades peak per-prompt performance for generality.
J Benchmarking on ASAP-SAS
- The ASAP-SAS benchmark covers diverse domains, and the experiments compare basic answer-plus-rubric and answer-plus-question input formats.Prior methods in the comparison use separate models for each question.Source: J Benchmarking on ASAP-SAS · p. 21
- Earlier feature-driven systems score answers using engineered lexical and semantic features, often with a separate model for each question.The described methods include lexical overlap and semantic similarity features, as well as lexical diversity, embeddings, and overlap with prompt or reference content.Source: J Benchmarking on ASAP-SAS · p. 21
- Prior neural approaches include answer-only CNN/LSTM scoring, rubric-aware alignment, and similarity-based comparisons to anchor answers from different levels.These methods respectively use prompt-specific training, rubric key-element attention, or maximum/average similarity to anchors.Source: J Benchmarking on ASAP-SAS · p. 21
- LLM encoders outperform MLM encoders on average on ASAP-SAS and generally maintain or improve mean performance when the question prompt is added.The comparison follows the official per-question QWK evaluation protocol.Source: J Benchmarking on ASAP-SAS · p. 22, Table 10 · p. 22
- Some prior methods achieve comparable performance with smaller models, but their prompt-specific training and tuning diverge from deployment on new questions without retraining.The authors frame joint rubric retrieval as more aligned with real-world ASAS use cases.Source: J Benchmarking on ASAP-SAS · p. 22
- Joint rubric-retrieval models use one model across prompts, trading maximal per-prompt performance for improved generality and practical applicability.Lightweight Llama-3.2-3B variants achieve mean QWK comparable to or exceeding most prior neural approaches despite joint training.Source: J Benchmarking on ASAP-SAS · p. 22