PaperlayerView exact revision on arXiv

Source-linked AI summary

Pass or Fail? Evaluating LLMs on Two Greek Examination Benchmarks

Panagiota Kyriazi, Eleni Kasoura, Prokopis Prokopidis

arXiv:2609.34800v1cs.CL

TL;DR

Independent LLM evaluations face transparency and reproducibility concerns, and Greek has limited benchmark coverage. This study evaluates models on two Greek exam benchmarks using established evaluation frameworks; results show targeted Greek adaptation can offset model size in humanities, while few-shot effects differ sharply by task format.

  • Problem

    Independent LLM evaluations face transparency and reproducibility concerns, while evaluation resources remain disproportionately focused on high-resource languages such as English.

    Source: §1 · p. 1, §9 · p. 9

  • Method

    The study evaluates four LLMs on two benchmarks using lm-eval and Inspect AI, including LLM-as-a-Judge assessment for open-ended responses.

    Source: §1 · p. 2, §3.3 · p. 4

  • Results

    Structured tasks consistently yield the lowest performance; larger models tend to dominate STEM, while KriKri-8B can outperform larger models in Greek Language and few-shot prompting affects task formats differently.

    Source: §5 · p. 7

  • Takeaways & Limitations

    Targeted linguistic adaptation and focused domain training can offset lower parameter counts in linguistically demanding humanities subjects.

    Source: §7 · p. 8

    A detailed ablation to isolate context-window overload from prompt-format confusion in few-shot structured-task degradation was deferred because of computational budget constraints.

    Source: §8 · p. 9

Abstract

from arXiv · show

The rapid advancement of Large Language Models (LLMs) imposes a thorough evaluation of their linguistic and analytical capabilities as well as constraints, particularly for a language with limited benchmark coverage such as Greek. To address the limited availability of comprehensive benchmarks in this domain, we introduce Prot-Ex and Pan-Ex, two benchmarks consisting of questions from entrance exams for Greek Model and Experimental schools as well as the Panhellenic exams (the Greek national university entrance examinations). These benchmarks are employed to assess the performance of text-only LLMs-including the Greek-adapted KriKri-8B-Instruct, Llama-3.1-8B, Gemma-4-26B, and Qwen-3-32B-across diverse academic disciplines (Modern Greek, Mathematics, Physics, etc.) and task formats (closed, structured, and open-ended), including textualized visual context (i.e., image descriptions). Our findings indicate the localized KriKri-8B significantly outperforms its base model, successfully rivalling much larger LLMs in linguistically demanding humanities tasks. By leveraging an LLM-as-a-Judge methodology, we expose the inadequacy of traditional lexical metrics for evaluating complex reasoning. Crucially, we uncover a few-shot prompting paradox: while synthetic examples improve accuracy in closed-ended questions, they severely overload the context window of 8B models in structured tasks, causing significant performance degradation. Ultimately, this study suggests targeted linguistic adaptation offsets lower parameter counts in specialized domains, despite the fragility of smaller models to prompt verbosity.

1 Introduction

The introduction motivates standardized, reproducible LLM evaluation and presents two Greek exam benchmarks as a way to assess models across academic subjects and task types. It frames questions about modality and subject performance, evaluation metrics, and few-shot prompting.

1 Introduction

  • Independent LLM evaluations face transparency and reproducibility concerns, including possible test-data contamination.The introduction presents unified frameworks and novel evaluations as a response to these concerns.Source: §1 · p. 1
  • Prot-Ex and Pan-Ex are Greek benchmarks built from entrance exams for model and experimental schools and universities.The exams cover subjects including Greek Language, Mathematics, Physics, Religious Studies, and a wider Panhellenic curriculum.Source: §1 · p. 2
  • The study evaluates four LLMs on both benchmarks using lm-eval and Inspect AI, asking how results vary by task format and subject, whether BERTScore aligns with LLM judging, and where few-shot examples matter.The questions distinguish closed-ended, structured, and open-ended task formats.Source: §1 · p. 2

2 Related Work

Existing Greek benchmarks span multiple domains, but the paper identifies limited coverage of educational exam formats beyond multiple choice. Prot-Ex and Pan-Ex extend that evaluation scope across formats and disciplines, with textualized visual context and reproducible evaluation infrastructure.

2 Related Work

  • Greek evaluation resources already span language, speech, educational, financial, legal, social-media, and multiple-choice assessment tasks.The related-work discussion situates the benchmarks in an expanding ecosystem of Greek-language resources.Source: §2 · p. 2
  • GreekMMLU covers many disciplines but is limited to multiple-choice questions, while Prot-Ex and Pan-Ex include structured and open-ended tasks across disciplines.The new benchmarks draw on Greek Model and Experimental school exams and national Panhellenic exams.Source: §2 · p. 2
  • The benchmarks add textual descriptions and transcriptions of diagrams and scientific figures to support evaluation of models that cannot process images directly.They also integrate with lm-eval and Inspect AI and use an LLM-as-a-Judge pipeline for complex reasoning.Source: §2 · p. 3

3 Methodology

The methodology builds two exam-derived Greek datasets, partitions public and private evaluation data, and assesses four LLMs across response formats using established evaluation frameworks. It also documents data availability constraints and the compute used.

3.1 Data Collection and Processing

  • Exam corpora were organized by subject and year, with questions parsed into JSON and solutions extracted into Markdown.Dataset records include identifiers, questions, context, choices, ground-truth answers, image metadata, and grading marks.Source: §3.1 · p. 3

3.2 Dataset Statistics and Taxonomy

  • Prot-Ex contains 1,766 entries, including 64 private 2019 entries, while Pan-Ex contains 1,540 entries, including 222 withheld 2026 entries.Prot-Ex has 1,484 closed-ended and 282 open-ended items; Pan-Ex has 454 closed-ended and 1,086 open-ended items.Source: §3.2 · p. 3

3.3 Evaluation Setup

  • Four models were evaluated using lm-eval-harness and inspect-ai; CEQ tasks used prompts requesting the correct identifier and were scored strictly with Exact Match accuracy.The models were Llama-KriKri-8B-Instruct, Llama-3.1-8B, Gemma-4-26B, and Qwen3-32B.Source: §3.3 · pp. 3–4

Computational Infrastructure and Compute

  • The study used distributed computing resources and estimates approximately 24 GPU hours across zero-shot and few-shot evaluation passes.The listed resources include local GPU and HPC systems, a paid Open-Router budget, and Gemini for visual-context preprocessing.Source: Computational Infrastructure and Compute · p. 4

3.4 Data Availability Limitations

  • Prot-Ex source files were unavailable or corrupted for 2015, some official solutions were missing, and essay components without official answers were excluded.Public datasets exclude 2019 Prot-Ex and 2026 Pan-Ex data; private-set rankings and task-format patterns closely mirrored public results.Source: §3.4 · p. 5

4 Experimental Results and Analysis

Across the benchmarks, performance varies with task format and subject: structured tasks are consistently difficult, larger models tend to lead in STEM, and Greek-adapted KriKri is competitive in Greek Language. The evaluation also finds BERTScore can misrepresent open-ended reasoning and that few-shot prompting helps closed tasks but can harm structured tasks, especially for 8B models.

4.1 Comparative Performance Across Exercise Types and Subjects

  • Structured-question tasks consistently produced the lowest performance, while closed- versus open-ended results varied by subject and benchmark.In Prot-Ex Mathematics, Qwen3-32B scored 88.91% on open-ended tasks versus 52.09% on closed-ended tasks; Pan-Ex Ancient Greek and Greek Language showed different format patterns.Source: §4.1 · p. 5
  • Larger models generally outperformed 8B models in technical and scientific reasoning, but KriKri-8B scored 77.54% on Prot-Ex Greek Language open-ended tasks, above Gemma-4-26B at 75.90% and Qwen3-32B at 74.75%.In Pan-Ex Greek Language, KriKri had the highest closed-ended score at 96.67% and tied Gemma in open-ended tasks at 82.02%.Source: §4.1 · pp. 5–6

4.2 BERTScore vs. LLM-as-a-judge

  • BERTScore often assigned high scores to incorrect open-ended answers, including 0.70 for a wrong Prot-Ex Mathematics answer that the LLM judge scored 0.0.In Pan-Ex, BERTScore rated Llama-8B at 65.68% while the LLM judge rated it at 30.81%; in Prot-Ex, Gemma-4-26B received 69.36% from BERTScore and 83.46% from the judge.Source: §4.2 · p. 6
  • 84.21% in Prot-Ex and 75.49% in Pan-Ex were Qwen3-32B’s aggregate LLM-Judge scores, while KriKri-8B nearly doubled Llama-8B’s Pan-Ex aggregate score, 59.33% vs. 30.81%.KriKri also reached 82.02% in the Pan-Ex Greek Language subject.Source: §4.2 · p. 6

4.3 Performance Across Baseline and Few-shot examples

  • Five-shot examples improved closed-ended performance across models and benchmarks, with KriKri-8B and Llama-8B gaining 8.5 and 6.7 percentage points in Pan-Ex, respectively.Open-ended results changed minimally, while structured-task performance fell sharply for 8B models; Pan-Ex Llama-8B dropped from 11.89% to 0.41%.Source: §4.3 · p. 6

5 Discussion

Performance varies substantially by task modality and subject: KriKri-8B excels in Greek Language, while few-shot prompting has sharply different effects across task types.

5 Discussion

  • Performance across both benchmarks depends heavily on task modality and subject, with structured tasks consistently yielding the lowest scores.Qwen3-32B reached 88.91% in Prot-Ex Mathematics open-ended generation, while KriKri-8B reached 96.67% in Pan-Ex Greek Language closed-ended tasks.Source: §5 · p. 7
  • Qwen3-32B achieved the highest Prot-Ex open-ended aggregate score at 84.21%, while KriKri-8B received higher Pan-Ex open-ended evaluations than Llama-8B by ∼20%.The open-ended evaluations were from the Gemma-3-27B judge.Source: §5 · p. 7
  • 5-shot prompting raised Pan-Ex closed-ended scores by ∼7 percentage points for KriKri-8B and Llama-8B, barely changed open-ended scores, and hurt smaller models on structured tasks.Llama-8B fell from 11.89% to 0.41% in Pan-Ex structured tasks; the authors suggest complex synthetic examples overload smaller models’ context windows.Source: §5 · p. 7

6 Resources

Prot-Ex and Pan-Ex are released as open-source resources, with code and evaluation tools publicly available; aggregate few-shot impact tables cover both benchmarks.

6 Resources

  • Prot-Ex and Pan-Ex are released as open-source benchmark resources.The datasets are available on Hugging Face.Source: §6 · p. 7
  • The accompanying codebase, including evaluation scripts and dataset-management commands, is publicly available in GitHub repositories.The repositories are provided for Prot-Ex and Pan-Ex.Source: §6 · pp. 7–8
  • Tables 5 and 6 report aggregate few-shot prompting impact across task types for Prot-Ex and Pan-Ex, respectively.Prot-Ex structured-task aggregates reflect only Greek Language because other subjects lack that format.Source: Table 5 · p. 8, Table 6 · p. 8

7 Conclusions

The conclusions emphasize that linguistic adaptation can let smaller models rival larger ones in Greek humanities, while evaluation methods and prompting remain imperfectly suited to all tasks.

7 Conclusions

  • KriKri-8B’s humanities performance shows that parameter size is not the sole determinant of success in specialized Greek domains.The authors conclude that targeted linguistic adaptation and focused domain training can offset lower parameter counts for specialized educational applications.Source: §7 · p. 8
  • BERTScore does not capture factual correctness and logical flow well in open-ended reasoning, while LLM-as-a-Judge evaluation can show leniency toward flawed answers.The authors call for further refinement through strict negative-constraint prompting.Source: §7 · p. 8
  • Few-shot prompting benefits closed-ended questions, yields negligible open-ended improvements, and degrades structured-task performance, especially for smaller models.Context overload can cause 8B models to lose structural focus and behave erratically; prompt engineering must account for task modality and model constraints.Source: §7 · p. 8
  • A future direction is testing native Vision Large Language Models on raw images rather than relying on textualized visual contexts.This could assess whether raw visual input changes performance on multimodal questions relative to text-only alternatives.Source: §7 · p. 8

8 Limitations

The benchmarks have gaps in their source data and exclude essay components, while some subject-level results are based on very small samples. The study also cannot rule out contamination or isolate the cause of few-shot degradation in structured questions.

8 Limitations

  • Prot-Ex has missing or unavailable exam files and excludes essay components because official answers are unavailable for those essays.The study says these constraints affect data composition and prevent assessment of models’ extensive writing capabilities.Source: §8 · pp. 8–9
  • Physics results are not statistically robust because that subject contains only 9 questions and should be interpreted with caution.Source: §8 · p. 9
  • Contamination cannot be entirely ruled out, although private holdouts withhold Prot-Ex 2019 and Pan-Ex 2026 exams from public releases.Source: §8 · p. 9
  • Ablations to distinguish context-window overload from prompt-format confusion as causes of few-shot structured-question degradation were deferred for lack of computational budget.Source: §8 · p. 9

9 Ethical Considerations

The authors limit use of the exam materials to noncommercial academic research and frame the benchmarks as addressing Greek’s underrepresentation in LLM evaluation. The supplied examples and table captions describe textualized visual inputs and subject-by-format distributions.

9 Ethical Considerations

  • The exam materials remain the intellectual property of their respective educational bodies, and the authors restrict their use to noncommercial academic research.Source: §9 · p. 9
  • The authors present Greek benchmarks as a contribution to reducing the disparity between evaluation resources for Greek and high-resource languages such as English.Source: §9 · p. 9

A Examples with LLM-generated image descriptions and transcriptions

  • For benchmark images, Gemini 3.1 Pro generates textual descriptions and transcriptions that serve as the text-only models’ visual input.The authors report manually checking the generated descriptions and transcriptions for accuracy.Source: A Examples with LLM-generated image… · p. 11

B Prot-Ex and Pan-Ex Public Benchmarks: Subject and Task Format Distribution

  • Table 7 presents public Prot-Ex and Pan-Ex subject distributions across closed-ended, structured, and open-ended questions.Source: Table 7 · p. 12

C Prot-Ex and Pan-Ex Private Test Sets: Subject and Task Format Distribution

  • Table 8 presents private Prot-Ex and Pan-Ex test-set subject distributions across closed-ended, structured, and open-ended questions.Source: Table 8 · p. 12

D Results on the Prot-Ex and Pan-Ex Private Test Sets

On the private test sets, overall model rankings and relative behaviors across question formats closely mirror the public-set results. The appendix also supplies judge rubrics and an example where a judge awards partial credit for an incorrect answer.

D Results on the Prot-Ex and Pan-Ex Private Test Sets

  • Overall model rankings and relative question-format behaviors on both private test sets closely mirror the primary public-dataset results.The private splits contain 64 Prot-Ex questions from 2019 and 222 Pan-Ex questions from 2026.Source: D Results on the Prot-Ex and Pan-Ex… · p. 12
  • Table 9 reports private Prot-Ex results and Table 10 reports private Pan-Ex results, with aggregate accuracy for closed and structured tasks and Inspect AI evaluations for open-ended questions.Source: Table 9 · p. 12, Table 10 · p. 12

E.1 Modern Greek Language (Pan-Ex)

  • The Modern Greek rubric grades answers from 0.0 to 1.0, considering spelling, grammar, syntax, vocabulary, and completeness of meaning.It assigns 0.75 when the meaning is correct but a minor expressive, syntactic, or spelling error remains.Source: §E.1 · p. 13

E.2 Mathematics (Prot-Ex)

  • The Mathematics rubric awards full credit for correct methodology and a matching final result, with partial credit for incomplete work or minor arithmetic errors.It uses a five-level scale from 0.0 to 1.0.Source: §E.2 · p. 14

F.1 Evaluator Leniency: Pan-Ex Ancient Greek

  • In the Ancient Greek example, Gemma-3-27B-it gives partial credit to an answer that discusses the passage but fails to identify the pronoun’s referent.The judge’s explanation says the response focuses on ἐπιστήμης rather than ἐπαγγελλόμενοί.Source: §F.1 · p. 14

F.2 Metric Discrepancy: Prot-Ex Mathematics

The examples show that lexical overlap can disagree with reasoning-based evaluation, while a Greek-adapted model can outperform its base model on a language-specific task. The comparison illustrates how contextual word choice affects judged answer quality.

F.2 Metric Discrepancy: Prot-Ex Mathematics

  • BERTScore assigns ∼0.71 to a mathematically incorrect answer that the LLM judge scores 0.0, illustrating lexical overlap’s mismatch with reasoning-based evaluation.The answer repeats vocabulary and numbers from the target without following a logical solution path.Source: §F.2 · pp. 14–15

F.3 Model Comparison: Prot-Ex Modern Greek

  • KriKri-8B selects a contextually appropriate Greek synonym and receives a higher judge score than Llama-8B, whose alternative changes the sentence’s meaning.The example concerns replacing a word in a sentence about operating hours changing without notice.Source: §F.3 · p. 15
  • In the Greek-language example, KriKri-8B’s τροποποιηθεί preserves the intended meaning, whereas Llama-8B’s μεταβεί suggests transit or movement.Source: §F.3 · p. 15

G Synthetic Examples used in Few-Shot Prompt

The appendix presents sample prompts and targets across Prot-Ex and Pan-Ex task formats, followed by examples of 5-shot failures in Latin, Biology, and Ancient Greek.

G.1 Prot-Ex: Modern Greek Language (Matching)

  • The Prot-Ex Modern Greek matching prompt asks students to assign five phrases from Aris’s story to family business, sports activity, or future career.Source: §G.1 · p. 16

G.2 Prot-Ex: Mathematics (Multiple Choice)

  • The Prot-Ex mathematics multiple-choice item asks which fifth exam score raises Maria’s grades of 14, 17, 15, and 16 to an average of 16; the target is C.Source: §G.2 · p. 16

G.3 Pan-Ex: Ancient Greek (Fill in the gaps)

  • The Pan-Ex Ancient Greek gap-fill asks for an etymologically related noun to λαμβάνοντας that completes a sentence about receiving new laboratory equipment.Source: §G.3 · p. 16

G.4 Pan-Ex: Computer Science (Fill in the gaps)

  • The Pan-Ex computer science item asks students to fill algorithm blanks so a boolean-controlled loop tests whether a 50×50 integer matrix is symmetric across its main diagonal.Source: §G.4 · p. 17

H.1 Pan-Ex Latin

  • In the Pan-Ex Latin example, Llama-8B 5-shot answered in English and labeled ancillae genitive rather than the target indirect object.Source: §H.1 · p. 17

H.2 Pan-Ex Biology

  • In the Pan-Ex Biology matching example, the model repeated choice γ for almost every item instead of respecting the prompt’s logical constraints.Source: §H.2 · pp. 17–18

H.3 Pan-Ex Ancient Greek

  • The 5-shot fill-in response ignored the required etymological relation to φέροντας and guessed φόβο instead of φέρσιμο.Source: §H.3 · p. 18
Loading 2609.34800v1…