Source-linked AI summary
BERT-as-a-Judge: A Robust Alternative to Lexical Methods for Efficient Reference-Based LLM Evaluation
Hippolyte Gisserot-Boukhlef, Nicolas Boizard, Emmanuel Malherbe, Céline Hudelot, Pierre Colombo
TL;DR
Lexical evaluation can confuse LLM problem-solving ability with formatting compliance, while LLM judges are computationally expensive. The paper studies this problem across models and tasks, introduces BERT-as-a-Judge using synthetically labeled triplets, and reports stronger evaluation alignment than lexical methods with performance comparable to much larger judges. Its evidence is limited to English benchmarks with objectively verifiable answers.
Problem
Lexical evaluation may conflate core problem-solving ability with output-format compliance, while avoiding expensive inference remains an open evaluation question.
Method
BERT-as-a-Judge trains an encoder-driven evaluator on synthetically labeled question-candidate-reference triplets for reference-based answer correctness.
Results
BERT-as-a-Judge consistently outperforms lexical evaluation and matches or surpasses much larger LLM judges across the evaluated task categories.
Takeaways & Limitations
BERT-as-a-Judge offers a lightweight approach for more reliable and efficient evaluation than rigid lexical assessment or expensive LLM judges.
Takeaways & Limitations
The study focuses on English benchmarks with objectively verifiable answers where correctness can be clearly defined.
Abstract
from arXiv · showhide
Accurate evaluation is central to the large language model (LLM) ecosystem, guiding model selection and downstream adoption across diverse use cases. In practice, however, evaluating generative outputs typically relies on rigid lexical methods to extract and assess answers, which can conflate a model's true problem-solving ability with its compliance with predefined formatting guidelines. While recent LLM-as-a-Judge approaches mitigate this issue by assessing semantic correctness rather than strict structural conformity, they also introduce substantial computational overhead, making evaluation costly. In this work, we first systematically investigate the limitations of lexical evaluation through a large-scale empirical study spanning 36 models and 15 downstream tasks, demonstrating that such methods correlate poorly with human judgments. To address this limitation, we introduce BERT-as-a-Judge, an encoder-driven approach for assessing answer correctness in reference-based generative settings, robust to variations in output phrasing, and requiring only lightweight training on synthetically annotated question-candidate-reference triplets. We show that it consistently outperforms the lexical baseline while matching the performance of much larger LLM judges, providing a compelling trade-off between the two and enabling reliable, scalable evaluation. Finally, through extensive experimentation, we provide detailed insights into BERT-as-a-Judge's performance to offer practical guidance for practitioners, and release all project artifacts to foster downstream adoption.
1 Introduction
Reference-based LLM evaluation must separate core problem-solving ability from compliance with rigid output formats. Lexical methods can misjudge semantically valid answers, while LLM judges reduce this issue at substantially higher computational cost; BERT-as-a-Judge is proposed as a lighter alternative.
- Formatting variations can prevent reliable answer extraction and make equivalent outputs appear different during comparison.Examples include “The answer is X” versus “Answer: X” and “2.00” versus “2$”.
- Lexical evaluation can conflate formatting adherence with factual knowledge, mathematical reasoning, or reading comprehension.Models that deviate from prescribed formats may receive artificially deflated downstream performance.
- LLM-as-a-Judge methods can credit semantically valid but structurally unconventional responses, but they add substantial computational overhead and variance.Performance can depend on the selected judge model and prompt design.
- The paper asks how to measure core problem-solving ability without relying on output formatting or expensive inference.
- BERT-as-a-Judge is introduced as an encoder-driven, reference-based evaluator intended to address lexical evaluation's limitations.The approach leverages bidirectional attention for text classification and is reported to outperform lexical evaluation while surpassing LLM-as-a-Judge under comparable inference conditions.
2 Experimental Protocol
The study evaluates reference-based answer correctness across objective benchmark tasks, 36 open-weight instruction-tuned models, and multiple evaluator baselines. BERT-as-a-Judge is trained on synthetically labeled question-candidate-reference triplets and compared with regex, generative judges, and encoder metrics.
- Tasks: The benchmark suite covers multiple-choice, context extraction, and open-form mathematics tasks with objectively verifiable answers.Named datasets include MMLU, MMLU-Pro, TruthfulQA, ARC, GPQA, SQuAD-v2, HotpotQA, DROP, and CoQA.
- Models: 36 open-weight instruction-tuned models spanning 135M to 70B parameters are evaluated.The study includes model families such as Llama-3, Qwen-3, Gemma-3, Falcon-3, Phi-4, SmolLM, OLMo-3, Ministral-3, LFM-2, and EuroLLM.
- Generation parameters: Responses are generated zero-shot with greedy decoding and a 2048-token maximum, using “Final answer: [answer]” to facilitate regex parsing.
- Labeling: Synthetic labels are produced by asking Nemotron-Super-v1.5 whether each candidate answer is correct given its question and reference answer.
- Labeling: Human validation comprises 3,212 annotations from 11 evaluators, with 97.5% average agreement with synthetic labels.
- BERT-as-a-Judge: BERT-as-a-Judge trains a BERT-style encoder on approximately 1M synthetically labeled triplets balanced across task categories and models.
- Baselines: Baselines include regex evaluation, LLM-as-a-Judge, and BLEURT-base and BLEURT-large encoder metrics.
- Evaluation: Evaluator accuracy is measured against synthetic correctness labels across in-domain test splits and tasks reserved for out-of-domain evaluation.
3 Limitations of Regex-Based Evaluation
Regex-based evaluation can make measured performance and model rankings depend on formatting and lexical matching rather than underlying capability. Failure rates vary substantially by model scale, family, and task type, with open-form mathematics especially difficult.
- Parsing failures: Regex parsing failure rates vary with model scale, family, and task type.Larger models generally produce fewer formatting errors, while Qwen-3 and Gemma-3 show near-perfect context-extraction compliance and smaller Llama-3 models fare worse.
- Parsing failures: Over 60% of Llama-3 70B outputs and around 20% of Qwen-3 32B outputs are incorrectly formatted on open-form mathematics.Multiple-choice and context-extraction tasks are much easier, with mid- to large-scale models often near zero failure rates.
- Performance distortion: Regex-based evaluation produces substantial negative measured-performance deltas relative to ground-truth labels.Underestimation can persist even with high formatting compliance when verbose outputs fail lexical matching.
- Performance distortion: Regex artifacts can shift leaderboard rankings without reflecting true capability differences.Qwen-3 32B drops 18 positions while Gemma-3 4B climbs 6 places on multiple-choice tasks.
4 Encoder-Based Evaluation
BERT-as-a-Judge is evaluated against lexical and generative judges across tasks, models, and compute budgets. It achieves strong alignment with human judgments while offering efficient and broad generalization.
- BERT-as-a-Judge consistently matches or surpasses generative judges up to 70 times larger and outperforms a fine-tuned model with three times more parameters.This trend holds across all three task categories.
- 98.6% on TruthfulQA, 88.1% on CoQA, and 95.3% on ASDiv show high accuracy on tasks excluded from the training mixture.
- Removing specific model generations from training has minimal impact on assessment quality for those excluded instances.The results support generalization to additional model families outside the training mixture.
- BERT-as-a-Judge matches the best-performing LLM judges while using substantially fewer inference FLOPs.LLM-judge performance saturates around the 10B parameter scale, and intermediate reasoning tokens do not improve accuracy.
5 Experimental Analysis
Complementary analyses examine BERT-as-a-Judge's training efficiency, threshold stability, calibration, hybrid deployment, context dependence, and formatting robustness. These experiments identify practical configurations and trade-offs for deployment.
- 100K training samples suffice for accurate multiple-choice and open-form mathematics assessment, with 2 GPU hours of training.Additional samples produce no significant improvement for those categories, while context extraction benefits more from larger budgets.
- A Regex+BERT-Judge fallback improves substantially over regex alone while reducing total compute fivefold when 20% of outputs fail regex parsing.
- Omitting the question causes a controlled accuracy decrease, while the question-free encoder remains better than regex and close to the full-prompt setup.The effect is particularly limited on multiple-choice and open-form mathematics tasks.
- Cross-format evaluation causes a slight performance drop, but the encoder still substantially outperforms regex on formatting variations.Training on free-form answers consistently outperforms training on formatted answers in the reported cross-format setting.
- Assessment accuracy remains stable across a broad range of decision thresholds for all task categories.This indicates strong class separation and supports deployment without task-specific threshold tuning.
- Temperature scaling with τ = 1.75 brings BERT-as-a-Judge's already reasonable soft probabilities to near-perfect calibration.
6 Related Work
Prior evaluation methods include structured-output lexical protocols and model-based metrics. These approaches address some limitations of exact matching but remain concerned with surface-form sensitivity and semantic correctness.
- Zero-shot generative evaluation commonly enforces structured outputs and uses metrics such as ROUGE, Math-Verify, or Code-Eval.
- Lexical overlap does not guarantee semantic equivalence, motivating neural metrics and task-specific evaluators.
7 Conclusion
The paper argues that standard evaluation can conflate problem-solving ability with formatting compliance and presents BERT-as-a-Judge as a lightweight, semantically oriented alternative. The method is positioned as more reliable and computationally efficient than lexical and LLM-judge approaches.
- Standard protocols can conflate underlying problem-solving ability with compliance with formatting constraints.
- BERT-as-a-Judge is a lightweight encoder framework designed to capture semantic correctness, align with human judgment, and avoid LLM-as-a-Judge's high computational cost.
8 Limitations and Future Work
The study is limited to English benchmarks with objectively verifiable answers, and future work targets broader task, language, and modality coverage.
- The evaluation setting is restricted to English benchmarks where answer correctness can be objectively defined.
- Future work proposes extending encoder-based evaluation to open-ended tasks including summarization, translation, code generation, and instruction following.
- Multilingual adaptation is identified as a route to improving applicability across diverse use cases.
- The authors also identify vision and speech inputs as promising targets for cross-modal evaluation.
Ethics Statement
The paper emphasizes fair, reliable, and scalable evaluation while examining how generation and assessment configurations affect measured performance.
- Ethics Statement: Evaluation metrics should align closely with human judgments so model comparisons reflect real-world capabilities across tasks.
- Ethics Statement: The study considers computational costs arising from more models, longer outputs, and an expanding number of benchmark tasks.
- Evaluation configurations: Three answer-production regimes are compared: log-likelihood, strict formatting, and soft or free generative prompting.
- Evaluation configurations: -22.1% on MMLU and -29.9% on ARC-Challenge are reported for likelihood-based evaluation relative to generative evaluation.
- Evaluation configurations: -11.8% on DROP and -30.5% on GSM8K are reported for strict prompting on tasks requiring more complex outputs.
C Human-Synthetic Label Agreement
Human and synthetic evaluators show high agreement, supporting synthetic labeling for scalable assessment, while disagreement analysis and an accuracy correction clarify remaining uncertainty.
- Agreement measurement: 97.5% of 3,212 annotated samples received matching human and synthetic labels.
- Discrepancy analysis: The 80 disagreement cases, representing 2.5% of samples, are categorized as interpretation, verbose answers, reading errors, or semantic closeness.
- Discrepancy analysis: Severe discrepancies account for under 1.25% of the entire dataset, indicating that systemic bias in synthetic-label distillation is rare.
- Accuracy correction: The human-label accuracy estimate AH is expressed using synthetic-label accuracy AS and the human-synthetic agreement rate ρ.
- Accuracy correction: When ρ = 0.5, the estimated human-label accuracy drops to random guessing regardless of AS.
D Multilingual Results
BERT-as-a-Judge remains competitive in multilingual evaluation, with English-only training already performing strongly and lightweight multilingual adaptation providing incremental gains.
- Multilingual evaluation: The multilingual study compares English-only training with an additional fine-tuning stage on 20k multilingual question-candidate-reference triplets.
- Multilingual evaluation: BERT-as-a-Judge matches or outperforms the much larger LLM judges evaluated.
- Multilingual evaluation: English-only fine-tuning is already sufficient for competitive assessment performance in the multilingual setting.
- Multilingual evaluation: Lightweight multilingual adaptation provides further incremental improvements in assessment accuracy.
- Additional analyses: The supplementary evaluation reports parsing failure rates, regex-based performance effects, and model-by-task comparisons across benchmark families.
- Additional analyses: Regex-based evaluation can fail when parsing correct answers or when added formatting prevents correct assessment against the reference.