PaperlayerView exact revision on arXiv

Source-linked AI summary

Codifying the Judge: Scalable Evaluation via Program Distillation

Tzu-Heng Huang, Shengqi Qiu, Frederic Sala

arXiv:2607.22561v1cs.AIcs.LG

TL;DR

LLM judges face scalability and reliability challenges from costly, latent, and opaque evaluation. PAJAMA distills their logic into executable program committees with confidence-aware LLM fallback, matching a 13B judge across five datasets and four model families while running 47.25× faster.

  • Problem

    LLM judges face four challenges that limit the scalability and reliability of automated evaluation.

  • Method

    PAJAMA synthesizes diverse rubric-based programs, calibrates and aggregates their verdicts, and routes uncertain cases to an LLM judge.

  • Results

    Across five preference datasets and four model families, programmatic judges match OLMO-2-13B-INSTRUCT accuracy while running 47.25× faster.

  • Takeaways & Limitations

    Programmatic judges provide fast, transparent evaluation and cheap reward signals without proprietary calls at evaluation time.

  • Takeaways & Limitations

    PAJAMA is best suited to straightforward evaluation because synthesized programs may fail to generalize on complex reasoning tasks such as mathematics or coding.

Abstract

from arXiv · show

LLM-as-a-judge has become the standard for automated evaluation, but it suffers from high cost, significant latency, and opaque decisions -- limitations that undermine its scalability and reliability. We address these with a simple, efficient alternative: program distillation. Instead of prompting an LLM at the evaluation time, we distill its decision logic into a committee of programs that score candidates directly. These programmatic judges offer transparency, are easily inspected or edited, and eliminate per-sample API costs. Building on this notion, we introduce PAJAMA, a system that synthesizes programs as judges, aggregates their decisions into a joint verdict, and incorporates a fallback mechanism to selectively escalate low-confidence cases to an LLM. Across five datasets and four model families, we show that programmatic judges can match the performance of a 13B-size LLM judge. When using program outputs as routing signals, PAJAMA improves both accuracy and throughput and advances the Pareto frontier. Beyond evaluation, programmatic judges produce cheap and effective reward signals: on RewardBench, a reward model distilled from programs' verdicts outperforms one trained on a proprietary LLM's labels at two orders of magnitude lower API cost.

1 Introduction

The introduction frames LLM judges as broadly useful but costly, opaque, and difficult to scale reliably. It proposes distilling their logic into reusable executable programs, with PAJAMA combining programmatic judges and selective LLM fallback.

  • Motivation: LLM judges support preference labeling, reward-model distillation, and rubric-based reinforcement-learning feedback, but their drawbacks increasingly constrain scalable and reliable evaluation.The introduction identifies inference cost and opaque decision logic as key challenges.
  • Program Distillation: Program distillation asks an LLM once to generate executable judging logic, after which programs locally score every candidate.This changes repeated per-sample model inference into a one-time synthesis investment followed by reusable local execution.
  • Program Distillation: Programmatic judges reduce dataset-scaled API costs, provide low-latency execution, and expose editable logic for inspection, refinement, or domain-knowledge injection.Programs can be stored and reused locally without additional API cost after synthesis.
  • PAJAMA: PAJAMA addresses limited generalization, repetitive synthesis, and incompatible noisy score scales by synthesizing, calibrating, aggregating, and routing among programmatic judges and LLM fallbacks.The system uses a curated evaluation set, diverse programmatic judges, joint verdict aggregation, and confidence-aware escalation.
  • Results: 47.25× faster, standalone programmatic judges match OLMO-2-13B-INSTRUCT accuracy across five preference datasets and four model families.The result compares standalone programmatic judges against OLMO-2-13B-INSTRUCT while reporting the measured speedup.
  • Results: +5.0% accuracy at 2.9× throughput with OLMO-2-7B-INSTRUCT, while pairing with QWEN2.5-3B-INSTRUCT yields +2.6% accuracy at 2.2× throughput.These confidence-aware routing results are presented as advances to the accuracy–throughput Pareto frontier.

2 Related Work

The work connects automated evaluation, weak supervision, and routing strategies. Prior research establishes LLM judges for human-aligned evaluation, aggregates noisy labeling functions into probabilistic decisions, and routes queries across models using task and cost considerations.

  • The paper situates itself at the intersection of automated evaluation, weak supervision, and routing strategies.
  • Automated Evaluation: LLM judges provide automated evaluations aligned with human preferences for ranking, pairwise comparison, and rubric-based scoring, and can support reward-model training.
  • Weak Supervision: Weak supervision combines noisy estimates from rules, domain knowledge, or pretrained models through labeling functions and probabilistic aggregation to create labeled datasets.
  • Routing Strategies: LLM routing directs queries among diverse models according to task difficulty, expected accuracy, and inference cost, using model-free or model-based approaches.

3 Framework

PAJAMA distills LLM judging into diverse Python programmatic judges that score candidate responses, calibrates and aggregates their verdicts, and routes uncertain cases to an LLM judge. The framework targets preference decisions between two responses to a query while improving transparency and reducing evaluation costs.

  • General Workflow: PAJAMA prompts an LLM to synthesize diverse Python judges, calibrates them on held-out data, selects effective programs, and aggregates their outputs into one preference decision.Diversity comes from varying prompts and evaluation criteria, while a text-similarity check filters repetitive programs.
  • Problem Setup: The task is to determine which of two candidate responses, generated by the same or different LLMs, is preferred for a given free-form-text query.Queries and responses are modeled as elements of Σ⋆, with Q and R denoting their respective spaces.
  • Programmatic Judges: Each synthesized judge directly scores the query and both responses, producing a scalar whose higher value indicates higher quality under its evaluation logic.This translates opaque, costly LLM judging into inspectable programs that assess inputs directly.
  • Program Output Modeling: Program outputs are min-max normalized, converted into thresholded votes with abstention, filtered below 50% validation accuracy, and aggregated through a top-k committee.Per-program thresholds can maximize validation accuracy, and abstention withholds weak signals before aggregation.
  • Fallback Mechanism: When all selected programs abstain or the committee lacks confidence, PAJAMA uses program-derived signals to route the input to an LLM judge.This fallback combines fast programmatic verdicts with LLM evaluation for uncovered or low-confidence cases.

4 Experiments

Across five preference datasets and four model families, PAJAMA’s programmatic judges match mid-sized LLM-judge accuracy while achieving dramatically higher throughput. Routing uncertain cases to LLMs improves the accuracy–throughput frontier, while program-derived labels reduce distillation costs and retain robustness to bias.

  • Experimental Setup: PAJAMA evaluates five pairwise preference datasets by synthesizing 80 candidate programs and retaining selected programs with coverage above 95.0%.The datasets are JudgeLM, PandaLM, MultiPref, Prometheus, and Preference-700K.
  • C1. Accuracy and Throughput: 78.11% average accuracy matches OLMO-2-13B-INSTRUCT and QWEN2.5-3B-INSTRUCT, while programmatic judges deliver throughput multiple orders of magnitude higher than standard LLM inference.On Prometheus, 8 programs reach 88.78%, matching OLMO-2-7B-INSTRUCT.
  • C2. Routing and Pareto Frontier: +5% accuracy at 2.9× higher throughput than LLM-only evaluation is achieved by aggregator-posterior routing with OLMO-2-7B-INSTRUCT.Across all twelve judges, PAJAMA’s hybrid frontier dominates the LLM-only frontier, with higher accuracy at every throughput.
  • C3. Cost-effective Distillation: 92.20% on Prometheus is reached with PAJAMA labels at roughly 50× lower API cost than proprietary GPT-4 labels.Out-of-domain, PAJAMA-trained reward models improve average REWARDBENCH scores by +1.67 points on Prometheus and +4.49 points on JudgeLM; synthesis costs O(1) versus GPT-4 labeling’s O(n).
  • C4. Robustness to Bias: PAJAMA achieves a lower average flip rate than all three LLM-judge families, with bias win rates comparable to QWEN2.5-7B-INSTRUCT and GEMMA-3-4B-INSTRUCT.Program reasoning is invariant to candidate order, and coding-agent calibration further reduces flip rate and bias win rate through editable program changes.

5 Conclusion · Appendix Roadmap

PAJAMA distills LLM evaluation logic into transparent programmatic judges, combining synthesized programs, calibration, weak-supervision aggregation, and fallback routing. The appendix documents prompts, datasets, reward-model distillation, compute resources, and additional empirical results.

  • 5 Conclusion: PAJAMA distills LLM evaluation logic into Python functions that directly score response quality from inputs.The framework is presented as the paper’s central contribution.
  • 5 Conclusion: Programmatic judges are synthesized from curated rubrics to create a diverse pool of evaluation programs.The programs form the basis of PAJAMA’s judge committee.
  • 5 Conclusion: PAJAMA calibrates program outputs and aggregates their verdicts through weak supervision.These steps produce the committee’s joint evaluation signal.
  • 5 Conclusion: The resulting evaluation system is fast, transparent, and reliable.These properties summarize the stated benefits of the programmatic-judge framework.
  • 5 Conclusion: A lightweight routing mechanism falls back to an LLM when the program committee is uncertain.The fallback selectively handles inputs lacking a confident committee decision.
  • Appendix Roadmap: Appendix A presents the judge-synthesis instruction and curated evaluation rubrics.These materials document how programmatic judges are guided during synthesis.
  • Appendix Roadmap: Appendix B describes the datasets, filtering procedure, and final data splits.It documents the data preparation used in the paper.
  • Appendix Roadmap: Appendix C covers reward-model distillation, compute resources, and additional empirical results in Appendix D.The roadmap identifies Appendix D as the location for further empirical findings.

A Prompt Collection

The prompt collection specifies how LLMs synthesize executable programmatic judges, supplies modular evaluation rubrics and validation examples, and defines prompts for LLM judging and bias-aware program calibration.

  • A.1 Prompt for Programmatic Judge Synthesis: The synthesis prompt combines an evaluation rubric with 10 randomly selected validation examples to guide Python programs that score candidate-response quality.Each synthesized program returns a value representing the candidate response’s quality.
  • A.1 Prompt for Programmatic Judge Synthesis: The generated judging function must return a numeric score, use the exact query-response signature, handle edge cases and errors, and discriminate quality levels.The prompt also requests standard Python operations, reasonable score ranges, and executable code only.
  • A.2 A Curated Set of Evaluation Rubrics: The curated rubric set covers relevance, readability, completeness, factual-accuracy indicators, logical coherence, clarity, reasoning transparency, uncertainty communication, organization, and evidence density.Rubrics can be swapped for customized or specialized domains.
  • A.3 Few-Shot Examples: Few-shot demonstrations are randomly drawn from the validation set and include a ground-truth preference verdict identifying Response A, Response B, or a tie.The examples show LLMs what real queries, responses, and quality differences look like in practice.
  • A.4 Used Prompt for LLM-as-a-Judge: The LLM-as-a-judge prompt asks an expert evaluator to choose which of two responses better serves the user’s needs and reply only with “A” or “B”.The query and two candidate responses are provided in explicit tagged fields.
  • A.5 Prompt for Programmatic Judge Calibration: Calibration seeks to preserve discriminative evaluation while preventing preference shifts caused by formatting artifacts, verbosity, emotional tone, or gendered framing unrelated to response substance.The stated goal is to avoid rewarding superficial or biased signals while evaluating correctness or usefulness.
  • A.5 Prompt for Programmatic Judge Calibration: Programmatic judges are executable programs that take a query and candidate response as input and return a numeric quality score.The calibration prompt asks LLMs to inspect and revise synthesized scoring logic for substantive quality.

B Dataset Description

The paper provides detailed descriptions of the datasets used in Sections 4.1 and 4.4.

  • B Dataset Description: Dataset descriptions are provided for the datasets used in Sections 4.1 and 4.4.The passage does not specify the dataset names or additional characteristics.

B.1 Preference Datasets

PAJAMA is evaluated on five preference datasets spanning varied annotation sources, task distributions, and scales. The evaluation retains decisive preference signals and uses capped test sets plus held-out data for program modeling.

  • B.1 Preference Datasets: Five datasets cover GPT-4 and GPT-3.5-Turbo judgments, crowdworker and expert annotations, rubric-based evaluation, and roughly 700K RLHF response pairs.The datasets are JudgeLM-100K, PandaLM, MultiPref, Prometheus, and Preference-700K.
  • B.1 Preference Datasets: Filtering removes ambiguous, tied, low-confidence, or non-consensus samples and requires a minimum score gap for LLM-scored datasets.Coding and mathematic examples are also excluded.
  • B.1 Preference Datasets: Up to 5,000 examples per dataset form test sets, while 500 additional held-out examples support threshold tuning, top-k program selection, and verdict aggregation.The held-out split is used for modeling program outputs.

B.2 Biased Samples

PAJAMA’s robustness to common evaluation biases is assessed using biased samples drawn from two existing benchmarks. The evaluation covers five bias categories across the two datasets.

  • Experimental setup: PAJAMA’s robustness is assessed against common evaluation biases using biased samples from two existing benchmarks.The biased samples are drawn from datasets in references and [18].
  • Bias categories: The dataset of covers position bias, rich content, gender bias, and reference bias.These four categories are evaluated using one existing benchmark dataset.
  • Bias categories: The dataset of [18] covers verbosity bias.Verbosity bias is evaluated separately from the four categories drawn from.

C Experimental Details

The experimental setup tunes program-specific abstention thresholds, distills reward models from programmatic or GPT-4 labels, and specifies shared compute and inference configurations.

  • Per-Program Threshold Tuning: Each program binarizes its continuous score difference d_i,j using an individual threshold τ_j ≥ 0.This produces a program-specific vote threshold.
  • Per-Program Threshold Tuning: A dead zone [−τ_j,τ_j] makes a program abstain when its score difference is too small to support a decisive vote.The abstention mechanism is intended to avoid noisy votes.
  • Per-Program Threshold Tuning: Thresholds are selected from τ ∈ {0.00,0.01,0.02,...,0.14} by maximizing validation accuracy over covered samples.The resulting per-program threshold and validation accuracy support selecting top-k programs.
  • Reward Model Distillation: 20,000 preference pairs from JUDGELM and PROMETHEUS train each reward model, using Qwen2.5-3B-Instruct, Bradley–Terry loss, one epoch, and learning rate 1×10^-4.Training uses batch size 2, gradient accumulation over 8 steps, and a cosine learning rate schedule.
  • Compute Resource: Experiments run on one NVIDIA A6000 GPU and a 13th Gen Intel Core i9-13900K CPU with 32 cores; throughput uses vllm, 64 concurrent requests, and 24 CPU threads for program calls.Aggregator prediction time is included in programmatic-judge throughput.

D Experimental Results · D.1 Effect of the Size of Validation Set

The experimental results report accuracy–throughput measurements across five preference datasets and four model families, alongside routing studies and RewardBench breakdowns. Validation-set ablations show that PAJAMA’s aggregator remains effective with substantially fewer labeled samples.

  • D Experimental Results: Table 4 reports accuracy–throughput results across five preference datasets and four model families.These results provide the main numerical evaluation across the paper’s experimental settings.
  • D Experimental Results: Routing benefits are further evaluated within the Qwen2.5 family.The Qwen2.5 routing results are presented in Figure 6.
  • D Experimental Results: Routing benefits are also further evaluated within the Gemma-3 family.The Gemma-3 routing results are presented in Figure 7.
  • D Experimental Results: Table 5 provides a performance breakdown on REWARDBENCH, complementing the results in §4.3.This extends the experimental analysis beyond the accuracy–throughput summary.
  • D.1 Effect of the Size of Validation Set: The validation-set ablation varies the number of samples N and measures the resulting evaluation performance.The aggregator is learned from validation-set samples, whose size is systematically reduced.
  • D.1 Effect of the Size of Validation Set: Testing accuracy stays essentially flat as the validation set shrinks from 100% to 5%.Across all five datasets, performance changes by less than half a point in absolute terms, without a consistent directional trend.
  • D.1 Effect of the Size of Validation Set: A few dozen labeled validation samples suffice to learn an effective aggregator for PAJAMA.The result indicates that the labeled-validation budget is not a bottleneck for the system.

E Discussion

PAJAMA makes potential evaluator bias more tractable through transparent programs and lightweight diagnostics, while acknowledging dependence on the underlying LLM and weaker suitability for complex reasoning tasks. The authors propose retrieval, expert knowledge, richer task descriptions, and LLM fallback as mitigations.

  • Broader Impact: Synthesized programs may inherit LLM biases and produce incorrect evaluations, but their transparent decision logic enables expert inspection of the code.Modern coding agents can also diagnose generated programs; a second-round calibration step improves robustness on biased samples.
  • Broader Impact: A small validation set can probe program behavior through coverage, conflict rate, and accuracy diagnostics that provide concrete insight into reliability.The paper contrasts these straightforward diagnostics with model-based evaluation methods, which typically lack comparable tools.
  • Limitation: PAJAMA’s labeling quality depends on the underlying LLM’s ability to comprehend tasks and generate effective programs.Suggested mitigations include retrieval, subject-expert knowledge, more detailed task descriptions, and other advanced program-synthesis techniques.
  • Limitation: PAJAMA is best suited to candidates admitting straightforward evaluation, because synthesized programs may fail to generalize on complex mathematics or coding tasks.The authors propose sending low-confidence samples to an LLM judge as a hybrid design, using program judges as an efficient first-pass checker.
Loading 2607.22561v1…