PaperlayerView exact revision on arXiv

Source-linked AI summary

ExecRubrics: Executable Tool-Augmented Rubrics for Verifiable and Efficient Long-Form Evaluation

Kaustubh D. Dhole, Charles L. A. Clarke, Eugene Y. Agichtein

arXiv:2608.22559v1cs.AIcs.CLcs.IR

TL;DR

Natural-language rubrics can be ambiguous, depend on black-box judges, and struggle to express interdependent evaluation logic. ExecRubrics represents rubric intent as executable scoring programs, and across three benchmarks its strongest variants outperform natural-language baselines while remaining transparent and judge-free. The paper positions tool-augmented executable rubrics as a promising intermediate representation for auditable, modifiable, and efficient evaluation.

  • Problem

    Natural-language rubrics are ambiguous, require black-box judges, and often use flat aggregation that cannot express dependencies, alternatives, penalties, or overrides.

  • Method

    ExecRubrics compiles natural-language rubric intent into deterministic, verifiable scoring programs that encode checks, branches, penalties, normalization, and tool calls.

  • Results

    The strongest executable variants outperform natural-language rubric baselines across HealthBench, ArgQuality, and HelpSteer without requiring a black-box LLM at evaluation time.

  • Takeaways & Limitations

    Executable rubrics provide a promising intermediate representation for auditable, modifiable, and efficient LLM evaluation.

  • Takeaways & Limitations

    Human interpretability is not directly evaluated, and more expressive rubric programs may become more complex to inspect and revise.

Abstract

from arXiv · show

Rubrics aim to make language-model evaluation transparent by decomposing response quality into interpretable criteria. However, natural-language rubrics are often ambiguous, require black-box LLM judges, and typically assume criteria aggregate independently through linear weighted sums, limiting their ability to capture dependencies, alternatives, penalties, and override conditions. We propose ExecRubrics, a framework for representing rubrics as compact executable programs. ExecRubrics encodes evaluation logic as verifiable Python scoring functions, giving natural-language rubric intent an operational semantics: a fixed decision procedure that can be inspected, executed, and edited. On three long-form response benchmarks-HealthBench, HelpSteer, and ArgQuality-we show that ExecRubrics can substitute for expensive black-box judges in ranking preferred over dispreferred responses, matching or improving NL rubric baselines with best preference accuracies of 53%, 78%, and 92%, respectively, while reducing evaluation latency by up to 320 times. We show that incorporating external logic and resources from text processing libraries such as NLTK and spaCy further improves preference accuracy. Our results suggest a novel way of looking at evaluation, by offering a faster, more explainable, and less ambiguous alternative to black-box rubric evaluation, particularly in high-stakes domains such as healthcare and banking where precision and auditability are critical.

1 Introduction

ExecRubrics addresses ambiguity and underspecified operational semantics in natural-language rubric evaluation by compiling rubric intent into deterministic, inspectable scoring programs. The framework is designed to express dependencies, alternatives, penalties, overrides, and tool-assisted checks.

  • Natural-language rubrics can be ambiguous and leave evaluation criteria's operational semantics underspecified.
  • Flat weighted checklists cannot naturally represent prerequisites, alternatives, penalties, and override conditions.The paper illustrates this with allergy-specific medical recommendations.
  • ExecRubrics compiles rubric intent into small symbolic scoring programs containing checks, branches, penalties, normalization, and tool calls.The programs are intended to provide a judge-free intermediate representation for preference evaluation.
  • Deterministic NLP tools, including tokenizers, lexicons, entity recognizers, negation detectors, readability measures, and n-gram metrics, serve as transparent primitives for atomic checks.
  • The paper asks whether LLMs can generate executable rubrics that distinguish high-quality from low-quality responses as effectively as natural-language rubrics.

2 Related Work

Prior work improves evaluation transparency through LLM judges, fine-grained criteria, adaptive rubric generation, and symbolic approximations. However, existing rubric systems largely retain weighted natural-language checklists, while symbolic approaches motivate explicit dependencies and safety-critical overrides.

  • LLM-as-judge and fine-grained evaluation: LLM-as-judge systems provide scalar, preference, or structured evaluations but offer limited diagnostic feedback about missed requirements.
  • Rubric-based and adaptive evaluation: Rubric-based benchmarks expose evaluation criteria, including physician-written query-specific criteria and expert-authored weighted rubrics in professional domains.
  • Rubric-based and adaptive evaluation: Adaptive systems generate, select, or retrieve rubric content for domain-specific and query-specific evaluation, but still largely use weighted linear sums of natural-language checklists.
  • Limitations of natural-language rubrics: Natural-language rubrics require clear granularity and often calibration or exemplars because vague criteria can produce inconsistent interpretations.
  • Interpretable symbolic approximations of neural models: Symbolic approximations support explicit rules, prior knowledge, and controllable constraints for deterministic scoring logic.

3 Rubric-Based Evaluation

Rubric-based evaluation scores candidate responses against explicit criteria, but natural-language criteria require judge-model interpretation and usually aggregate independently through weighted sums. ExecRubrics replaces that mediation with executable scoring functions that can represent richer evaluation logic.

  • 3 Rubric-Based Evaluation: Rubric-based evaluation chooses a preferred response y+ over a dispreferred response y− for query x by scoring both through rubric criteria.
  • 3 Rubric-Based Evaluation: Effectiveness is measured as preference accuracy across many query and preference pairs D.
  • 3.1 Natural Language Rubrics: Natural-language rubric systems represent criteria as weighted items, where positive points reward desirable properties and negative points penalize undesirable properties.
  • 3.1 Natural Language Rubrics: Because natural-language criteria are not directly computable, a separate judge model M determines whether each response satisfies each criterion.
  • 3.1 Natural Language Rubrics: The final natural-language rubric score is a weighted sum of satisfied criteria normalized by total positive points.
  • 3.1 Natural Language Rubrics: Although individual criteria are inspectable, natural-language rubric semantics remain mediated by M and assume criteria contribute independently and linearly.
  • 3.2 Executable Rubrics: An executable rubric is a symbolic scoring program that approximates each criterion with an explicit scoring function fri and may call deterministic tools.
  • 3.2 Executable Rubrics: Executable rubrics directly encode conditional logic, prerequisites, alternative satisfaction paths, penalties, and override rules in a logical form such as Python code.

4 Experimental Setup

The experiments compare human-written natural-language rubrics with executable variants across HealthBench, HelpSteer, and ArgQuality using preference accuracy. Executable rubrics are generated as Python scoring functions, optionally augmented with deterministic text-processing tools, and compared with LLM-based and agentic baselines.

  • Experimental design: The experimental setup compares human-written rubrics with translated executable rubrics, with and without external logic.
  • Benchmarks: Preference accuracy is evaluated on HealthBench, HelpSteer, and ArgQuality, which provide rubric information and preference-style supervision.
  • Benchmarks: Preference Accuracy measures the fraction of queries for which the positive response is preferred over the negative response.
  • Rubric generation methods: Human-written rubric evaluations use 4 to 14 corpus-level criteria, including properties such as cogency, emotional appeal, and instruction following.
  • Rubric generation methods: Qwen3-30B-Instruct-A3B predicts criterion satisfaction for the natural-language rubric evaluations.
  • Rubric generation methods: Executable rubric variants translate natural-language criteria into Python scoring functions whose summed scores evaluate query-response pairs.
  • Baselines: A no-rubric black-box agentic baseline uses GPT 5.5 to generate preferred responses for each row of a query file.
  • Tool augmentation: Tool augmentation exposes functions for tokenization, lemmatization, entity extraction, phrase matching, profanity, negation, and lexical frequency analysis.

5 Results and Analysis

ExecRubrics recover substantial preference signal without an LLM judge at evaluation time and outperform natural-language rubric baselines across all three benchmarks. Their gains vary by benchmark, with tool augmentation helping on HelpSteer but remaining less reliable on HealthBench.

  • Executable rubrics recover substantial preference signal without invoking an LLM judge at evaluation time.
  • The best executable systems reach .529 on HealthBench, .915 on ArgQuality, and .780 on HelpSteer, exceeding natural-language rubric results of .522, .904, and .623.
  • Natural-language rubric performance depends on the interpreting judge model, matching direct GPT-5.5 on ArgQuality (.904) but trailing it on HealthBench (.522 vs. .640) and HelpSteer (.623 vs. .790).
  • Deterministic tools improve GPT-5.5 and Claude-generated rubrics to .780 on HelpSteer, close to the direct GPT-5.5 baseline of .790.
  • Tool use is less reliable on HealthBench and does not close the gap with direct GPT-5.5 evaluation.

6 Conclusion and Future Work

ExecRubrics turns natural-language rubric intent into explicit deterministic scoring programs that can encode complex evaluation logic. Across three benchmarks, the framework recovers preference signals without black-box evaluation and points toward auditable, modifiable, efficient evaluation, while future work targets domain adaptation and broader applications.

  • ExecRubrics turns natural-language rubrics into executable scoring programs with explicit deterministic evaluation logic.
  • Executable rubrics can encode complex logic rather than requiring an LLM or human judge to interpret each criterion.
  • Across HealthBench, ArgQuality, and HelpSteer, executable rubrics recover substantial preference signals without a black-box LLM at evaluation time.
  • Deterministic NLP tools provide useful and transparent primitives for evaluation.
  • Future work could adapt rubric granularity to domain needs through parsing, claim extraction, entity recognition, or typed intermediate representations.
  • Future work should examine human inspection, revision, debugging, cross-domain robustness, and uses in dataset filtering, reward modeling, and preference optimization.

7 Limitations

The paper identifies limitations involving interpretability, gaming, and possible memorization effects. More expressive executable rubrics may be harder to inspect, explicit checks may be exploitable, and benchmark performance may partly reflect memorized content.

  • The paper does not directly evaluate human interpretability, which may be subjective.
  • More expressive rubric programs can capture dependencies, alternatives, penalties, and overrides but may become more complex for humans to inspect and revise.
  • Executable rubrics may be vulnerable to gaming when response generators learn to trigger textual, structural, or semantic checks without satisfying the underlying intent.
  • This gaming risk is especially important when executable rubrics serve as reward signals for reinforcement-learning optimization.
  • Performance on datasets such as HelpSteer and ArgQuality may be influenced partly by memorized content.

8 Ethical Considerations

The paper frames executable scoring logic as a way to make rubric-based evaluation more responsible in high-stakes settings. It emphasizes accountability and inspectability, while the reported experiments and writing process rely on specific automated tools.

  • LLM judges are black boxes whose scoring behavior can be difficult to inspect, reproduce, or contest.
  • This opacity raises concerns about accountability, bias, and overreliance on automated judgments in high-stakes settings.
  • The work aims to replace repeated black-box criterion interpretation with explicit executable scoring logic.
  • The experiments were mostly conducted using GPT 5.5 Codex2 and Gemini CLIs.
  • Some paragraphs were grammatically corrected using ChatGPT 5.5.

W James Popham. 1997. What’s wrong-and what’s

This section lists prior work spanning evaluation, rule execution, interpretability, and professional-domain benchmarks. The cited works include both methodological papers and domain-specific studies.

  • The references include work on behavioral testing and rule-based explanations for language-model systems.
  • The cited literature covers neural rule execution tracking and symbolic approximations of model behavior.
  • The bibliography also includes HelpSteer, FLASK, ProfBench, Qwen3, and clinical decision-support studies.

A ArgQuality Evaluation with Different LLM Judges

Preference Accuracy over ArgQuality varies with both the evaluation model and the prediction format.

  • Preference Accuracy over ArgQuality varies according to the evaluation model and whether predictions use Binary Satisfaction (1/0) or Scalar (1-10) scores.

B Rubric Types

The paper reports benchmark rubric criteria and test-example counts, and compares evaluation speed between executable and natural-language rubrics on HealthBench.

  • Table 3 reports rubric criteria and the number of test examples for each benchmark.
  • Executable-rubric evaluation is considerably faster than NL Rubrics for HealthBench, including on NVIDIA H200.Table 4 reports average per-query preference evaluation time in seconds.

Executable Rubric Generated by GPT 5.5 for ArgQuality

Figure 2 presents a GPT 5.5-generated executable rubric for 14 ArgQuality dimensions, including shown functions for cogency, local acceptability, and global sufficiency. The rubric combines marker-based scoring rules into a bounded score and returns explanatory reasons.

  • Rubric structure: Figure 2 shows a GPT 5.5-generated executable rubric for evaluating 14 dimensions of ArgQuality long-form arguments.The displayed functions include cogency, local acceptability, and global sufficiency.
  • Rubric structure: The final score_response function calls all 14 dimension functions and sums their outputs into the evaluation score.
  • Cogency: The cogency function scores argument length, conclusions, premise markers, logical relations, evidence-like support, counterarguments, query overlap, and repetitive reasoning.It returns a score, a maximum possible score of 10, and reasons after bounding the score between 0 and 10.
  • Global sufficiency: The global sufficiency function scores objections, rebuttals, evidence-like support, conclusions, qualification, length, and absolute claims.Its score is bounded between 0 and a maximum possible score of 10, with reasons returned alongside the score.
  • Local acceptability: The local acceptability function rewards evidence markers, cautious wording, quantitative detail, and penalizes absolute, insulting, or vague language.It counts these textual features before applying score adjustments and producing reasons.

Executable Rubrics Augmented With Deterministic Tools Generated by Claude Sonnet 4.6 for HealthBench

Claude’s HealthBench rubric decomposes medical-response quality into deterministic checks and combines keyword, grounding, uncertainty, safety, and task-specific signals through explicit scoring branches.

  • Calibration: Overclaiming is assessed by counting absolute claims and uncertainty language, with medical responses receiving explicit calibration penalties.The rubric identifies phrases such as “always works” or “completely safe,” then adjusts scores according to overclaim and uncertainty counts.
  • Grounding: Factual grounding rewards evidence-related language, disclaimers, and professional-care referrals through thresholded scoring branches.Grounding coverage, disclaimer presence, and consultation terms determine whether the response receives 3, 2, 1, or no points.
  • Consistency and output: The implementation adds deterministic checks for contradiction, hallucination signals, false reassurance, and final score normalization.It applies penalties or credits for contradiction and fabricated-certainty markers, then clamps and rounds the score before returning reasons.
  • Completeness: Completeness scoring combines substance, red-flag handling, clinical-step coverage, and response length with task-sensitive thresholds.The rubric distinguishes brief requests, medical safety omissions, medication or diagnosis queries, symptom queries, and word-count ranges.
  • Safety handling: The rubric checks medical red flags and rewards urgent escalation while penalizing false reassurance or omitted escalation.It detects red-flag symptoms, searches for urgent-care language, and applies explicit score changes for escalation or omission.
Loading 2608.22559v1…