PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Scores: Understanding LLM-as-a-Judge Mechanisms in Summarization Evaluation

Himil Vasava, Ming Jiang

arXiv:2609.01604v1cs.CLcs.LG

TL;DR

LLM-based NLG evaluators are widely used, but how they compute ratings remains poorly understood. The paper studies this mechanism with controlled summarization perturbations and causal analyses, finding a structured two-stage pipeline in which attention routes error signals and upper-layer MLPs write ratings. Fine-tuning modifies an existing base-model substrate by installing stage separation and advancing crystallization depth.

  • Problem

    Existing work mainly studies behavioral agreement and failure modes, leaving the internal computation of LLM-judge ratings largely unexamined.

  • Method

    The study combines eight Readability and Adequacy attacks, controlled clean/corrupt summary pairs with token maps, and four mechanistic experiments on Themis and Prometheus.

  • Results

    Both evaluators use lower-layer attention for error identification and routing, upper-layer MLPs for rating integration, and sharp late-layer crystallization at L = 26 on Themis and L = 25 on Prometheus.

  • Takeaways & Limitations

    Fine-tuning sculpts an existing routing and crystallization substrate by adding stage separation and advancing crystallization depth.

  • Takeaways & Limitations

    Perturbations are generated by GPT-4o, so generator-independent characterization remains outside the study’s scope.

Abstract

from arXiv · show

LLM-based evaluators of natural language generation (NLG) quality are widely deployed as scoring tools and as automated training signals, yet the internal procedure by which they assign a rating remains poorly understood. We investigate this procedure mechanistically through an eight-attack perturbation taxonomy across the Readability and Adequacy dimensions of NLG quality, a generation pipeline that produces paired clean and corrupt summaries with controlled error intensity and explicit token-level modification maps, and a four-experiment battery of causal tracing, logit-lens vocabulary projection, and attention-head knockout applied to Themis (Llama-3-8B) and Prometheus (Mistral-7B). Both evaluators implement a structured, coherent evaluation pipeline operating in two stages: below layer 15, attention performs local error comparison and routes the result to the final input position; above it, the MLP cascade integrates the signal and writes the rating, with the decision crystallizing in the residual stream at a sharp late layer (L = 26 on Themis, L = 25 on Prometheus). Furthermore, a base-model control at the same scale (Llama-3-8B) reproduces the routing architecture and crystallization but not the stage separation, isolating the two mechanisms that fine-tuning specifically installs, suppression of below-L15 MLP contribution at the last position and a two-layer advance of the crystallization depth, indicating that fine-tuning sculpts an existing substrate rather than building the pipeline from scratch. We release the source code and data at https://github.com/himil-v/judge-mech

1 Introduction

The paper addresses the limited understanding of how LLM-based NLG evaluators compute ratings by studying summarization mechanistically. It develops controlled perturbations and identifies structured evaluation stages, criterion-specific attention, evaluator differences, and fine-tuning effects.

  • Motivation: Behavioral studies characterize what LLM judges score, but the internal computation of ratings remains largely unexamined.This gap limits understanding of evaluator reliability and anticipated failure cases.
  • Research question: The study asks whether judges identify specific quality defects or instead produce ratings consistent with training-distribution patterns.Summarization is used because it offers mature perturbation methods, benchmark datasets, and a structured article-summary format.
  • Approach: The analysis combines eight Readability and Adequacy attacks with paired clean/corrupt summaries, controlled error intensity, and token-level modification maps.These controls enable localized mechanistic tracing of perturbation effects.
  • Key findings: Both evaluators use a two-stage pipeline: lower-layer attention identifies and routes errors, while upper-layer MLPs integrate signals and write ratings near layers 25–26.The decision crystallizes sharply in the residual stream at late layers.
  • Key findings: Readability errors recruit broader contextual attention, whereas Adequacy attacks remain concentrated on perturbed tokens.The finding suggests criterion-specific attention strategies.
  • Key findings: Fine-tuning preserves a base-model routing substrate while installing stage separation and advancing crystallization from L = 28 to L=26.The comparison indicates that fine-tuning modifies rather than creates the evaluation pipeline.

2 Related Work

Prior work has established behavioral evaluation, perturbation benchmarks, and mechanistic interpretability methods, but has not directly characterized the internal procedure of LLM-based NLG judges.

  • LLM-based NLG evaluation: LLM-based evaluation tools are widely used, while prior studies mainly measure agreement, criterion confusion, prompt sensitivity, positional bias, and length bias.These studies describe evaluator behavior rather than internal computation.
  • Perturbation-based evaluation: Table 1 consolidates perturbation categories from prior NLG-evaluation benchmarks into eight taxonomy nodes spanning Readability and Adequacy.The taxonomy links the paper’s controlled interventions to established benchmark categories.
  • LLM-based NLG evaluation: Existing factuality and faithfulness benchmarks provide an empirical foundation for evaluator development but remain behavioral in scope.They characterize what judges score rather than how scores are produced.
  • Mechanistic interpretability: Mechanistic interpretability supplies activation patching, residual-stream projection, and attention-head analysis techniques for localizing transformer computations.These methods have primarily been applied to narrowly scoped behaviors such as factual recall and indirect-object identification.
  • Perturbation-based evaluation: Prior perturbation resources isolate NLG failure modes but generally lack the position-level metadata required for mechanistic analysis.The present framework extends behavioral perturbation work with token-localizable interventions.

3 Data Perturbation

The paper constructs a perturbation framework tailored to causal tracing by controlling error intensity and recording exact token locations across eight Readability and Adequacy attack types.

  • Design requirements: Existing adversarial datasets use sentence- or document-level errors, so they lack the token-level maps needed to trace perturbation effects.The framework fixes the number of perturbed tokens through intensity k and records their positional indices.
  • Taxonomy design: The eight perturbations are selected for prior-benchmark grounding, token-localizable failure modes, and coverage of both evaluation criteria.Stylistically entangled and coarse noise interventions are excluded.
  • Taxonomy design: The taxonomy merges more than 35 initial error categories into eight perturbation types.Readability attacks degrade fluency while preserving factual content, whereas Adequacy attacks target content-related quality.
  • Data generation: GPT-4o generates clean/corrupt summary pairs for CNN/DailyMail and XSum by injecting exactly k instances of each category-specific error.The same pipeline supports a cross-domain generalization corpus.
  • Quality control: Approximately 19 to 20 of 20 manually inspected pairs per attack typically pass checks for injection, token-map accuracy, and declared perturbation type.Structural filtering also removes invalid JSON, intensity violations, and insufficient rating shifts.

4 LLM-judge Interpretation

The interpretation framework uses activation patching, normalized causal effects, logit-lens projections, and head knockout to localize how evaluators process errors and form ratings.

  • Activation patching: Activation patching substitutes clean activations into corrupt runs to measure recovery of clean rating behavior at selected layer-position sites.The study traces both MLP sublayers and attention output projections.
  • Causal effect: Causal effects are normalized so 1.0 recovers clean rating behavior and 0.0 matches the unmodified corrupt baseline.Last-token tracing and head knockout use logit differences, while window-mode tracing uses expected rating.
  • Tracing modes: Window-mode tracing restores a six-token window from relative positions +0 through +5, whereas last-token tracing restores only the final input position.The two modes localize input processing and decision-assembly depth, respectively.
  • Logit lens: The logit lens projects each layer’s final-position residual stream into a distribution over five rating tokens.Sweeping layers reveals both crystallization depth and the ultimately committed rating.
  • Head knockout: Attention-head knockout zeros each of 1024 heads and measures the resulting normalized causal effect at the final position.Positive effects identify heads driving corrupt ratings, while negative effects identify heads whose ablation releases clean signal.

5 Experiments

Experiments reveal a shared two-stage evaluation pipeline across Themis and Prometheus: attention identifies and routes errors below layer 15, while upper-layer MLPs integrate the signal and write ratings that crystallize late. The evaluators share this scaffold but differ in attention-band breadth, criterion-specific recruitment, top-layer writing dynamics, and the effects installed by fine-tuning.

  • Shared evaluation pipeline: Eight localized attacks expose early MLP registration at perturbed tokens, followed by attention-based comparison and routing toward the final input position.The perturbed token’s MLP output locally encodes corrupted content across criteria and evaluators; downstream attention routes the verdict to the last position.
  • Criterion- and attack-specific mechanisms: Preposition attacks spread attention across adjacent positions, whereas Entity and Numerical attacks remain localized; other attacks rely on distant-token comparison outside the local window.These patterns indicate that attention recruitment depends on the information required by each error type.
  • Evaluator differences: Themis concentrates last-position routing at L13-L15 and writes the rating in a dominant L31 MLP step, while Prometheus distributes routing and writing more broadly and applies an opposite-direction L31 correction.Prometheus’s routing spans L14-L17, L19-L21, and L29-L30, whereas Themis shows MLP bands at L17-L18 and L25-L27 before its L31 peak.
  • Late rating commitment: The rating crystallizes at L=26 on Themis and L=25 on Prometheus across attacks, although their post-crystallization rating trajectories differ in severity and confidence.Themis assigns Entity and Antonym Rating 2 and Tense Rating 4; Prometheus assigns Entity and Antonym Rating 1, with most other attacks at Rating 4 and less confident trajectories.
  • Attention-head organization: L10-L11 contains a narrow alternating band of disruptive and suppressive Themis heads, while Readability recruits additional L7-L9 heads and Adequacy remains sparse there.The shared evaluation band implements interacting roles rather than relying on one dominant head.
  • Fine-tuning effects: A Llama-3-8B base-model control preserves routing and upper-cascade magnitude but lacks strong stage separation and crystallizes two layers later than Themis.Fine-tuning suppresses below-L15 MLP contribution at the last position and advances crystallization from L=28 to L=26.

6 Conclusion

Across four mechanistic experiments, Themis and Prometheus implement a structured two-stage evaluation pipeline: attention routes error information below L15, while upper-layer MLPs integrate it and write the rating at a sharp late layer.

  • Attention performs local error comparison and routes the result to the final position below L15.
  • Above L15, an MLP cascade integrates the routed signal and writes the rating into the residual stream.
  • The rating crystallizes at L = 26 on Themis and L = 25 on Prometheus.
  • The base-model control retains routing and crystallization but lacks stage separation, indicating that fine-tuning installs localized modifications on an existing transformer substrate.The reported modifications are suppression of below-L15 MLP contribution at the last position and a two-layer advance of crystallization depth.

7 Limitations

The study’s circuit-level conclusions are bounded by the evaluated models, task, prompts, perturbation conditions, behavioral filtering, decoding approximation, and GPT-4o-generated perturbations.

  • Model and task scope: The analysis covers Themis and Prometheus on English CNN/DailyMail summaries with one fixed prompt per evaluator.
  • Model and task scope: Generalization to dialogue, story generation, external factuality assessment, larger or multilingual models, and prompt variation remains unverified.
  • Behavioral-filter selection: Only samples whose evaluator rating changed under perturbation were analyzed, excluding cases where the evaluator missed the perturbation.
  • Single intensity and single-attack samples: Experiments used single-attack samples at intensity k = 1, leaving higher intensities and mixed attacks uncharacterized.
  • Cross-family claims rest on two evaluators: Shared or family-specific patterns rely on n = 2 evaluators, one per architecture, rather than full model-family coverage.
  • Logit-lens approximation: The logit lens may under-report intermediate rating probability, so absolute pre-crystallization probabilities are treated as qualitative rather than calibrated.
  • Generator-model dependence: GPT-4o-generated perturbations may introduce systematic stylistic bias, while generator-independent characterization remains outside scope.

8 Ethics Statement

The work used public datasets and evaluator models, generated controlled perturbations through GPT-4o, and disclosed AI-assistant use for implementation and language editing.

  • Experiments used the public CNN/DailyMail dataset and publicly released Themis and Prometheus models.
  • No personally identifiable information was collected, and no human subjects were involved beyond the datasets’ original construction.
  • GPT-4o generated controlled syntactic and semantic alterations of public news summaries through standard API access.
  • The released artifacts could be repurposed for adversarial attacks, but the authors assess this risk as limited.
  • Claude and ChatGPT supported code implementation, debugging, proofreading, and stylistic polishing, not research questions, experiment design, or result interpretation.

A Evaluation Prompts

The evaluation prompts differ by evaluator: Themis uses a single-instruction prompt, whereas Prometheus uses five-shot examples spanning ratings 1–5 and a shared accuracy-completeness rubric.

  • A.1 Themis Prompt: Themis uses the single-instruction prompt shown in Figure 7.
  • A.2 Prometheus Prompt: Prometheus uses a few-shot prompt because this reliably produces the rating token at the first generation position.
  • A.2 Prometheus Prompt: The Prometheus examples_block contains five article-summary-rubric-rating exemplars spanning the full 1–5 rating range.
  • A.2 Prometheus Prompt: The shared Prometheus rubric rates how accurately and completely a summary captures an article’s main facts.
  • A.2 Prometheus Prompt: The rubric defines 1 as highly inaccurate or off-topic and 5 as fully accurate and comprehensive.
  • A.2 Prometheus Prompt: Prompt templates use runtime-substituted placeholders for the article, summary, criteria, and Prometheus examples block.
  • A.2 Prometheus Prompt: The five examples pair rating 4 with a partially captured interest-rate story and rating 2 with a vague EU–Vietnam trade summary.
  • A.2 Prometheus Prompt: They also pair ratings 5, 1, and 3 with comprehensive battery coverage, a factually mismatched Mars summary, and a concise earthquake summary, respectively.

B Window-Mode Causal Tracing: Methodology Details

The window-mode causal-tracing analysis applies alignment and filtering controls before robustly aggregating per-layer, relative-position effects across attacks and seeds.

  • Scope: The analysis documents window-mode causal tracing separately from last-token tracing and head knockout, which do not require the same per-token alignment controls.The distinction reflects window-mode’s dependence on per-token alignment and the other methods’ unconstrained per-sample yield.
  • Filtering: Samples are filtered for rating-token probability and a minimum clean–corrupt score difference before causal-tracing aggregation.The score-difference threshold is |Eclean−Ecorrupt| ≥0.05; retained pass rates are roughly 25–40% for Readability and 50–70% for Adequacy attacks.
  • Alignment: A difflib-derived clean-source-position mapping corrects token alignment when corrupt sequences change length under BPE tokenization.This handling is especially relevant to Spelling, Numerical, and Reordering perturbations.
  • Aggregation: Per-cell effects pool three seeds and use a 20% trimmed mean, with cells having N < 10 masked as NaN.Well-behaved attacks contribute approximately 400–500 samples per cell, whereas BPE-unstable attacks can retain as few as approximately 20.
  • Visualization: Four independent symmetric color scales preserve within-criterion visibility while retaining the magnitude difference between Readability and Adequacy effects.The scales separate MLP and attention rows as well as the two quality criteria.

B.1 Verification Protocol for Injected Perturbations

Injected perturbations are manually checked for presence, correct token-span mapping, declared attack type, and naturalness before structural filtering and analysis.

  • Verification criteria: Each attack receives manual inspection of 20 clean/corrupt pairs for injection success, token-map correctness, and perturbation-type fidelity.The token map anchors every causal intervention, so an incorrect span would invalidate the trace.
  • Filtering outcomes: Typically 19 to 20 of 20 pairs pass all three checks, while unmodified outputs are removed automatically through an empty-new_tokens structural filter.The remaining corrupted summaries are also read for fluency and plausibility.
  • Cross-evaluator context: Prometheus knockout heatmaps place Readability attacks in denser head activity than Adequacy attacks across the broader L10-L15 evaluation band.This reverses the per-criterion sparsity pattern reported for Themis.
  • Design scope: The eight attacks are not severity-matched; they are matched on perturbation intensity k = 1 so severity remains the variable under study.A formal human annotation of severity is left to future work.

C Model Size, Computational Budget, and Infrastructure

The study analyzes two similarly structured open-source evaluators using substantial GPU resources distributed across generation, tracing, logit-lens, and head-knockout experiments.

  • Model scale: Themis uses Llama-3-8B and Prometheus-7Bv2.0 uses Mistral-7B; both have 32 layers, 32 attention heads per layer, and hidden dimension 4096.Perturbations are generated with GPT-4o, whose parameter count is not publicly disclosed.
  • Infrastructure: Most mechanistic-analysis experiments run on 16 RTX 2080 Ti GPUs, with the remaining 20% on one NVIDIA H100.Inference and activation patching use bfloat16 with model-parallel sharding.
  • Computational budget: Approximately 800 GPU-hours support the full experimental battery.The budget covers perturbation generation, causal tracing, logit-lens analysis, and attention-head knockout.
  • Perturbation generation: Approximately 5000 GPT-4o API calls across eight attacks, 200 samples, and three seeds cost approximately $100.
  • Experiment allocation: Causal tracing uses 250 GPU-hours, while logit-lens analysis uses 150 GPU-hours.Causal tracing is dominated by per-layer and per-position activation patching across 32 layers and both sublayers.
Loading 2609.01604v1…