Source-linked AI summary
StableJudge: A Lightweight Adversarial Defense Framework for LLM-as-a-Judge
Anning Hu, Yangyu Li, Yuqi Huang, Shengchao Hu, Junchi Yan
TL;DR
LLM judges can have their scoring signals hijacked by instructions embedded in evaluated content, while defenses tied to a judge may need renewed adaptation when backends change. StableJudge places a compact Sentry before scoring to localize and sanitize injected spans, with adversarially evolving training. The reported evaluations support score recovery across backends and attacks, alongside cross-domain localization.
Problem
Score-hijacking instructions embedded in evaluated content can corrupt LLM-judge scores and the rankings or decisions driven by them.
Source: §1 · p. 1Method
StableJudge uses a compact 0.13B Sentry to localize and sanitize injected spans before scoring, with mutual red–blue evolution to train the defense.
Source: §1 · p. 2Results
StableJudge evaluations support backend transfer, independent-attacker score recovery, and cross-domain localization; precise localization reaches 99.21%/99.97% character IoU on seen/unseen attacks.
Source: §4.2 · p. 7, §5 · p. 8Takeaways & Limitations
A compact reusable Sentry separates defense from judge choice while preserving scoring evidence through span localization and verification.
Source: §5 · p. 8The main scoring experiments use abstracts rather than full papers; end-to-end score recovery for full-paper judging remains future work.
Source: Limitations · p. 9
Abstract
from arXiv · showhide
Large language models (LLMs) are increasingly used as judges to assign scalar scores that drive downstream decision-making, including tasks such as paper review and recommendation. However, LLM-as-a-Judge pipelines are vulnerable to score-hijacking prompt injection embedded in evaluated content, which corrupts scoring signals and distorts induced rankings. Existing defenses often enforce robustness by modifying the judge model itself, making safety dependent on a specific model's scale and training and requiring repeated effort when the judge is updated or replaced. We propose StableJudge, a lightweight defense framework that externalizes prompt-injection mitigation as a reusable pre-scoring capability. StableJudge centers on a compact 0.13B Sentry that localizes injected spans and performs targeted sanitization prior to evaluation, and is trained using adversarially evolving attacks to improve robustness beyond a fixed prompt set. To stabilize measurement, StableJudge derives deterministic, fine-grained scalar scores from the judge's output distribution. Across held-out scientific abstracts and out-of-distribution attack variants, StableJudge improves score recovery under attack while largely preserving general-purpose performance, demonstrating that robust LLM-based judging can be achieved by decoupling safety from judge scale via reusable defense capabilities. Code and data will be released.
1 Introduction
StableJudge addresses score-hijacking injections that can corrupt judge scores and downstream rankings by moving mitigation into a reusable pre-scoring capability. Its compact Sentry sanitizes evaluated content, and the framework evaluates frozen-Sentry transfer and adversarially evolving training.
1 Introduction
- Score-hijacking injections embed short directives in evaluated content to inflate scores and corrupt signals used for rankings or downstream decisions.The evaluated content is also an entry point for instructions targeting the judge.Source: §1 · p. 1
- When mitigation is tied to a judge’s weights or behavior, changing the backend entails renewed adaptation or validation.StableJudge instead externalizes mitigation as a compact callable capability reusable when the judge changes.Source: §1 · p. 1
- A fixed-input evaluation across three judge backends tests score recovery using the same frozen Sentry and sanitized texts.This tests the reusable pre-scoring defense without modifying the judge.Source: §1 · pp. 1–2, Figure 1 · p. 2
- The 0.13B Sentry localizes and sanitizes injected instructions before an unchanged judge scores the text.StableJudge is designed to retain evaluation evidence while removing adversarial instructions.Source: §1 · pp. 1–2
- Mutual red–blue evolution uses localization misses to guide new attacks and effective attacks to train Sentry.The resulting defense is evaluated on held-out attacks and an independent attack source.Source: §1 · p. 2, Figure 1 · p. 2
2 Related Work
Related work covers judge reliability, prompt injection, and defenses ranging from model training to external guard models. StableJudge is positioned as a compact tagger–verifier for targeted sanitization, with evaluation of score recovery across judge backends.
LLM-as-a-Judge
- Prior work identifies judge biases and reliability issues, while system-level ranking quality can diverge from per-instance judgment accuracy [Zheng et al., 2023; Ye et al., 2024; Godfrey et al., 2025; Liu et al., 2025].JuStRank motivates system-centric evaluation; StableJudge addresses the complementary risk that adversarial content hijacks scoring and corrupts induced rankings [Maloyan and Namiot, 2025].Source: §2 · p. 2
Prompt injection
- Prompt injection can place malicious directives in inputs or context, including external resources processed at inference time [Perez and Ribeiro, 2022; Zou et al., 2023; Greshake et al., 2023; Keuper, 2025].Recent benchmarks systematize attack and defense protocols and quantify robustness across tasks and models [Liu et al., 2023b; De Stefano et al., 2024; Zhan et al., 2024; Nasr et al., 2025].Source: §2 · p. 2
Defenses
- Defenses range from instruction-hierarchy training [Wallace et al., 2024] to external guard models such as Llama Guard [Inan et al., 2023].Guardrails also face over-defense, where benign inputs are falsely flagged due to trigger-word bias [Li and Liu, 2024].Source: §2 · p. 2
- LLAMOS purifies adversarial text with a prompted defense agent, whereas StableJudge trains a compact tagger–verifier for targeted span sanitization [Lin et al., 2024].StableJudge tests whether the same frozen defense recovers clean-input scores across judge backends; an external tool interface supports reuse without retraining each backend [Karpas et al., 2022].Source: §2 · pp. 2–3
3 Method
StableJudge defines score recovery as restoring the same judge’s clean score after sanitizing injected content, using a reusable Sentry before scoring. Its method combines localized span removal, evolving attack training, and a deterministic score readout.
3.1 Problem Setup and Objective
- StableJudge aims to minimize score-recovery residuals on effective attacks while preserving non-injected content, with clean, attacked, and defended comparisons holding the judge protocol fixed.Attack effectiveness and defense success are calibrated against judge variability using a 3σ tolerance.Source: §3.1 · p. 3
3.2 StableJudge Overview
- The reusable pre-scoring Sentry sanitizes injected spans into minimally modified text that the judge evaluates through a fixed readout.The same frozen Sentry can be reused across judge backends.Source: §3.2 · p. 3, §3.5 · p. 5
3.3 StableJudge Sentry: Localize and Sanitize
- The 0.13B Sentry pairs a tagger and verifier to remove localized instructions while retaining useful evaluation content.The compact DistilBERT encoders support fine-grained sanitization rather than rejecting an entire input.Source: §3.3 · p. 3
3.3.1 Token-Level Localizer (Tagger)
- The token tagger predicts O, B, and I labels for non-injection tokens and the beginning and continuation of injected spans.It combines contextual token embeddings with lightweight pattern features, including style consistency, instruction density, and keyword frequency.Source: §3.3.1 · pp. 3–4
3.3.2 Span Verification and Sanitization
- A verifier filters candidate spans before minimal redaction, while bounded head, tail, and radar-guided interior windows extend the process to long inputs.Character-offset mappings return selected-window spans to the original document, leaving text outside accepted intervals unchanged.Source: §3.3.2 · p. 4, Figure 2 · p. 4
3.4 Red–Blue Adversarial Evolution Training
- Mutual red–blue training uses tagger misses to guide attack generation and score-impact filtering to select effective attacks for detector updates.The blue team trains on injected examples, with paper-ID splits preserved across rounds.Source: §3.4 · pp. 4–5, Figure 3 · p. 5
3.5 Tool-Augmented Scoring Pipeline
- The scoring pipeline invokes Sentry before each judge call, then reuses the frozen Sentry across backends while changing only the judge scoring interface.Source: §3.5 · p. 5
3.6 Deterministic Logit-Based Scoring
- The deterministic readout takes the expected value over normalized probabilities for allowed score labels, yielding finer-grained scores than greedy integer decoding.The method requires access to complete label probabilities and uses the expected score as the scalar score throughout.Source: §3.6 · pp. 5–6
4 Experiments
StableJudge is evaluated on held-out attacks using detection, score-recovery, scoring-readout, and capability-retention measures. The experiments report precise localization, improved recovery and human-score agreement, and largely retained general capabilities.
4.1 Experimental Setup
- The evaluation uses abstract-only inputs, paper-ID train/validation/test splits, and unseen attacks as the default test setting.It measures localization, paired score recovery, and scoring-readout agreement separately; the primary judge is abstract-scoring-adapted Qwen3-8B, with two additional backends evaluated without Sentry-specific judge training.Source: §4.1 · p. 6
4.2 Prompt Injection Detection
- 99.21%/99.97% character IoU on seen/unseen attacks accompanies 1.50%/0.50% text damage and no observed clean false positives.A separate cue-triggered full-paper test reports 98.89% detection across head, middle, and tail placements, with 96.67% middle-position detection and no edits to 30 clean controls.Source: §4.2 · p. 7
4.3 Red–Blue Adversarial Training
- The full Sentry pipeline retains high attack-removal recall while leaving all 372 clean controls unchanged; on an independent attacker set, it recovers 43/50 scores within ϵ and edits 0/50 matched clean inputs.Red–blue evolution guides attack generation with detector feedback and trains the detector on the resulting effective-attack pool.Source: §4.3 · pp. 7–8
4.4 Score Recovery and Scoring Readout
- With inputs and sanitized texts fixed, frozen Sentry reduces mean absolute deviation from clean scores on all three judge backends.The backend-transfer evaluation uses 400 fixed unseen inputs.Source: Table 4 · p. 7, §4.4 · p. 8
- Mean absolute deviation from clean scores falls from 1.10 to 0.36 across 298 effective attacks, with 69.5% of defended scores within ϵ.Figure 4b compares the primary judge’s clean, attacked, and defended scores; defended mean score is 7.47 versus 7.82 clean and 8.92 attacked.Source: Figure 4 · p. 7, §4.4 · p. 8
- With the judge and abstracts fixed, logit-based scoring raises Pearson r from 0.0213 to 0.2026 and Kendall τ from 0.0146 to 0.1052 against human paper-level scores.The comparison changes the scoring readout alone.Source: §4.4 · p. 8
4.5 General Capability
- General capabilities are largely retained: GPQA is unchanged, MMLU rises by 0.26 percentage points, and GSM8K falls by 1.44 points.The comparison is between base Qwen3-8B and the aligned StableJudge scoring agent equipped with Sentry.Source: §4.5 · p. 8
5 Conclusion
StableJudge separates defense from judge choice using a compact, reusable Sentry. Its reported architecture combines span localization and verification, adversarial evolution, and a logit-based readout.
5 Conclusion
- StableJudge separates defense from judge choice through a compact, reusable Sentry.The conclusion connects span localization and verification, mutual red–blue evolution, and logit-based scoring with backend transfer, independent-attacker recovery, and cross-domain localization.Source: §5 · p. 8
Limitations
Because repeated judge evaluations are computationally costly, the main score-recovery experiments use abstracts rather than full papers. Full-paper experiments assess injection localization, while end-to-end score recovery on full papers is left for future work.
Limitations
- Repeated judge-evaluation costs limit the main scoring experiments to abstracts rather than full papers.The primary judge is trained on abstract-conditioned review supervision to align its scoring with the available input.Source: Limitations · p. 9
- Full-paper experiments assess injection localization, but extending end-to-end score recovery to full-paper judging remains future work.Source: Limitations · p. 9
Ethical considerations
StableJudge uses generated attacks for defense training and evaluation, creating a dual-use risk. The paper recommends authorized robustness testing, human oversight for consequential decisions, and privacy-conscious data reuse.
Ethical considerations
- Attack-generation techniques used for defense training and evaluation could also be misused to manipulate automated assessments.Source: Ethical considerations · p. 9
- The paper recommends restricting attack-generation techniques to authorized robustness testing and keeping humans involved in consequential decisions such as paper screening.Automated scores should support, not replace, human judgment.Source: Ethical considerations · p. 9
- Reuse of submission–review data should respect source access conditions and reviewer privacy.Source: Ethical considerations · p. 9
- The study uses existing text corpora and model-generated attacks, with no new human-subject experiments.Source: Ethical considerations · p. 9
A.1 Scope and score tolerance
The primary task measures recovery of the same judge’s clean-input score after score-inflation attacks, including transfer settings. Evaluation pairs a clean-input-derived tolerance with localization measures and continuous score residuals.
A.1 Scope and score tolerance
- The primary task is recovering the same judge’s clean-input score after an injected score-inflation attempt, with backend transfer, independent-attacker transfer, and carrier-domain localization evaluated using one frozen Sentry.Source: §A.1 · p. 11
- The scoring tolerance is ϵ = 0.407007, set to 3σ from repeated clean-input scoring and retained for additional backends.Signed and absolute residuals complement thresholded success as continuous measures of score recovery.Source: §A.1 · p. 11
- Table 1 evaluates localization on 400 seen attacks, 400 unseen attacks, and 372 clean controls, using character IoU, removal damage, clean false positives, and token-based removal recall.Perfect token removal requires at least 95% recall; full-paper character coverage measures the fraction of gold-injection characters covered.Source: §A.1 · p. 11
- Localization metrics quantify edits, while paired score residuals quantify their effect on scoring.Source: §A.1 · p. 11
A.2 Verifier, features, and sanitization
The verifier checks proposed injected spans, and accepted overlapping intervals are merged before sanitization. The described training and attack-generation procedures set the conditions for the final verifier and tagger.
A.2 Verifier, features, and sanitization
- The verifier classifies proposed span text using the injected-class softmax probability and a fixed threshold τ = 0.5.Positive examples are injected spans; negative examples are clean sentences.Source: §A.2 · p. 11
- The tagger and verifier are separate DistilBERT encoders, with a combined deployment size of approximately 0.13B parameters.Source: §A.2 · p. 11
- The tagger is trained for three epochs on 2,860 training and 358 validation examples; the verifier is also trained for three epochs.The tagger uses batch size 16 and maximum length 512; the verifier uses batch size 32 and maximum length 128.Source: §A.2 · p. 11
- Accepted overlapping intervals are merged, and verified spans are replaced with the fixed marker [REDACTED SECURITY THREAT] in primary and cross-backend scoring.Source: §A.2 · p. 11
- After each blue round, tagger misses inform a category-conditioned red-side attack report, while novelty is handled separately from score-shift filtering.The recorded novelty weight is 0.2, with a minimum novelty of 0.08.Source: §A.2 · p. 11
A.3 Frozen-Sentry backend transfer
These subsections examine frozen-Sentry transfer across judges and attack sources, carrier domains, long papers, and tagger encoders. Reported findings include score recovery under independently generated attacks, full-paper localization, and encoder-control results.
A.3 Frozen-Sentry backend transfer
- Frozen-Sentry transfer used the same 400 held-out unseen inputs and fixed removal predictions across three judges, with effective subsets selected separately for each judge.The fixed-cohort mean absolute attack shift includes upward and downward movements; signed residuals show the direction of remaining score displacement.Source: §A.3 · p. 11
A.4 Independent attack generator
- 43/50 defended scores were within ϵ of clean scores against independently generated, cue-avoiding attacks, and none of 50 clean inputs was edited.The Sentry and its predictions remained fixed; defended scores used the [REDACTED] placeholder.Source: §A.4 · p. 11
A.5 Carrier-domain transfer
- Carrier-domain transfer injected 200 held-out payloads into 200 of 805 AlpacaEval outputs to assess localization and clean-text preservation.The detector, attack pool, and thresholds stayed fixed; perfect removal meant at least 95% payload-token recall.Source: §A.5 · p. 12, Table 9 · p. 12
A.6 Selective full-paper scanning
- 86.00% mean character-level IoU was reported for selective full-paper scanning.The scan includes the first and last 2,000 characters and bounded windows extending 500 characters around cue matches.Source: §A.6 · p. 12
A.7 Verifier threshold and encoder controls
- Across τ ∈{0.3, 0.5, 0.7}, detection was 249/250, perfect removal ranged from 98.8% to 99.2%, and no clean false positives occurred among 372 controls.Both tagger encoders achieved high perfect-removal rates, with a lower clean false-positive rate for DistilBERT.Source: §A.7 · p. 12