Cheating Automatic LLM Benchmarks: Null Models Achieve High Win Rates

1 INTRODUCTION

• Automatic benchmarks are popular because they are cheap, scalable, and timely, while high win rates can provide promotional benefits.They use LLM-based auto-annotators and correlate highly with Chatbot Arena.Source: §1 · p. 1
• Existing controls for output length and style reduce some gameability but do not prevent structured cheating responses.The paper identifies intentional auto-annotator manipulation as an unresolved risk.Source: §1 · pp. 1–2
• Less than 1% win rates show that persuasive constant responses alone do not reliably fool AlpacaEval 2.0’s auto-annotator.The null model always returns “Pick me!” across all 805 instructions.Source: §1 · p. 2
• 76.8% LC win rate is achieved by a manually crafted structured cheating response on AlpacaEval 2.0.The response exploits auto-annotator confusion during syntactic analysis.Source: §1 · p. 2
• The method adds a transferable prefix optimized on public UltraFeedback instructions and tests it across three automatic benchmarks without access to their private instructions.The evaluation includes AlpacaEval 2.0, Arena-Hard-Auto, and MT-Bench.Source: §1 · p. 2

2 PRELIMINARIES

• An auto-annotator maps an instruction and one or two model responses to an evaluation distribution, producing scores or pairwise judgments.MT-Bench uses scores from 1 to 10, whereas AlpacaEval 2.0 uses binary model identifiers.Source: §2 · p. 2
• The threat model gives the cheater API query access but withholds auto-annotator parameters and test instructions.The goal is to make a constant, non-informative null-model response beat the reference response.Source: §2 · p. 2
• The experiments evaluate AlpacaEval 2.0, Arena-Hard-Auto, and MT-Bench using their official templates and benchmark-specific metrics.Reported metrics include length-controlled and raw win rates, reference-model win rates, and MT-Bench first-turn scores.Source: §2 · pp. 2–3
• The study focuses on LLM-based auto-annotators rather than ground-truth automatic scoring of LLMs.This bounds the scope of the analyses.Source: §2 · p. 2

Experimental setup

• The benchmark setup uses GPT-4-1106-Preview as the auto-annotator, with GPT-4-1106-Preview and GPT-4-0314 as reference models for AlpacaEval 2.0 and Arena-Hard-Auto.Table 1 summarizes the benchmark details.Source: Table 1 · p. 5

Crafting adversarial prefix by random search (RS)

• An optimized adversarial suffix alone is ineffective on AlpacaEval 2.0 when GPT-4 is the auto-annotator.This motivates combining optimization with a structured cheating response.Source: §3 · p. 3, Figure 3 · p. 5

Structured cheating responses

• 76.8% LC win rate is achieved by the structured response alone on AlpacaEval 2.0.The response is designed to remain effective when the compared outputs are swapped.Source: §3 · p. 5
• The structured response overwrites the original instruction-output triplet with a counterfeit one and exploits positional behavior to steer the predicted winner.The default and swap settings induce opposite model identifiers.Source: Figure 2 · p. 4, §3 · p. 5

Crafting adversarial prefix by random search (RS)

• Random search optimizes one adversarial prefix by minimizing aggregated losses over public instructions, making it transferable across private benchmark inputs and positions.The search samples modifications and retains variants with lower aggregated loss.Source: §3 · p. 5

Experimental setup

• The experiments used GPT-4-1106-Preview as auto-annotator and varied random-search training instructions and optimization steps across the three benchmarks.The settings used N values of 10, 8, and 4 and T values of 384, 96, and 64 for AlpacaEval 2.0, Arena-Hard-Auto, and MT-Bench, respectively.Source: §4 · p. 5

The effectiveness of our structured response

• The structured response achieved the lowest −log p(winner = NullModel) among 16 persuasive responses in the null-response comparison.The comparison used default and swapped response positions.Source: §4 · p. 5, Figure 4 · p. 6
• GPT-4-1106-Preview generally favored the first response under the default configuration, revealing a position bias in its pairwise judgments.The observed preference is consistent with the reported position-bias analysis.Source: §4 · pp. 5–6

Empirical results

• 86.5% LC win rate on AlpacaEval 2.0, 83.0% on Arena-Hard-Auto, and 9.55 on MT-Bench were achieved after random-search optimization.These results were reported for the structured response combined with random search and compared favorably with verified state-of-the-art results.Source: §4 · p. 6, Table 2 · p. 6

Sanity check

• The Llama-3 auto-annotators were validated against 2.5K human annotations, with Llama-3-8B-Instruct comparable to ChatGPT and Llama-3-70B-Instruct matching GPT-4.This sanity check supported their use as evaluation models.Source: §5 · p. 7, Table 3 · p. 7

Is the structured response useful on open-source auto-annotators?

• The structured response was considerably less effective on Llama-3 than on GPT-4, especially for Llama-3-8B-Instruct, although Llama-3-70B-Instruct showed stronger swap-setting effects.The authors suggest that instruction-following capability may explain part of this difference.Source: §5 · p. 7, Figure 5 · p. 8
• Llama-3-8B-Instruct showed little position bias, whereas Llama-3-70B-Instruct displayed a stronger preference for the last output under the swapped setting.The larger model behaved more similarly to GPT-4 in its response to positional bias and the structured response.Source: §5 · p. 7, Figure 5 · p. 8

Is random search effective on open-source auto-annotators?

• 95.4% LC and 86.3% raw win rates were achieved on Llama-3-8B-Instruct, while Llama-3-70B-Instruct reached 95.1% LC and 91.6% raw after random search.Without random search, the corresponding LC win rates were 2.9% and 0.4%.Source: §5 · p. 7

Does searching on the test instructions directly help?

• Direct optimization on test instructions raised Llama-3-8B-Instruct to 99.8% LC and 99.4% raw, and Llama-3-70B-Instruct to 99.4% LC and 98.2% raw.These results represent the upper-bound setting with test-instruction access.Source: §5 · pp. 7–8

Can our method be combined with normal responses?

• With GPT-3.5-0613, initial win rates were already high, while the null, non-informative model began much lower and steadily increased to competitive win rates as optimization progressed.The null setting was more challenging because it offered no relevant information to input queries.Source: §5 · p. 8

Anti-cheating strategies

• Benchmark developers are urged to integrate specialized detectors targeting adversarial manipulations of LLM-based evaluation systems.The recommendation is presented as an immediate proactive measure for improving benchmark safety and integrity.Source: §6 · p. 8
• Table 4 frames test-instruction visibility as an upper bound, with direct access producing nearly perfect win rates for the Llama-3 targets.This comparison emphasizes the remaining advantage of optimizing directly on benchmark instructions.Source: Table 4 · p. 9

Template paraphrasing

• 92.1% LC win rate was achieved on the unseen official template after optimization on paraphrased templates.This result shows that keeping the real template private while releasing paraphrased versions was insufficient.Source: §6 · p. 9, Table 5 · p. 10

PPL filter

• A windowed PPL filter failed to consistently identify adversarial outputs, including a structured response with a 76.8% LC win rate below the filtering threshold.The threshold was set to the maximum perplexity observed on reference-model outputs.Source: §6 · p. 9

LLM-based evaluation

• LLM judges have been used for AI feedback, summarization, hallucination detection, and broad capability benchmarks including MT-Bench, Chatbot Arena, AlpacaEval, and ArenaHard.These systems address the cost and time required for human evaluation of open-ended generation.Source: §7 · p. 10

Attacking LLM-based evaluations

• Prior analyses identify position bias, verbosity bias, self-enhancement bias, and limited grading capability as limitations of LLM-as-a-Judge systems.These documented weaknesses motivate robustness analyses of automatic evaluation.Source: §7 · p. 10
• Adversarial attacks on language models have been studied across question answering, document classification, sentiment analysis, and toxicity detection.Related work also finds that LLMs can be distracted by irrelevant context.Source: §7 · p. 10

• Null models generating irrelevant yet structured outputs can exploit automatic benchmarks to achieve deceptively high win rates.Source: §8 · p. 10
• Reliable and fair automatic LLM evaluation requires anti-cheating mechanisms and reconsideration of benchmark design.Source: §8 · p. 10

• The experiments used publicly available benchmarks and involved no human subjects or private data, while the methods and results were disclosed to improve evaluation security.Source: ETHICS STATEMENT · p. 11
• Effectiveness on less-studied benchmarks remains uncertain, and the approach relies heavily on manually crafted structured responses.Source: ETHICS STATEMENT · p. 11
• Current defenses targeting output length and style are insufficient against structured responses, motivating more robust anti-cheating mechanisms.Source: ETHICS STATEMENT · p. 11

• LLM-based evaluators reduce the cost and time of judging open-ended generation and are used in benchmarks including MT-Bench, Chatbot Arena, AlpacaEval, ArenaHard, WildBench, and MixEval.Source: A RELATED WORK · p. 17
• Existing research identifies evaluator vulnerabilities including position, verbosity, self-enhancement, limited grading capability, irrelevant-context distraction, and adversarial prompt manipulation.Source: A RELATED WORK · p. 17
• This paper differs by using a completely non-informative null model that emits the same irrelevant output for every instruction, rather than appending adversarial suffixes to meaningful responses.Source: A RELATED WORK · p. 17
• The approach emphasizes universal transferable prompts that do not require access to test instructions and targets widely used benchmarks such as AlpacaEval 2.0 and Arena-Hard-Auto.Source: A RELATED WORK · p. 17
• Jailbreaking research provides related methodology because carefully crafted manual or automated prompts can induce unintended outputs from aligned LLMs.Source: A RELATED WORK · pp. 17–18

B IMPLEMENTATION DETAILS

• Universal Random Search iteratively replaces a randomly selected prefix token, evaluates losses across judge prompts, and returns the prefix with the best loss.Source: B IMPLEMENTATION DETAILS · p. 18
• The implementation includes official and rewritten prompt templates for AlpacaEval 2.0, Arena-Hard-Auto, and MT-Bench, alongside structured cheating-response variants with different complexities.Source: Figure 8 · p. 19, Figure 10 · p. 21, Figure 11 · p. 22

C.1 COMPARISION AGAINST MORE BASELINES

• The structured cheat combined with random search performs better than other attack methods on AlpacaEval 2.0 against GPT-4-1106-Preview.Source: Table 7 · p. 30
• Existing baseline methods yield near-zero win rates, while sufficiently complex structured responses achieve high win rates, showing that response structure is crucial for cheating effectiveness.Source: §C.1 · p. 32

C.2 EVALAUTION AGAINST VARIOUS DEFENSES

• Perplexity defense uses windowed GPT-2 perplexity with a threshold set from clean GPT-4-1106-Preview outputs to filter responses with higher perplexity.Source: §C.2 · p. 32
• Self-Reminder only slightly reduces structured-cheat win rates, whereas 20% Insert and Swap perturbations reduce them to near zero and Patch is also effective.Source: Table 8 · p. 30, §C.2 · p. 33
• Higher SmoothLLM perturbation rates generally improve defense effectiveness, but even 1.25% perturbation drives clean GPT-4 Omni responses to near-zero win rates, making the defense impractical for realistic use.Source: Figure 18 · p. 31, Figure 19 · p. 31, §C.2 · p. 33

C.3 RESULTS ON ADDITIONAL OPEN-SOURCE LLMS

• The strategy remained effective across the additional open-source architectures evaluated, extending the demonstration beyond Llama-3 models.The experiments targeted Mistral-7B-Instruct5 and SOLAR-10.7B-Instruct6.Source: §C.3 · p. 33

C.4 JUDGE TRANSFER

• Direct transfer of a structured response optimized for GPT-4 to GPT-3.5 produced no significant success.The result is reported in Table 10 and motivates developing more transferable responses across judge architectures.Source: §C.4 · p. 33
• For different auto-annotators, stronger instruction-following ability corresponds to a lower structured-response success log-probability objective.Figure 21 relates official AlpacaEval 2.0 LC win rates to the objective −log p(winner = NullModel).Source: Figure 21 · p. 34