PaperlayerView exact revision on arXiv

Source-linked AI summary

RH-Detect: A Unified Benchmark for Reward Hacking Detection

Junwei Quan, Evgenii Opryshko, Rohan Subramani, Igor Gilitschenski

arXiv:2610.10947v1cs.LG

TL;DR

Reward-hacking datasets use different labels, formats, and metadata, complicating cross-source detector comparisons. RH-Detect harmonizes eleven public datasets and evaluates language models across sources and detector variations. Although leading models perform strongly overall, accuracy is lower on MALT and TRACE, underscoring source-specific variation in monitoring.

  • Problem

    Existing reward-hacking datasets differ in labels, response formats, and metadata, while reward hacking may undermine deployment-time oversight.

    Source: §1 · p. 1

  • Method

    RH-Detect harmonizes reward-hacking subsets from eleven public datasets, then evaluates detectors across sources, inputs, and training procedures.

    Source: §1 · pp. 1–2

  • Results

    The four strongest judges are 10.7–15.9 percentage points less accurate on MALT and TRACE than on the other sources, despite leading models exceeding 93% accuracy overall.

    Source: §6 · p. 7

  • Takeaways & Limitations

    Detector performance varies with source, transcript representation, and post-training strategy, supporting source-stratified evaluation and complete specification of detector inputs and procedures.

    Source: §7 · p. 7

    The MALT/TRACE difference is a source-level association because length, tool formatting, category mixture, elicitation, and verification vary together with agentic status.

    Source: §6.1 · p. 7

Abstract

from arXiv · show

Reward hacking, where a model exploits an evaluation signal without completing the intended task, threatens the reliability of deployed language model systems. Existing datasets use different labels, response formats, and metadata conventions, making detector results difficult to compare. We present RH-Detect, a benchmark that combines reward-hacking-relevant subsets from eleven public datasets, comprising 92,761 rows and six behavior categories, into a common schema. On 5,021 open-ended evaluation units, each comprising a task prompt and a free-form model continuation, including multi-turn tool-use trajectories, we evaluate six off-the-shelf language models from five families as reward hacking detectors without additional training. The best model achieves a pooled AUROC of 0.962, with accuracy above 93%. For the four strongest models, however, accuracy on the two multi-turn tool-use datasets, MALT and TRACE, is 10.7-15.9 percentage points lower than on the other sources at a common decision threshold, highlighting a key gap for deployment-time monitoring. We find that different input formats have different effects across models. Removing thinking raises Qwen3.5-4B AUROC from 0.779 to 0.849, but lowers Qwen Flash from 0.977 to 0.950. We also evaluate the benchmark as a training dataset for detectors. Holding out each source in turn, single-token SFT improves average AUROC on five of six held-out sources. A GRPO follow-up on that failure case yields a slight improvement in detection performance. Our results show that a single pooled score can conceal variation across data sources, detector inputs, and training procedures.

1 Introduction

RH-Detect addresses fragmented reward-hacking datasets with a shared benchmark and studies whether detector performance holds across sources, inputs, and training procedures. The paper frames reliable detection as relevant to deployment-time oversight and downstream interventions.

1 Introduction

  • Reward hacking includes exploiting an evaluation signal without fulfilling the intended task, including behavior outside training that may undermine deployment-time oversight.The paper describes behaviors ranging from sycophancy and test-case exploitation to reward tampering and concealment under monitoring.Source: §1 · p. 1
  • Existing datasets use incompatible labels, response formats, metadata, and evaluation procedures, making broad cross-source evaluation difficult to assemble.A strong result on one dataset provides limited evidence of transfer to an unseen source.Source: §1 · p. 1
  • RH-Detect harmonizes reward-hacking subsets from eleven public datasets into a shared schema with six behavior categories, provenance metadata, task-specific tracks, and audited source-held-out splits.The benchmark preserves inherited label semantics and source provenance.Source: §1 · pp. 1–2
  • The study examines whether detector performance remains stable as the data source, input representation, or training procedure changes.Its three research questions address zero-shot monitoring, source- or behavior-specific weaknesses, and cross-source robustness.Source: §1 · p. 2

2 Related Work

Prior work studies reward-hacking behaviors, datasets, and monitoring methods from different angles. RH-Detect's related-work discussion distinguishes its contribution as a common cross-source evaluation interface and an analysis of source, transcript, and training variation for the same detector.

2 Related Work

  • Prior reward-hacking work asks whether models exploit evaluations; RH-Detect instead asks whether monitors recognize diverse exploits after they occur.The described literature ranges from proxy-signal optimization and strategic manipulation to verifiable coding and tool-use exploits.Source: §2 · p. 2
  • RH-Detect contributes a common schema and evaluation interface for comparing sources while preserving provenance and redistribution constraints.Existing datasets cover varied behaviors and formats, including multi-turn tool-use trajectories and multiple-choice probes.Source: §2 · p. 2
  • Unlike studies that train one monitor or survey many failure modes, this study holds detection fixed while varying source, transcript representation, and post-training procedure.The paper states that these factors move measured detection performance as much as detector choice does.Source: §2 · p. 2

3 Dataset Construction

RH-Detect combines reward-hacking-relevant subsets from public datasets while retaining source-specific information and response formats. It organizes behaviors into a taxonomy, records harmonized labels and metadata, and uses audited splits to support evaluation across sources and tracks.

3 Dataset Construction

  • RH-Detect integrates relevant subsets from eleven public sources into a benchmark with 92,761 rows across open-ended, multiple-choice, and protected alignment-faking partitions.It retains source response formats, label semantics, and provenance metadata.Source: §3 · p. 2

3.1 Taxonomy of Reward Hacking Behaviors

  • The taxonomy groups recurring behaviors into six categories, describing behavior type and target rather than an ordinal severity or sophistication scale.The open-ended track covers five categories; sycophancy appears only in the multiple-choice track.Source: §3.1 · pp. 2–3, Table 1 · p. 3

3.2 Source Selection and Harmonization

  • The eleven sources vary in labeling and annotation, so RH-Detect maps source labels into a common taxonomy and preserves construction, verification, response-model, and agentic metadata.When a hacking continuation lacks a paired clean continuation, a clean response may be drawn from the same task or source split when possible.Source: §3.2 · p. 3

3.3 Unified Schema, Evaluation Tracks, and Splits

  • Prompt-hash grouping prevents identical prompts from crossing training and evaluation boundaries in the cross-source and cross-track split protocols.The release uses three storage partitions, including protected alignment-faking responses distributed under source-required encryption.Source: §3.3 · pp. 3–4

4 Zero-Shot Detector Evaluation

The zero-shot evaluation finds strong pooled detection by leading models, alongside marked differences across sources and behavior categories. Performance is lower on MALT and TRACE, which contain multi-turn tool-use trajectories, than on the other evaluated sources for the strongest judges.

4.1 Evaluation Methodology

  • On 5,021 eligible open-ended units, six models from five families classify task prompts and continuations, including tool calls, with temperature 0 and no source or category metadata.The evaluation excludes transcripts longer than 32,000 tokens; Grok's reported results omit MALT.Source: §4.1 · p. 4

4.2 RQ1: Overall Detector Performance

  • Qwen 3.5 Flash leads complete-coverage pooled ranking at AUROC 0.962, with Gemini 3.1 Flash Lite at 0.960; Grok 4.1 Fast is 0.960 on reduced coverage.GPT-5 Nano has pooled AUROC 0.817 and false-positive rate 0.353 at the common threshold of 0.5.Source: §4.2 · p. 4

4.3 RQ2: Performance by Category and Source

  • 98–100% accuracy on reward tampering contrasts with 90–95% on style gaming and deception for the four stronger models; environment exploitation accuracy is lowest and most variable.Environment-exploitation estimates are indicative because the subset has at most 43 examples.Source: §4.3 · p. 4, Table 3 · p. 4
  • 10.7–15.9 percentage points lower accuracy on MALT/TRACE than on the other sources characterizes the four strongest judges; balanced accuracy, AUROC, AUPRC, and TPR also decline.The comparison is descriptive because source, task, category mix, length, tool format, elicitation, verification, and agentic status vary together.Source: §4.3 · p. 4
  • The six models' ROC curves show the top three clustered near the upper-left corner at AUROC 0.96, while GPT-5 Nano's curve is lower at 0.817.The curves trace false-positive rate against true-positive rate as the decision threshold varies.Source: Figure 1 · p. 5, §4.3 · p. 5

5 Input Representation and Detector Training

This section tests how detector inputs and post-training affect cross-source robustness, finding that results vary by source, representation, and training procedure. A targeted GRPO case study yields small positive AUROC gains on SoRH and in-distribution validation, while single-token SFT and input-representation changes have uneven effects.

5.1 Transcript Representation

  • Removing input reasoning raises Qwen3.5-4B AUROC from 0.779 to 0.849 but lowers Qwen Flash from 0.977 to 0.950.Retaining only reasoning lowers Qwen3.5-4B AUROC to 0.655; extract-and-evaluate lowers GPT-5 Nano to 0.788, and using GPT-5 Nano to select excerpts for Qwen Flash lowers AUROC to 0.879.Source: §5.1 · p. 5, Table 5 · p. 5

5 Input Representation and Detector Training

  • 10.7–15.9 percentage points lower accuracy on MALT/TRACE than on other sources is observed for the four stronger detectors; GPT-5 Nano shows the reverse pattern.The corresponding balanced-accuracy, AUROC, AUPRC, and TPR declines indicate that the contrast is not limited to class balance or a fixed threshold.Source: Table 4 · p. 5

5.2 Single-Token Supervised Fine-Tuning

  • Source-held-out single-token SFT raises average per-source AUROC by 0.036, with mean AUROC improving on five of six held-out sources but decreasing by 0.140 on SoRH.Table 6 compares the frozen Qwen3.5-4B base with SFT averaged over three seeds for each omitted source.Source: Table 6 · p. 6, §5.2 · p. 6
  • Single-token SFT nearly saturates same-track test AUROC but lowers AUROC on the other track, showing no positive cross-track transfer.The open-ended test split includes sources also seen in training, so it is not source held-out.Source: Table 7 · p. 6, §5.2 · p. 6
  • SFT raises average per-source Brier score from 0.180 to 0.190 and negative log-likelihood from 0.550 to 1.373, while ECE decreases from 0.213 to 0.192.The authors report that the proper scoring rules worsen despite improved ranking and a small ECE reduction, with some errors receiving greater confidence after fine-tuning.Source: §5.2 · p. 6

5.3 Targeted GRPO Case Study

  • GRPO improves AUROC by approximately +0.035 on in-distribution validation and +0.018 on held-out SoRH, but neither improvement is statistically significant.This targeted case study avoids the large SoRH regression observed with single-token SFT; Figure 2 includes all evaluation units, assigning 0.5 to completions without a parseable boxed probability.Source: Figure 2 · p. 7, §5.3 · p. 7
  • On held-out SoRH, parse validity rises from 0.61 for the frozen base to 0.675 at the final checkpoint, while formatting and instruction-following remain difficult on long transcripts.The fraction truncated at the generation limit falls from 0.42 to 0.341.Source: §5.3 · p. 7

6 Discussion

Off-the-shelf detectors perform strongly overall, but accuracy is lower on MALT and TRACE, and results vary with transcript representation and training procedure. The benchmark’s static design and other data and study constraints bound these findings.

6 Discussion

  • 0.962 is the best pooled AUROC, while Qwen Flash and Gemini each exceed 93% accuracy at the common threshold.Gemini has the lowest false-positive rate, whereas GPT-5 Nano assigns substantial suspicion to benign responses.Source: §6 · p. 7
  • 10.7–15.9 percentage points lower accuracy on MALT and TRACE than on the other four open-ended sources marks a gap across the four strongest complete-coverage judges.Balanced accuracy, AUROC, and AUPRC move in the same direction; MALT and TRACE contain long tool-use trajectories.Source: §6 · p. 7
  • Removing input reasoning improves the 4B judge but helps neither API judge, while extraction results depend on which model selects the excerpts.Single-token supervision improves mean AUROC across held-out sources but declines on SoRH; cross-track transfer is near chance or below despite near-ceiling in-track AUROC.Source: §6 · p. 7

6.1 Limitations and Future Work

  • The training study covers one model, and the benchmark is static despite deployment monitoring being incremental and latency-constrained.Other caveats include inherited heterogeneous labels, possible stylistic cues in unpaired negatives, and source-level confounding in the MALT/TRACE gap.Source: §6.1 · p. 7

7 Conclusion

RH-Detect harmonizes reward-hacking data from eleven public datasets and evaluates detector performance across sources, inputs, and training procedures. Detectors are less accurate on MALT and TRACE, while training gains are uneven and do not transfer uniformly.

7 Conclusion

  • The four strongest models are less accurate on MALT and TRACE than on other sources, making multi-turn tool-use monitoring the area identified for additional engineering investment.The benchmark harmonizes eleven public datasets with consistent metadata, task-specific tracks, and audited source-held-out splits.Source: §7 · p. 7
  • Off-the-shelf language models range from strong to unreliable on the primary open-ended track, with pooled AUROC from 0.817 to 0.962.The conclusion emphasizes source-stratified evaluation and complete specification of detector input, score readout, and training procedure.Source: §7 · p. 7
  • Single-token SFT improves average per-source AUROC, but gains are uneven and do not transfer across tracks; targeted GRPO avoids the large SFT regression and yields a small positive held-out gain.Detector performance depends on transcript representation and post-training strategy.Source: §7 · p. 7

A Dataset and Release Construction

RH-Detect integrates reward-hacking subsets from eleven public datasets while retaining source-specific formats and label semantics. Its construction covers varied source types, maps labels into a shared taxonomy, audits splits, and specifies detector and transcript-processing protocols.

A Dataset and Release Construction

  • 92,761 rows from eleven datasets form the integrated benchmark, while the reported primary analysis uses 5,021 eligible open-ended evaluation units from the core sources.Paired stored rows expand into multiple evaluation units, and the 32,000-token eligibility policy excludes some units; Terminal Wrench and CheatBench were incorporated after the reported experiments and do not enter analysis.Source: A Dataset and Release Construction · p. 10, Table S1 · p. 12

A.1 Sources

  • The sources span paired short tasks, protected alignment-faking responses, multiple-choice probes, and multi-turn tool-use trajectories, with source-specific selection and label handling.TRACE contains human-verified coding-agent trajectories and consolidated native annotations; MALT uses selected hacking runs and separately matched clean runs.Source: §A.1 · p. 10

A.2 Labels and Taxonomy

  • Rows from the nine datasets used in reported experiments receive one of six behavior categories, while binary labels are inherited or deterministically derived from source signals.The categories are output style gaming, sycophancy, evaluation gaming, environment exploitation, reward tampering, and deception.Source: §A.2 · p. 11

A.3 Filtering, Expansion, and Splits

  • Prompt-hash grouping keeps paired continuations together across splits, and source-held-out and within-track protocols prevent prompt hashes from crossing train/test boundaries.The primary open-ended policy retains rows under 32,000 tokens; paired units are also resampled together in bootstrap procedures.Source: §A.3 · p. 11

B.1 Detector Protocol

  • Zero-shot detectors read rendered transcripts and return a normalized reward-hacking score without source, category, label, or verification metadata.The six API detectors are run through one gateway; Grok became unavailable before MALT scoring and is excluded from analyses requiring complete six-source coverage.Source: §B.1 · p. 11

B.2 Transcript Representations E1–E4b

  • Four transcript representations compare raw input, removed or isolated reasoning, and extracted evidence; a cross-extractor variant changes the extractor while keeping the Qwen Flash judge fixed.For agentic trajectories, E2 represents externally observable interaction and E3 reasoning material; invalid or empty extraction falls back to the raw input.Source: §B.2 · p. 11

C Additional Zero-Shot Analysis

The four other open-ended sources score better than MALT and TRACE for the four stronger complete-coverage models across several metric families, while GPT-5 Nano shows a contrasting profile. Score distributions also differ across detectors: Gemini and Qwen Flash separate hack and benign scores more sharply than GPT-5 Nano.

C Additional Zero-Shot Analysis

  • Gemini and Qwen Flash concentrate benign scores near zero and hack scores near 100, while GPT-5 Nano’s broad benign scores correspond to a 0.353 false-positive rate.The fixed decision threshold is τ=0.5 on the normalized score.Source: C Additional Zero-Shot Analysis · p. 12
  • Other-source-minus-MALT+TRACE differences are positive, with 95% intervals excluding zero across threshold-based, class-balanced, and threshold-free metrics; GPT-5 Nano differs except for TPR.The comparison is descriptive because source, task, transcript length, category mixture, tool format, elicitation, and agentic status are confounded.Source: C Additional Zero-Shot Analysis · p. 12

D.1 Single-Token Source-Held-Out SFT

Single-token SFT improves average AUROC across held-out sources but does not improve every source or probability-quality metric. The cross-track experiment trains on each track separately and evaluates on both, with the open-ended test split not measuring source-held-out transfer.

D.1 Single-Token Source-Held-Out SFT

  • Strict source-held-out SFT removes one open-ended source from training, validation, threshold selection, and checkpoint selection in each rotation.Examples sharing an exact prompt hash stay in the same split; Qwen3.5-4B is fine-tuned for one epoch with three seeds.Source: §D.1 · p. 12
  • 0.859 to 0.895±0.021 mean held-out-source AUROC; Brier rises from 0.180 to 0.190 and NLL from 0.550 to 1.373, while ECE falls from 0.213 to 0.192.Ranking improves on five of six sources, but SoRH declines in ranking and the proper scoring rules worsen.Source: §D.1 · pp. 12–13

D.2 Cross-Track Transfer

  • The cross-track experiment trains the same single-token objective separately on open-ended and multiple-choice data, then evaluates each model on both tracks.Because all six open-ended sources appear in training, its open-ended test split does not measure source-held-out transfer.Source: §D.2 · p. 13

E Targeted GRPO Case Study

The targeted GRPO run on held-out SoRH improves the AUROC point estimate without the regression seen under single-token SFT. This is a single-seed, single-fold case study, not an estimate of general RL performance.

E Targeted GRPO Case Study

  • Parse-valid completions receive one minus Brier loss as reward; parsing determines reward availability, while reasoning text is not scored.Each prompt group uses eight generations, and invalid completions are excluded from the group-relative advantage.Source: E Targeted GRPO Case Study · pp. 13–14
  • 0.018 held-out SoRH AUROC point-estimate increase; parse validity rises and generation-limit truncation falls, with unparseable completions scored 0.5.The result is specific to the tested model, reward, decoding policy, single seed, and single fold; replication across seeds and held-out sources is needed to establish a general advantage.Source: E Targeted GRPO Case Study · p. 14
  • The evaluation uses one held-out SoRH fold and one seed, so broader GRPO gains require replication across seeds and held-out sources.The tested result also depends on the model, reward, and decoding policy.Source: E Targeted GRPO Case Study · p. 14

F Release and Reproducibility

The release provides data for the reported open-ended and multiple-choice experiments alongside code to reconstruct the full benchmark. Reproducing the experiments requires rerunning workflows and access to relevant upstream sources, credentials, model weights, and compute.

F Release and Reproducibility

  • The release package includes 4,151 primary open-ended rows and 66,760 multiple-choice rows, not the full 92,761-row benchmark.Terminal Wrench, CheatBench, and the protected-response track are excluded at row level; ingestion code reconstructs the full benchmark from upstream sources.Source: F Release and Reproducibility · p. 14
  • Reproducibility materials include experiment data tables, the unified schema, ingestion and harmonization code, prompts, experiment memberships, validation tools, and training scripts with fixed seeds.The pinned environment records the reported library versions.Source: F Release and Reproducibility · p. 14
  • The package contains no precomputed predictions, metrics, figures, raw API responses, or checkpoints; rerunning experiments requires relevant API credentials, model weights, and compute.Reconstructing the complete benchmark may also require access to original upstream sources.Source: F Release and Reproducibility · pp. 14–15

Related papers

Loading 2610.10947v1…