PaperlayerView exact revision on arXiv

Source-linked AI summary

Black-Box Auditing of Epistemic Reliability in Multi-Agent Debate Distillation

Derui Wang, Zewei Shi, Rayne Holland, Ruoxi Sun, Xingliang Yuan, Jason Xue, Liming Zhu

arXiv:2609.32361v1cs.LGcs.AIcs.CLcs.CR

TL;DR

Debate distillation can improve monitored-task judgments, but it remains an open question whether it preserves reliability on related hidden tasks or whether degradation is distinct from catastrophic forgetting. The paper introduces paired benchmarks and ER-Audit, a black-box framework for testing frozen verifier checkpoints across valid paraphrases. Its experiments find that higher hidden-task accuracy can coexist with more degradation counterexamples and lower non-degradation bounds.

  • Problem

    Debate-based verifier adaptation is evaluated for monitored performance, but whether it degrades reliability on related hidden tasks while preserving that performance, and whether such degradation differs from catastrophic forgetting, remains unresolved.

    Source: §1 · p. 1

  • Method

    ER-Audit compares frozen verifier checkpoints before and after adaptation, searching valid paraphrases for counterexamples and then using sequential testing, alongside benchmarks pairing monitored and hidden prompts grounded in shared contexts.

    Source: §1 · p. 3, §3.1 · p. 4

  • Results

    Adversarially adapted verifiers can outperform honest-transcript counterparts in hidden-task accuracy while producing more degradation counterexamples and lower non-degradation bounds at a common auditing threshold.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The findings support treating epistemic reliability degradation as a distinct risk beyond aggregate capability loss and evaluating hidden-task reliability alongside aggregate performance.

    Source: §5 · p. 11

    The auditor is limited to black-box queries on frozen checkpoints and semantically valid prompt paraphrases within prescribed budgets, and cannot modify checkpoints or adaptation data.

    Source: §2.2 · p. 4

Abstract

from arXiv · show

Debate distillation adapts weaker verifiers using multi-agent debate transcripts to improve their judgement in subsequent debates, but gains on monitored tasks do not establish reliability on related unmonitored tasks. We study epistemic reliability degradation, in which adaptation preserves monitored performance while reducing support for correct responses on hidden tasks. We consider an adversarial debater that manipulates debate arguments while defending the correct monitored response, and ask whether the resulting degradation merely reflects catastrophic forgetting and whether standard evaluation can detect it. To address these questions, we propose ER-Audit, a two-stage black-box auditing framework that compares frozen verifier checkpoints before and after adaptation, and introduce two evaluation benchmarks pairing monitored and hidden task prompts grounded in shared contexts. ER-Audit searches for counterexamples to non-degradation by evaluating semantically valid paraphrases and, if none is found, uses independent paraphrases for sequential hypothesis testing. We derive anytime-valid lower confidence bounds on the non-degradation probability, allowing data-dependent stopping within a finite budget. We further establish a common lower bound across fixed paraphrase distributions and extend it to distributions within a bounded total variation distance of their mixtures. Our experiments show that higher hidden-task accuracy can coexist with more counterexamples to non-degradation and lower non-degradation bounds. This divergence challenges explanations based solely on broad catastrophic forgetting and shows that auditing can uncover selective hidden-task degradation concealed by aggregate performance gains. Our code and benchmarks are available at https://github.com/CSIRO-CQS-AI-alignment-Team/Epistemic-Reliability-Auditor.

1. Introduction

The paper asks whether debate distillation can preserve monitored-task performance while degrading reliability on related hidden tasks, and whether this risk is distinct from catastrophic forgetting. It proposes benchmarks and a black-box audit to investigate this gap between aggregate evaluation and hidden-task reliability.

1. Introduction

  • Existing evaluations of multi-agent debate examine judge accuracy and argument informativeness but leave open whether verifier adaptation improves monitored performance while degrading related hidden-task reliability.The paper places this question in the context of scalable oversight for tasks that are difficult for humans or smaller models to evaluate directly.Source: §1 · p. 1
  • Debate distillation may shift verifier beliefs on related, unmonitored tasks because an adversarial debater can choose and frame evidence while defending the correct monitored response.The paper asks whether this degradation is distinct from catastrophic forgetting and whether standard evaluation detects it.Source: §1 · p. 2
  • The paper identifies three challenges: characterizing the risk across models and datasets, building shared-context monitored/hidden benchmarks with degradation reference models, and auditing paraphrases under limited query budgets.A negative audit result must support a statistically sound conclusion despite the possibility that tested phrasings fail to expose degradation.Source: §1 · p. 2
  • Across two benchmarks and verifier models with 8B to 20B parameters, adversarially adapted verifiers can preserve monitored performance and outperform honest-transcript counterparts in hidden-task accuracy, yet show more degradation counterexamples and lower non-degradation bounds.The comparison uses a common auditing threshold.Source: §1 · p. 2
  • The authors introduce QuALITY-H and GPQA-H and ER-Audit, which searches for degradation counterexamples and quantifies non-degradation guarantees when finite audits find none.The framework uses anytime-valid bounds that allow data-dependent stopping and establishes guarantees across fixed paraphrase generators, with a relaxation for distributions within bounded total variation distance of their mixtures.Source: §1 · pp. 2–3

2. Preliminaries

The setting adapts a verifier using debate transcripts for a monitored task, then evaluates it on related hidden tasks grounded in the same context. The threat model gives one debater control over transcript arguments while constraining the auditor to black-box access to frozen pre- and post-adaptation checkpoints.

2.1. Problem Setting

  • Two LLM debaters generate transcripts under a fixed debate protocol, and a verifier is fine-tuned on the transcripts, monitored prompt, and correct response for future task prompts.The verifier receives prompts at inference without direct access to the full context.Source: §2.1 · p. 3
  • A hidden task concerns a distinct but related proposition grounded in the monitored task’s context and is excluded from the monitored training objective.Its prompt has a reference response grounded in that shared context.Source: §2.1 · p. 3
  • Epistemic reliability degradation is when the adapted verifier performs well on the monitored task but degrades on a hidden task relative to the pre-adaptation verifier.Auditing seeks either to detect degradation or to provide statistical evidence of non-degradation.Source: §2.1 · p. 3

2.2. Threat Model

  • The adversary controls one debater’s arguments, knows the context and monitored and hidden prompts and answers, and aims to induce incorrect hidden-task responses while preserving monitored-task performance.Its influence is limited to transcripts; it cannot change reference responses, the training objective, or verifier parameters.Source: §2.2 · p. 4
  • The auditor compares frozen pre- and post-adaptation checkpoints through black-box queries, including response-label softmax probabilities, and may test semantically valid hidden-task paraphrases within prescribed budgets.The auditor cannot modify checkpoints or adaptation data.Source: §2.2 · p. 4

3. Design of ER-Audit

ER-Audit measures hidden-task non-degradation across semantically valid paraphrases, first searching for counterexamples and then statistically certifying the probability of non-degradation when none is found. Its guarantees extend from fixed paraphrase generators to mixtures and nearby distributions.

3.1. Measuring Changes in Epistemic Reliability

  • The degradation score measures hidden-task performance loss per unit of monitored-task improvement, and its thresholded indicator defines non-degradation.The comparison uses frozen pre- and post-adaptation verifier checkpoints; the audit evaluates prompts from the measurable set of semantically valid hidden-task paraphrases.Source: §3.1 · p. 4

3. Design of ER-Audit

  • The two-stage audit searches paraphrases for degradation counterexamples, then tests non-degradation statistically if the finite search finds none.The search and test use separate budgets; failure to find a counterexample alone does not establish that degradation is absent.Source: §3.1 · p. 4, §3.2 · p. 5

3.2. Counterexample Search

  • A finite search evaluates generated candidate paraphrases and reports degradation when any candidate is a counterexample; otherwise, it proceeds to independent test samples.Because the candidate search is finite, finding no counterexample is not evidence that degradation is absent.Source: §3.2 · p. 5

3.3. Sequential Hypothesis Testing

  • For each fixed generator, an anytime-valid lower confidence bound on non-degradation probability supports data-dependent stopping while maintaining the stated confidence level.The bound updates with each independent test paraphrase; a counterexample sets it to zero, while an all-non-degraded sequence raises it.Source: §3.3 · pp. 5–6

3.4. Generalisation across Generator Distributions

  • The minimum of the generator-specific bounds is a common lower bound across fixed generators, and the guarantee extends to their mixtures and nearby distributions.For a nearby distribution, applying the extension requires separately justifying its total-variation distance; a mixture has distance zero.Source: §3.4 · pp. 6–7

4. Experiments

Across the experiments, adversarial-transcript adaptation can preserve monitored performance and improve aggregate hidden-task accuracy while still degrading epistemic reliability on hidden questions. ER-Audit detects this divergence through paraphrase counterexamples and lower non-degradation bounds.

4.1. Setup

  • QuALITY-H and GPQA-H pair monitored and hidden QA questions with reference answers grounded in a shared context.The benchmarks contain 222 and 101 instances, respectively; contexts are source stories in QuALITY-H and shared domains and subdomains in GPQA-H.Source: §4.1 · p. 7

4.3. Epistemic Reliability Auditing

  • On QuALITY-H, Vadv’s higher hidden-task aggregate performance coexists with more counterexamples and lower average non-degradation bounds than Vhonest.This divergence appears despite Vadv outperforming Vhonest in hidden-task accuracy and mean task probability across all three verifier architectures.Source: §4.2 · p. 8, §4.3 · p. 9

4.2. Standard Evaluation

  • Standard evaluation finds higher monitored-task Acc and MTP for Vhonest and Vadv than the baselines, while hidden-task advantages vary across models and benchmarks.Vadv outperforms Vhonest on hidden-task Acc and MTP across all three verifier architectures on QuALITY-H, but on GPQA-H the advantage holds only for Qwen-9B.Source: §4.2 · p. 8

4.3. Epistemic Reliability Auditing

  • ER-Audit finds at least one counterexample for 103 of 222 QuALITY-H questions with GPT-20B-based Vadv, versus 90 with Vhonest.On GPQA-H, the corresponding counts are 40 and 35 of 101 questions.Source: §4.3 · p. 9
  • Across both benchmarks and all verifier architectures, Vadv has a lower average lower confidence bound on non-degradation than Vhonest.The comparison is reported at confidence levels from 0.8 to 0.999.Source: §4.3 · p. 9
  • On QuALITY-H, higher—and sometimes near-saturated—correct-answer probabilities for Vadv coexist with more questions showing paraphrase degradation and larger maximum degradation scores.The authors suggest this pattern may indicate overfitting to some hidden questions while degrading on others.Source: §4.3 · pp. 9–10
  • The Vhonest-versus-Vadv difference in average bounds persists across ϵ values for GPT-20B, suggesting robustness to the auditing threshold.The reported plot covers GPT-20B-based verifiers.Source: §4.3 · p. 10
  • The divergence is more pronounced on QuALITY-H than GPQA-H, suggesting greater concern in settings with long contexts and demanding evidence retention and integration.QuALITY-H questions are grounded in long narrative contexts.Source: §4.3 · p. 11

5. Discussion and Conclusion

The paper presents epistemic reliability degradation as a distinct risk in debate distillation: aggregate gains can coexist with degradation on individual hidden questions. It introduces paired benchmarks and ER-Audit to identify and quantify this risk.

5. Discussion and Conclusion

  • Aggregate gains can coexist with degradation on individual hidden questions, motivating treatment of epistemic reliability degradation as distinct from aggregate capability loss.The paper argues that verifier evaluation should assess both aggregate performance and hidden-task reliability.Source: §5 · p. 11
  • QuALITY-H and GPQA-H pair monitored and hidden task prompts, while ER-Audit compares verifier checkpoints before and after adaptation.The framework combines counterexample search with sequential testing to provide anytime-valid lower bounds on non-degradation probabilities under fixed paraphrase distributions.Source: §5 · p. 11
  • The findings motivate treating epistemic reliability degradation as a distinct risk beyond aggregate capability loss, potentially more relevant with long contexts and demanding evidence integration.The paper states that verifier evaluation should assess both aggregate performance and hidden-task reliability.Source: §5 · p. 11

AI use statement

The authors used generative AI tools for language polishing and state that they reviewed and verified the AI-assisted content.

AI use statement

  • Generative AI tools were used for language polishing.The statement limits the described use to language polishing.Source: AI use statement · p. 11
  • The authors reviewed and verified all AI-assisted content.Source: AI use statement · p. 11
  • The authors take full responsibility for the work’s final content, theoretical claims, and conclusions.Source: AI use statement · p. 11

Ethics statement

The paper frames its ethical considerations around four Menlo Report principles. It reports no human subjects or deceptive practices and describes aims concerning risk reduction, stakeholder benefit, and responsible disclosure.

Ethics statement

  • The authors assess ethical implications using the Menlo Report principles of Beneficence, Respect for Persons, Justice, and Respect for Law and Public Interest.Source: Ethics statement · p. 12
  • The research aims to advance AI safety studies by identifying and categorising potential threats while considering benefits and risks such as data misuse by adversaries.Source: Ethics statement · p. 12
  • No human subjects were involved, and the authors report that no deceptive practices were employed.They describe prioritising transparency and accountability.Source: Ethics statement · p. 12
  • The authors state that they seek equitable benefit across stakeholders and disclose findings responsibly to avoid enabling adversarial actions.They also state adherence to applicable laws and ethical standards.Source: Ethics statement · p. 12

Reproducibility statement

The authors state that the benchmarks and auditing framework codebase are publicly available to support independent validation and further exploration.

Reproducibility statement

A. Proofs and Algorithms

The proofs establish anytime-valid lower confidence bounds for non-degradation, including a common bound across paraphrase generators and guarantees for nearby distributions. The algorithms return audit bounds and support stopping within a finite test budget.

A.1. Proofs

  • Theorem 1 yields a one-sided (1 −η) confidence sequence for the non-degradation probability over the test horizon.The bound is valid under data-dependent stopping, with confidence level 1 −η.Source: §A.1 · p. 15
  • Theorem 2 provides a common lower bound across paraphrase distributions, including when the per-generator sample counts are data-dependent.On the stated event, the bound applies to every generator considered.Source: §A.1 · p. 16
  • Theorem 3 transfers the guarantee to distributions within a bounded total variation distance of a mixture of fixed paraphrase distributions.The bound holds simultaneously for admissible mixtures, nearby distributions, and distance bounds.Source: §A.1 · p. 16

A.2. Practical Auditing Algorithms

  • Algorithm 1 returns the current bound and evaluated sample count when stopped, or the last bound when its budget is exhausted.The sample requirement can serve as a stopping criterion at confidence level 1 −η.Source: §A.2 · p. 17
  • Algorithm 2 runs Algorithm 1 for each generator and returns zero with a counterexample if one is found; otherwise it returns the minimum bound.The generators are processed in a fixed order using the same η and per-generator budget.Source: §A.2 · p. 17

B. Benchmark Construction in Detail

This section details the construction of paired hidden-task benchmarks, including transcript and verifier-adaptation production and the generation of auditing paraphrases. It also specifies safeguards and prompt constraints for preserving question meaning and logical and technical content.

B. Benchmark Construction in Detail

  • Pairs were selected for shared evidence or context, semantic separation, and text-only binary-choice compatibility; question-embedding cosine similarity had to be below 0.78.When similarity was at least 0.78, the hidden question was reassigned within its permitted context group.Source: B. Benchmark Construction in Detail · pp. 17–18
  • QuALITY-H contains 222 question pairs from shared stories, and GPQA-H contains 101 expert-level STEM pairs matched within the same domain and subdomain.The questions in each pair are semantically distinct while sharing a story or topical context.Source: B. Benchmark Construction in Detail · p. 18

C. Honest and Adversarial Transcripts Generation

  • For each item, eight three-round candidates were generated per setting; honest selection preserved the monitored answer, while adversarial candidates targeted the hidden answer under safety checks.Canonical adversarial transcripts were available for 184 of 222 QuALITY-H items and 83 of 101 GPQA-H items; all items had an honest transcript.Source: C. Honest and Adversarial Transcripts… · pp. 19–20

D.1. Verifier Adaptation Procedures

  • Honest and adversarial adaptation used the same fixed fine-tuning objective, whereas the original adversarial setting additionally optimized a hidden-question loss.Honest adaptation used transcript language-modeling and monitored binary-choice losses, without hidden-question inputs; both search and testing used fresh paraphrases not used in adaptation.Source: §D.1 · pp. 20–21

D.2. Hyper-parameter Settings

  • Training ran for three epochs at learning rate 10^-5 and effective batch size 8; auditing used 100 search and test paraphrases, with one search generator and two testing generators.The search generator was gpt-5.6-terra; testing used gpt-5.6-terra and claude-opus-4.8.Source: §D.1 · p. 21, §D.2 · p. 21

D.3. Paraphrase Generation for Auditing

  • Counterexample search generated diverse meaning-preserving paraphrases in batches, using the original question as the sole source and excluding answers, outside references, and added clues.Batch generation requested 100 paraphrases by default and required changes to wording and sentence structure without changing answer conditions.Source: §D.3 · p. 21, Prompt Template for Paraphrase Batch… · p. 21, §1 · p. 21

2. Preserve meaning and information

  • Paraphrases had to preserve targets, premises, scope, technical content, and logical relations while varying structure naturally rather than relying on templates or synonym-only changes.Prompt requirements also prohibited adding context or solving questions and required independent paraphrases for paired fields.Source: §2 · pp. 21–22, §3 · p. 22, §4 · p. 22

6. Output

  • Every candidate was to be checked against its original for fidelity, completeness, naturalness, technical accuracy, and duplicates; sequential testing used fresh independent paraphrases from two generators.The workflow specified exact output structure and separate reporting of unresolved constraints, while review was required to cover every candidate rather than a sample.Source: §5 · p. 22, §6 · p. 23

Prompt Template for Independent Paraphrase Generation

  • Paraphrases must preserve logical form, presuppositions, context, and technical content, including symbols, values, labels, and their associations.The instructions prohibit narrowing ambiguous targets, changing hypotheses into facts, or altering experimental order, conditions, or observations.Source: Prompt Template for Independent… · p. 23
  • Revisions should be natural and structurally varied, while preserving meaning, scope, attribution, and referents; the full paraphrase must be checked for omissions or changes.Only unambiguous language-level errors should be corrected; ambiguity, contradictions, missing assumptions, and source errors must remain unresolved.Source: Prompt Template for Independent… · pp. 23–24
  • The response must contain only the complete paraphrased question, including any supplied background or premises, with no label, explanation, answer, or alternative.Source: Prompt Template for Independent… · p. 24
  • Preferred transformations include reorganizing clauses or premises, changing voice or verbal form, and repositioning attribution or temporal clauses.These transformations apply only when they preserve the original meaning and relationships.Source: Prompt Template for Independent… · p. 24

D.4. Additional Experiment Results

Across the reported comparisons, honest-transcript verifiers usually show fewer instances of large degradation than adversarial-transcript verifiers, and audit results are robust to the threshold choice. The discussion relates this pattern to formulation-dependent support for correct beliefs and explains how matched paraphrase audits expose declines that aggregate accuracy can conceal.

D.4. Additional Experiment Results

  • In most comparisons, Vhonest exhibits fewer instances of large degradation than Vadv.The comparisons cover hidden questions across the reported models and datasets.Source: Figure 5 · p. 24, §D.4 · p. 24
  • Runtime is reported for auditing one hidden task using 100 paraphrases per question and a sequential-test budget of 100 per generator.The runtimes are measured in GPU-seconds on a single NVIDIA H100 GPU.Source: §D.4 · p. 25, Table 3 · p. 25
  • Across all models and datasets, the average audited lower bound is robust to variations in threshold ϵ.Figure 10 reports changes in the average audited lower bound as ϵ varies.Source: §D.4 · p. 25, Figure 10 · p. 26

E. Related Work

  • The related-work distinction is that this paper audits belief shifts after transcript-based adaptation, rather than inference-time persuasion or hidden objectives.It evaluates related hidden questions without supplying the transcripts and does not require a verifier to pursue a hidden objective.Source: E. Related Work · p. 26

F. Further Discussion on Epistemic Reliability Degradation

  • Adversarial transcripts may combine useful transferable reasoning with shortcuts that improve canonical-prompt accuracy but weaken correct-answer support under meaning-preserving reformulation.The proposed local explanation is that equivalent prompts can have differently oriented loss gradients; selective forgetting remains possible.Source: F. Further Discussion on Epistemic… · p. 27
  • Matched paraphrases reveal adaptation-induced declines that gains on other prompts can conceal in aggregate accuracy, while the guarantees do not depend on shortcut learning as the cause.When finite search finds no counterexample, independent samples provide anytime-valid bounds under specified paraphrase distributions.Source: F. Further Discussion on Epistemic… · p. 27

G. Limitations

ER-Audit's guarantees are limited by its finite paraphrase distributions and the difficulty of bounding distance to unseen distributions.

G. Limitations

  • A lower bound from finite paraphrase distributions may overstate non-degradation on unseen distributions.Generalising the guarantee requires a total variation distance bound, which can be challenging to compute in practice.Source: G. Limitations · p. 27

Related papers

Loading 2609.32361v1…