PaperlayerView exact revision on arXiv

Source-linked AI summary

Conformity Breaks Conformal Prediction

Yibo Hu, Hanyu Su

arXiv:2609.04445v1cs.LGcs.CL

TL;DR

Clean-calibrated conformal prediction may not cover correct answers when an LLM sees peers asserting a wrong answer, even though the question is unchanged. The paper evaluates this score-mechanism shift in multi-agent LLM settings. Coverage drops from 90% to 74%, with a sharper loss on the targeted subgroup.

  • Problem

    Clean conformal calibration does not establish how an LLM will score answers when peer messages change its context, despite the question remaining the same.

    Source: §1 · p. 1

  • Method

    The paper evaluates clean-calibrated conformal prediction under peer pressure, treating the question as the input and peer transcripts as context that generates scores.

    Source: §1 · p. 1, §2 · p. 2

  • Results

    90% to 74% marginal coverage under unanimous-wrong peers, while coverage on the attacker-targeted subgroup falls from 87% to 47%.

    Source: §1 · pp. 1–2

  • Takeaways & Limitations

    An attacker controlling the peer channel can invalidate a clean-calibrated certificate without changing the question, and the paper reports that standard conformal fixes do not restore exchangeability under an adaptive adversary.

    Source: §1 · p. 1, §4.2 · p. 7, §8 · p. 11

    The effect is model-dependent and not claimed to be universal; condition-aware recalibration requires pressure-labeled data and widens prediction sets until the layer escalates nearly everything.

    Source: §8 · p. 11

Abstract

from arXiv · show

A conformal certificate can be valid when an LLM answers alone and invalid when the same LLM sees peers that unanimously assert a wrong answer. The question is unchanged; the model's score for the correct answer changes. We call this a score-mechanism shift: clean calibration certifies how the model scores answers alone, but not how it scores them under peer pressure. We show that this shift silently breaks conformal prediction in multi-agent LLM systems. Across open-weight models and multiple-choice QA tasks, coverage falls from a calibrated 90% to 74% under unanimous-wrong peers at the standard alpha = 0.10 operating point. The average hides a sharper failure: by targeting the low-confidence items the certificate still covers, an attacker nearly halves coverage on that subgroup, from 87% to 47%, while the monitored average remains much higher. The failure also reaches the decision layer: a system that should escalate when uncertain can instead become confident enough to act on the attacker's wrong answer. Standard conformal fixes do not solve the problem, because the question distribution has not changed; the model's scoring behavior has.

1 INTRODUCTION

Clean-calibrated conformal prediction can fail under peer pressure even when the question is unchanged, because peers can shift the model’s score for the correct answer. The paper shows that this failure includes hidden subgroup coverage loss and failures of an act-vs-escalate defense.

1 INTRODUCTION

  • An attacker controlling the peer channel can trigger the shift silently without changing the input.The paper contrasts this with covariate shift, which conformal prediction’s standard toolkit is designed to address.Source: §1 · p. 1
  • Coverage on the attacker-targeted low-confidence subgroup falls from 87% to 47%, while the monitored average remains 76%.An adversary without the answer key can target a chosen 10% of items, taking 40 pp from the subgroup while moving the overall average by 4.Source: §1 · pp. 1–2
  • Under unanimous-wrong peers, marginal coverage falls from 90% to 74% because the model’s score for the correct answer changes while the question stays identical.The paper names this adversarially controllable score change a score-mechanism shift.Source: §1 · p. 1
  • The clean-calibrated act-vs-escalate layer commits to the attacker’s answer on 12% of flipped items for the pooled predictor and 71% for Qwen2.5-32B.The closest prior work, Wang et al. (2026), uses conformal prediction to defend multi-agent debate, but treats peer influence as benign and defers the conditional case.Source: §1 · pp. 1–2

2 SETUP: CONFORMAL PREDICTION UNDER PEER PRESSURE

The setup calibrates a conformal prediction set on solo answers, then evaluates the same questions under peer pressure, treating the transcript as context rather than part of the question. It measures how peer-induced score changes affect coverage across models, tasks, and pressure conditions.

2 SETUP: CONFORMAL PREDICTION UNDER PEER PRESSURE

  • Conformal prediction keeps each candidate answer whose nonconformity score is at most a calibrated threshold, chosen as the (1−α)-quantile of correct-answer scores.The paper uses split conformal prediction with the LAC score and requires calibration and test data to be exchangeable for its coverage guarantee.Source: §2 · p. 2
  • The question is x, while the peer transcript is context that generates the model’s score rather than part of x.Thus, calibration and test use the same question even when peer pressure changes the score mechanism.Source: §2 · p. 2
  • Under unanimous-wrong peers, Qwen2.5-7B has an answer-flip rate of about 75%, distinct from the rate at which scores cross the coverage threshold.The paper reports that these rates closely track each other under the studied unanimous-wrong condition.Source: §2 · p. 2
  • The study records solo and peer-pressured answer distributions across six conditions, using four core open models on ARC-Challenge and TruthfulQA.It also includes two further models and five more task types in two of three seeds; headline numbers use the no-authority unanimous-wrong condition.Source: §2 · pp. 2–3
  • Coverage is the fraction of test items whose correct-answer probability meets the calibrated threshold, averaged over 2,000 stratified calibration/test splits.The splits are 50/50 and stratified by model and dataset.Source: §2 · p. 3

3 MAIN RESULTS

Unanimous-wrong peers break clean-calibrated coverage, with the sharpest loss hidden on low-confidence items the certificate still covers; the shift can also defeat an act-vs-escalate policy.

3.1 UNANIMOUS-WRONG PEERS BREAK MARGINAL COVERAGE

  • At α = 0.10, coverage under unanimous-wrong pressure is 73.9% ± 0.3%, versus 90.0% ± 0.0% for the round-1 certificate.The reported mean is over three seeds.Source: Table 1 · p. 3, §3.1 · p. 3
  • Coverage drops −16.1 pp at α = 0.10 and −8.6 pp at α = 0.05, but the drop vanishes at α = 0.01; unanimous-correct peers improve coverage.At α = 0.01, the threshold makes the prediction set nearly vacuous.Source: §3.1 · p. 3, Figure 1 · p. 4

3.2 TARGETED PRESSURE CREATES A HIDDEN CONDITIONAL COLLAPSE

  • For the pooled predictor, targeted-band coverage falls from 87.3% to 47.4% while marginal coverage remains 75.7%, near the monitored 76%.The band is the decile of solo confidence immediately above the calibrated threshold; every model drops under pressure.Source: Figure 2 · p. 4, §3.2 · p. 4
  • Three adversaries without answer keys target subsets using deployment confidence, prediction-set size, or an outside model, reducing their coverage by 36.4–40.2 pp.Attacking a random 10% costs 14.3 pp; targeted attacks on 10% move overall coverage from 90.1% to 86.0%.Source: §3.2 · pp. 4–5
  • The targeted-band collapse holds for every model under LAC; APS avoids it here only with sets averaging 4.0 of 4 options.The pooled predictor falls further than three of the four models, whose shifts are common-mode; a wider bottom-20% band understates the attack.Source: §3.2 · p. 5

3.3 THE CLEAN-CALIBRATED ACT-VS-ESCALATE DEFENSE FAILS

  • For Qwen2.5-32B, targeted-band coverage falls from 87.2% to 30.3%, and the singleton-act-on-wrong rate reaches 71.4%.These results are for the largest open model tested.Source: §3.3 · p. 5
  • Tightening α to 0.05 lowers the singleton-ACT rate from 12.1% to 1.9%, but leaves a 21.3 pp targeted-band shortfall and reduces autonomous action.The average pressured set grows from 2.44 to 3.31 of four options; no single offline-calibrated α closes both failures.Source: §3.3 · pp. 5–6
  • Among conformity-flipped items, the clean-calibrated act-vs-escalate layer commits the wrong answer without escalation on 12.1% of pooled cases.The policy acts on singleton sets and escalates when set size is at least two.Source: §3.3 · p. 5, Table 2 · p. 6

4 WHY THE FAILURE OCCURS AND WHY STANDARD FIXES DO NOT SUFFICE

The failure comes from peer pressure changing scores at a fixed question, not from a change in the input distribution. The section traces the resulting coverage loss, tests standard repairs, and finds that restoring coverage requires pressure-aware calibration with a substantial autonomy trade-off.

4.1 THE SHIFT IS IN THE SCORE MECHANISM, NOT THE INPUT

  • For a fixed question, peer pressure changes the score distribution; the coverage gap equals net threshold-crossing flow, with 19.8% pushed out and 3.2% pulled in under unanimous-wrong peers.The resulting net loss is 16.6 pp; under mixed and unanimous-correct peers, inflow exceeds outflow and the gap is negative.Source: §4.1 · pp. 6–7

4.2 WEIGHTED CONFORMAL PREDICTION TARGETS THE WRONG SHIFT

  • Weighted conformal prediction is the identity when the question is unchanged, and cannot separate pressured from clean items using a likelihood ratio over questions.The paper also notes that an adaptive attacker leaves no fixed pressured-context distribution for reweighting.Source: §4.2 · p. 7

4.3 THE SHIFT SURVIVES PER-MODEL CALIBRATION

  • At α = 0.10, per-model Mondrian calibration still leaves three of four models below target; at α = 0.05, it largely rescues Qwen and Mistral.Under the stricter threshold at α = 0.10, Llama reaches ≈72%, Qwen 61%, and Mistral ≈74%; at α = 0.05, Mondrian lifts Qwen to ≈90% and Mistral to ≈88%.Source: §4.3 · p. 7

4.4 RESTORING COVERAGE REQUIRES THE PRESSURE LABELS THE ATTACK DENIES

  • Under-pressure calibration restores marginal coverage to 90.0% and targeted-band coverage from 47.4% to 83.9%, but requires advance knowledge of which items will be pressured.That knowledge is unavailable in the stated attack setting, where pressure labels are denied.Source: §4.4 · p. 8
  • The coverage repair widens prediction sets from 2.4 to 3.1 of 4 options and reduces autonomous action to 3.6% of items, versus 44.6% with clean calibration and no pressure.Coverage is bought by escalating almost everything under the act-vs-escalate rule.Source: §4.4 · p. 8

5 ROBUSTNESS AND GENERALIZATION

The collapse varies by model and tracks score movement more closely than solo confidence. A flip-rate stress test can rank vulnerability across tasks, and the coverage decline also appears across seven task types.

5 ROBUSTNESS AND GENERALIZATION

  • The collapse persists under per-model calibration, while weighted conformal prediction is the identity for a fixed question and pressured-distribution calibration restores 90.0% marginal coverage.The restored coverage comes with an act-on-wrong rate decrease from 12.1% to 2.6%.Source: §5 · p. 8

5.1 VULNERABILITY TRACKS SCORE MOVEMENT, AND A STRESS TEST CAN RANK IT

  • Under per-model thresholds, Qwen-7B coverage falls to 61%, Llama-8B to 72%, Mistral-7B to 74%, while Gemma-9B holds at 90%.The shared threshold makes Llama appear to reach 99.3%, an artifact of global calibration rather than a model property.Source: §5.1 · p. 8
  • Gemma and Qwen have nearly identical solo confidence, but Qwen’s ground-truth probability drops by 0.747 under pressure versus 0.151 for Gemma.The paper identifies score movement, rather than starting confidence, as the driver of vulnerability.Source: §5.1 · p. 8

5.2 THE COLLAPSE GROWS ACROSS SEVEN TASK TYPES

  • Across six models and seven task types, unanimous-wrong coverage falls to 69.8%, a −20.2 pp gap, and every task type degrades by −5 to −28 pp.The answer-flip stress test also transfers across task types, though geometric shapes is the exception to its cross-task prediction.Source: §5.2 · p. 9, §5.1 · p. 9

6 THREAT MODEL AND DEPLOYMENT IMPLICATIONS

The threat model gives an attacker control of peer messages and item selection, while denying access to the target model’s internals or calibration data. The deployment discussion recommends stress-testing candidate models and suppressing autonomous action on singleton sets when peer influence is present.

6 THREAT MODEL AND DEPLOYMENT IMPLICATIONS

  • An attacker can inject fabricated unanimous-wrong peer responses, choose items without an answer key, and act at inference time after the certificate is frozen.The attacker controls orchestration or compromises an agent but has no access to model weights, logits, or calibration data.Source: §6 · p. 9
  • Clean offline calibration followed by test-time peer injection breaks exchangeability without changing the question; pressure-aware recalibration requires labels the attacker denies.Calibration on the pressured distribution returns marginal coverage to 90.0%, but that is not the stated realistic deployment order.Source: §6 · p. 9
  • Shared message buses make false peer responses realizable, but a full exploit inside a deployed framework is left to future work.The paper recommends ranking candidate models with a cross-task answer-flip stress test and suppressing autonomous action on singleton sets where peer influence exists.Source: §6 · pp. 9–10

7 RELATED WORK

The related work connects the paper to conformal validity, LLM applications, social conformity, and multi-agent debate. Its distinction is an adversarial shift in peer communication that breaks a certificate while leaving the input untouched.

7 RELATED WORK

  • Prior conformal work establishes exchangeability-based validity, covariate-shift reweighting, and limits on exact distribution-free conditional coverage; the paper locates its failure in score changes instead.Weighted CP and Mondrian calibration do not absorb this shift, while APS appears robust because its sets degenerate to all four options.Source: §7 · p. 10
  • LLM conformal work spans language modeling, multiple-choice QA, factuality, abstention, sampling, and risk control, with newer work extending to agent pipelines and LLM judges.The cited agent-pipeline and judge work assumes clean calibration.Source: §7 · p. 10
  • The paper connects its attack to documented human conformity and LLM sycophancy, distinguishing behavioral answer flips from realized score-based coverage loss.Hao et al. (2026) show raw flip rate overstates conformity; the paper uses flips to compare models and coverage gaps to measure harm.Source: §7 · p. 10
  • Unlike input-perturbation defenses, this attack targets the communication channel; unlike prior debate work stopping at accuracy, it demonstrates a certificate-level consequence.Wang et al. (2026) study a marginal act-vs-escalate defense and defer the conditional case that this paper examines.Source: §7 · pp. 10–11

8 SCOPE, LIMITATIONS, AND FUTURE WORK

The tested conformity attack invalidates clean-calibrated coverage, with the sharpest loss on a targeted subgroup, but its magnitude varies across models and the authors do not claim universality. They identify pressure-labeled recalibration as a costly fix and call for practical defenses and broader tests.

8 SCOPE, LIMITATIONS, AND FUTURE WORK

  • The effect was consistent across tested models, tasks, and three runs, but its magnitude depended on pressure-induced changes in correct-answer probability and is not claimed to be universal.The reported scores are token probabilities; verbalized-score runs showed the same collapse but could not be reproduced from released data.Source: §8 · p. 11
  • Condition-aware recalibration fixes the failure but requires pressure-labeled calibration data and widens prediction sets until the system escalates nearly everything.The authors note that defenders usually cannot obtain the needed labels in advance.Source: §8 · p. 11
  • Proposed next steps include detecting pressured items, selective abstention, synthetic-context calibration, worst-case conditioning, and online or drift-detection conformal prediction.Further work should test safety-domain data, a systematic wrong-peer-count dial, and an end-to-end deployed exploit.Source: §8 · p. 11
  • The paper’s broader conclusion is that multi-agent certificates must account for social context, not just the input.Source: §8 · p. 11

A ROBUSTNESS AND GENERALIZATION

The robustness decomposition attributes model differences under unanimous-wrong pressure to changes in the correct-answer probability, rather than to starting confidence or buffer. Experiments hold questions and solo answers fixed while resampling peer transcripts.

A ROBUSTNESS AND GENERALIZATION

  • Across seeds, questions and solo answers stay fixed while fabricated peer transcripts are resampled, so seed spreads measure sensitivity to transcript variation.Seed 0 covers four models and two datasets; seeds 1–2 cover six models and seven task types.Source: A ROBUSTNESS AND GENERALIZATION · p. 14
  • The robustness gap is driven by pressure-induced change in correct-answer probability, not static confidence or buffer.Under per-model thresholds, the four models’ coverage under pressure is reported as 61%, 72%, 74%, and 90%; the shared threshold makes Llama appear immune.Source: A ROBUSTNESS AND GENERALIZATION · p. 14

B FULL CONDITIONAL-COVERAGE RESULTS

The conditional-coverage results show that the attack is concentrated in the confidence band immediately above the calibrated threshold, while pressure-dose analyses illustrate how partial pressure can affect coverage.

B FULL CONDITIONAL-COVERAGE RESULTS

  • Coverage falls most sharply in the decile just above the calibrated threshold; below threshold, it rises from 13.4% to 23.6% because many items are already confidently wrong.The bottom-20% band averages these two regimes and understates the attack; the authors therefore report the above-threshold band.Source: B FULL CONDITIONAL-COVERAGE RESULTS · p. 14
  • Gemma and Qwen begin with similar round-1 confidence, but their answer-flip rates are 16% and 76%, and their pgt drops are −0.15 and −0.75, respectively.The robustness gap is attributed to ∆pgt rather than static confidence or buffer.Source: Figure 3 · p. 15
  • Per-model Mondrian calibration still leaves three of four models below 90% coverage under unanimous-wrong pressure at α = 0.10, while Gemma remains near 90%.At α = 0.05, Mondrian largely rescues vulnerable models, with coverage of 88–92%.Source: Figure 4 · p. 15

C EXTRA PROBE: PRESSURE DOSE-RESPONSE

  • A no-answer-key adversary attacking 10% of selected items costs 36–40 percentage points on those items, versus 14.3 points for random targeting, while overall coverage shifts by at most 4 points.Ranking by pgt instead selects a decile already excluded by the threshold, where coverage starts at 13.4% and the attack raises it.Source: Figure 5 · p. 16

C.1 HOW MUCH PRESSURE IS NEEDED TO BREAK COVERAGE?

  • On a synthetic logit-interpolation axis, Qwen falls below 80% coverage at t ≈ 0.65 and Mistral at t ≈ 0.80; the curve is illustrative, not an attacker-controlled dial.Gemma and Llama do not fall below 80% on this axis; the authors leave the real wrong-peer-count dial for future work.Source: C EXTRA PROBE: PRESSURE DOSE-RESPONSE · p. 15, §C.1 · p. 16

D VALIDATING THE PRE-DEPLOYMENT STRESS TEST

The stress-test checks support using applied peer pressure to rank candidate models, but they do not establish a safe flip-rate cutoff. Qwen size comparisons also caution against using model size as a proxy for conformity susceptibility.

D VALIDATING THE PRE-DEPLOYMENT STRESS TEST

  • Dynamic predictors that apply peer pressure rank pressured coverage, whereas static predictors do not and can reverse direction between regimes.The stress-test evaluation uses per-model thresholds with clean coverage of 89.9–90.2%; Monte Carlo error is at most 0.4 percentage points.Source: D VALIDATING THE PRE-DEPLOYMENT STRESS… · pp. 16–17
  • Across different task types, flip-rate rankings transfer out of sample with median ρ = −0.49 and 35 of 42 comparisons in the right direction; a sign test gives p = 7.5 × 10^-6.The authors describe this as a ranking, not a cutoff: six models per regime and four model families are too few to establish a safe threshold or settled ranking.Source: D VALIDATING THE PRE-DEPLOYMENT STRESS… · p. 17

E QWEN SIZE SCALING: MODEL SIZE IS NOT A SAFE PROXY

F CANONICAL NUMBERS AND REPRODUCIBILITY

G PROOF OF PROPOSITION 1

The proof decomposes the coverage gap into threshold crossings and shows that if pressured scores satisfy F1(τ̂) ≤ F0(τ̂), coverage under pressure is at most 1 − α.

G PROOF OF PROPOSITION 1

  • The coverage gap Δ equals the net flow of items crossing the clean calibrated threshold.Down-crossings from covered to uncovered raise Δ; up-crossings from uncovered to covered lower it, while non-crossings cancel.Source: G PROOF OF PROPOSITION 1 · p. 19
  • Covered items pushed above the threshold raise Δ, while uncovered items pulled below it reduce Δ.Items that do not cross the threshold cancel in the decomposition.Source: G PROOF OF PROPOSITION 1 · p. 19
  • If F1(τ̂) ≤ F0(τ̂), then pressured coverage is at most 1 − α.The calibrated threshold is the (1 − α)-quantile of clean scores; stochastic dominance implies the inequality but is not required.Source: G PROOF OF PROPOSITION 1 · p. 19
Loading 2609.04445v1…