PaperlayerView exact revision on arXiv

Source-linked AI summary

Jailbreaks for Black-Box Uncertainty Quantification in Large Reasoning Models

Lucas Biechy, Cédric Eichler, Adrien Boiret, Nicolas Anciaux

arXiv:2609.35350v1cs.AIcs.CLcs.LG

TL;DR

Black-box uncertainty methods for LRMs remain limited, with existing approaches offering little advantage over repeated sampling. The paper introduces relaxation operators and J4U, a jailbreak-derived prompt transformation method, and reports improved calibration over repeated sampling and the evaluated state-of-the-art baseline. It also identifies reliance on jailbreak-style prompts as a limitation because providers may mitigate them.

  • Problem

    Black-box access restricts LRM uncertainty quantification to prompt-level methods, and the effectiveness of existing methods developed for LLMs on LRMs is poorly understood.

    Source: §1 · p. 1

  • Method

    The paper introduces relaxation operators that approximate a higher-KL-regularized policy and proposes J4U, which applies jailbreak-style prompt transformations to benign QA for black-box self-consistency.

    Source: §1 · p. 2, §4 · p. 4

  • Results

    15.8%, 14.5%, and 4.8% reductions in ECE, NLL, and Brier, respectively, over repeated stochastic sampling, compared with 3.1%, 2%, and a 2% increase for the strongest evaluated state-of-the-art approach.

    Source: §1 · p. 2

  • Takeaways & Limitations

    J4U empirically reproduces relaxation's predicted behavioral signatures while improving confidence calibration in black-box LRM QA.

    Source: §6 · p. 9

    The method relies on jailbreak-style adversarial prompts that providers may mitigate, creating a risk that specific jailbreaks become ineffective over time.

    Source: §6 · p. 9

Abstract

from arXiv · show

While Large Reasoning Models (LRMs) excel at complex reasoning, alignment through reinforcement learning often induces systemic overconfidence. In production environments, where logits may be unavailable, robust black-box uncertainty quantification (UQ) is essential for trustworthiness and safety. Focusing on question-answering for LRMs, we show that existing black-box methods, such as paraphrase-based self-consistency and confidence verbalization, offer little to no improvement over simple repeated sampling, suggesting that alignment suppresses useful output variability. We introduce prompt-level relaxation operators that broaden the model's effective output distribution by approximating the effect of an optimal policy obtained with a stronger KL-regularization parameter, hence closer to the reference model. Theoretically, we demonstrate that relaxation improves calibration. We propose Jailbreak for Uncertainty (J4U), a jailbreak-derived technique for UQ that empirically reproduces the behavioral signatures predicted by our relaxation theory. Across 3 datasets and 4 LRMs, including a closed-source production model, J4U's improvement over repeated sampling achieves statistical significance in up to 6 times more LRM-dataset-metric settings than the strongest black-box UQ state-of-the-art baseline we evaluate, with average ECE reductions up to 5 times larger. These results provide a practical tool for UQ in black-box LRM deployment.

1 INTRODUCTION

Black-box access limits LRM uncertainty quantification to prompt-level methods, whose effectiveness on LRMs remains poorly understood. The paper introduces relaxation operators and J4U to improve calibration by restoring useful output variability.

1 INTRODUCTION

  • Existing black-box UQ methods are comparable to repeated sampling on LRMs, which inherit overconfidence and limited predictive variability associated with alignment.The paper links this limited variability to reinforcement learning that can sharpen output distributions and favor deterministic reasoning paths.Source: §1 · pp. 1–2
  • Relaxation operators transform prompts to approximate an optimal policy with a higher KL-regularization parameter, closer to the reference policy, and are shown theoretically to lower ECE.This is proposed as a way to restore output variability suppressed by alignment.Source: §1 · p. 2
  • J4U repurposes jailbreak-style prompt transformations for black-box self-consistency UQ on strictly benign questions, sampling multiple traces across suffix, progressive, and format-manipulation variants.The three instantiations are J4U-SUFFIX, J4U-PROG, and J4U-ART.Source: §1 · p. 2
  • 15.8%, 14.5%, and 4.8% reduction of ECE, NLL, and Brier, respectively, over repeated stochastic sampling; the strongest evaluated state-of-the-art approach achieves 3.1%, 2%, and a 2% increase.These are averages over 4 LRMs and 3 datasets for the best J4U variant.Source: §1 · p. 2
  • J4U reproduces relaxation's predicted signatures of flatter output distributions and more diverse reasoning traces.The introduction also positions black-box confidence calibration as important for LRMs deployed as decision-makers in high-stakes QA settings.Source: §1 · pp. 1–2

2 RELATED WORK

Prior black-box QA uncertainty methods estimate uncertainty from sampled answers, perturbations, or verbalized confidence, while conformal prediction uses confidence or nonconformity scores to form prediction sets. The paper positions J4U as a complementary approach that changes the answer-sampling distribution using jailbreak-style transformations on benign questions.

2 RELATED WORK

  • J4U modifies the answer-sampling distribution, complementing semantic consistency methods that instead assess agreement between responses.For the multiple-choice and exact-match settings studied, semantic clustering reduces to exact-match frequency of the extracted answer.Source: §2 · p. 2
  • Repeated sampling uses answer variability as an uncertainty proxy, while semantic consistency addresses the difficulty of comparing agreement across free-form responses.Semantic methods compare agreement at the level of meaning.Source: §2 · p. 2
  • On LRMs, perturbation-based consistency and confidence verbalization are rarely statistically distinguishable from repeated sampling in accuracy or calibration.These methods were developed and validated primarily on LLMs, motivating methods that restore usable uncertainty signals for LRMs.Source: §2 · p. 3
  • Conformal prediction converts confidence or nonconformity scores into prediction sets with finite-sample coverage guarantees, making it complementary rather than competing with this work.Its framework is agnostic to how the input score is produced.Source: §2 · p. 3
  • Jailbreak prompting is typically evaluated for safety bypass on harmful prompts, whereas this work applies such transformations to benign QA to counteract RL-induced sharpening and recover uncertainty signals.The paper describes its proposal as the first method, to its knowledge, that leverages jailbreaks to improve UQ.Source: §2 · p. 3

3 PROBLEM FORMULATION AND PRELIMINARIES

The paper formalizes black-box multiple-choice QA by treating reasoning traces as latent and mapping each trace to a unique answer. It estimates confidence from sampled answers and frames calibration as alignment between confidence and accuracy under RL-based overconfidence.

3 PROBLEM FORMULATION AND PRELIMINARIES

  • The model is treated as a black box that accepts prompts and returns textual answers, with no access to traces, logits, hyperparameters, or hidden states.The task is multiple-choice QA, with questions and answers drawn from a joint distribution.Source: §3 · p. 3
  • A latent reasoning trace T is mapped deterministically to its final answer by f, while the reasoning policy induces a marginal distribution over answers.The model's reasoning traces are policy-distributed, and the answer is uniquely identified by the realized trace.Source: §3 · p. 3
  • The framework distinguishes a pretrained reference policy from an RL-aligned policy, whose overconfidence motivates input transformations that increase KL regularization toward the reference.The adjustment acts on the input because access is black-box.Source: §3 · pp. 3–4
  • Self-consistency draws K independent traces, selects the majority answer, and uses that answer's sample frequency as its confidence estimate.The frequency is computed over the sampled traces' final answers.Source: §3 · p. 4
  • Calibration requires reported confidence to match conditional success probability, and ECE measures misalignment between predicted probabilities and empirical accuracy.The paper defines overconfidence as confidence exceeding accuracy and notes that it increases ECE.Source: §3 · p. 4

4 THEORETICAL RELAXATION FOR UQ

Relaxation operators transform inputs so an aligned policy behaves like an optimal policy with a higher KL-regularization parameter, shifting it toward the reference policy. Under aggressive RL alignment, suitable relaxation reduces ECE, strictly in the asymptotic limit.

4 THEORETICAL RELAXATION FOR UQ

  • A relaxation operator maps an input question to a modified prompt whose induced policy is equivalent to an optimal policy with a higher KL-regularization parameter.The aligned policy uses β2, while the relaxed policy uses β1 > β2.Source: §4 · p. 4
  • Increasing KL regularization pulls the optimal policy closer to the reference policy because it increases the penalty on KL divergence relative to reward.The operator therefore shifts the aligned policy back toward the pretrained reference policy.Source: §4 · p. 5
  • Under aggressive RL alignment, there exists a higher-regularization range in which relaxation reduces ECE for a finite number of sampled traces, and the reduction is strict as K approaches infinity.The result assumes an overconfident policy and a relaxation parameter β1 above β2 but no greater than β′.Source: §4 · p. 5
  • Relaxation improves calibration when it smooths rewards without changing the theoretical mode that maximizes reward.The proof sketch decomposes ECE in an overconfident regime and bounds finite-sample confidence-estimator bias, which vanishes as K grows.Source: §4 · p. 5
  • As β increases, the theoretical mode's probability decreases and probability mass shifts toward the tails, flattening the predictive distribution and increasing entropy.This motivates expecting J4U instantiations to move predictions toward the higher-entropy reference policy.Source: §4 · p. 5

5 JAILBREAK-STYLE TRANSFORMATIONS FOR UQ: CALIBRATION AND BEHAVIORAL EVIDENCE OF RELAXATION

The section introduces J4U as jailbreak-style prompt perturbations for black-box uncertainty estimation and evaluates their calibration and behavioral signatures across LRMs and datasets. J4U variants improve calibration more often and by larger average margins than the evaluated baselines, while producing the output and reasoning diversity predicted by relaxation.

5.1 J4U AS CANDIDATE INSTANTIATIONS OF RELAXATION

  • J4U applies stochastic prompt transformations that preserve the question payload while mitigating alignment constraints; experiments compare three variants with repeated sampling, verbalized confidence, and rephrasing.The evaluation uses four LRMs, three datasets, and K = 10 samples per question.Source: §5.1 · pp. 5–6, §5.2 · p. 6

5.3 J4U IMPROVES CONFIDENCE CALIBRATION

  • J4U-ART significantly improves calibration in 18/36 settings, versus REPHRASE improvements in 3/36 and degradations in 3/36; VC has no significant calibration gains.No jailbreak transformation significantly decreases accuracy.Source: §5.3 · p. 7, Table 1 · p. 7
  • J4U-PROG reduces ECE by 15.8% on average, while J4U-ART reduces Brier by 4.8% and NLL by 14.5%.J4U-ART's average ECE reduction is 13.8%; J4U-PROG's average Brier and NLL reductions are 4.4% and 14.4%.Source: §5.3 · p. 7

5.4 RELAXATION-OPERATOR PREDICTIONS AND OBSERVED BEHAVIORS IN J4U

  • Across LRMs and datasets, J4U increases predictive entropy, with average increases of 11.8% for SUFFIX, 64.6% for PROG, and 47.7% for ART.The shifts are consistent with the flattening predicted for relaxation operators.Source: §5.4 · p. 8
  • J4U-ART and J4U-PROG produce larger entropy increases than SUFFIX, while responses vary by model, including a +102.9% entropy increase for Qwen3-4B with PROG.The section tests entropy and reasoning-trace diversity as observable implications of relaxation.Source: §5.4 · p. 8
  • J4U reduces reasoning-trace cosine similarity, averaging 3.5% for SUFFIX, 14.6% for PROG, and 28.0% for ART; this supports, but does not establish, the relaxation interpretation.GPT-5.6 Luna is excluded from the trace analysis because its reasoning traces are inaccessible.Source: §5.4 · p. 9, Table 3 · p. 9

6 CONCLUSION AND DISCUSSION

The conclusion presents J4U as a black-box uncertainty method using jailbreak-based prompt perturbations, motivated by the possibility that RL alignment suppresses predictive variability. It reports theoretical and empirical support for relaxation while noting the method's reliance on prompts providers may mitigate.

6 CONCLUSION AND DISCUSSION

  • J4U uses jailbreak-based prompt perturbations for black-box uncertainty quantification, and empirically reproduces relaxation's predicted signatures while improving calibration.The conclusion frames suppressed predictive variability under RL alignment as motivation for the approach.Source: §6 · p. 9
  • The authors caution that jailbreak-style prompts may introduce generic variability rather than reflect true uncertainty; on GSM8K, J4U does not significantly degrade accuracy or ECE.All four tested LRMs achieve 95% accuracy on that benchmark.Source: §6 · p. 9
  • Reliance on adversarial jailbreak prompts creates a risk that providers' mitigation efforts could make specific prompts ineffective over time.The authors report that J4U remains effective across jailbreak families.Source: §6 · p. 9
  • The authors identify access to internal weights and gradients as a future route to analyzing alignment and targeting components responsible for predictive entropy collapse.They suggest this could lead to more robust calibration techniques.Source: §6 · p. 9
  • The work repurposes existing jailbreak techniques only for strictly benign questions and states that it neither proposes new attacks nor improves harmful-content elicitation.Source: ETHICS STATEMENT · p. 10

A.1 PROOF OF THE PROPOSITION 4.2

  • Under the stated policy assumptions, increasing β moves the optimal policy closer to the reference policy because it increases the KL penalty relative to reward.The proof shows the KL divergence from the reference policy decreases as β increases, when reward is non-constant on the reference policy's support.Source: §A.1 · pp. 15–16

A.2 PROOF OF THE THEOREM 4.3

The proof decomposes ECE into expected confidence minus accuracy under overconfidence, then shows that relaxation lowers confidence while accuracy remains stable over a range where the theoretical mode is unchanged. Its conclusion relies on an overconfident, aggressively aligned policy and a sufficiently separated mode.

A.2 PROOF OF THE THEOREM 4.3

  • Relaxation lowers ECE by reducing expected confidence under overconfidence, using the decomposition of ECE into expected confidence minus accuracy.Source: §A.2 · pp. 16–18
  • The proof assumes an overconfident policy produced by excessive RL-based alignment and considers increases in β within a range where the theoretical mode is unchanged.Source: §A.2 · pp. 16–17
  • Aggressive alignment amplifies reward gaps so the highest-reward reasoning's answer dominates, and moderate increases in β can lower confidence without changing that mode.Source: §A.2 · p. 16
  • The accuracy comparison assumes the theoretical mode remains unchanged and its probability gap over the closest competitor is uniformly bounded below.Source: §A.2 · p. 19

B DETAILED EXPERIMENTAL RESULTS

Experiments use four LRMs and three challenging datasets, with repeated samples used to estimate confidence and calibration. Temperature access is specific to the reproducible open-weight setup, not a requirement of J4U's text-in/text-out target setting.

B DETAILED EXPERIMENTAL RESULTS

  • The evaluation includes gpt-oss-20b, Qwen3-4B, DeepSeek-R1-32B, and GPT-5.6 Luna, with no fine-tuning and a 4096-token generation limit.Source: B DETAILED EXPERIMENTAL RESULTS · p. 19
  • Temperature access is an artifact of the open-weight setup rather than a J4U assumption; GPT-5.6 Luna is evaluated through a text-in/text-out API without temperature control.Source: B DETAILED EXPERIMENTAL RESULTS · p. 19
  • The three datasets are SuperGPQA, Humanity’s Last Exam, and SynthPAI; datasets exceeding 1,000 instances are randomly subsampled to 1,000.Source: B DETAILED EXPERIMENTAL RESULTS · p. 20
  • With K=10 samples, confidence is the fraction supporting the selected answer; ECE uses bins for each attainable confidence, while NLL probabilities are clipped at 10^-15.Source: B DETAILED EXPERIMENTAL RESULTS · p. 20

B.1 ACCURACY AND CONFIDENCE CALIBRATION PER LRM AND DATASET

J4U-PROG and J4U-ART improve calibration on the production model without degradation, though significant gains are limited to two settings. Across tested transformations, the simplified suffix variant underperforms the other two.

B.1 ACCURACY AND CONFIDENCE CALIBRATION PER LRM AND DATASET

  • On GPT-5.6 Luna, J4U-PROG and J4U-ART improve ECE and NLL on every dataset, but significant gains occur only for J4U-ART on SynthPAI ECE and NLL.Source: §B.1 · p. 20
  • J4U-SUFFIX consistently underperforms J4U-PROG and J4U-ART; the authors attribute this to its simplified, randomly sampled suffix design.Source: §B.1 · p. 20
  • Tables 4 and 5 compare UQ techniques with repeated sampling for Qwen3-4B and gpt-oss-20b; bold marks statistically significant improvements.Source: Table 4 · p. 21, Table 5 · p. 21
  • The authors suggest that jailbreak effectiveness may correlate with operator strength, with higher-quality jailbreaks inducing stronger calibration effects.Source: §B.1 · p. 21

B.2 DETAILED EVALUATION ON HLE - OPEN-ENDED QUESTIONS

On exact-match open-ended HLE, J4U-ART improves ECE in three of four settings, while iteration analysis shows calibration gains from additional samples level off after about five. Accuracy remains largely stable as sampling increases.

B.2 DETAILED EVALUATION ON HLE - OPEN-ENDED QUESTIONS

  • J4U-ART improves ECE in 3 of 4 HLE exact-match settings, with statistically significant gains on Qwen3-4B, gpt-oss-20b, and DeepSeek-R1-32B.It shows no improvement on GPT-5.6 Luna; no method achieves statistically significant NLL improvements in this setting.Source: §B.2 · pp. 21–22, Table 8 · p. 23

B.3 ACCURACY AND CONFIDENCE CALIBRATION AS A FUNCTION OF THE NUMBER OF ITERATIONS

  • At K = 1, self-consistency methods assign 100% confidence, while additional samples enable frequency-based confidence estimates and reduce overconfidence.The paper reports all results at K = 10; Figure 1 plots accuracy and calibration for K = 1, . . . , 10 on four representative LRM–dataset pairs.Source: §B.3 · pp. 22–23
  • Increasing K leaves accuracy largely stable while ECE and Brier decrease sharply for small K, with marginal calibration gains after approximately K ≈5.NLL decreases more gradually, and it remains unclear whether it has plateaued at K = 10.Source: §B.3 · p. 23

B.4.1 FLATTENING OUTPUT DISTRIBUTION

J4U-PROG and J4U-ART broaden output diversity more consistently than suffix and rephrase prompts, while J4U keeps sampling independent and parallelizable with modest token overhead. Rephrase can increase diversity alongside degraded reliability in a reported configuration.

B.4.1 FLATTENING OUTPUT DISTRIBUTION

  • J4U-PROG and J4U-ART significantly increase entropy in 12 and 11 of 12 settings, respectively, sometimes doubling it relative to RS.REPHRASE reaches the largest relative increase (×2.6) on SuperGPQA–DeepSeek-R1-32B, alongside significant accuracy, Brier, and NLL degradation; the authors hypothesize that it weakens question semantics.Source: §B.4.1 · pp. 23–24

B.4.2 DIVERSIFICATION OF REASONING TRACES

  • J4U-ART reduces reasoning-trace cosine similarity in 8 of 9 settings and J4U-PROG in 7 of 9, while suffix and rephrase often diversify little.On SynthPAI, J4U-ART approximately halves cosine similarity across all LRMs; exceptions occur where DeepSeek-R1-32B already has low baseline similarity on SuperGPQA and HLE.Source: §B.4.2 · p. 25

C PROMPT EXAMPLES

  • The prompt variants alter question presentation through a sampled suffix, rephrasing, fragment reconstruction in J4U-PROG, or replacing a word with ASCII art in J4U-ART.J4U-PROG asks the model to predict from reconstructed text semantics, while J4U-ART asks it to infer the masked word from the ASCII art and context.Source: §C.1 · p. 26, §C.3 · p. 27, §C.4 · p. 27

D.1 COMPUTATIONAL COMPLEXITY AND INFERENCE COST

  • J4U retains O(K) complexity with K independent queries that can run in parallel, reducing wall-clock latency relative to sequential approaches.Even the most expensive variant, J4U-ART, incurs less than half the total-token overhead of VC and REPHRASE; J4U-SUFFIX and J4U-PROG reduce wall-clock latency.Source: §D.1 · p. 29

D.2 ALL EVALUATED LRMS EXHIBIT SYSTEMIC OVERCONFIDENCE IN OUR SETTINGS

Repeated sampling reveals overconfidence across evaluated LRM–dataset settings, including among incorrect answers. In a high-certainty gsm8k setting, J4U does not significantly degrade accuracy or calibration, unlike random-noise perturbations.

D.2 ALL EVALUATED LRMS EXHIBIT SYSTEMIC OVERCONFIDENCE IN OUR SETTINGS

  • Reliability-diagram bins lie consistently below the perfect-calibration diagonal across settings, indicating systemic overconfidence.Only three lowest-confidence bins show minor crossings, which do not alter the overall trend.Source: §D.2 · p. 29, Figure 2 · p. 30
  • Approximately 60% of incorrect Qwen3-4B answers and 65% of incorrect GPT-5.6 Luna answers on SynthPAI receive confidence above 0.9.Across models and datasets, erroneous answers are concentrated near confidence 1.Source: §D.2 · p. 29, Figure 3 · p. 30

D.3 J4U DOES NOT SIGNIFICANTLY DEGRADE ACCURACY OR CONFIDENCE CALIBRATION IN A HIGH-CERTAINTY SETTING

  • On gsm8k, where evaluated LRMs achieve approximately 95% accuracy and RS has ECE ≈0.04, J4U does not significantly degrade accuracy or ECE.Unlike J4U, NOISE 20% consistently degrades both accuracy and calibration, while NOISE 10% degrades calibration only.Source: §D.3 · p. 31, Table 12 · p. 31

Related papers

Loading 2609.35350v1…