PaperlayerView exact revision on arXiv

Source-linked AI summary

RefusalBench: Generative Evaluation of Selective Refusal in Grounded Language Models

Aashiq Muhamed, Leonardo F. R. Ribeiro, Markus Dreyer, Virginia Smith, Mona T. Diab

arXiv:2510.10390v1cs.CLcs.AIcs.LG

TL;DR

RAG models must decide whether flawed or insufficient context supports an answer, but static benchmarks can be exploited and become obsolete. RefusalBench creates diagnostic tests through controlled linguistic perturbations. Evaluations reveal substantial selective-refusal gaps, including among frontier models on multi-document tasks.

  • Problem

    RAG models must decide whether insufficient or unreliable context supports an answer, while static benchmarks are vulnerable to dataset-specific artifacts and obsolescence.

    Source: §1 · p. 1

  • Method

    RefusalBench dynamically perturbs answerable QA pairs using 176 strategies across six uncertainty types and three intensity levels, with generator-verifier consensus quality control.

    Source: §1 · pp. 1–2, §3.2 · p. 3, §3.4 · p. 4

  • Results

    Evaluation of 30+ models reveals severe selective-refusal gaps; even frontier models identify the underlying refusal reason correctly less than 50% of the time in multi-document scenarios.

    Source: §1 · pp. 1–2

  • Takeaways & Limitations

    Selective refusal is a trainable, alignment-sensitive capability that scales independently from answer accuracy.

    Source: §1 · p. 2

    The released benchmarks focus on English text and isolate the language model’s generator component rather than evaluating the full RAG pipeline.

    Source: §6 · p. 9

Abstract

from arXiv · show

The ability of language models in RAG systems to selectively refuse to answer based on flawed context is critical for safety, yet remains a significant failure point. Our large-scale study reveals that even frontier models struggle in this setting, with refusal accuracy dropping below 50% on multi-document tasks, while exhibiting either dangerous overconfidence or overcaution. Static benchmarks fail to reliably evaluate this capability, as models exploit dataset-specific artifacts and memorize test instances. We introduce RefusalBench, a generative methodology that programmatically creates diagnostic test cases through controlled linguistic perturbation. Our framework employs 176 distinct perturbation strategies across six categories of informational uncertainty and three intensity levels. Evaluation of over 30 models uncovers systematic failure patterns: refusal comprises separable detection and categorization skills, and neither scale nor extended reasoning improves performance. We find that selective refusal is a trainable, alignment-sensitive capability, offering a clear path for improvement. We release two benchmarks -- RefusalBench-NQ (single document) and RefusalBench-GaRAGe (multi-document) -- and our complete generation framework to enable continued, dynamic evaluation of this critical capability.

1 Introduction

RAG models must decide whether context supports an answer, yet selective refusal remains difficult and static benchmarks can be exploited. RefusalBench uses controlled perturbations to evaluate this capability and reveals substantial gaps, including among frontier models.

1 Introduction

  • Less than 50% of the time, even frontier models identify the underlying refusal reason correctly in multi-document scenarios; evaluation of 30+ models reveals severe gaps.Selective refusal is described as trainable and alignment-sensitive, scaling independently from answer accuracy.Source: §1 · pp. 1–2
  • Static benchmarks invite exploitation of dataset-specific artifacts and can become obsolete as models evolve, motivating fresh, controlled test instances.Incorrect answers based on flawed information pose risks in high-stakes domains.Source: §1 · p. 1
  • RefusalBench applies 176 perturbation strategies across six uncertainty categories and three intensity levels to evaluate selective refusal.Its generator-verifier consensus pipeline achieves 93.1% human agreement.Source: §1 · pp. 1–2, Figure 1 · p. 2
  • The study identifies selective refusal as a trainable, alignment-sensitive capability that scales independently from answer accuracy.The authors present this as a forward path toward more reliable systems.Source: §1 · p. 2

2 Related Work

Prior work addresses abstention through unanswerable-question benchmarks, targeted failure-mode datasets, taxonomies, and RAG evaluations. RefusalBench instead uses controlled linguistic perturbations to diagnose selective refusal at varying intensities.

2 Related Work

3 The RefusalBench Methodology

RefusalBench turns QA datasets into dynamic diagnostic benchmarks using a linguistic uncertainty taxonomy, controlled perturbations, and multi-model quality control. Its theory, perturbation levels, and verification process are designed to support evaluation as models evolve.

3 The RefusalBench Methodology

  • The methodology combines a formal uncertainty taxonomy, a perturbation engine, and multi-model quality control to turn QA datasets into dynamic diagnostic benchmarks.The pipeline is illustrated in Figure 1.Source: §3 · pp. 2–3

3.1 Generative Evaluation: Theory and Advantages

  • When contamination reaches epsilon, the static error bound becomes vacuous, whereas fresh generative samples retain consistent error bounds as models evolve.Static benchmarks also face distribution drift, adaptive overfitting, and test saturation.Source: §3.1 · p. 3

3.2 A Linguistic Taxonomy of Informational Uncertainty

  • The taxonomy defines six uncertainty types: ambiguity, contradiction, missing information, false premises, granularity mismatch, and epistemic mismatch.Each type corresponds to an expected refusal category when the defect prevents a reliable response.Source: §3.2 · p. 3

3.3 The Perturbation Engine: Levers and Intensity Control

  • The perturbation engine provides 176 linguistic levers across 18 type-intensity combinations, with low, medium, and high levels testing answerability and refusal.Experts authored lever conditions; three LLMs generated examples accepted only by unanimous agreement, followed by human validation. Low intensity should be answered, while medium and high should be refused.Source: §3.3 · pp. 3–4

3.4 Quality Control: The Generator-Verifier Pipeline

  • Multiple generators create perturbations and all models verify them against seven criteria; only unanimous approvals enter the final dataset.The consensus filter is intended to remove ambiguous or model-specific artifacts and maintain quality at scale.Source: §3.4 · p. 4

4 Experiments and Results

Experiments use RefusalBench-NQ and RefusalBench-GaRAGe to assess selective refusal, revealing weak multi-document refusal performance alongside distinct generation, detection, and alignment patterns.

4.1 Experimental Setup

  • RefusalBench-NQ contains 1,600 balanced samples and GaRAGe 1,506 naturally imbalanced samples, built from answerable questions and audited by human experts.Human pass rates were 93.1% for NQ and 88.3% for GaRAGe.Source: §4.1 · p. 4

RQ1: How effective is the generative

  • On NQ, generator self-evaluation averaged 91.0% pass rate versus 82.1% for cross-evaluation, while verifier agreement was poor.Inter-verifier κ fell as low as 0.061; generator quality ranked Deepseek-R1 > Claude-4-Sonnet > GPT-4o > Nova-Pro.Source: Figure 3 · p. 5, RQ1: How effective is the generative… · p. 5
  • Generators readily create explicit logical flaws but struggle with implicit reasoning tasks, especially ambiguity and missing information.Explicit-flaw categories exceed 95% pass rates in cited cases; top NQ generators reach 75–85% on harder categories, while weaker GaRAGe generators fall below 55%.Source: RQ1: How effective is the generative… · pp. 5–6, Figure 4 · p. 6

RQ2: How can we characterize the selective refusal capabilities of current models?

RQ3: What factors influence performance?

  • Answer and refusal accuracy scale independently, while DPO consistently improves refusal accuracy across scales and domain specialization affects GaRAGe performance.Extending Claude-4-Sonnet reasoning to 4096 tokens improves refusal accuracy by less than 1pp; no model leads in both answer quality and refusal accuracy within any domain.Source: RQ3: What factors influence performance? · pp. 7–8, Figure 9 · p. 8

4.3 Discussion; 5 Conclusion and Future Work; A Extended Related Work

  • The authors frame selective refusal as a trainable, alignment-sensitive capability and generative evaluation as a way to sustain measurement as models evolve.RefusalBench modifies existing answerable pairs through controlled perturbations, distinguishing it from approaches that synthesize new questions.Source: §4.3 · pp. 8–9, §5 · p. 9

6 Limitations; 7 Ethical Considerations

  • The benchmarks use programmatic English-text perturbations and isolate the language model rather than evaluating retrieval’s dynamic interaction with generation.The authors also note that verifier consensus cannot rule out shared model blind spots, and that lever patterns could be adapted for attacks or benchmark overfitting.Source: §6 · p. 9, §7 · pp. 9–10

A.1 Static Benchmarks for Unanswerability and Abstention

Static benchmarks span simple abstention tests to broader uncertainty scenarios, but their fixed instances can be memorized. RefusalBench instead evaluates controlled context defects, distinct from policy refusal, hallucination mitigation, and verbalized uncertainty.

A.1 Static Benchmarks for Unanswerability and Abstention

A.2 Holistic Taxonomies and Modern Generative Approaches

  • Fixed benchmark instances can be memorized or overfit by rapidly evolving models, motivating evaluation that generates fresh cases.Prior approaches include static collections, synthetic worlds, and synthesized or complex RAG questions; RefusalBench instead perturbs verified answerable pairs along flaw type and severity axes.Source: §A.1 · p. 12, §A.2 · p. 13

A.3 Distinguishing Selective Refusal from General Refusal Capabilities

  • Selective refusal concerns epistemic unreliability in supplied RAG context, not policy compliance or knowledge gaps in the model’s parameters.RefusalBench measures the answer-or-abstain decision and separately examines confidence calibration.Source: §A.3 · p. 14

B.1 Notation and Formal Setup

  • The formal setup scores test instances with f(x) in [0, 1] and tracks the expected score under each round’s relevant-instance distribution.The framework compares a static estimator based on instances drawn once from D0 with a generative estimator using new samples at each round.Source: §B.1 · p. 14

B.2 Proof of Theorem 3.1

  • The theorem bounds static-estimator error using sample size and distribution drift, while the generative-estimator bound uses fresh per-round sample counts.The proof applies Hoeffding’s inequality and a union bound across rounds; the static bound becomes trivial when drift reaches the error tolerance.Source: §B.2 · pp. 14–15

B.3 When Static Benchmarks Fail

The analysis establishes when contamination undermines static evaluation and describes how RefusalBench generates fresh tests. It also details benchmark curation and reports human checks of generated cases.

B.3 When Static Benchmarks Fail

  • When ∆T ≥ ϵ, static error exceeds ϵ with probability at least 1 − 2 exp(−2n(∆T − ϵ)^2), tending to 1 as n grows.Thus, static benchmarks can fail under contamination even as their sample size increases.Source: §B.3 · p. 15

B.4 Practical Implications for RefusalBench

  • Generative evaluation needs fresh samples each round, whereas static evaluation requires a large curated set and low contamination to remain reliable.RefusalBench defines its evaluation distribution through perturbation functions, generates samples on demand, and maps perturbations to ground-truth behavior.Source: §B.4 · pp. 15–16

C.1 Detailed Benchmark Construction

  • RefusalBench-NQ samples 100 answerable NaturalQuestions instances; GaRAGe samples 100 across five domains and normalizes each context to 10 passages.The source pools are filtered for answerability and solvability; GaRAGe contexts combine relevant signal passages with noise passages.Source: §C.1 · p. 16

C.2 Human Validation

  • Human validation pass rates were 93.1% for RefusalBench-NQ and 88.3% for RefusalBench-GaRAGe.One expert assessed a stratified random sample of 180 perturbations per benchmark against criteria including lever fidelity, intended intensity, uncertainty induction, and linguistic soundness.Source: §C.2 · pp. 16–17

C.3 Benchmark Composition Details

The benchmark’s composition reflects both deliberate balancing and the results of unanimous verification. GaRAGe also has comparable representation across its five selected domains.

Generator Contributions

  • 400 samples from each of four generators make RefusalBench-NQ’s final composition exactly balanced.This balance was enforced during sampling to limit bias from any single generator’s style.Source: §C.3 · p. 17
  • RefusalBench-GaRAGe contributions were Claude 406, Deepseek 385, GPT 370, and Nova 345 after unanimous verification.The counts reflect generator success passing the filter; Nova contributed relatively more P-FalsePremise examples than P-Ambiguity examples.Source: §C.3 · p. 17

Domain Distribution

  • GaRAGe’s largest domain, Health (22.9%), and smallest, Finance (16.4%), differ by 6.5 percentage points.The five domains have comparable representation, limiting disproportionate influence from any single subject area.Source: §C.3 · p. 17

D Detailed Evaluation Metrics

The evaluation metrics separate answer quality, refusal behavior, and calibration. Extended generator-verifier analyses report low verifier agreement, intensity-dependent patterns, and category-specific generation difficulty.

D Detailed Evaluation Metrics

  • Refusal Accuracy requires correct refusals with appropriate categorization, while Detection F1 and Category Accuracy separately assess refusal decisions and predicted reasons.The Hierarchical Refusal Score multiplies Detection F1-Score by Category Accuracy; the Calibrated Refusal Score averages Answer Accuracy and Refusal Accuracy.Source: D Detailed Evaluation Metrics · pp. 18–19

E.1 Inter-Verifier Agreement Analysis

  • RefusalBench-GaRAGe’s highest calculable verifier agreement was κ=0.230, while NQ’s mean off-diagonal agreement was 0.190.GaRAGe’s scores were lower overall; Nova-Pro’s κ values were NA because it approved nearly all perturbations, leaving insufficient variance.Source: §E.1 · p. 19, Figure 14 · p. 19

E.2 Generator Performance across Intensity Levels

  • Deepseek-R1’s RefusalBench-NQ pass rates were 91.0% LOW, 94.9% MEDIUM, and 96.5% HIGH; Nova-Pro’s were 71.1%, 69.0%, and 73.9%.Rankings remained stable across intensity levels; pass rates often rose at HIGH intensity because its flaws were more explicit, while LOW required subtle, plausible modifications.Source: §E.2 · p. 20

E.3 Overall Perturbation Class Ranking

  • Ambiguity was RefusalBench-NQ’s hardest category at 72.5%, while EpistemicMismatch was easiest at 97.8%.In GaRAGe, Ambiguity (73.4%) and MissingInfo (72.5%) were the most difficult categories; ambiguity was challenging across both benchmarks.Source: §E.3 · p. 20

E.4 Detailed Self-Evaluation Bias Analysis

Self-evaluation bias varies by model and task, and multi-document contexts widen the gap between detecting a need to refuse and identifying its reason.

E.4 Detailed Self-Evaluation Bias Analysis

  • Self-evaluation bias is model- and task-dependent: Claude-4-Sonnet is consistently self-critical, while Nova-Pro and GPT-4o show positive bias.On GaRAGe, the bias intensifies to −26.3pp for Claude-4-Sonnet and +43.0pp for Nova-Pro; challenging Ambiguity and MissingInfo cases often exceed ±30pp deviations.Source: §E.4 · p. 20

F Extended Frontier Model Analysis (Supporting RQ2)

  • HIGH-intensity perturbations often have higher pass rates than LOW-intensity ones, while MissingInfo and Ambiguity are harder than FalsePremise, Contradiction, and EpistemicMismatch.The pattern suggests explicit flaws may be easier to generate than subtle ones.Source: Figure 15 · p. 21, Figure 16 · p. 21

F.1 Refusal Detection vs. Categorization on RefusalBench-GaRAGe

  • On RefusalBench-GaRAGe, the detection-categorization gap widens, and Claude-4-Opus leads categorization despite lower detection scores.The increased scatter and lower overall performance indicate that multi-document complexity challenges both refusal decisions and understanding why to refuse.Source: §F.1 · p. 21

F.2 Calibration Analysis

Calibration, refusal behavior, and error profiles vary across models and perturbations. Multi-document tasks expose persistent difficulty in balancing accurate refusal decisions with answer quality.

F.2 Calibration Analysis

  • 73–99% of predictions occur at maximum confidence, yet all models are miscalibrated; Claude-4-Sonnet has the best ECE (0.286), and GPT-4.1 the worst (0.546).At 95% confidence, Claude-4-Sonnet is correct 68.5% of the time; GPT-4.1’s highest-confidence predictions succeed 40.6% of the time.Source: Figure 19 · p. 22, §F.2 · pp. 22–23

F.3 Refusal Intensity Curves

  • Refusal rates rise monotonically with perturbation intensity, with the largest average increase—47pp—at the LOW-to-MEDIUM transition.GPT-4o refuses 62.8% at LOW intensity on NQ, while its GaRAGe rate rises only 1pp from MEDIUM to HIGH.Source: §F.3 · p. 23, Figure 20 · p. 24

F.4 Perturbation Performance Heatmaps

  • REFUSE_GRANULARITY has the lowest performance, with a 31.1% maximum on NQ, while REFUSE_INFO_MISSING reaches 76–98% on NQ.On GaRAGe, DeepSeek-R1 scores 77.7% on REFUSE_FALSE_PREMISE; GPT-4o reaches 98.2% on NQ REFUSE_INFO_MISSING but scores below 52% on other types.Source: §F.4 · p. 23, Figure 21 · p. 24

F.7 Comprehensive Performance Dashboards

  • 13.6pp separates Claude-4-Sonnet’s calibrated refusal score on NQ (65.3%) from its GaRAGe score (51.7%).The report describes the multi-document setting as a decline in calibrated refusal performance.Source: §F.7 · p. 24

F.5 Error Rate Analysis

  • 62.8% false refusals (GPT-4o) accompany 4.3% missed refusals on NQ, while o4-mini has 17.8% false refusals and 21.5% missed refusals.On GaRAGe, Nova-Premier’s missed refusal rate reaches 53.7%.Source: §F.5 · p. 23, Figure 22 · p. 25

F.6 Refusal Accuracy Ranking - RefusalBench-GaRAGe

  • 47.4% refusal accuracy (DeepSeek-R1) leads GaRAGe, but Claude-4-Opus leads the hierarchical score at 50.3% versus 49.1%, with a 21.2pp best-to-worst accuracy spread.The hierarchical score combines detection F1 and category accuracy.Source: §F.6 · pp. 23–24, Figure 23 · p. 25

F.8 Response Distribution Analysis

Across response categories, the main failures concern deciding whether to answer or refuse, rather than producing low-quality answers. Answer-quality measures and confusion patterns further distinguish these failure modes.

F.8 Response Distribution Analysis

  • Under 3.0% of NQ and 3.4% of GaRAGe responses are incorrect or low-quality answers; multi-document tasks amplify missed refusals for answer-oriented models and wrong refusals for conservative ones.GPT-4o has the highest wrong-refusal rates (34.2% NQ; 38.0% GaRAGe), while answer-oriented models exceed 32.9% missed refusals on GaRAGe.Source: §F.8 · p. 25, Figure 26 · p. 27

F.9 RefusalBench-GaRAGe Answer Quality Analysis

  • On GaRAGe answer attempts, eligibility exceeds 91% and RAF ranges from 83.4% (o4-mini) to 95.9% (GPT-4o), while the latter answers only 49 questions.GPT-4o’s 49 attempts are 13.7% of Nova-Premier’s 357; the authors locate the primary challenge in the answer-versus-refuse decision.Source: §F.9 · p. 27

F.10 Individual Model Confusion Matrices

  • REFUSE_INFO_MISSING attracts misclassifications from nearly every other category, while REFUSE_GRANULARITY is especially difficult and is often mistaken for missing information.Even Claude-4-Sonnet reaches only 25% accuracy on REFUSE_GRANULARITY; GaRAGe confusion matrices have lower diagonal values than NQ.Source: §F.10 · pp. 27–28, Figure 28 · p. 28

G Statistical Analysis Details

The paper uses bootstrap resampling to quantify uncertainty in its primary evaluation metrics.

G Statistical Analysis Details

  • 1,000 non-parametric bootstrap resamples produce standard errors and 95% confidence intervals for primary metrics.Variance is low across most evaluations.Source: G Statistical Analysis Details · p. 28
  • Below 2.0% is the reported standard error for main refusal-accuracy metrics on both benchmarks.Claude-4-Sonnet’s NQ refusal accuracy is 73.0% with SE 1.7%; DeepSeek-R1’s GaRAGe accuracy is 47.4% with SE 1.9%.Source: G Statistical Analysis Details · pp. 28–29
  • Figure 29’s lower confusion-matrix diagonal values on GaRAGe indicate increased difficulty in multi-document contexts.Source: Figure 29 · p. 29

H Extended Analysis of Influential Factors (Supporting RQ3)

This section examines influential factors in RefusalBench evaluation, including domain-specific answer-quality and refusal-accuracy patterns and the effects of reasoning length. It also describes the prompts and generation constraints used to create, verify, present, and judge cases, including whether they remain answerable or should elicit a coded refusal.

H Extended Analysis of Influential Factors (Supporting RQ3)

  • Nova-Premier wins answer quality in 4 of 5 domains, while DeepSeek-R1 leads refusal accuracy in 3; no model tops both metrics in any domain.DeepSeek-R1's refusal accuracy ranges from 40.0% to 51.6%, while Nova-Premier's answer quality ranges from 54.7% to 82.8%.Source: H Extended Analysis of Influential… · p. 29, Figure 30 · p. 30
  • Business & Industrial is the hardest overall domain (0.634) and Law & Government the easiest (0.528), a 10.6% spread; difficulty patterns differ by task.Business & Industrial is hardest for answering and Science for refusal; Law & Government is easiest for answers but difficult for refusals, while Science shows the opposite pattern.Source: H Extended Analysis of Influential… · pp. 29–30, Figure 31 · p. 30
  • Claude-4-Sonnet's refusal accuracy improves only 0.91pp at 1024 thinking tokens, then returns to baseline or degrades; answer accuracy falls from 57.7% to 56.1%.The test used 0, 1024, 2048, and 4096 thinking tokens on RefusalBench-NQ; the results indicate longer reasoning traces do not improve selective refusal.Source: Figure 32 · p. 30, H Extended Analysis of Influential… · p. 30

I.1 RefusalBench-NQ Prompts

  • RefusalBench-NQ perturbs single-passage contexts; its generator applies exactly one lever to a target at a specified intensity, and its verifier checks implementation fidelity, intensity, target, and induced uncertainty.The NQ setup also evaluates refusal classification and answer correctness against reference answers.Source: §I.1 · p. 30, §I.1.1 · pp. 30–31, §I.1.2 · p. 31

I.1.3 Model Evaluation Template

  • The NQ evaluation prompt requires context-grounded answers or a refusal code, while its judge classifies responses and scores answer attempts from 1 to 5 against references.Refusal cases or answers without references receive N/A for quality scoring.Source: §I.1.3 · p. 32, §I.1.4 · p. 32

I.2 RefusalBench-GaRAGe Prompts

  • RefusalBench-GaRAGe perturbs multi-passage contexts with relevant and irrelevant passages and combines eligibility, factuality, relevance-aware factuality, and refusal classification.It tests long-form question answering in complex multi-source contexts.Source: §I.2 · p. 33

I.2.1 Generator Template

  • For GaRAGe ambiguity perturbations, the generator is instructed to make ambiguity unavoidable by modifying multiple signal passages, the query, or all signal passages, rather than relying on clearer alternatives.The prompt identifies signal passages as modification targets and noise passages as read-only unless the strategy is noise injection.Source: §I.2.1 · p. 33

I.2.2 Verifier Template

  • The verifier checks lever fidelity, intensity, target and strategy compliance, uncertainty induction, answer constraints, expected RAG behavior, and refusal-reason alignment.Its critical test asks whether a careful RAG system can derive the answer confidently or should refuse for insufficient support.Source: §I.2.2 · p. 34

I.2.3 Model Evaluation Template

  • The model prompt requires answering only when a faithful response follows from the passages and otherwise returning a refusal code, with precedence rules for overlapping reasons.The codes distinguish issues including ambiguity, missing information, and false premises.Source: §I.2.3 · pp. 34–35

I.2.4 Judge Template

  • The judge classifies responses and scores answer eligibility plus factual support against all context and relevant context separately.Eligibility and factuality are scored only for answer attempts; refusals receive a refusal code.Source: §I.2.4 · p. 35

I.3 Template Variables and Dynamic Content

  • Dynamic variables specify perturbation class, intensity, modification target, expected ground truth, class-intensity levers, and application strategy.Targets are assigned by class: false premises modify queries, contradictions and missing information modify context, ambiguity may affect either, and granularity or epistemic mismatch targets the query-context interaction.Source: §I.3 · pp. 35–36

I.4 Answer Constraints by Intensity Level

  • Intensity constraints preserve answer derivability at LOW intensity, but make confident answering unreliable and require an appropriate refusal at MEDIUM and HIGH.For GaRAGe, the constraints refer to passage-based answer derivation in the multi-passage context.Source: §I.4 · p. 36

J Software, Models, and Packages Used

This appendix documents the resources and evaluation setup for RefusalBench, then presents representative perturbation levers organized by uncertainty class and intensity. It reports the evaluation’s scale, cost, and runtime.

J Software, Models, and Packages Used

  • The full RefusalBench generation and evaluation pipeline cost less than $10,000 and completed within one week.The total covers data generation for RefusalBench-NQ and RefusalBench-GaRAGe and comprehensive model evaluation.Source: J Software, Models, and Packages Used · p. 36
  • Evaluations covered 30+ proprietary and open-source models, generally using default temperature=1.0 and top_p=1.0, except Gemini 2.5 Pro at temperature=0.1.Table 3 lists the evaluated models, identifiers, and access platforms.Source: J Software, Models, and Packages Used · p. 36, Table 3 · p. 37
  • The pipeline used Transformers and Datasets for model loading and data handling, Pandas and NumPy for processing, and cloud APIs or local 4× NVIDIA A100 GPUs for inference.LiteLLM unified cloud-provider access, while vLLM served local models.Source: J Software, Models, and Packages Used · p. 36

K Representative Perturbation Lever Catalogue

  • Table 4 presents 90 representative perturbation levers—five per intensity level across six classes—while the full catalogue contains approximately 10 levers per class-intensity combination.The examples illustrate test cases across varying complexity levels.Source: Table 4 · p. 38, K Representative Perturbation Lever… · p. 38
  • The catalogue examples distinguish expected answering at low intensity from refusal at medium and high intensity for false-premise, granularity-mismatch, contradiction, and missing-information perturbations.The listed refusal labels vary by uncertainty type, including REFUSE_FALSE_PREMISE, REFUSE_GRANULARITY, REFUSE_CONTRADICT, and REFUSE_MISSING.Source: K Representative Perturbation Lever… · pp. 41–43
Loading 2510.10390v1…