PaperlayerView exact revision on arXiv

Source-linked AI summary

Knowing When Not to Answer: Abstention and Refusal Reasoning in Vision--Language Models

Karan Dua, Amit Agarwal, Hitesh Laxmichand Patel, Hansa Meghwani, Jyotika Singh, Ranjeet Gupta, Graham Horwood, Tao Sheng, Avi Sil, Sujith Ravi, Dan Roth

arXiv:2609.05540v1cs.CVcs.AI

TL;DR

ASD diagnosis requires behavioral and developmental evidence, yet VLMs are increasingly queried about medical judgments from facial images. This paper introduces PARITY, a synthetic identity-controlled audit of abstention and expression effects, and finds refusal-first and speculative model regimes, with clinical guardrails and single-image framing increasing abstention.

  • Problem

    VLMs are increasingly queried about medical judgments from facial images, although ASD diagnosis requires behavioral evidence and cannot be inferred from a static photograph.

    Source: §1 · p. 1

  • Method

    The paper introduces PARITY, a synthetic identity-controlled portrait-pair benchmark with neutral–neutral controls to audit abstention and expression effects on unanswerable ASD queries.

    Source: §1 · p. 1, §3.2 · p. 4

  • Results

    VLMs divide into refusal-first and speculative regimes, certain expressions disproportionately trigger harmful selections in speculative models, and clinical guardrails and single-image framing increase abstention.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Paired comparisons can elicit diagnosis-like attributions without evidence, so guardrails and framing are useful probes but not complete mitigations.

    Source: §4.7 · p. 8

    The audit covers one ASD-themed prompt family and facial expression as a spurious factor, uses primarily synthetic stimuli, and leaves open-ended conversational audits for future work.

    Source: §3.2.1 · p. 4, Limitations and Future Work · pp. 8–9

Abstract

from arXiv · show

Many medical conditions require diagnosis through detailed, multi-context clinical assessment rather than from visual appearance alone. Despite this, vision-language models (VLMs) are increasingly queried to interpret images in ways that touch on medical or diagnostic judgments, raising safety concerns when such inferences are unsupported. ASD diagnosis requires behavioral and developmental evidence, not static facial photographs. We audit whether VLMs abstain from this unanswerable paired-image query, and whether expressions sway non-abstaining choices. We introduce PARITY (Paired Assessment with Reused Identity), a synthetic, demographically balanced set of identity-controlled neutral/expression portrait pairs with neutral-neutral controls. All identities are synthetic and have no ASD status; because the query is unanswerable from images, any non-abstaining selection is treated as a harmful attribution. Across contemporary VLMs, we find a clear split between refusal-first models and speculative models; in the latter, certain expressions disproportionately trigger harmful selections. Clinical guardrails and single-image framing substantially increase abstention, suggesting actionable mitigations in both prompting and interface design

1 Introduction

The paper audits whether vision–language models abstain from clinically inappropriate ASD judgments based on facial images and whether expression changes sway unsupported selections. It introduces PARITY and measures both harmful attributions and the quality of refusals.

1 Introduction

  • ASD diagnosis requires behavioral evidence across contexts, so a static facial photograph cannot support the diagnosis-like inference studied here.The paired query is deliberately ill-posed: abstention is correct, and any image selection is treated as harmful attribution.Source: §1 · p. 1
  • PARITY is a synthetic, demographically balanced benchmark using identity-controlled, expression-parallel portraits to audit diagnosis-like attributions.Pairs hold pose, background, and camera parameters fixed; the framework also includes neutral–neutral controls and an explicit abstention option.Source: §1 · p. 1
  • The evaluation operationalizes abstention-aware behavior with Abstention Rate (AR), Harmful Attribution Rate (HAR), and order-invariant Expression-Driven Bias Ratio (EDBR).EDBR measures edited selections in paired decisions.Source: §1 · p. 1
  • Refusal Quality Score (RQS) assesses whether abstentions give explicit, clinically grounded justifications rather than generic or evasive refusals.A clinically grounded rationale states that ASD cannot be inferred from an image and directs to behavioral or clinical assessment.Source: §1 · pp. 1–2
  • VLMs split into refusal-first and speculative regimes, with certain expressions disproportionately triggering harmful attributions in speculative models.Clinical guardrails and single-image framing can substantially increase abstention.Source: §1 · p. 2

2 Related Work

Prior work studies VLM facial analysis, social stereotyping, and medical inferences from appearance, while benchmarks have largely emphasized demographic bias. This paper addresses the gap by auditing refusal behavior on clinically unanswerable diagnostic queries.

2 Related Work

  • VLM facial-analysis research studies emotion understanding, but models may hallucinate affective states from irrelevant visual cues (Bhattacharyya and Wang, 2025).Emotion-focused pipelines also emphasize identity-preserving expression edits and reporting identity fidelity and expression calibration (Zhang et al., 2025; Schuhmann et al., 2025).Source: §2 · p. 2
  • Prior studies report VLM associations between facial features and social stereotypes (Konavoor et al., 2025), alongside disputed claims of ASD detection from facial images.Those ASD claims have been criticized for methodological confounds and ethical risks, including stigmatization and false positives.Source: §2 · p. 2
  • Most benchmarks emphasize demographic bias rather than whether VLMs avoid physiognomic or diagnostic inference; this work audits refusal on clinically unanswerable queries.The cited discussion notes that validated synthetic data can replace real images while mitigating ethical risks (Borsukiewicz et al., 2025).Source: §2 · p. 2

3 Methodology

PARITY uses synthetic identity-matched portraits to test whether VLMs abstain from an image-based ASD query that is unanswerable by construction. The evaluation varies expression and framing while tracking harmful selections, refusal reasoning, and controls.

3.1 Identity-Controlled Data Synthesis

  • PARITY makes image-based diagnostic speculation measurable by varying expression within identity-controlled portraits while holding other visual conditions approximately fixed.The contribution is the counterfactual evaluation design, not the image generator itself.Source: §3.1 · p. 2

3.1.1 Attribute Sampling and Identity Construction

  • The dataset uses 1,000 stratified synthetic adult identities; demographic attributes support coverage and analysis, not clinical labeling.Each identity reuses one neutral anchor across expression edits, while non-essential factors may vary across identities.Source: §3.1.1 · p. 2, §3.1.2 · p. 3

3.1.3 Expression Editing with Identity

  • Six expression edits per neutral anchor are checked for identity preservation and expression fidelity before inclusion.Validation combines embedding similarity, expression classification, AU increases of at least 30%, and a bounded manual artifact review; PARITY contains 7,000 images.Source: §3.1.3 · p. 3, §3.1.4 · pp. 3–4

3.2.1 Primary Task: Identity-Controlled Paired Comparisons

  • The primary query compares a neutral anchor with an edited image from the same synthetic identity and offers abstention; any selection is harmful by construction.The main evaluation comprises 6,000 trials; CFD is supplementary because its scale and expression coverage constrain identity control and systematic perturbation.Source: §3.2.1 · p. 4, §3.2 · p. 4

3.2.2 Prompting and Framing Variants

  • Robustness variants add a clinical guardrail, reverse image order, or present one edited image; neutral–neutral pairs test baseline selection and side bias.In neutral–neutral trials, a choice reflects comparison-induced bias rather than evidence in the input.Source: §3.2.2 · p. 4, §3.2.3 · p. 5

3.2.4 Models and Inference Protocol

  • The study evaluates contemporary VLMs spanning architectures, sizes, and training paradigms using deterministic decoding and identical batching.Temperature is set to 0 to preserve presentation order.Source: §3.2.4 · p. 5

3.3.1 Primary Rates

  • Abstention Rate (AR) and its complement, Harmful Attribution Rate (HAR), quantify refusal and non-abstention; EDBR measures edited-image selection among non-abstaining trials.Edited-image selection is tracked independently of left/right position.Source: §3.3.1 · p. 5, §3.3.2 · p. 5

3.3.3 Refusal Quality Score

  • Refusal Quality Score (RQS) measures whether abstention rationales reject image-based ASD inference and direct readers to behavioral or clinical assessment.The analysis also checks neutral–neutral behavior, side bias, sensitivity to validation thresholds and generator seeds, and five questions about abstention, reasoning, expression susceptibility, edited preference, and framing.Source: §3.3.3 · p. 5, §3.3.4 · p. 5, §3.4 · p. 5

4 Results

The results reveal a split between models that abstain and models that speculate on the unanswerable paired-image query. Among models that answer, expression and framing affect selections, while refusal frequency does not by itself capture refusal quality.

4.1 RQ1: Do VLMs abstain on an ill-posed diagnosis-like request?

  • Several VLMs abstain at or near ceiling, while speculative models often comply despite an explicit abstention option and no diagnostic evidence.Because abstention is the only appropriate response, harmful-attribution rate measures a safety failure mode rather than accuracy.Source: §4.1 · p. 6, Figure 4 · p. 6

4.2 RQ2: Refusal quality: Abstention rate vs. clinical justification

  • Abstention frequency and refusal quality diverge: some near-ceiling abstainers have low RQS, while some mixed models provide more clinically specific justifications.The authors caution that abstention rate alone can overstate safety; appropriate behavior also requires explaining why image-based ASD attribution is inappropriate.Source: §4.2 · p. 6, Table 1 · p. 6

4.3 RQ3: Which expressions act as spurious triggers for harmful selection?

  • Surprise, Fear, Anger, and often Disgust yield the largest edited-selection rates in speculative or partially speculative models, while Happy and Sad remain lower.Figure 5 summarizes aggregate edited-selection probability by expression; model-level breakdowns are in Appendix G.4.Source: §4.3 · p. 6, Figure 5 · p. 6

4.4 RQ4: Do models prefer the edited face conditional on attributing?

  • Among models with enough attributions to report EDBR, values above 0.5 indicate disproportionate selection of the edited portrait once the model answers.Models with fewer than 20 non-abstaining trials are omitted to avoid refusal-first edge cases.Source: §4.4 · p. 7, Figure 6 · p. 7

4.5 RQ5: Framing effects

  • Neutral–neutral trials still elicit non-trivial selections from several speculative models, while single-image framing and clinical guardrails increase abstention.The neutral controls show that pairwise comparison can elicit fabricated distinctions; guardrails do not make the paired baseline reliably trigger refusal.Source: §4.5 · p. 7
  • Order swaps produce comparatively small abstention changes, and Llama Guard 4 classifies every baseline paired prompt as safe.The external classifier check suggests a general-purpose safety classifier alone may not reliably block this failure mode.Source: §4.5 · p. 7

4.6 Qualitative Analyses

  • The qualitative rationale analysis uses keyword-seeded clustering followed by manual verification.Inventories and representative samples are provided in Appendix I.Source: §4.6 · p. 7

1. Refusals

Refusals range from clinically grounded explanations to shallow disclaimers. The distinction is whether the refusal explicitly explains why a photograph cannot support ASD attribution.

1. Refusals

  • Clinically grounded refusals state that ASD cannot be determined from a photo and requires behavioral or clinical evaluation.Source: §1 · p. 7
  • The refusal categories distinguish merely declining from explaining why image-based ASD attribution is inappropriate.Source: §1 · p. 7
  • Shallow or deflective refusals rely on generic disclaimers such as “I cannot diagnose” without explicit clinical justification.Source: §1 · p. 7

2. Speculative reasoning

Speculative explanations invoke either surface affect or social–behavioral stereotypes, while refusals can remain shallow. The authors treat framing variants as diagnostic probes and describe PARITY as reusable for other unsupported face-based inferences.

2. Speculative reasoning

  • Speculative selections invoke affective cues such as wide eyes or furrowed brows, or social–behavioral stereotypes such as eye contact, engagement, and social withdrawal.The broader rationale analysis contrasts these speculative explanations with shallow refusals and clinically grounded explanations.Source: §2 · pp. 7–8

4.7 Discussion and Broader Implications

  • Clinical guardrails increase abstention for several models but do not fully mitigate the effect; the authors treat these variants as diagnostic probes, not complete mitigations.Figure 7 reports abstention-rate changes under a clinical guardrail, order swap, and single-image framing, excluding baseline abstainers with AR ≥95%.Source: §4.7 · p. 8, Figure 7 · p. 8
  • PARITY can be reused to audit unsupported face-based inferences beyond ASD by varying controlled visual cues and measuring abstention or speculation.The authors state they will release code and accompanying artifacts for regenerating the benchmark and analyses.Source: §4.7 · p. 8

5 Conclusion

The paper presents an identity-preserving audit of VLMs asked to make a clinically inappropriate paired-image ASD judgment. It reports distinct refusal and attribution patterns and introduces a controlled synthetic pipeline for studying when such attributions emerge.

5 Conclusion

  • The audit separates refusal behavior from harmful attribution and uses EDBR to measure edited-face preference conditional on attribution.The conclusion also introduces RQS, which reveals that many abstentions are shallow and lack medically appropriate justification.Source: §5 · p. 8
  • Guardrail prompts and single-image framing substantially increase abstention, while reverse-order swaps have limited impact when preference is measured with EDBR.Source: §5 · p. 8
  • The synthetic generation and evaluation pipeline uses quantitative controls to create meaningful inter-person variation while preserving identity across expression edits.This controlled setting supports analysis of interface-induced speculation, superficial visual cues, and refusal behavior.Source: §5 · p. 8

Limitations and Future Work

The audit is limited to ASD-themed prompts and facial expression as a visual factor, with no ASD ground truth. Its primarily synthetic stimuli and controlled scope leave questions about robustness and real-world generalization open.

Limitations

  • The audit covers one family of sensitive prompts and one visual factor, does not evaluate autism detection, and assumes no ASD ground truth.Results may also vary with prompt paraphrases, stochastic decoding, response formats, and model versions.Source: Limitations and Future Work · p. 8
  • Primarily synthetic stimuli enable identity-controlled perturbations, but distribution shift and limited expression coverage in real datasets remain considerations.The study includes embedding- and AU-based validation and a real-image sanity check using CFD.Source: Limitations and Future Work · p. 9

Future Work

  • Future work could test non-visual cues, other health- or disability-related attributes, and task structures such as ranking, open-ended generation, and conversational follow-ups.The authors also identify richer real-image benchmarks and stronger refusal policies and safety routers as directions for future work.Source: Limitations and Future Work · p. 9

Appendices

The appendix reviews facial-analysis VLM research, biases and dataset challenges in facial expression recognition, and audit and mitigation approaches. It situates the paper among broader work on measuring and addressing model and data bias.

A Extended Related Work

  • Facial-analysis VLM work spans emotion recognition, explainability, and data auditing, while reported challenges include hallucinated affect from irrelevant cues and the need to validate synthetic expression edits.Examples include context-aware fine-tuning, large-scale pre-training, and identity-preserving edits.Source: A Extended Related Work · p. 13
  • Facial analysis can reproduce demographic and social stereotypes, including biased estimates of gender, race, and age and inferred links between facial features and traits.The cited examples include “American” being associated with “White” and minority groups treated as deviations from a norm.Source: A Extended Related Work · p. 13
  • FER faces demographic disparities, label noise, semantic ambiguity, and domain shift, prompting fairness, AU-supervision, causal, adaptation, and synthetic-data approaches.The appendix also notes audits of demographic bias and facial-property effects, alongside data- and model-level mitigation strategies.Source: A Extended Related Work · p. 13

B Extended Methodology

The extended methodology uses synthetic identity-matched portraits to isolate expression effects, then evaluates abstention and harmful attribution under controlled prompts and validation procedures. It also describes demographic stratification, image checks, and supplemental analyses of controls and model-level expression sensitivity.

B Extended Methodology

  • Expression edits are compared within the same identity and visual context, avoiding cross-identity confounds and prioritizing causal interpretability over population realism.Synthetic portraits support a neutral anchor and expression edits while keeping facial structure and other attributes as similar as possible.Source: B Extended Methodology · pp. 13–14

C Demographic Attribute Sets and Stratification

  • The 1,000 identities are allocated as evenly as possible across a 9×2×6 race, perceived-gender, and age grid, with 9–10 identities per cell.The study uses these coarse attributes for diversity and descriptive slicing, excludes minors, and does not prompt for clinical traits.Source: C Demographic Attribute Sets and… · p. 14

D Generation and Editing Prompts

  • Paired and single-image prompts request diagnosis-like judgments with expression-focused instructions; expression wording is a diagnostic probe, while a clinical guardrail instructs models to abstain.Removing expression-focused wording slightly increases abstention but reduces the ability to characterize speculative reasoning.Source: D Generation and Editing Prompts · p. 15

E Identity Preservation and Identity Uniqueness via Face Embeddings

  • Identity edits are filtered by within-identity embedding similarity, while new identities are screened for between-identity uniqueness; thresholds are calibrated using a human-verified reference subset.The authors use 512-dimensional InsightFace embeddings and report thresholds and post-filtration acceptance rates in Table 4.Source: E Identity Preservation and Identity… · p. 16, Table 4 · p. 16

F Human Verification Protocol

G.1 Bootstrap details and resampling unit

  • The reported confidence intervals use identity-level bootstrap resampling, preserving the dependence among trials from each identity.Each resample draws identities with replacement and includes all corresponding trials.Source: §G.1 · p. 17

G.2 Neutral–neutral controls: coercion and side bias

  • Neutral–neutral controls show that selections can reflect coerced choice and/or side bias; conditional edited-selection rates should be interpreted with their attributed-trial counts.The authors recommend flagging models with nattr < 30.Source: §G.2 · p. 17, §G.3 · p. 17

G.4 Model-Level Expression Sensitivity

  • Refusal-first models remain near zero in edited selections, whereas speculative and mixed models show larger expression-dependent shifts.Figure 10 provides the model-level breakdown behind the aggregate expression pattern.Source: §G.4 · pp. 17–18

H Real-Image Replication on the Chicago Face Database (CFD)

The appendix extends paired evaluation to real CFD images as a sanity check, then describes qualitative rationale clustering and a rule-based refusal-quality measure. It also visualizes variation between abstention frequency and the clinical grounding of refusals.

H Real-Image Replication on the Chicago Face Database (CFD)

  • The CFD replication uses the same prompt schemas, but its limited expression taxonomy, demographic balance, and identity count make it a sanity check rather than a replacement benchmark.The real images are used for evaluation without re-identification or conclusions beyond the dataset’s scope.Source: H Real-Image Replication on the… · p. 18, Table 7 · p. 18

I.1 Keyword inventory and clustering procedure

  • Rationale clustering surfaces two refusal subtypes—clinically grounded and shallow or deflective—and two speculative subtypes: affect-based heuristics and stereotype-laden inferences.N-grams are embedded and agglomeratively clustered; manual review validates and interprets clusters rather than labeling all rationales exhaustively.Source: §I.1 · p. 18

I.2 Refusal Quality Score (RQS) computation

  • RQS flags an abstention as positive when its rationale matches a fixed refusal-pattern inventory, but missed paraphrases count as negative, making the score a lower-bound proxy.The patterns include statements that diagnosis cannot be made from a photograph and references to clinical or behavioral evaluation.Source: §I.2 · p. 18

I.3 RQS Visualization

  • Models with higher abstention rates do not necessarily provide more clinically grounded refusals: some abstain frequently but give generic responses, while others abstain less consistently but offer more grounded justifications.Figure 13 plots abstention rate against RQS for each model.Source: §I.3 · p. 18, Figure 13 · p. 20
Loading 2609.05540v1…