HalluPrism: When Multimodal Uncertainty Should Diagnose, Not Decide

• Scalar confidence can indicate whether to abstain but cannot explain whether an error reflects weak visual evidence, image-independent answer persistence, or cross-modal relation failure.Existing multimodal errors are heterogeneous, spanning object hallucination, visual illusion, attribute and spatial errors, and answerability failure [3] [4].Source: §1 · p. 1
• HalluPrism computes (V, L, A) from visual perturbation, blank-image replacement, and grounding or relation probes to diagnose visual sensitivity, image-removal retention, and binding instability.The signature is behavioral rather than a causal decomposition: high L does not prove image nonuse, while high V or A records sensitivity rather than an isolated mechanism.Source: §1 · p. 2, Figure 1 · p. 2
• Only 18 of 48 ruined-condition checks are diagonally aligned, showing that V, L, and A are entangled behavioral coordinates rather than independent failure causes.Visual inputs pass 11/16 matching checks, compared with 2/16 for language and 5/16 for alignment inputs.Source: §1 · pp. 2–3
• The signature’s prevalence is not diagnostic: L dominates 99.7% of samples, while A is the strongest tree-based feature for failure-family prediction.The analysis covers 58,440 examples from three MLLMs across HallusionBench, POPE, VSR, and VizWiz-VQA.Source: §1 · p. 2
• AUROC rises from 0.634 to 0.769 on HallusionBench and from 0.707 to 0.817 on VizWiz with (V, L, A) over scalar confidence; pooled XGBoost reaches 0.97 when confidence is added.Gains are smaller on POPE and VSR; pooled AUROC increases from 0.78 with scalar confidence to 0.95 with (V, L, A).Source: §1 · pp. 2–3
• Matched routing preserves 87.6% accuracy and lowers answer breakage to 2.7% on Qwen3-VL/VSR, whereas generic prompting drops accuracy to 76.2% and breaks 15.5% of correct answers.The routes select visual re-grounding, anti-prior prompting, or relation checking according to the signature.Source: §1 · pp. 2–3

Hallucination Evaluation

• HalluPrism measures a behavioral Perturbation-Response Signature over visual-perturbation sensitivity, image-removal confidence retention, and grounding/relation-probe instability, extending hallucination evaluation beyond heterogeneous error categories.Existing benchmarks cover object presence, visual illusion, attributes, relations, perception, reasoning, and instruction following [3] [4] [14].Source: §2 · p. 3

Uncertainty and Abstention

• Uncertainty and abstention research treats confidence as a decision object, but HalluPrism narrows the critique: thresholded scores need not diagnose risk in new open-world instances [15].Related signals include calibration, language-model uncertainty [6] [7], and multimodal visual or evidential uncertainty [11].Source: §2 · p. 3

Probing and Attribution

• Following behavioral testing rather than attribution, HalluPrism probes output changes under pressure on visual evidence, question-only priors, and cross-modal binding.CheckList, contrast sets, and counterfactually augmented data expose local failures through controlled edits, while attribution methods localize visual or token-level influence.Source: §2 · p. 3

Mitigation and Routing

• Unlike fixed mitigation rules, HalluPrism is diagnosis-first: it profiles a risky prediction and selects a matched prompt-based correction [28] [33] [34].Prior methods use visual contrast, decoding penalties, image-grounded amplification, post-hoc revision, inter-modality calibration, attention calibration, or visual-aware sparsification [28] [33] [34].Source: §2 · p. 3

Methodological objective

• The method separates correctness ranking, failure-source diagnosis, and correction-path choice because a diagnostic signature can explain failures without improving answer-versus-abstain ranking.Scalar confidence is treated as a correctness-ranking surrogate, not a grounded reliability certificate.Source: §3 · p. 4

Prediction and Perturbation-Response Signatures

• The signature comprises V for visual-perturbation sensitivity, L for image-removal confidence retention, and A for grounding or relation-probe instability.These are behavioral probe responses rather than hidden causal mechanisms, and failure families may occupy different regions of their joint space.Source: §3 · p. 4

Perturbation Probes

• V is measured by answer flips and confidence shifts under fixed-prompt Gaussian blur, center cropping, brightness reduction, and Gaussian noise.Changed answers receive greater weight because they provide the primary signal of visual fragility.Source: §3 · pp. 4–5
• L measures confidence retention after replacing the image with a uniform blank image, while remaining a behavioral marker rather than evidence that the model causally ignores images.Because the score can saturate, diagnosis uses the full (V, L, A) vector; the main results use Lconf unless stated otherwise.Source: §3 · p. 5
• A applies grounding re-checks to all datasets and relation swaps when explicit relations are available, weighting relation changes more heavily because they impose sharper binding pressure.Raw A magnitudes are not compared across relation-bearing and non-relation-bearing datasets; routing instead uses within-setting normalized dominance.Source: §3 · p. 5

Interpreting And Using Perturbation-Response Signatures

• For interpretation and routing, the method uses within-setting z-normalized dominance to prevent saturated coordinates such as L from controlling every decision.It tests, rather than assumes, diagonal source-target alignment because interventions can change unintended coordinates at least as much as intended ones.Source: §3 · p. 6
• The evaluation distinguishes diagnosis, abstention scalarization, and routing, testing failure-family prediction, three source-aware abstention scores, and matched correction policies against generic reasoning.Routing maps normalized dominance to visual re-grounding, anti-prior prompting, or relation checking, and reports accuracy, fix rate, and break rate without retraining.Source: §3 · p. 6

Experimental Aim

• Each model output receives scalar confidence c and the Perturbation-Response Signature s(x) = (V, L, A), tested for diagnosis, abstention, and intervention routing.The protocols are matched across perturbation conditions, with the signature evaluated alongside scalar confidence.Source: §4 · pp. 6–7

Datasets and Models

• The evaluation suite spans four MLLMs and four benchmarks covering visual grounding, object presence, spatial relations, and answerability.Gemma 4, Qwen3-VL, LLaVA-Mistral, and LLaVA-Vicuna are evaluated on HallusionBench, POPE, VSR, and VizWiz-VQA [37].Source: §4 · p. 7

Diagnostic and Decision Tasks

• Failure diagnosis uses benchmark-structured labels, while separate metrics evaluate abstention ranking and correction effects.The study reports diagnosis AUROC and Macro-F1, abstention AUROC and risk-coverage, and routing accuracy, fix rate, and break rate using logistic regression, XGBoost, and LightGBM with stratified 5-fold cross-validation.Source: §4 · p. 7

Controls, Abstention, and Routing

• Controls test dataset alignment, alternative L formulations, removal of the relation-eligible dataset, and matched intervention choices for routing.Routing compares no intervention, a generic reasoning prompt, and source-dominance-selected image enhancement, anti-prior instructions, or grounding/relation checking.Source: §4 · p. 7

Human Audit

• A blind 200-example human audit provides only a limited semantic plausibility check, not ground truth, supervision, calibration, or causal-source evidence.Three annotators label six failure categories, adjudicating 36 disagreements; Fleiss’ κ is 0.236 for full labels and 0.48 for source versus non-source labels.Source: §4 · p. 7

Interventions Expose Entanglement

• Only 18/48 source-target checks show diagonal dominance, so (V, L, A) is a joint perturbation-response signature rather than three causal labels.Raw L dominance reaches 58,287/58,440 samples (99.7%), but Lpersist reduces L dominance to 51.6% while A and V dominate 27.9% and 20.4%, respectively.Source: §5 · pp. 8–9

Signatures Diagnose Failure Families

• (V, L, A) improves AUROC over scalar confidence from 0.634 to 0.769 on HallusionBench, 0.707 to 0.817 on VizWiz, 0.823 to 0.839 on POPE, and 0.722 to 0.724 on VSR.In pooled XGBoost, AUROC rises from 0.78 with scalar confidence to 0.95 with (V, L, A) and 0.97 when confidence is added.Source: §5 · p. 10

Probe-Design Controls

• Diagnosis gains persist when L variants are substituted or VSR is removed, and alignment contributes 0.50 of pooled XGBoost importance.Source: §5 · p. 10

Visual-Probe Robustness

• Removing individual visual perturbations keeps diagnosis AUROC between 0.822 and 0.824, while stronger perturbations increase mean V from 0.132 to 0.252.With Lpersist, diagnosis AUROC remains 0.785, 0.786, and 0.783 across weak, default, and strong settings, versus 0.632 for scalar confidence.Source: §5 · p. 10

Naive Scalarization Can Hurt

• Weighted, geometric, and max-penalty scalarizations underperform scalar confidence for abstention, with Qwen3-VL dropping from 0.81 to 0.73 AUROC.The findings support using source-aware uncertainty diagnostically rather than assuming direct scalarization improves answer-versus-abstain ranking.Source: §5 · p. 10

Diagnosis-Triggered Routing

• Matched routing beats generic prompting in 10/16 settings, but pooled accuracy is 75.9% versus 75.7%, below 76.6% without intervention.On Qwen3-VL/VSR, routing maintains 87.6% accuracy and reduces breakage from 15.5% to 2.7%; a probe-informed policy improves pooled accuracy by 3.54 points and reduces break rate by 5.12 points versus generic CoT.Source: §5 · pp. 10–11

• Across 58K+ examples, perturbation-response signatures improve failure-family diagnosis, while matched routing reduces harmful breakage in selected high-risk settings.The signatures characterize visual sensitivity, image-removal confidence retention, and alignment instability.Source: §6 · p. 11
• The three tested direct signature-based scalarizations do not improve correctness ranking and can perform worse than scalar confidence.This establishes a boundary between diagnosing failure structure and scoring abstention decisions.Source: §6 · p. 11
• HalluPrism should diagnose whether behavior reflects visual sensitivity, image-removal confidence retention, or alignment instability before determining whether generic correction is safe.Learned or calibrated scalarizers remain an open direction for converting diagnostic signatures into decision scores.Source: §6 · p. 11

Scope of diagnosis

• HalluPrism is a behavioral diagnostic layer, not a causal explanation of hidden MLLM mechanisms.Its scores depend on a specific set of probes and perturbation strengths, although tested ablation and severity trends remain stable.Source: Limitations · p. 12

Generalization and coverage

• Cross-dataset generalization remains unresolved because diagnostic gains vary across benchmarks, despite controlled and fixed-dataset analyses.The evaluation covers four public benchmarks and four open-weight MLLM families, excluding video, multi-image, medical, document-heavy expert, and latency-critical settings.Source: Limitations · p. 12

Deployment constraints

• The implementation requires 7 to 8 forward passes per example, while routing offers conditional actionability rather than universal mitigation.Broader probe choices and deployment settings remain avenues for future evaluation.Source: Limitations · p. 12

Validation and abstention

• The human audit is small and tests semantic plausibility rather than causal source recovery, while abstention evaluation covers only three direct hand-designed scalarizations.Learned or calibrated scalarization strategies are outside scope; statistical guarantees such as selective prediction, calibration, risk control, and conformal prediction still require validation-to-deployment assumptions and do not alone provide instance-grounded reliability.Source: Limitations · p. 12

Ethics Statement

• HalluPrism is intended for model auditing, not standalone safety certification or deployment approval, because its scores are behavioral signatures rather than causal explanations.The main ethical risk is over-trust in a compact source label as a complete reliability guarantee.Source: Ethics Statement · p. 12

A Theoretical Analysis

• Perturbation-Response Signatures support failure-family diagnosis, whereas scalar confidence ranks correctness, scalarization may distort abstention, and routing requires both fix and break rates.The framework is related to reject-option classification and selective prediction [15].Source: A Theoretical Analysis · p. 17

A.1 Scalar Confidence Does Not Identify Failure Source

• Scalar confidence cannot diagnose failure source when multiple failure families share confidence values; zero-error scalar diagnosis requires failure labels to be determined by confidence.This motivates evaluating (V, L, A) against scalar confidence for failure-type prediction.Source: §A.1 · p. 18

A.2 Interventional Separability and Entanglement

• Off-diagonal intervention responses show that probe families may entangle sources, so (V, L, A) should be interpreted as a joint signature rather than one-to-one causal coordinates.A visual-ruined condition affecting language-prior reliance remains informative rather than merely nuisance variation.Source: §A.2 · pp. 18–19

A.3 Diagnosis And Abstention Are Different Prediction Problems

• Failure diagnosis predicts error type, while abstention ranks correctness; these are distinct prediction problems with different information requirements.The distinction separates predicting T or F from ranking examples by P(Z = 1 | ·).Source: §A.3 · p. 19
• The Perturbation-Response Signature can strictly improve failure-type prediction while scalar confidence remains at least as informative for correctness ranking.Under the stated conditions, adding S cannot improve the Bayes-optimal correctness score beyond C.Source: §A.3 · p. 19
• Theorem 1 formalizes role separation: confidence alone can match Bayes-optimal abstention, while adding (V, L, A) can strictly improve Bayes-optimal failure-family classification.This justifies evaluating diagnosis and abstention as separate tasks rather than treating abstention as the sole test of uncertainty usefulness.Source: §A.3 · p. 19

A.4 Why Naive Source-Aware Scalarization Can Hurt Abstention

• Source-aware penalties can reverse correct scalar-confidence rankings when penalty variation is weakly aligned with correctness and large enough to overcome confidence margins.This applies to multiplicative weighted, geometric, and max-penalty scores, including penalties based on V, L, and A.Source: §A.4 · p. 20
• Near-saturated image-removal confidence retention can make small, weakly informative penalty differences flatten or distort rankings without improving correctness.Thus, perturbation-response signatures may improve failure diagnosis even when source-aware scalar confidence underperforms scalar confidence for selective prediction.Source: §A.4 · p. 20

A.5 Routing Depends On Fix And Break Rates

• Post-intervention accuracy equals a(1 − Breakπ) + (1 − a)Fixπ, so routing must evaluate both repaired incorrect answers and damaged correct answers.A generic intervention can improve some failures yet reduce overall accuracy when it breaks too many originally correct examples.Source: §A.5 · pp. 20–21
• When baseline accuracy a > 0.5, an equal-sized break-rate reduction improves accuracy more than a fix-rate increase because the break coefficient a exceeds 1 − a.This makes reducing harmful overcorrection especially safety-relevant in high-accuracy settings.Source: §A.5 · p. 21

A.6 Limits Of Scalar Abstention Under Open-World Deployment

• Abstention is a designer-imposed guardrail over a score or rule, not an intrinsic inference-time capability or automatically grounded verdict for new inputs.Reliable selective answering requires both score sufficiency for the grounded safe-answer label and transfer of validation risk to deployment.Source: §A.6 · p. 21
• A scalar abstention policy is perfectly reliable if and only if the grounded safe-answer label is measurable from that scalar; mixed labels within a score region force positive error.The result applies across confidence, entropy, calibrated probabilities, uncertainty, verifier, judge, conformal, and learned rejection scores.Source: §A.6 · p. 21
• Without a validation-to-deployment transfer condition, threshold abstention has no distribution-free inference-time guarantee because deployment can concentrate accepted inputs among unsafe examples.Validation curves and calibration guarantees therefore warrant accepted regions only under their stated support and transfer assumptions.Source: §A.6 · p. 22
• A scalar answer-versus-abstain interface is not source-complete when grounding state or the safe correction action is not measurable from the score.It may support selective answering under assumptions, but cannot identify failure sources, establish grounding, or determine whether generic correction is safe.Source: §A.6 · p. 22

B Dataset Details

• Four benchmarks isolate complementary risks: HallusionBench visual hallucination, POPE object presence, VSR spatial relations, and VizWiz visual insufficiency and answerability.The suite is designed to expose different diagnostic stresses rather than create a pooled hallucination leaderboard.Source: B Dataset Details · p. 22

B.6 Failure-Family Mapping

• The evaluation pools model-output instances across datasets, assigns correctness and coarse benchmark-based failure-family labels, and reports both pooled and within-dataset diagnosis.Correct predictions map to no failure, while incorrect predictions follow each benchmark’s primary stress factor and available metadata.Source: §B.6 · p. 25, §B.8 · p. 25

C Complete Prompt Catalog

• Diagnostic probing keeps prompts reproducible while applying visual degradation, blank-image replacement, grounding re-checks, and VSR-only relation swaps to compute V, L, and A.Answer normalization is applied consistently across clean, perturbed, scalarization, and intervention evaluations.Source: C Complete Prompt Catalog · p. 25, §B.7 · p. 25, §C.2 · p. 26

D.1 Image-Removal Saturation Controls

• 0.952 AUROC remains for (V, Lpersist, A), rising to 0.967 with scalar confidence, while stricter L variants preserve the diagnostic pattern despite reducing raw L dominance from 99.7% to about 51.6–51.7%.The default L therefore measures confidence retention under image removal, not answer persistence or a causal source label.Source: §D.1 · p. 30

D.2 Alignment Availability Controls

• Excluding VSR still improves AUROC from 0.761 to 0.839 and Macro-F1 from 0.329 to 0.422, showing diagnosis is not driven by relation-swap availability or its weighting.Relation swapping remains a task-valid alignment stress for relation-bearing inputs, while A is not directly comparable across heterogeneous datasets.Source: §D.2 · p. 30

D.3 Visual-Perturbation Robustness

• Diagnosis remains robust to probe choice and severity: removing individual visual probes keeps AUROC at 0.822–0.824, while weak, default, and strong settings yield 0.785, 0.786, and 0.783 with Lpersist.Instance rankings correlate 0.892 for weak and 0.924 for strong perturbations, and all paired-bootstrap intervals for AUROC and Macro-F1 gains exclude zero.Source: §D.3 · p. 31

D.4 Full Source Profiles

• L is high across settings, but raw arg max(V, L, A) selects L for 58,287/58,440 instances; diagnosis therefore uses the full vector and normalized dominance for routing.Mean L ranges from 0.914 to 1.000, whereas V and A show stronger task dependence.Source: §D.4 · p. 31

D.5 Interventional Entanglement Across Models

• Only 18 of 48 intervention checks are diagonally aligned—11/16 visual, 2/16 language, and 5/16 alignment—so (V, L, A) is a joint behavioral signature, not causal decomposition.Visual ruin is often magnitude-selective, but signed and off-target responses vary across models, including shifts in A under visual or language interventions.Source: §D.5 · pp. 31–32

E Human Audit Protocol; E.2 Sampling Strategy; E.3 Annotator Recruitment And Payment; E.4 Annotator View; E.5 Annotation Labels; E.6 Annotation Instructions

• The human audit samples 200 balanced benchmark examples, uses six visible-evidence labels, and excludes confidence, source scores, dominant-source indicators, routing decisions, and model explanations.Examples are stratified by normalized dominant source when possible, while correct and incorrect predictions are both retained; annotators were locally recruited students aged 18–22 without cash compensation.Source: §D.9 · pp. 35–36, §E.2 · p. 40, §E.3 · pp. 40–41

D.9 Human Audit Results

• Human adjudication assigns 71 alignment, 20 language-prior, and 16 visual clear-source labels, contrasting with near-universal raw L-dominance and supporting alignment’s diagnostic value.Among 140 incorrect examples, 71 are alignment, 32 unclear, 19 language-prior, 16 visual, and 2 mixed.Source: §D.9 · p. 36

D.8 Dataset-Identity And Within-Dataset Controls

• In the dataset-ID-controlled diagnosis, dataset ID alone reaches AUROC 0.976 and Macro-F1 0.558; adding V + L + A to dataset and model identifiers yields Macro-F1 0.783.Under stricter outcome-family control, the combined representation reaches AUROC 0.853 and Macro-F1 0.348, compared with baseline Macro-F1 0.145 for dataset and model identifiers.Source: §D.8 · p. 34

D.9 Human Audit Results; E Human Audit Protocol; E.1 Audit Objective

• Raw dominance agrees poorly with clear-source human labels, reaching accuracy 0.165 and Macro-F1 0.094, while normalized dominance improves accuracy to 0.363 and Macro-F1 0.307.The audit therefore supports using the full signature as a behavioral diagnostic object rather than treating arg max(V, L, A) as a causal source label.Source: §D.9 · p. 37

D.10 Abstention And Source-Aware Scalarization

• Scalar confidence retains pooled correctness-ranking AUROC 0.783 versus 0.767 for the best source-aware scalarization, with source-aware penalties also worsening low-coverage risk.At 5% coverage, scalar confidence risk is 0.018 versus 0.023 for weighted scalarization and 0.290 for max-penalty.Source: §D.10 · pp. 37–38, Table 32 · p. 41, Table 34 · p. 42

D.8 Dataset-Identity And Within-Dataset Controls

• AUROC rises from 0.634 to 0.769 on HallusionBench and from 0.707 to 0.817 on VizWiz when (V, L, A) replaces scalar confidence under fixed-dataset controls.Combined representations reach AUROC 0.801 and 0.825, respectively; gains are smaller on POPE and VSR.Source: §D.8 · p. 35, Table 25 · p. 38

D.11 Intervention Routing

• Matched routing is conditional rather than universal: it beats generic intervention in 10/16 model-dataset settings, reduces break rate in 10/16, and improves accuracy over generic CoT by 3.54 points.In the pooled comparison, no intervention reaches 76.6% accuracy, generic intervention 75.7%, and matched routing 75.9%; routing lowers pooled break rate from 4.1% to 3.9%.Source: §D.11 · pp. 38–39, Table 35 · p. 43, Table 36 · p. 43

E.7 Agreement and Adjudication

• Three annotators independently label each example, adjudicate disagreements using only visible evidence, and report full-label Fleiss’ κ 0.236 versus source/non-source κ 0.480.Exact all-annotator agreement is 20.5%, while at least two annotators agree in 82.0%; model-human comparison is restricted to clear-source cases, with mixed and unclear cases separate.Source: §D.9 · p. 36, §E.7 · p. 42

E.8 Reporting Metrics

• HalluPrism uses eight audit quantities covering annotator agreement, source coverage, model-human agreement, and mixed, unclear, and no-failure rates.The reported agreement measures include raw agreement, Fleiss’s κ, collapsed Fleiss’s κ, and model-human agreement.Source: §E.8 · p. 43
• Clear-source auditing maps Visual, Language-prior, and Alignment labels to V, L, and A coordinates and reports agreement on adjudicated clear-source examples.The audit also reports a confusion matrix over clear-source examples and separately counts mixed and unclear cases.Source: §E.8 · pp. 43–44

E.9 Interpretation

• Human auditing tests semantic plausibility of the Perturbation-Response labels, not causal recovery of model-internal sources.Low model-human agreement would indicate that normalized dominant source is not always human-interpretable as a single source, while remaining consistent with the entanglement claim.Source: §E.9 · p. 44, Table 39 · p. 45

F Computational Cost And Resource Use

• The core Perturbation-Response computation requires 7 forward passes without relation-swap probing and 8 for VSR, trading overhead for source-aware diagnostic information.Each evaluation uses the clean answer, four visual perturbations, one blank-image answer, and one or two alignment-probe answers.Source: F Computational Cost And Resource Use · p. 45

F.1 Inference Budget

• 58,440 model-output instances were evaluated across four datasets and four model families.HallusionBench, POPE, and VizWiz-VQA use 7 passes per model-example; VSR uses 8 because relation-swap probing is available.Source: §F.1 · p. 45, Table 41 · p. 46
• Routing adds two additional branches per model-output instance: one generic intervention and one matched intervention.The reported full diagnosis-plus-routing budget excludes optional self-reported confidence baselines and implementation-level retries caused by model-serving failures.Source: Table 41 · p. 46, §F.1 · p. 46

F.2 Training Cost

• The diagnostic classifiers are lightweight and analysis-only, using at most four features without fine-tuning or updating the evaluated MLLMs.Their cost is negligible relative to MLLM inference.Source: §F.2 · p. 46

F.3 Decoding And Runtime Controls

• Greedy decoding with temperature 0, disabled sampling, and a 128-token limit reduces stochastic variation and makes probe costs easier to interpret.Perturbation-Response cost scales linearly with the number of probes, which can be selectively disabled, although the full protocol is retained for the joint-signature analysis.Source: §F.3 · p. 47

F.4 Hardware And Wall-Clock Reporting

• Forward-pass counts, decoding configuration, checkpoints, and prompt regimes are the primary reproducibility quantities because wall-clock runtime varies with hardware and serving conditions.Hardware and runtime are reported in Table 42.Source: §F.4 · p. 47, Table 42 · p. 47

F.5 Resource Use And Environmental Reporting

• HalluPrism avoids fine-tuning costs but increases inference cost through multiple perturbation branches, so the paper reports fixed forward-pass budgets rather than unmeasured carbon or power estimates.GPU-hours or energy use will be reported only if available.Source: §F.5 · p. 47

F.6 Practical Deployment Tradeoff

• The full protocol is intended for audit-time, high-risk, or routing use rather than as a cheap scalar for every query, with selective deployment preserving source-aware diagnosis before correction.Latency-critical systems can reserve it for uncertain, high-impact, or audit-selected examples.Source: §F.6 · p. 47

G.1 Dataset Artifacts; G.3 Derived Artifacts

• Released materials exclude new user images, benchmark images, and model weights, instead providing derived metadata, prompts, normalized outputs, source scores, and aggregate results unless licenses permit raw-data redistribution.VizWiz-VQA raw images receive additional care because they may contain user-provided visual contexts.Source: G Artifact, License, and Data Statement · p. 48, §G.1 · p. 48, §G.3 · p. 49

G Artifact, License, and Data Statement; G.1 Dataset Artifacts; G.2 Model Artifacts

• The study uses public benchmark datasets and accessible model checkpoints under their stated terms, with dataset access and license summaries reported in Table 43 and model artifacts in Table 44.The evaluated models are used only for deterministic inference; their weights are not fine-tuned, modified, or redistributed.Source: §G.1 · p. 48, Table 43 · p. 48, §G.2 · p. 48

G.4 License Compliance

• Users remain responsible for complying with original dataset and model terms, while the released implementation includes its own license file and documents external dependencies separately.This applies to HallusionBench, POPE, VSR, VizWiz-VQA, Gemma 4, Qwen3-VL, LLaVA-Mistral, and LLaVA-Vicuna.Source: §G.4 · p. 49

G.5 Data Sensitivity

• VizWiz-derived raw images are not redistributed because they may expose private environments, personal objects, or accessibility-sensitive contexts.Downstream reuse should follow dataset licenses, privacy notes, access terms, and intended-use statements; derived artifacts support reproducibility without raw images or model weights.Source: §G.5 · p. 49, Table 45 · p. 50

G.6 Statement Of Use

• The artifacts support diagnosis-first evaluation, not deployed-model training, private-attribute inference, person identification, or decisions about real users.Any audit-label release should omit annotator-identifying information and retain only example identifiers, adjudicated labels, and aggregate agreement statistics.Source: §G.6 · p. 50

I Ethical Considerations and Responsible Use

• HalluPrism measures behavioral responses under controlled probes but does not establish causal model internals or guarantee safety on new open-world instances.It is intended for model analysis, audit support, and safer correction-path selection rather than standalone certification or deployment approval.Source: I Ethical Considerations and… · p. 50

I.1 Intended Use

• Appropriate deployment requires specifying the task, user population, validation distribution, risk threshold, fallback policy, and monitoring plan before routing corrections.The method is intended for benchmark comparison, failure-family analysis, and correction-path selection only when validation shows routing reduces harmful breakage.Source: §I.1 · pp. 50–51, Table 46 · p. 51

I.2 Non-Intended Use

• HalluPrism should not replace human judgment or certify high-stakes outputs without domain-specific validation, expert review, and post-deployment monitoring.Its source scores are bounded diagnostic signals under a fixed protocol, not ground-truth causes or complete reliability guarantees.Source: §I.2 · p. 51

I.3 Risks From Diagnostic Misuse

• The primary misuse risk is over-trust because signatures are bounded by evaluated datasets, models, prompts, perturbations, and routing choices, leaving out-of-distribution harms possible.Potentially missed risks include expertise gaps, cultural context, ambiguous intent, missing background information, and harms outside measured source coordinates.Source: §I.3 · p. 51
• Correction routing can help one model-dataset condition while harming another, so systems should treat it as conditional actionability and log outcomes beyond accuracy.Recommended logs include the original and corrected answers, routed action, fix and break indicators, and human-review outcomes.Source: §I.3 · p. 51

I.4 Risks For Affected Users

• For blind or low-vision users, unsupported visual answers and generic corrections can cause practical harm, making accessibility-aware design and human fallback essential.HalluPrism is an audit aid for identifying risk structure, not a substitute for user consent, domain-specific evaluation, or accessibility-aware system design.Source: §I.4 · p. 51

I.5 Data And Privacy Considerations

• The main experiments use public benchmark datasets and collect no new user images.Any human audit samples evaluated benchmark outputs and requires only standard annotation logistics.Source: §I.5 · p. 52
• Annotators do not see source scores or routing decisions, limiting score-induced bias during the audit.Source: §I.5 · p. 52
• Public image datasets may contain sensitive visual content, private environments, or user-specific contexts.Source: §I.5 · p. 52
• Released artifacts should document dataset sources, licenses, access constraints, and which fields are original, derived, or model-generated.Source: §I.5 · p. 52

I.6 Responsible Deployment Guidance

• Deployment planning should specify the target population, validation distribution, expected shifts, acceptable risk, fallback policy, trigger conditions, logging fields, and drift monitoring.Source: §I.6 · p. 52
• In high-impact or latency-critical settings, reserve the full protocol for audit-selected, uncertain, or high-risk examples.Source: §I.6 · p. 52
• Perturbation-Response Signatures should guide review or guarded correction rather than automatically certifying answers as safe.Source: §I.6 · p. 52