PaperlayerView exact revision on arXiv

Source-linked AI summary

PAIQ: Patch-Aligned Semantic Injection via Residual Rotation

Pinze Ren, Yuwei Zhang, Hao Chen, Linghao Meng, Chang Li, Qiankun Li

arXiv:2609.37685v3cs.CV

TL;DR

Visual-language understanding needs semantic recognition and local detail, but fusion does not automatically preserve distinctions between related patches. PAIQ matches and aggregates complementary encoder features, then injects them with an orthogonal residual update; across evaluated backbones and benchmarks, it reports higher Accuracy and lower Hallucination than matched single-encoder baselines.

  • Problem

    The central challenge is enriching local visual features with complementary semantics while preserving distinctions between patches that may share a category but differ in attributes, parts, or spatial relations.

    Source: §1 · p. 1

  • Method

    PAIQ uses DINOv3 patch features as a base, aggregates content-matched SigLIP features, and injects their difference through a shared orthogonal transformation.

    Source: §1 · pp. 1–2

  • Results

    Across five language backbones and four benchmarks, PAIQ achieves higher Accuracy than matched single-encoder baselines in 19 of 20 settings and lower Hallucination in all 20 settings.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    The evaluated interface integrates complementary visual representations while retaining a 196-token input to the language model.

    Source: §1 · p. 2, §5 · p. 9

    The separability statements are conditional on projected features and do not guarantee lossless spatial-information preservation.

    Source: §3.3 · p. 6

Abstract

from arXiv · show

Language-aligned and self-supervised visual encoders offer complementary strengths in semantic abstraction and spatial detail. Harnessing this complementarity requires enriching local features while retaining distinctions between semantically related patches. We introduce PAIQ, a patch-aligned semantic injection framework that combines content-based cross-encoder matching with orthogonally constrained residual updates. Using DINOv3 patch features as the spatial base, PAIQ aggregates complementary SigLIP features through joint source allocation and injects the aggregate--base differences through a shared orthogonal transformation Q. This rotation adapts update directions while preserving residual norms and pairwise angles. For fixed projected features, we derive conditions for patch separability under similar semantic aggregates and show that rotation adds a nonnegative separation term over direct interpolation when the aggregate is shared. Only the projection and fusion parameters are trained; both visual encoders and the language model remain frozen, and fusion retains 196 visual tokens. Across diverse language backbones, PAIQ yields broad gains in judge-assessed correctness and reductions in hallucination severity over single-encoder interfaces on image description and visual question answering. On the 2B and 9B Qwen backbones, this compact interface outperforms the strongest evaluated fusion or token-compression baselines by about 2.9 correctness points on average.

1 INTRODUCTION

PAIQ addresses how to enrich patch-level visual features with complementary semantics without losing distinctions between locations. It combines content-based matching and orthogonal residual updates, and reports broad gains over single-encoder interfaces.

1 INTRODUCTION

  • Visual understanding requires semantic recognition alongside local detail, while existing fusion designs do not by themselves constrain how patch distinctions change.Patches sharing a category such as clothing may still differ in attributes, parts, and spatial boundaries.Source: §1 · p. 1
  • For fixed projected features, residual rotation preserves update geometry and can retain patch distinctions even when positions receive the same or similar aggregates.Under shared aggregates, rotation adds a nonnegative separation term over direct interpolation; the separability statements are conditional, not guarantees of lossless spatial-information preservation.Source: §1 · p. 2
  • Across four datasets and language backbones up to 27B, PAIQ improves judge-assessed correctness and reduces hallucination severity over matched single-encoder interfaces.With 2B and 9B Qwen backbones, it achieves the highest correctness scores on every dataset; a matched Qwen3.5-2B ablation reports a 31.81-point task-macro correctness improvement over direct interpolation.Source: §1 · p. 2
  • PAIQ enriches DINOv3 patch features by matching and aggregating SigLIP features, then injecting the aggregate–base difference through a shared orthogonal transformation.The approach combines content matching, joint source allocation, and residual updates while retaining a fixed-length visual sequence.Source: §1 · pp. 1–2, Figure 1 · p. 2

2 RELATED WORK

Related work spans visual encoders with different representational emphases, multi-encoder fusion interfaces, and intervention-based analyses of visual evidence use.

2 RELATED WORK

  • Language-aligned encoders such as SigLIP and self-supervised encoders such as DINOv2 and DINOv3 learn complementary visual representations, though neither family is limited to only one type of information.Related approaches also align DINO features with text or adapt granularity within a single DINOv3 stream.Source: §2 · p. 3
  • Multi-encoder interfaces either align features on common grids and fuse channels or aggregate features into learned queries, while token-reduction methods compress the visual sequence.The cited approaches include common-grid fusion, query-based fusion, and query-based compression with image-text pre-fusion.Source: §2 · p. 3
  • Intervention-based studies assess visual evidence use through answer changes, since attention weights alone do not establish predictive dependence.Figure 3’s diagnostic readouts include fixed-answer likelihood differences after local residual removal.Source: §2 · pp. 3–4, Figure 3 · p. 4

3 METHOD

PAIQ projects frozen DINOv3 and SigLIP features into a shared space, matches and aggregates SigLIP features for each DINOv3 position, and injects the difference through a shared orthogonal transform. The method also gives conditional results on preserving patch distinctions and defines diagnostics for source mixing, update size, and residual removal.

3.1 REPRESENTATIONS AND LEARNING OBJECTIVE

  • Frozen DINOv3 and SigLIP features are independently projected, and the fused 196-token sequence replaces image placeholders while only projection and fusion parameters are trained.The autoregressive objective supervises answer and end-of-sequence tokens; gradients pass through the frozen language model, without auxiliary transport loss, correspondence labels, or an orthogonality penalty.Source: §3.1 · pp. 4–5

3.2 CONTENT-BASED MATCHING AND AGGREGATION

  • Content-based cosine matching learns associations across the encoders’ different patch grids, and finite-step row–column normalization couples source allocation across output positions.Matching is instruction-independent and has no coordinate-distance or local-window constraint; five normalization iterations need not satisfy exact transport marginals, and a source feature may contribute to multiple positions.Source: §3.2 · p. 5

3.3 ORTHOGONAL RESIDUAL ENRICHMENT

  • A shared Cayley-parameterized orthogonal transform rotates the aggregate–base residual before it is added to the base, preserving residual norms and angles.The transform changes residual directions relative to the base; the orthogonality constraint applies to updates, not to the final fused features or trainable projections.Source: §3.3 · p. 5
  • For fixed projected features, distinct bases remain distinct when ∥e_jk∥2 < (1−λ)∥x_jk∥2/λ; with a shared aggregate, rotation adds a nonnegative separation term over direct interpolation.The separation term is strictly positive when Qx_jk ≠ x_jk; these are conditional statements, not guarantees of lossless spatial-information preservation.Source: §3.3 · p. 6
  • The method separately diagnoses source mixing, relative update size, and the effect of removing local residuals using fixed-answer log-likelihood differences.These diagnostics do not enter training; evaluating the likelihood differences requires additional teacher-forced evaluations.Source: §3.3 · p. 6

4 EXPERIMENTS

The experiments compare matched visual interfaces across four datasets and multiple language backbones using Judge-scored Accuracy and Hallucination. PAIQ generally improves correctness and reduces hallucination severity, with scaling, sample-level, compute, and ablation analyses.

4.1 EXPERIMENTAL SETUP

  • The evaluation covers image description and visual question answering across four datasets, using matched single-encoder interfaces and Judge scores for Accuracy and Hallucination.FLUX is an in-domain description diagnostic; the other datasets assess transfer to different image and question distributions.Source: §4.1 · p. 6

4.2 MAIN RESULTS

  • Across backbones, PAIQ improves correctness and generally reduces hallucination severity over matched single-encoder interfaces, achieving highest Accuracy on three of four InternVL tasks.Table 1 compares mean Judge scores across four language backbones and four tasks.Source: §4.2 · p. 6, Table 1 · p. 7

4.3 SCALING AND COMPUTE

  • At 27B, PAIQ remains ahead of both single-encoder interfaces on every dataset and metric; generation is limited to 512 tokens versus 128 at smaller scales.The scaling study compares method-matched Qwen3.5 models.Source: §4.3 · p. 7, Table 2 · p. 8

4.4 SAMPLE-LEVEL SCORE DISTRIBUTIONS

  • 67.85% of pooled Qwen3.5-2B responses receive Accuracy scores of at least 80, compared with under 21% for either single-encoder interface.Figure 5 pools four equally sized datasets, with 4,000 responses per method; its distributions also shift Accuracy upward and Hallucination downward for PAIQ.Source: §4.4 · p. 7, Figure 5 · p. 9
  • PAIQ exceeds a post-hoc best-of-two selector in all four displayed FLUX backbone settings, although the normalized analytical-compute proxy excludes fusion.The selector chooses the completed single-encoder output with higher Accuracy for each example.Source: §4.4 · p. 8, Figure 4 · p. 8
  • PAIQ retains 196 visual tokens versus 576 for SigLIP-only and 772 for concatenation, with analytical compute at 133% of DINOv3-only at 2B and 103–108% for larger backbones.The compute figures are analytical comparisons normalized to DINOv3-only.Source: §4.4 · p. 8, Table 3 · p. 8

4.5 ABLATION AND DIAGNOSTICS

  • Learned residual rotation improves Qwen3.5-2B task-macro Accuracy by 31.81 score points over fixed-Q = I interpolation, with gains across all four datasets.Fixing Q = I also reduces Accuracy and increases Hallucination on all four datasets.Source: §4.5 · p. 8, Figure 6 · p. 9
  • Paired intervals favor PAIQ on both metrics in 18 of 20 settings; in the FLUX case, answer sensitivity concentrates near hair and dress while update magnitudes are more diffuse.These spatial correspondences are diagnostic and do not establish phrase-level causality.Source: §4.5 · p. 9, Figure 6 · p. 9

5 CONCLUSION & LIMITATION

PAIQ integrates DINOv3 spatial features with SigLIP semantics through orthogonal residual injection, retaining a fixed 196-token visual sequence. Across five language backbones and four benchmarks, it reports higher Accuracy in 19 of 20 settings and lower Hallucination in all 20, while the study is limited to one encoder pair.

5 CONCLUSION & LIMITATION

  • PAIQ achieves higher Accuracy than matched single-encoder baselines in 19 of 20 settings and lower Hallucination in all 20.These results span five language backbones from 2B to 27B and four benchmarks.Source: §5 · p. 9
  • The interface aligns DINOv3 spatial features with SigLIP semantics through orthogonal residual injection while keeping the visual sequence at 196 tokens.The conclusion says scaling, ablation, and diagnostic analyses support the alignment and residual-injection design under this fixed token budget.Source: §5 · p. 9
  • The study evaluates one pair of complementary visual encoders; higher-resolution and video inputs, additional encoder families, and adaptive alignment remain unevaluated.The listed appendices cover supplementary method analysis, implementation and diagnostics, experimental settings, additional results, compute accounting, qualitative cases, and evaluator prompts.Source: §5 · p. 9, APPENDIX CONTENTS · p. 12

A SUPPLEMENTARY METHOD ANALYSIS

The supplementary analysis qualifies PAIQ’s matching and residual geometry, distinguishing finite stabilized computation from idealized transport and bounding what orthogonality guarantees. It also notes that the algebraic claims concern projected features and exact arithmetic.

A SUPPLEMENTARY METHOD ANALYSIS

  • The algebraic results concern projected features and exact arithmetic, while the idealized transport interpretation omits clipping and numerical stabilizers.The appendix first establishes properties used in the main method, then gives execution and diagnostic details.Source: A SUPPLEMENTARY METHOD ANALYSIS · p. 12

A.1 MATCHING NORMALIZATION AND SOURCE ALLOCATION

  • Five row-then-column iterations produce ten factor updates, and the finite clipped, stabilized coefficients need not satisfy the ideal marginal constraint.The stabilizer makes implemented coefficient row sums less than one; setting δ = 0 recovers a row-conditioned barycentric average.Source: §A.1 · p. 12
  • Without clipping and δ, row conditioning cancels global scaling and yields the same conditional coefficients as probability-marginal updates.The comparison is derived for the idealized updates; the actual finite clipped computation need not satisfy the marginal constraint.Source: §A.1 · p. 13
  • The marginal constraint balances total source use rather than imposing one-to-one matching, and its task-level effect is empirical.Balancing may discourage concentration on a few sources but may also increase the weight of irrelevant sources.Source: §A.1 · p. 13

Cayley orthogonality and residual geometry

  • The Cayley transform is a continuous orthogonal rotation, preserving residual distances and angles, but does not bound updates relative to base-feature norms.The cross term changes with update direction, and trainable content projections can change both feature scales.Source: Cayley orthogonality and residual… · pp. 13–14

A.3 PATCH DIFFERENCES: PROOF AND EXTENSIONS

The patch-difference analysis gives sufficient conditions for distinct outputs when aggregates are shared or sufficiently close, and refines the separation bound using the learned rotation. Under stated projection assumptions, direct interpolation and PAIQ represent the same function class, so the explicit rotation is a training parameterization rather than an expanded class.

A.3 PATCH DIFFERENCES: PROOF AND EXTENSIONS

  • If xjk ≠ 0 and ||ejk||2 ≤ κ||xjk||2 for κ < (1 − λ)/λ, the output distance is at least (1 − λ − λκ)||xjk||2 > 0.The lower bound applies when aggregate differences are sufficiently small relative to base differences.Source: §A.3 · p. 14
  • The rotation-aware refinement is never weaker than the earlier bound and permits a larger aggregate discrepancy when Qx ≠ x.This is a feature-level guarantee, not a claim that training realizes such rotations or improves task accuracy.Source: §A.3 · p. 15
  • For a shared aggregate, rotation adds a nonnegative separation term over direct interpolation; it is strictly positive when Qxjk ≠ xjk.The result is conditional on fixed base and aggregate features.Source: §A.3 · p. 15

A.4 EQUIVALENT PARAMETERIZATIONS WITH FREE PROJECTIONS

  • With unrestricted affine content projections and linear matching maps, no obstructing intervening activation or normalization, and fixed 0 < λ < 1, PAIQ and Q = I represent the same function class.The equivalence assumes the same matching and aggregation rule and exact arithmetic.Source: §A.4 · p. 15
  • Function-class equivalence does not imply identical initialization, gradients, optimization trajectories, or finite-budget results.The explicit orthogonal layer specifies a training parameterization under the stated assumptions.Source: §A.4 · p. 16

B IMPLEMENTATION AND INTERNAL DIAGNOSTICS

The implementation fuses frozen-encoder features into 196 visual vectors, training only projection and fusion parameters. The section also defines diagnostics for source mixing, update size, and answer-likelihood sensitivity, with limits on their interpretation.

B.1 FEATURE EXTRACTION AND EXECUTION

  • 196 fused vectors replace image-placeholder embeddings, while both visual encoders and the language model remain frozen and only projection and fusion parameters are trained.Answer-token and EOS loss is backpropagated through the frozen language model to the visual input; no auxiliary correspondence or orthogonality loss is used.Source: §B.1 · p. 16

B.2 PARAMETER COUNTS AND COMPUTATIONAL SCOPE

  • The fixed 196-token output limits the visual prefix but does not establish end-to-end latency, memory, or FLOPs savings.Table 4 counts trainable parameters by hidden dimension, including both content projections and excluding frozen visual and language parameters.Source: §B.2 · p. 17, Table 4 · p. 17

B.3 AGGREGATION AND RESIDUAL DIAGNOSTICS

  • Source concentration summarizes the contextualized SigLIP mixing coefficients, but is not exact Shannon entropy, semantic uncertainty, or the fraction of an answer attributable to image pixels.The row sum is below one and the logarithm includes a stabilizer.Source: §B.3 · p. 17
  • The relative-norm map locates updated output positions, not necessarily the image regions supplying their aggregate content.Its ratio can exceed one or become large with a small base norm, and is undefined at zero base norm.Source: §B.3 · p. 17
  • Blockwise likelihood sensitivity measures the answer-level effect of restoring updated features to projected DINOv3 bases, not phrase-level causality or image-region attribution.Other fused features, parameters, and text remain fixed; the same generated answer is scored under teacher forcing without regeneration or length normalization.Source: §B.3 · pp. 17–18

C.1 LANGUAGE BACKBONES, TRAINING, AND GENERATION

The experiments attach the same visual fusion to frozen language-side configurations and use a shared optimization recipe. Generation limits differ at 27B, and the clustering comparator is a fixed-10 adaptation.

C.1 LANGUAGE BACKBONES, TRAINING, AND GENERATION

  • The experiments use one training seed, and the named backbone configurations refer to frozen language-side weights rather than unmodified official multimodal systems.The InternVL configuration uses its language weights and tokenizer, not its original vision encoder and projector.Source: §C.1 · p. 18
  • The Qwen3.5-27B archive uses a 512-token generation limit, unlike the 128-token limit at smaller scales, so the scaling comparison is not controlled for a common generation limit.Equal limits within a backbone do not imply equal realized response lengths.Source: §C.1 · p. 18
  • The Granulon-style comparator appends 10 fixed-K cluster means to 196 DINOv3 patch features, producing 206 visual tokens, and trains only its two-layer projector.The fixed-10 adaptation does not include a text-conditioned granularity controller; the PAI comparator sets Q = I while retaining two-stream matching and aggregation.Source: §C.1 · p. 18

C.2 EVALUATION SLICES

The evaluation uses fixed dataset slices and separate image-description and question-conditioned VQA scoring. Scores come primarily from automatic evaluators, with limited Qwen recovery for audited VQA response failures; coverage checks do not establish evaluator calibration or validity.

C.2 EVALUATION SLICES

  • The four evaluation datasets cover image description on FLUX and Caption and VQA on SEED-Bench and A-OKVQA, using fixed 1,000-record slices per cell.FLUX shares its source dataset with training and is an in-domain diagnostic; the other datasets provide cross-source evaluation without implying that all pretraining overlap is excluded.Source: §C.2 · pp. 18–19

C.3 SCORING PROTOCOL AND INTERPRETATION

  • Accuracy and Hallucination are separate judge scores in [0, 100], and task-macro means are equal-weight descriptive summaries rather than a common accuracy scale across task rubrics.Descriptions are judged against the image, while VQA is judged against the image and a particular question with an authoritative reference.Source: §C.3 · p. 19
  • The recorded archive retains evaluator returns for all 330 empty or whitespace-only outputs, including 52 of 71 empty VQA candidates that violate the executed prompts’ zero-score rule.The VQA scoring rules describe the rubric, not guaranteed compliance of each recorded judgment.Source: §C.3 · p. 20, Table 7 · p. 20, Table 8 · p. 20

C.4 EVALUATOR ROUTES AND REQUEST VALIDATION

  • Audited terminal empty responses in SEED and A-OKVQA can receive Qwen recovery, but a valid GPT-4o score is not replaced by recovery.Failed evaluator requests are not converted to zero or imputed, and record validation does not establish semantic correctness of evaluator decisions.Source: §C.4 · p. 21

C.5 EVALUATOR-SOURCE COVERAGE

  • Of 68,000 finalized VQA scoring axes, 207 use Qwen recovery, so the results are hybrid-evaluator rather than exclusively GPT-4o scores.Matching prompts and schemas preserves the request definition but does not establish calibration between evaluators or human-level scoring validity.Source: §C.5 · p. 22

D.1 QWEN3.5-2B VQA RESULTS AND RECOVERY

The dedicated Qwen3.5-2B archive contains complete VQA score groups, and paired intervals favor PAIQ across its comparisons. The intervals are conditional on recorded outputs and scoring, while the best-of-two reference is only a post-hoc accuracy upper bound among observed outputs.

D.1 QWEN3.5-2B VQA RESULTS AND RECOVERY

  • The archive contains 20 complete method–dataset–axis groups of 1,000 records each, with no missing or duplicate keys or terminal failures.GPT-4o supplies 19,961 of 20,000 axes; the remaining 39 audited terminal empty-response keys receive valid Qwen recovery results.Source: §D.1 · p. 22
  • The paired-difference intervals favor PAIQ in all 16 comparisons against four comparators across two VQA datasets and two score axes.These are paired score differences, not marginal table intervals, and quantify resampling uncertainty conditional on recorded outputs and evaluator scores.Source: §D.1 · pp. 22–23, Table 12 · p. 22

D.2 AGGREGATION AND PAIRED UNCERTAINTY

  • The reported intervals are record-level and conditional on one training seed, fixed evaluator records, and the selected comparator; they omit training and evaluator variability and do not model image clustering.The selection step is not repeated within bootstrap replicates, and the intervals are unadjusted for multiplicity.Source: §D.2 · pp. 23–24

D.3 POST-HOC BEST-OF-TWO REFERENCE

  • The post-hoc best-of-two reference is an accuracy upper bound among the two observed complete outputs, not a learned router or a bound on representations combining both inputs.Hallucination is reported for the same selected output and is not minimized independently.Source: §D.3 · p. 24

E ANALYTICAL COMPUTE ACCOUNTING

The appendix reports analytical compute coordinates using approximate 2PT accounting, but the accounting omits parts of fusion and is not a complete FLOPs estimate or latency measurement.

E ANALYTICAL COMPUTE ACCOUNTING

  • Approximate 2PT accounting uses parameter count P and token budget T, with 201 DINOv3 tower tokens, 577 SigLIP tower tokens, and 64 text tokens.The accounting excludes fusion operations, uses 303M and 316M visual-tower parameter counts, and considers 196- or 576-token language prefixes.Source: E ANALYTICAL COMPUTE ACCOUNTING · pp. 24–25
  • The average-efficiency reference evaluates both visual towers and averages the costs of the 196- and 576-token language branches, but is neither a complete FLOPs estimate nor a latency measurement.The accounting excludes transport normalization, the full Cayley solve, and all implemented fusion operations; Equation (43) separately states fusion’s asymptotic costs.Source: E ANALYTICAL COMPUTE ACCOUNTING · p. 25
  • 14,107.7, 34,791.5, 24,692.9, and 14,472.4 GFLOPs are reported for DINOv3-only, SigLIP-only, average efficiency, and PAIQ, respectively.These are analytical estimates for the archived Qwen3.5-27B parameter count, not device measurements.Source: E ANALYTICAL COMPUTE ACCOUNTING · p. 25
  • Device-matched timing, memory, and any benefit from caching Q are outside the reported measurements.The average-efficiency coordinate is an analytical reference accompanying a post-hoc comparison, not the execution cost of selecting between two generated answers.Source: E ANALYTICAL COMPUTE ACCOUNTING · p. 25

F.1 TRACING INJECTED EVIDENCE TO ANSWER RELIANCE

This section describes diagnostics for examining where PAIQ’s visual evidence appears relevant to answers, alongside question-conditioned VQA hallucination scoring and image-description protocols. It notes that the diagnostics are case-specific and documents limits on reproducing historical templates.

F.1 TRACING INJECTED EVIDENCE TO ANSWER RELIANCE

  • PAIQ’s residual-removal scores peak around the woman’s hair and floral dress, while the relative-update map spans more of the image.The score compares summed token log-likelihood for the same teacher-forced answer with and without residuals in each of 49 grid blocks; it does not isolate answer phrases.Source: §F.1 · p. 25
  • The residual diagnostic displays only positive likelihood changes, depends on the fixed model’s parameterization, and this single case cannot establish prevalence or phrase-level causality.Spatial labels connect boxes to phrases for inspection, but R(B) sums likelihood over the entire answer.Source: §F.1 · p. 25

G EXTENDED QUALITATIVE CASES WITH DECODER ATTENTION

  • The six qualitative comparisons are selected examples, and attention overlays are normalized independently, so they neither estimate error frequency nor establish comparable visual reliance or localization.Overlays average teacher-forced decoder attention at layers 11, 15, 19, and 23; they are distinct from cross-encoder weights Π and residual-removal scores R(B).Source: §F.2 · p. 25, G EXTENDED QUALITATIVE CASES WITH… · p. 26
  • In three FLUX cases, PAIQ identifies grapes, a skull tattoo, and a rendered virus, unlike the single-encoder descriptions, but also makes unsupported or erroneous details.It calls the grape cluster spherical, adds unsupported clothing coverage to the tattoo scene, and the case comparison concerns virus recognition, not biological species identification.Source: §G.1 · p. 26, §G.2 · p. 27, §G.3 · p. 28
  • In three Caption cases, PAIQ recognizes a zipper and climbing scene where single encoders describe an e-bike, comb, globe, or ball game, yet it also makes unsupported claims.The zipper is called partly open with an unsupported track direction; the climbing description says the lower person is climbing although only the upper person is clearly shown.Source: §G.4 · p. 29, §G.5 · p. 30, §G.6 · p. 31

H.1 QUESTION-CONDITIONED VQA PROMPTS

  • The VQA evaluator scores question-conditioned correctness continuously against the authoritative reference, while assessing hallucination independently rather than requiring exact-match wording.A clearly correct answer receives 100 even if brief; unsupported explanation is assessed separately unless it contradicts or undermines the answer.Source: §H.1 · pp. 32–33
  • The evaluator treats the reference answer as authoritative, permits reasonable inference, excludes unselected distractors, and scores empty VQA responses 0 because omission is not a false claim.Unsupported claims beyond the image, question, reference, or reasonable inference count as hallucinations.Source: §H.1 · pp. 34–36
  • Hallucination scoring judges unsupported or contradictory claims in answers and explanations by severity and prominence; correct answers alone score 0, while wrong answers contribute.The evaluator does not calculate hallucination as 100 minus accuracy.Source: §H.1 · pp. 34–35

H.2 IMAGE-DESCRIPTION PROTOCOL ARTIFACTS

  • Image-description evaluation used distinct templates and separate 0–100 accuracy and hallucination scores; exact prompts are omitted, and historical empty-output scoring remains unspecified.The archive links each score to its sample, response, evaluator route, rendered prompt, provider response ID, and attempt history; syntactic validation does not establish semantic judgment validity.Source: §H.2 · p. 37

Related papers

Loading 2609.37685v3…