PaperlayerView exact revision on arXiv

Source-linked AI summary

Symmetry-Aware Likelihood-Orbit Aggregation for Selective Left-Right Claim Verification

Zhouzhi Xiong, Chuxi Zhang, Weizhen He, Yi Chen, Qi Li, Donglian Qi

arXiv:2609.17004v1cs.CV

TL;DR

Frozen VLMs can reverse left–right relations, while raw claim likelihoods may not reliably rank verification errors. Relation-Orbit organizes likelihoods from a fixed reflection by relation symmetry into a parameter-free contrast. It improves coverage over Orbit-Max across all eight VSR and GQA dataset–backbone settings, while structural diagnostics indicate a systematic but not uniquely optimal advantage.

  • Problem

    Frozen VLMs can reverse left–right relations, and after reflection is fixed, one-sided scoring leaves inverse-relation and entity-exchange structure unused.

    Source: §1 · p. 1

  • Method

    Relation-Orbit uses a parameter-free contrast that organizes reflection-induced likelihoods into supporting and counterfactual roles through inverse relation and entity exchange.

    Source: §1 · p. 1, §2.2 · p. 2

  • Results

    Relation-Orbit improves coverage over Orbit-Max in all eight VSR and GQA settings; its smallest gain over Orbit-Max is +1.18 pp on VSR/Qwen-7B.

    Source: §3.2 · p. 3

  • Takeaways & Limitations

    The partition diagnostic supports a systematic, but not uniquely optimal, structural advantage for the relation-symmetry organization.

    Source: §3.3 · p. 4, §4 · p. 4

    The method requires likelihood access and eight view–formulation scores, and reflection may be invalid when flip-sensitive cues are present.

    Source: §3.3 · p. 4

Abstract

from arXiv · show

Frozen vision-language models (VLMs) remain unreliable on fine-grained left-right claims, and raw claim likelihoods need not reliably rank verification errors. After a horizontal-reflection intervention is fixed, how should its induced likelihood measurements be combined into a selective verification signal? We introduce Relation-Orbit, a closed-form contrast with no learned fusion parameters that assigns eight normalized likelihoods to query-supporting and counterfactual roles determined by reflection, inverse relation, and entity exchange. A claim is asserted only when the signed contrast exceeds a threshold selected on held-out data using pointwise Clopper-Pearson upper confidence bounds. On VSR and GQA across four frozen VLMs, Relation-Orbit yields higher mean test coverage at a 10% selective-risk calibration target than an all-eight Orbit-Max baseline in all eight dataset-backbone settings; gains over a nearly abstain-all one-sided intervention score are reported separately. A separate LLaVA-1.5/COCO evaluation, reduced-orbit controls, and a two-sided partition diagnostic further characterize the structural advantage.

1. INTRODUCTION

Fine-grained left–right errors make average accuracy insufficient when verification must abstain at an explicit error target. Relation-Orbit organizes likelihoods induced by a fixed reflection using relation symmetry, then evaluates the resulting score against alternative aggregation strategies.

1. INTRODUCTION

  • Left–right reversals persist even when a VLM identifies both objects, making abstention important when evidence cannot meet an explicit error target.Improving average accuracy alone is insufficient for downstream decisions based on spatial claims.Source: §1 · p. 1
  • With reflection fixed, one-sided scoring leaves inverse-relation and entity-exchange structure among induced likelihoods unused.The paper asks how to organize all likelihood observations induced by that intervention.Source: §1 · p. 1
  • Relation-Orbit organizes reflection-induced likelihoods into query-supporting and counterfactual roles, comparing the score with one-sided, Orbit-Max, and reduced-orbit controls.The paper also uses a separate structural diagnostic to compare alternative four-versus-four partitions.Source: §1 · p. 1
  • The work distinguishes fixed-intervention aggregation from evidence retrieval, post-training consistency, and grounded-view approaches, and reports no universal visual involution or end-to-end guarantee.Its score has no learned parameters but requires labeled held-out calibration under the common protocol.Source: §1 · pp. 1–2

2. RELATION-ORBIT AGGREGATION

The method treats a left–right claim and its transformed formulations as linked observations from a frozen VLM. It contrasts supporting and counterfactual likelihoods, then selects an assertion threshold using held-out calibration.

2.1. Problem setting

  • The task is to assert or abstain on a query claim using a frozen VLM’s token likelihoods under a fixed horizontal reflection.The claim and its entity-swapped, inverse-relation formulation are equivalent; two other formulations express the opposite relation.Source: §2.1 · p. 2

2.2. Closed-form signal construction

  • Reflection determines which of the eight view–formulation likelihoods support the query claim and which support its counterfactual.The score is a parameter-free contrast, with larger values favoring the query claim and negative values favoring its counterfactual.Source: §2.2 · p. 2
  • The contrast is invariant to joint entity exchange and relation inversion, antisymmetric to relation inversion alone, and cancels view- and formulation-specific offsets.These are properties of the constructed score and do not assume that the VLM itself is equivariant.Source: §2.2 · p. 2

2.3. Selective-risk calibration

  • Held-out calibration chooses the highest-coverage threshold whose one-sided Clopper–Pearson upper bound meets the target; if none qualifies, the method abstains on the test split.Calibration labels count false assertions but do not enter the score; test labels are used only for retrospective reporting, and the score is not a calibrated probability.Source: §2.3 · p. 2

3. EXPERIMENTS

Experiments compare Relation-Orbit with one-sided, Orbit-Max, and reduced-orbit controls under grouped evaluation, then separately test orbit structure and characterize scope. Relation-Orbit improves test coverage over both primary baselines in all eight VSR/GQA dataset–backbone settings.

3.1. Protocol and baselines

  • Across VSR and GQA, 300 seeded image-grouped splits evaluated four frozen backbones using shared prompts and threshold search.Image grouping keeps paired claims and orbit observations within partitions; no VLM or fusion parameters are trained.Source: §3.1 · pp. 2–3

3.2. Main results

  • Relation-Orbit improves test coverage over one-sided scoring and Orbit-Max in all eight dataset–backbone settings, with paired intervals above zero.Orbit-Max uses the same eight likelihoods and orbit membership but changes mean pooling to max pooling.Source: Table 1 · p. 3, §3.2 · p. 3, §3.1 · p. 3
  • Relation-Orbit beats sctr in 8/8 settings and ssym/stext in 7/8, but trails them on VSR/InternVL3-2B by −3.10/−1.26 pp, respectively.The reduced-orbit controls use partial subsets of the likelihood structure.Source: §3.2 · p. 3, §3.1 · p. 3
  • On LLaVA-1.5/COCO, Relation-Orbit gains 34.39 pp over one-sided and 13.08 pp over matched-input; all eight mean risks are below 10%, though some splits exceed it.Across the protocol, the fraction of test splits meeting the target ranges from 76.3% to 85.7%; GQA/Qwen-7B asserts only 1.91% of claims.Source: §3.2 · p. 3

3.3. Structure diagnostic and limitations

  • In a separate two-sided diagnostic, the correct orbit exceeds alternative-partition benchmarks in all 12 settings, with average rank 1.61/35, though some post-hoc alternatives do better.The label-independent alternatives hold observation count fixed; this diagnostic evaluates two-sided reliability rather than the main assertion protocol.Source: §3.3 · p. 4
  • Relation-Orbit requires likelihood access and eight view–formulation scores; flip-sensitive cues can invalidate reflection, and vertical-reflection pilots showed no stable matched-budget gains.Domain shift requires transformation checks and recalibration; stringent targets may yield abstain-all.Source: §3.3 · p. 4

4. CONCLUSION

Relation-Orbit is a parameter-free contrast that organizes reflection-induced likelihoods by inverse-relation and entity-exchange roles. Under the evaluated held-out calibration protocols, it improves coverage over one-sided and Orbit-Max controls, remains favorable to reduced-orbit alternatives in most settings, and shows a systematic but not uniquely optimal structural advantage.

4. CONCLUSION

  • Relation-Orbit improves coverage over one-sided and Orbit-Max controls under the evaluated held-out calibration protocols.It organizes reflection-induced likelihoods according to inverse-relation and entity-exchange roles.Source: §4 · p. 4
  • Relation-Orbit remains favorable to reduced-orbit alternatives in most evaluated settings.The comparison is under the evaluated held-out calibration protocols.Source: §4 · p. 4
  • A label-independent partition diagnostic supports a systematic, but not uniquely optimal, structural advantage.Source: §4 · p. 4

Related papers

Loading 2609.17004v1…