PaperlayerView exact revision on arXiv

Source-linked AI summary

Can the Parts Fool the Test? Counterfactual Pairing Cycles for Relational OOD

Yibo Zhou, Yirong Xiang

arXiv:2610.23248v1cs.CVcs.LG

TL;DR

Relational OOD is difficult to evaluate because plausible images and captions can still be paired incorrectly, while benchmark shortcuts may avoid testing their coupling. Counterfactual pairing cycles hold components fixed and change only their assignments, canceling additive component cues. Across the evaluated settings they distinguish coarse matching from stricter relational structure, and cycle training outperforms shortcut training on exact COCO cycles.

  • Problem

    Relational OOD concerns whether an image and caption are correctly paired, but benchmark differences can let models succeed using component cues rather than coupling.

    Source: §1 · p. 1

  • Method

    Counterfactual pairing cycles compare original and swapped pairings of the same images and captions, canceling image-only, text-only, and additive component scores.

    Source: §1 · p. 2

  • Results

    Across synthetic controls, COCO, and GQA, cycles distinguish coarse matching from stricter spatial and role-binding structure; on shortcut-contaminated COCO, Cycle OE scores 0.9270 versus shortcut OE 0.8817 exact CycleAcc.

    Source: §5.5 · p. 8, §7 · p. 10

  • Takeaways & Limitations

    Cycle evaluation provides evidence about pair-dependent information, but high CycleAcc is not a certificate of full semantic understanding of the intended relation.

    Source: §2.2 · p. 4, §7 · p. 10

    The exact cancellation guarantee covers additive component-level nuisances, not non-additive pair-dependent artifacts, and the population identity describes unfiltered rather than filtered cycles.

    Source: §2.2 · p. 4

Abstract

from arXiv · show

Relational out-of-distribution errors are easy to describe and hard to evaluate cleanly. An image may be natural, a caption may be fluent, and all mentioned objects may be common, yet the caption may describe the wrong relation in the image. Many evaluations intended to test this behavior can be solved without checking the pairing at all, because negatives differ in caption style, template, length, object frequency, or image statistics. We propose counterfactual pairing cycles, a blocked evaluation primitive that reuses the same two images and the same two captions on both sides of the comparison, changing only which image is paired with which caption. This makes image-only, text-only, and additive component shortcuts cancel exactly in finite samples. The same contrast also gives a training loss for relational outlier exposure and a diagnostic for bilinear vision-language scores. Across synthetic controls, GQA, and COCO, cycles separate coarse image-caption matching from directional role-binding failures, while also identifying geometric size relations that remain strongly visible in global features. In a shortcut-contaminated COCO outlier-exposure stress test, a model selected by a near-perfect shortcut validation AUROC retains substantial relational signal but remains below cycle training on exact cycles; cycle training improves CycleAcc from 0.882 to 0.927. The main lesson is simple: to test whether a model understands a pairing, the benchmark should keep the parts fixed, change the pairing, and ask only about coupling. The official implementation is available online.

1 Introduction

Relational OOD occurs when plausible images and captions are paired incorrectly, but ordinary benchmark differences can let models succeed without checking that pairing. Counterfactual pairing cycles hold the same images and captions fixed while swapping their assignments, isolating evidence about coupling.

1 Introduction

  • Relational OOD is an anomaly of image-caption coupling: each component can look plausible even when the caption assigns the wrong attribute, relation, or event.Models may recognize the objects but miss who does what, which object is where, or which caption belongs to the image.Source: §1 · p. 1
  • Benchmarks can permit shortcuts when negative captions differ in style, template, length, or object frequency, or negative images differ in style or category distribution.High performance can mislead when evaluation leaves easier component-level signals available.Source: §1 · pp. 1–2
  • Counterfactual pairing cycles compare original and swapped assignments of the same two images and captions, retaining swaps only when construction rules verify them as invalid.This makes the benchmark ask whether the original pairing is preferred rather than whether either side looks unusual.Source: §1 · p. 2
  • The paper formalizes cycles as a blocked evaluation primitive, characterizes the balance condition for canceling additive component scores, and uses the contrast for relational outlier exposure and bilinear-score diagnostics.Evaluations on synthetic controls, GQA, and COCO separate coarse matching from role-binding failures and show that shortcut validation can overstate relational transfer.Source: §1 · p. 2

2 The cycle test

The cycle test compares valid pairs with their swapped alternatives in a complete block, retaining only swaps marked invalid for the intended relation. Its signed balance cancels additive image- and caption-only scores exactly, while the population interpretation requires care when benchmarks filter cycles.

2.1 Cycles and cycle accuracy

  • A cycle starts from two valid pairs and retains the swapped pairs only when annotations or construction filters mark them invalid for the intended relation.The half-tie convention makes additive scores exactly chance on each exact-cycle split in exact arithmetic.Source: §2.1 · pp. 2–3

2.2 Why the shortcut disappears

  • A linear comparison cancels every additive image-only and text-only score if and only if each image and caption has zero signed degree.The 2 × 2 cycle is the smallest nondegenerate block satisfying the balance principle, and its contrast is unique up to scale.Source: §2.2 · p. 3
  • On every exact cycle, an additive score s(u, v) = a(u) + b(v) has CycleAcc = 1/2 under the half-tie convention.The result is finite-sample and holds within each retained block.Source: §2.2 · p. 4
  • The population identity applies to unfiltered iid recombinations; filtering for invalid swaps changes the retained-cycle distribution but preserves blockwise additive cancellation.The paper therefore distinguishes coarse object-caption cycles from stricter relation cycles where both objects are present and the swapped relation is false.Source: §2.2 · p. 4

3 Learning and diagnostics

Cycle loss trains on the same pairing contrast used for evaluation, while bilinear factorization and rank sweeps diagnose which interactions are accessible to the evaluated probes.

3 Learning and diagnostics

  • The cycle loss trains by penalizing the cycle contrast, using the same comparison that supports cycle-based evaluation.The supplied passage identifies the loss as softplus of the negative cycle contrast.Source: §3 · p. 5
  • Unlike four-entry BCE, cycle loss is unchanged by arbitrary image-only or text-only score heads, which therefore cannot improve its objective.Four-entry BCE depends on absolute scores and can reward component shifts that carry no relational information.Source: §3 · p. 5
  • Bilinear scores factor the cycle contrast into image- and text-embedding differences, with rank controlling the interaction directions available to the probe.The supplied passage introduces the bilinear factorization and its cycle contrast.Source: §3 · p. 5
  • Increasing bilinear rank can help when relevant differences are spread across directions, but cannot recover differences absent from the representations.Rank sweeps distinguish probe-accessibility limits from missing representational information.Source: §3 · p. 5

4 Constructing exact cycles

Exact cycles retain only complete four-entry blocks whose two swaps are verified invalid under conservative, dataset-specific rules.

4 Constructing exact cycles

  • Each retained cycle contains two valid pairs and their two swaps; blocks with missing entries, unavailable annotations, or unverifiable swaps are discarded.The retained labels are 1, 1, 0, 0.Source: §4 · p. 5
  • For GQA, both swapped scene-graph fact pairings must be invalid under an annotation-derived predicate; COCO object swaps require a mentioned category to be absent from the target image.COCO object categories use a fixed synonym table.Source: §4 · p. 5
  • COCO relation cycles use instance boxes: the source relation must meet a fixed geometric margin, and strict swaps retain both object categories while falsifying that relation.Construction annotations filter blocks but are not given to the detector.Source: §4 · p. 5
  • Reusing the same two images and captions on both sides matches their marginals within each block and after aggregation; caption-length marginals also match when strings are reused.The passage notes that total-variation audits are reported in an appendix.Source: §4 · p. 5

5 Experiments

The experiments use exact cycles to audit component shortcuts, probe relational information in frozen vision-language features, and test transfer from shortcut validation. Results distinguish recoverable pair-dependent signals from strict directional binding failures and show that cycle training outperforms shortcut-contaminated outlier exposure on exact-cycle accuracy.

5 Experiments

  • The experiments first audit signed-degree balance, then test what frozen CLIP-like features expose, and finally assess transfer from shortcut-selected outlier exposure.The raw-image setup uses frozen CLIP ViT-B/32 features and evaluates zero-shot cosine, interaction probes, and low-rank bilinear scores.Source: §5 · pp. 5–6

5.1 Sanity checks: the audit behaves as intended

  • Synthetic controls show additive probes at chance on pure interactions but perfect after a marginal shortcut is injected; bag representations discard binding information recovered by pair-aware scores.Rank-constrained interaction probes rise from 0.6725 at rank 1 to 0.7658 at rank 8, then dip to 0.7575 at rank 16.Source: §5.1 · p. 6

5.2 Exact-cycle diagnostics on GQA and COCO

  • Across GQA and COCO, global features support coarse matching and object-level shifts, while strict left/right binding stays near chance and size retains the strongest pair-dependent signal.The GQA binding family is omitted because it falls below the prespecified minimum of 30 validation cycles.Source: §5.2 · p. 6

5.3 Strict spatial relations

  • In strict COCO cycles, left/right remains near chance for CLIP cosine and rank-64 probes, while size reaches 0.763 and 0.787, respectively.Above/below reaches 0.659 with the trained rank-64 probe, and same-pair relation changes reach 0.599.Source: §5.3 · p. 6, Figure 1 · p. 7

5.4 Rank sweeps: bilinear recoverability

  • Rank sweeps improve some interactions but leave strict present left/right near chance even at rank 64, indicating limited accessibility to the evaluated bilinear probes.Strict same-pair relation differences improve modestly, while strict size shows a large recoverable interaction signal.Source: §5.4 · p. 7, Table 3 · p. 8, Figure 2 · p. 8

5.5 Shortcut-contaminated outlier exposure

  • Shortcut-contaminated OE trains and validates on captions marked with an OOD prefix, whereas its test uses unmarked exact cycles; the preserved caption semantics leave relational signal in its interaction branch.The marker creates a text-marginal shortcut, and positives remain valid image-caption pairs.Source: §5.5 · pp. 7–8
  • Cycle OE reaches 0.9270 CycleAcc on exact-cycle tests, above shortcut OE at 0.8817 and four-entry BCE at 0.8853.Shortcut OE nearly solves its own validation task, but its validation signal overstates relational transfer.Source: §5.5 · p. 8, Table 4 · p. 9

5.6 Transfer alignment

  • Cycle-difference alignment is strongly associated with transfer in the reported diagnostics, but the correlations are descriptive because ordered family pairs share families and trained probes.Alignment is computed on validation cycles and transfer performance on held-out test cycles.Source: §5.6 · p. 9

6 Related work

Relational OOD asks whether individually plausible components are paired correctly, so component shortcuts can obscure what an evaluation measures. Counterfactual pairing cycles distinguish this work through exact marginal balance within each comparison.

6 Related work

  • Relational OOD tests whether plausible components are paired correctly, unlike conventional OOD framing that asks whether an example looks unlike the training distribution.Caption style, length, object frequency, or image-distribution artifacts can act as shortcuts.Source: §6 · p. 9
  • Related attribute-recognition work identifies label imbalance, co-occurrence bias, and feature resampling as factors in whether models learn intended attributes or correlated proxies [35].The cited analysis also asks when spurious-correlation removal is learnable [38].Source: §6 · p. 9
  • Compositional benchmarks test word order, binding, and relations with paired comparisons [28] [12], while this work requires exact finite-sample marginal balance.Theorem 1 characterizes the condition: every image and caption must have zero signed degree in the comparison.Source: §6 · p. 9

7 Conclusion

The conclusion presents counterfactual pairing cycles as a balanced test of pair-dependent information and a shared basis for training and diagnostics. Across the evaluated settings, the diagnostics distinguish coarse matching from stricter relation structure, while cycle training outperforms the shortcut-contaminated comparison on exact cycles.

7 Conclusion

  • Cycle OE obtains 0.9270 ± 0.0075 exact CycleAcc, versus 0.8817 ± 0.0021 for shortcut-contaminated BCE in the marker-controlled COCO study.The protocols differ in objective, validation criterion, and marker exposure, so the comparison does not isolate a single causal factor.Source: §7 · p. 10
  • Across synthetic controls, COCO, and GQA, cycles separate coarse object compatibility from stricter spatial and role-binding structure; strict left/right is weak while apparent size is more visible.The conclusion also identifies coarse COCO cycles as easy.Source: §7 · p. 10
  • Exact CycleAcc indicates sensitivity to pair-dependent structure, not full semantic understanding, and its conclusions depend on the evaluated cycle predicates and representations.The marker-controlled comparison is not evidence that one protocol component alone caused the difference.Source: §7 · p. 10

A Proofs of the cycle-test properties

The appendix characterizes the cycle contrast as an interaction test: it compares matched with cross-pairs while canceling additive component effects. It also connects this blockwise property to bilinear scoring and exact marginal audits.

A.1 Notation

  • The cycle contrast defines exact CycleAcc: scores are 1 for Δ>0, 1/2 for Δ=0, and 0 for Δ<0; reported results use a fixed absolute numerical tie tolerance, with numerical ties receiving half credit.Source: §A.1 · p. 14

A.2 The cycle contrast is the atomic interaction contrast

  • The cycle contrast compares two matched pairs with two cross-pairs; its zero-sum rows and columns remove row-wise and column-wise main effects while retaining interaction.Every candidate is used once on each side.Source: §A.2 · p. 14

A.3 Additive and unimodal scores lie in the cycle null space

  • For additive scores, every cycle ties and exact CycleAcc is 1/2; image-only and text-only scores are therefore exactly at chance on any nonempty exact-cycle set.The cancellation holds regardless of how accurately either score describes marginal features.Source: §A.3 · p. 15

A.4 Bilinear cycle factorization

  • The bilinear cycle identity tests whether the learned operator couples changes across modalities, so strong unimodal features may coexist with little weight on a relevant cross-modal difference.A model can assign little or no weight to that direction.Source: §A.4 · p. 15

A.5 Exact marginal audits

  • Matched and swapped component measures are identical in the empirical audit, giving total variation distance exactly zero.This follows from reusing the same images and captions on both sides.Source: §A.5 · p. 16

B Interaction-space view

  • All cycle contrasts together form the finite-dimensional interaction test after candidate-specific main effects are quotiented out.Source: B Interaction-space view · p. 16

C Low-rank bilinear accessibility

Low-rank bilinear scores can access only cross-modal differences captured by their latent coupling directions. Rank sweeps show partial recovery, with some relations remaining weak even as others are strongly visible.

C Low-rank bilinear accessibility

  • A rank-r bilinear model expresses only cross-modal difference interactions captured by its latent directions; increasing rank can help representable interactions but does not guarantee optimization will allocate capacity to a difficult relation.Source: C Low-rank bilinear accessibility · p. 16
  • The synthetic rank control reaches 0.7658 ± 0.0031 at rank 8, then saturates or decreases slightly rather than returning to near-ceiling performance.Performance improves from rank 1 to rank 8 in the reported control.Source: C Low-rank bilinear accessibility · p. 17
  • Strict size reaches 0.763 in the frozen-feature diagnostic and 0.787 in the separately evaluated rank-64 relation probe, while strict left/right remains near chance.Source: C Low-rank bilinear accessibility · p. 17

D Dataset construction and exact audits

Exact cycles retain two valid pairs and their invalid swaps, balancing component marginals within each block. The construction spans different relation families, with audits and evaluation settings defining the scope of the diagnostics.

D.1 Cycle-construction principle

  • Each retained block contains two valid matched pairs and their two exchanged, invalid pairs, with candidates drawn from the same stratum.Construction checks matched validity, cross-pair invalidity, and marginal balance.Source: §D.1 · p. 17

D.2 COCO cycle families

  • COCO cycles range from coarse object compatibility through matched-object and box relations to strict left/right and size predicates.These families test different levels of pair dependence while retaining the same four-pair form.Source: §D.2 · p. 17

D.3 GQA cycle families

  • GQA families test attribute, relation/object, and binding structure, but binding has only 18–19 validation cycles and is unavailable.The paper therefore does not treat the binding set as a stable diagnostic.Source: §D.3 · p. 17

D.4 Exact audit summary

  • All accepted validation cycles have exact marginal balance and no validity violations; GQA binding is omitted because it fails the minimum-cycle requirement.The audits check both marginal balance and whether matched and cross-paired candidates satisfy their intended predicates.Source: Table 5 · p. 18, §D.4 · p. 18

D.5 Split sizes and uncertainty sources

  • COCO uses 10,000 train2017 and 2,744 val2017 images; GQA uses 8,000 images, and family estimates share image pools.Diagnostics report variability across three independently constructed cycle splits and runs; zero-shot CLIP is deterministic conditional on each split.Source: §D.5 · p. 18

E.1 Frozen representations

  • The diagnostics use frozen CLIP ViT-B/32 representations, so learned probes isolate interaction structure in those features rather than representation learning.The encoders are not fine-tuned.Source: §E.1 · p. 18

E.2 Exact-cycle evaluation

  • A fixed absolute tie tolerance leaves cycle contrasts and tie decisions unchanged when additive image-only or text-only terms are added, in exact arithmetic.Evaluation uses one tolerance across blocks, models, and runs.Source: §E.2 · p. 19

E.3 Shortcut-contaminated OE

In shortcut-contaminated COCO outlier exposure, Cycle OE scores higher than Shortcut OE on exact cycles, while the latter retains substantial relational signal. The broader diagnostics vary by relation family, and the paper frames reliable measurement as the main practical benefit.

E.3 Shortcut-contaminated OE

  • Cycle OE reaches CycleAcc 0.9270, compared with 0.8817 for Shortcut OE on exact cycles.The three-seed differences are all positive, with a mean difference of 0.0453; Shortcut OE nevertheless retains substantial relational compatibility.Source: §E.3 · p. 19, Table 7 · p. 19

E.4 Additive and unimodal audits

  • Additive and unimodal controls receive exactly half credit on every exact cycle, even when marginal information predicts ordinary pair classification.The controls verify that marginal information is invisible to CycleAcc under exact cycles.Source: §E.4 · p. 19, Table 8 · p. 19

E.5 Broader interaction probes

  • The synthetic rank control is trained for 40 epochs and attains its best result at rank 8; subsequent behavior is saturation or a slight decrease, not full recovery of the directly observable interaction signal.Source: §E.5 · p. 20
  • COCO diagnostics find coarse, matched-object, and box-relation cycles easy, strict left/right near chance, and strict size more visible at 0.763 and 0.787.The two strict-size values are for the frozen-feature diagnostic and rank-64 relation probe, respectively.Source: §E.5 · p. 20

F Broader impact

  • The framework aims to improve multimodal measurement reliability, while its diagnostics could also help identify benchmark weaknesses or design harder shortcuts.The paper notes this dual-use risk and emphasizes exact audits and explicit reporting of near-chance or unavailable probes.Source: F Broader impact · p. 20

G Limitations

The empirical claims are bounded by the frozen encoder, two benchmarks, specific atomic cycles, and annotation-dependent construction. Some cycle families are unavailable, and the reported Shortcut OE transfer gap is not presented as universal.

G Limitations

  • The empirical analysis uses frozen CLIP ViT-B/32 on COCO and GQA, so its findings do not establish how every multimodal encoder behaves.Source: G Limitations · p. 20
  • CycleAcc tests a specific atomic 2×2 interaction, and one cycle family cannot capture every form of compositionality.Different cycle predicates may probe different visual or linguistic evidence.Source: G Limitations · p. 20
  • Cycle construction depends on annotation quality, and exact audits do not remove all semantic ambiguity in natural captions or scene graphs.Source: G Limitations · p. 20
  • The GQA binding family is unavailable because too few validation cycles survive construction and minimum-size requirements, so no binding-specific empirical claim is made.Source: G Limitations · p. 20
  • The Shortcut OE experiments support a modest, consistent transfer gap in this setting, not a universal conclusion across models, objectives, or contamination mechanisms.Source: G Limitations · p. 20
Loading 2610.23248v1…