Flow-CLIP: Image-Conditioned Residual Scoring for CompositionalImage-Text Matching

1. Introduction

• Compositional matching must distinguish descriptions that name the same objects and attributes but assign or relate them differently.Benchmarks document persistent errors on attribute binding, relations, and word order.Source: §1 · p. 1
• CLIP-based models can prefer hard negatives that share words with positives but alter object identity, attribute assignment, or spatial relations.Targeted negatives and other compositional training methods address these distinctions; LABCLIP provides a learned global-alignment baseline for measuring local refinement.Source: §1 · p. 1
• Flow-CLIP combines GCC sentence-level alignment with ICER, which uses visual patch context to correct token queries and adjust global compatibility by signed changes in pooled evidence.The components are jointly trained for caption ranking with frozen encoders.Source: §1 · pp. 1–2
• CIMR compares the same positive–negative caption margin on a matched image and a shared donor image, encouraging preference to depend on the matched image.It uses full-caption supervision alongside hard negatives.Source: §1 · p. 2
• Experiments separate binding gains from global calibration’s sentence-level gains and assess how cross-image supervision mitigates reliance on language priors.The analyses also examine corrected binding errors, new spatial errors, image-replacement effects, and variation across training sources and encoders.Source: §1 · p. 2

2.1. Compositional Evaluation

• Compositional benchmarks test controlled changes in composition, including attribute assignments, relations, word order, and spatial relations.Caption-construction controls address linguistic artifacts, while image replacement tests whether preference depends on the visual match with captions fixed.Source: §2.1 · pp. 2–3

2.2. Image–Text Matching

• Image–text matching methods range from pooled global scoring to fine-grained or structured comparisons, while Flow-CLIP uses contextual token and patch states without explicit scene-graph or subimage decomposition.LABCLIP transforms pooled text embeddings; ICER updates individual token queries using image patches, and GCC retains sentence-level transformation for comparison.Source: §2.2 · p. 3

2.3. Training Objectives

• Training objectives differ in which relationships they constrain: Flow-CLIP trains global and local terms with full-caption hard negatives, while CIMR compares the same caption pair across matched and donor images.Holding caption identity fixed constrains score change induced by image replacement.Source: §2.3 · p. 3

3. Method

• Flow-CLIP adds an ICER local term to GCC global compatibility, returning their sum as the image–caption score.The main configuration freezes both encoders; setting the local scale to zero recovers the global branch.Source: §3 · pp. 3–4

3.1. Global Compatibility Calibration

• GCC learns a linear transformation of the sentence embedding for global cosine similarity, jointly trained with the token and residual maps in Flow-CLIP.The maps initialize so the score equals pretrained global cosine similarity.Source: §3.1 · p. 4

3.2. Image-Conditioned Evidence Refinement

• ICER builds visual context from each token’s patch responses, projects it off the original query, and uses a learned residual update to produce a corrected query.Above-mean patch responses weight context features; the original and corrected queries are evaluated on the same patches, and their signed pooled-evidence change is averaged across valid tokens.Source: §3.2 · pp. 4–5

3.3. Training Objective

• The bidirectional hard-negative contrastive objective trains image–caption matching, with positive captions retained as image-retrieval queries and shared image identities accommodating multiple positives.The hard-negative objective is abbreviated HNB.Source: §3.3 · p. 5
• CIMR compares the same positive–negative caption margin on a matched image and a different-identity donor, encouraging a larger margin on the matched image.Using the same caption pair cancels additive text-only terms; the complete objective combines HNB and CIMR losses.Source: §3.3 · p. 5

4.1. Experimental Setup

• The experiments train scoring maps on COCO captions and hard negatives, compare a CC3M training source, and evaluate attribution, relations, caption order, binding, retrieval, and classification.The main configuration freezes CLIP ViT-B/32 and trains M, Mtok, and W; the supplied setup also describes encoder-adapted variants and multiple benchmark protocols.Source: §4.1 · pp. 5–6

4.2. Main Results

• Frozen COCO-trained Flow-CLIP reaches 78.15% on ABC, 3.39 percentage points above FSC-CLIP, and an ARO average of 85.94%.With CC3M training, it reaches an ARO average of 84.01% and exceeds LABCLIP on all four ARO tasks, Swap-Attribute, ABC, and GQA-Spatial, but both binding swap scores remain below FSC-CLIP.Source: §4.2 · p. 6, Table 1 · p. 7

4.3. Ablation Studies

• On COCO, Global-M raises the 11-task mean from 55.95% to 67.73%, while Flow-CLIP reaches 71.10%; the seven-task binding mean rises from 54.63% to 62.64%.The binding mean covers Swap-A, Swap-O, ABC, COCO-Spatial, GQA-Spatial, PUG, and the CLEVR 449-pair split; global calibration adds 3.70 points and the joint scorer another 4.30 points.Source: §4.3 · p. 7, Table 2 · p. 7
• The local branch improves controlled assignment accuracy under both HNB and CIMR, while Flow-CLIP scores 1.38 points lower than Global-M on COCO-Spatial.On CLEVR’s 449-pair split and PUG, HNB raises accuracy from 44.99% to 53.67% and from 50.63% to 54.20%, respectively; on the 4,449-pair CLEVR pool, Flow-CLIP reaches 56.06% versus 51.29% for HNB-trained Global-M. The COCO encoder-adaptation gains are 0.52 points on ARO, despite decreases on caption-order tasks, Swap-Object, and COCO-Spatial.Source: §4.3 · pp. 7–8

4.4. Image Replacement

• Adding CIMR lowers ARO accuracy under image replacement while nearly preserving matched-image accuracy, indicating less retained gold-caption preference when visual correspondence is disrupted.For Flow-CLIP, matched-image ARO is 85.75% with HNB and 85.91% with CIMR; under Shuffle and Fixed, CIMR yields 35.66% and 37.21%, drops of 13.08 and 12.96 points from HNB. The six-task average remains 50.38% under Shuffle.Source: §4.4 · p. 9, Table 4 · p. 9

4.5. Encoder Comparisons

• Flow-CLIP’s effect varies by encoder: it improves SigLIP’s ARO mean but lowers its binding mean, while BLIP shows the opposite pattern.Across CLIP ViT-B/32, ViT-B/16, and ViT-L/14, Flow-CLIP raises the five-task binding mean over Global-M by 2.91, 1.12, and 2.30 percentage points, respectively; ViT-L/14 also improves both retrieval directions despite decreases on several ARO tasks.Source: §4.5 · pp. 9–10, Table 5 · p. 10, Table 6 · p. 10

4.6. Classification and Retrieval

• Compared with pretrained CLIP, COCO- and CC3M-trained Flow-CLIP improve text-to-image retrieval, while top-1 classification declines on all three evaluated datasets.Relative to Global-M, COCO-trained Flow-CLIP improves all four retrieval scores, whereas CC3M-trained Flow-CLIP reduces them; COCO-trained Flow-CLIP nearly preserves COCO image-to-text retrieval versus pretrained CLIP.Source: §4.6 · p. 10, Table 7 · p. 10

4.7. Qualitative Analysis

• In qualitative examples, LABCLIP and Flow-CLIP both rank the gold caption first, with larger Flow-CLIP margins in the aircraft and plate cases; Flow-CLIP also gives positive margins on displayed PUG:SPAR assignments where CLIP gives negative margins.The aircraft example tests word order, the plate example reverses an object relation, and the PUG:SPAR examples exchange color assignments while holding objects and scene templates fixed.Source: Figure 5 · p. 11, §4.7 · p. 12
• In four CREPE examples, Flow-CLIP selects the gold caption in three cases where CLIP chooses a foil, resolving clothing, object, or location distinctions.In the pavement case, the gold caption rises from fourth to second but remains below an alternative describing a different pavement feature.Source: §4.7 · p. 12
• Patch sensitivity uses first-order changes in gold–foil evidence increments along centered patch-feature directions, highlighting regions relevant to caption distinctions.Across attribution cases, selected regions cover the door, building, wall, and pillow; adjacent margins show the final caption preference.Source: §4.7 · p. 12
• 47 of 48 illustrative gold–foil margins increased; correct top-1 selections rose from 17 to 45 of 46 cases after excluding two duplicate-gold order cases.There were 28 repairs and no correct-to-incorrect reversals, though one COCO Order case remained incorrect and the order-task candidate sets differ from the full benchmark.Source: §4.7 · pp. 12–13, Figure 8 · p. 13

• Experiments show improved binding beyond learned global alignment and reduced reliance on language priors.Source: §5 · p. 13
• GCC calibrates global alignment, while ICER adjusts the score using image-conditioned changes in token–patch evidence.Source: §5 · p. 13
• CIMR supervises caption-pair margins across matched and replacement images to address binding ambiguities that persist after sentence-level alignment.Source: §5 · p. 13