PaperlayerView exact revision on arXiv

Source-linked AI summary

RePair: Turning Retrieval Failures into Counterfactual Hard Pairs

Siyi Liu, Xiaorong Zhu, Enjun Du, Xinyu Zuo, Lisheng Duan, Haijin Liang, Jin Ma, Junfu Pu, Yongqi Zhang

arXiv:2608.29604v1cs.IRcs.CV

TL;DR

Vision-language retrieval remains vulnerable to near misses that differ from the correct match in localized semantic details, while existing mining and augmentation do not directly repair the retriever’s own failures. RePair minimally edits false positives into hard positives and paired hard negatives, yielding stronger retrieval on Flickr30K and COCO30K with 107K synthetic samples, 26%–75% fewer than comparable methods.

  • Problem

    Vision-language retrieval often fails on localized semantic distinctions, while mining cannot construct corrected counterparts and failure-agnostic augmentation may target irrelevant dimensions of hardness.

  • Method

    RePair mines false positives bidirectionally, applies minimal counterfactual editing under Validity, Minimality, and Locality, and trains with local hard-pair contrastive supervision.

  • Results

    107K synthetic samples yield consistently stronger R@1 retrieval than error-agnostic baselines on Flickr30K and COCO30K, using 26%–75% fewer samples than existing methods.

  • Takeaways & Limitations

    Failure-conditioned counterfactual repair is more data-efficient than brute-force scaling and turns retrieval errors into opportunities for local boundary repair.

  • Takeaways & Limitations

    RePair is evaluated on CLIP dual encoders, while applicability to cross-attention or fusion-based retrieval architectures remains unexplored.

Abstract

from arXiv · show

Vision-language retrieval with CLIP-style dual encoders achieves strong cross-modal performance, yet practical accuracy often hinges on localized semantic distinctions where top-ranked near misses differ from the true match by a single critical detail. Hard-sample mining can select confusable candidates but cannot construct corrected counterparts; synthetic augmentation can generate novel samples but, without conditioning on actual model failures, targets irrelevant dimensions of hardness. We observe that a top-ranked false positive is a counterfactual scaffold---sharing most of the query's semantics while differing in a localized failure-causing residual. Minimally correcting this residual yields a hard positive of the ground truth in the same modality; the corrected and unedited versions form a hard negative pair that straddles the decision boundary, producing complementary pull--push supervision. We introduce RePair, guided by three principles---Validity, Minimality, and Locality---which mines false positives bidirectionally, applies LLM-guided counterfactual editing, and trains with a local hard-pair contrastive objective. On Flickr30K and COCO30K, RePair outperforms controlled augmentation baselines with only 107K synthetic samples---26\%--75\% fewer than comparable methods---confirming failure-conditioned repair is more data-efficient than error-agnostic augmentation.

1 Introduction

CLIP-style retrieval often fails on localized semantic distinctions, especially top-ranked near misses that match most query content but miss a critical detail. RePair converts these failures into minimally edited hard pairs for local boundary repair and improves retrieval with fewer synthetic samples.

  • Motivation: Localized distinctions in attributes, relations, and word order remain persistent sources of vision-language retrieval failure.Examples include group membership, object identity, and action roles.
  • Motivation: Top-ranked near misses are especially informative because they preserve most query semantics while revealing the retriever’s failure-causing residual.Their confusable structure identifies where semantic understanding breaks down.
  • Limitations of Existing Methods: Hard-sample mining selects confusable existing candidates but cannot construct a corrected neighbor in the same semantic region.Mining and reweighting alter selection or weighting rather than pair content.
  • Limitations of Existing Methods: Most synthetic augmentation methods are not conditioned on the retriever’s failures and may change irrelevant details, requiring large synthesis volumes.These pipelines commonly use templates or external generators rather than the model’s current confusion factor.
  • RePair: RePair minimally edits a false positive’s failure-causing residual, producing a hard positive of the ground truth and a hard negative pair with the unedited error.The framework uses Validity, Minimality, and Locality, including bidirectional mining, semantic auditing, post-edit selection, and local pull–push training.

2 Related Work

Related work strengthens contrastive vision-language retrieval through hard negatives, filtering, distillation, and generated hard samples. However, existing augmentation often operates in a single modality and emphasizes negatives without balancing repulsion with attraction.

  • Hard-Sample Training: Contrastive vision-language learning is dominated by hard negatives, while simply increasing batch size yields diminishing returns.Hard-negative mining, filtering, and distillation are established strategies for improving data efficiency and robustness.
  • Synthetic Augmentation: Generative methods synthesize hard samples across modalities, including compositional hard-negative captions and hard-positive captions.Examples include NegCLIP, SugarCrepe, and LaCLIP.
  • Open Gap: Most existing methods operate in a single modality and skew toward negatives without balancing repulsion from hard negatives with attraction to hard positives.This leaves a gap in complementary hard-pair supervision.

3 Method

RePair constructs failure-conditioned hard pairs by mining and validating bidirectional false positives, minimally editing their failure-causing residuals, and contrasting the resulting examples locally.

  • Failure-Driven Mining and Validation (VALIDITY): RePair mines top-Kmine false positives in both text-to-image and image-to-text retrieval, then validates them as genuine confusions before editing.Validation excludes annotation noise and benign alternative matches.
  • Counterfactual Editing (MINIMALITY): Each retained false positive preserves a shared semantic scaffold while differing from the query in a localized failure-causing residual.This scaffold–residual structure makes the false positive a counterfactual editing seed.
  • Counterfactual Editing (MINIMALITY): An instruction generator specifies a minimal edit, a modality-specific editor generates candidates, and post-edit selection chooses the least-disruptive candidate.LLMs edit text, diffusion models edit images, and selection balances query alignment with scaffold preservation.
  • Counterfactual Editing (MINIMALITY): The edited counterpart becomes a hard positive of the ground truth, while the edited and original false positives form a hard negative pair.The synthesis pipeline outputs the grounded tuple (q, pgt, pfp, pedit).
  • Bidirectional Hard-Pair Construction: RePair applies the synthesis process bidirectionally, producing complementary hard pairs for text-to-image and image-to-text retrieval.For each direction, the corrected item is a hard positive and its original false positive is part of the hard negative pair.
  • Local Hard-Pair Contrastive Training (LOCALITY): A 3 × 3 local grid combines an anchor pair with two failure-derived pairs; diagonal entries are positives and six off-diagonal pairs are hard negatives.The local objective keeps hard-pair interactions together, while the global objective uses original anchor pairs to preserve feature-space uniformity.

4 Experiments

RePair is evaluated on Flickr30K and COCO30K under a unified CLIP fine-tuning protocol, where bidirectional failure-conditioned synthesis delivers stronger retrieval with fewer synthetic samples. Ablations and analyses attribute gains to failure conditioning, minimal editing, local paired supervision, and robust—but editor-sensitive—counterfactual quality.

  • Experimental Setup: RePair is evaluated on Flickr30K and COCO30K using full-rank I2T and T2I retrieval with Recall@K and MRR.The main experiments use CLIP ViT-B/32 and the standard Karpathy split.
  • Main Results: 107K synthetic samples—26%–75% fewer than baselines—yield the best R@1 across both datasets and retrieval directions.On Flickr30K I2T, RePair reaches 90.13 versus SugarCrepe’s 88.95.
  • Main Results: Combining I2T and T2I synthesis improves R@1 in all four dataset–direction settings, while each single-direction variant also improves both directions.The two directions provide complementary supervision for the shared embedding space.
  • Ablation and Analysis: Failure conditioning is the dominant ablation factor: removing it drops I2T by −2.10 and T2I by −2.42, whereas using mined false positives without repair yields only marginal gains.These results distinguish targeted error-region supervision from selection-only hard negatives.
  • Ablation and Analysis: Removing quality control, minimal-change selection, or local paired supervision consistently hurts performance, and push-only training lacks the complementary pull signal from hard positives.Dispersing synthetic items into the global batch produces the largest single-component drop after failure conditioning.
  • Generalization and Sensitivity: RePair remains beneficial with larger training data, a stronger backbone, and compositional evaluation, while image-editor fidelity and scaffold drift constrain performance.More than 92% of 800 manually checked retained records form valid hard contrasts, with higher quality on I2T.

5 Conclusion

RePair treats top-ranked retrieval errors as scaffold–residual counterfactuals and repairs them through minimal editing into paired hard positives and negatives. On Flickr30K and COCO30K, this framework outperforms error-agnostic baselines at R@1 while using substantially fewer synthetic samples.

  • Conclusion: RePair uses the scaffold–residual view to convert each top-ranked false positive into a hard positive and grounded hard negative through minimal counterfactual editing.Validity, Minimality, and Locality govern seed selection, editing, and training.
  • Conclusion: RePair consistently outperforms error-agnostic baselines, particularly at R@1, using only 107k synthetic samples—26%–75% fewer than existing methods.The conclusion identifies failure-conditioned counterfactual repair as more data-efficient than brute-force scaling.

Limitations

RePair’s current scope is limited by external editor and quality-control fidelity, single-snapshot failure mining, dataset and transfer coverage, and evaluation only on dual-encoder CLIP models.

  • Scope Boundaries: The synthesis and quality-control pipeline relies on external models, so RePair’s performance ceiling is set by their current fidelity.The framework is modular and editor-agnostic, but component quality affects outcomes.
  • Scope Boundaries: The current study mines failures from one retrieval snapshot and does not evaluate iterative re-mining, webscale noisy corpora, or cross-dataset transfer.These are identified as natural extensions rather than evaluated settings.
  • Scope Boundaries: Applicability beyond dual-encoder CLIP retrieval remains unexplored for cross-attention or fusion-based architectures.The reported evaluation is restricted to dual-encoder CLIP models.

Ethics Statement

The paper uses public Flickr30K and MS-COCO-derived data, standard evaluation and training setups, and pretrained models rather than training large generative models from scratch. It acknowledges dataset bias, possible misuse of editing techniques, and use of an LLM in synthesis and manuscript preparation.

  • Potential risks and misuse: Caption and image editing techniques could be misused to generate misleading content, but the pipeline produces training data on public benchmarks rather than user-facing content.The authors frame this as limiting the scope of potential misuse.
  • Data and bias: The paper notes that source datasets may contain societal biases, while edits target fine-grained semantic distinctions rather than sensitive demographic attributes.The authors state that the method does not address or amplify these biases.
  • Computational considerations: The approach leverages existing pretrained models and moderate standard-hardware computation instead of training large generative models from scratch.Synthetic image generation uses Flux2-9B under shared settings across relevant baselines.
  • Use of AI assistants: An LLM supports caption editing and quality filtering, while additional AI assistants were used for manuscript language editing and proofreading.The authors attribute the research ideas, design, analyses, and claims to themselves.
  • Data and evaluation: RePair is evaluated on publicly available Flickr30K and COCO30K data, with COCO30K sampled from MS-COCO to match Flickr30K’s training scale.COCO30K uses 29,000 sampled training images and preserves five captions per image.
  • Data and evaluation: The experiments use standard benchmark splits, full-ranking cosine-similarity retrieval, and a pretrained CLIP ViT-B/32 backbone.Flickr30K uses a 1,000-image test split, while MS-COCO uses a 5,000-image test split.

F Full Experimental Results

RePair constructs dense local confusion neighborhoods around genuine retrieval failures, pairing hard negatives with minimally edited positives. Examples show that this structure targets localized activity, subject, and attribute-binding errors while concentrating supervision on difficult boundaries.

  • Full retrieval results: RePair’s advantage over baselines is concentrated in R@1 and MRR across both Flickr30K and COCO30K retrieval directions.The expanded results cover R@1, R@5, R@10, and MRR for Image-to-Text and Text-to-Image retrieval.
  • 3 × 3 local confusion grids: Each 3 × 3 grid centers on a ground-truth pair with a high-ranking false positive and adds synthesized or paired positive counterparts.The construction forms a dense, discriminative neighborhood around the failure anchor.
  • Flickr30K example: Flickr30K examples target errors in activity type and subject composition by contrasting cricket versus soccer and boy-and-girl versus two-kids descriptions.The edited counterpart preserves the cricket context while aligning the subject description to the hard-negative image.
  • COCO30K example: COCO30K examples target attribute binding by distinguishing red stools from red cabinets and correcting descriptions of the counter, chairs, and table.The synthesized positive precisely describes the hard-negative image’s wooden counter and black chairs.
  • Interpretation: The resulting hard negatives are genuine retrieval failures, while synthesized positives are minimal edits designed to remain maximally confusable yet semantically distinct.This structured supervision concentrates gradients on challenging decision boundaries.

H Statistical Analysis of RePair Synthetic Samples

RePair produces sparse hard positives and heavy-tailed hard-negative connectivity, with COCO30K showing broader negative tails than Flickr30K. The resulting pattern supports concentrating synthesis on semantically dense confusion hubs rather than uniformly augmenting samples.

  • Hard-positive distribution: Most image anchors receive 1–5 hard-positive captions, whereas most caption anchors receive exactly one hard-positive image.These patterns reflect retention counts dI2T=5 and dT2I=1, respectively.
  • Hard-negative distribution: Hard-negative connectivity is heavy-tailed: typical anchors have 10–50 negatives, while hubs reach approximately 250 on Flickr30K and 400 on COCO30K.Hubs arise in semantically dense regions such as generic kitchen or sports scenes.
  • Cross-dataset comparison: COCO30K has wider hard-negative distribution tails than Flickr30K, while hard-positive distributions remain consistent across datasets.The text attributes the wider COCO30K tails to broader scene diversity in MS-COCO.
  • Implications: The combination of sparse high-precision positives and dense long-tail negatives supports RePair’s precision-over-volume strategy.The framework concentrates generative capacity on anchors intersecting multiple confusion boundaries.

I Influence of Synthetic Sample Size

Across Flickr30K synthesis budgets from 20k to 100k, RePair remains the strongest method in both retrieval directions. It improves from low budgets, whereas non-failure-driven baselines often saturate or regress as synthetic volume increases.

  • Budget scaling: 20k–100k: RePair traces the upper envelope of Recall@1 performance for both Image-to-Text and Text-to-Image retrieval on Flickr30K.It already provides clear gains at 20–40k and continues improving as more samples are added.
  • Baseline comparison: Non-failure-driven baselines show early saturation or occasional regression at higher synthesis budgets.The passage attributes this pattern to redundant or misaligned training signals offsetting marginal gains.
  • Baseline comparison: RePair’s advantage widens with budget size, supporting error-driven targeting over simply increasing synthetic-data volume.This comparison is reported for the budget sweep on Flickr30K.

J Synthesis Pipeline Efficiency Analysis

RePair combines quality-controlled failure mining with LLM-guided editing and post-edit selection to produce targeted hard pairs. Its synthesis pipeline uses multiple stages, but reports moderate generation overhead and consistently stronger retrieval across sample budgets.

  • Pipeline stages: The pipeline quality-controls 1,000 mined failure cases per direction and dataset before generating edit instructions and modality-specific edits.Quality control checks ground-truth consistency and whether a false positive is already a benign alternative match.
  • Token usage: Quality Control consumes the most LLM tokens, at approximately 1,953K for Flickr30K and 1,930K for COCO30K combined.Instruction Generation is the second-largest consumer, while caption Editing is lighter and T2I image editing uses zero LLM tokens.
  • Token usage: I2T uses more total LLM tokens than T2I: 2,475K versus 1,470K on Flickr30K.I2T requires all three LLM stages, whereas T2I skips text-based editing and instead incurs image-editor GPU compute.
  • Generation overhead: RePair uses 355K LLM calls and 180K image-generation invocations, lower than quality-controlled generation baselines under the aligned comparison.Its per-sample breakdown is five LLM calls for I2T and two LLM calls plus three Flux image edits for T2I.
  • Quality control: The retained synthesis pipeline is supported by human checks: QC reaches 90.2% precision, while accepted edited pairs achieve 92.5% valid hard contrasts.QC recall is 83.9%, reflecting conservative filtering that sometimes rejects valid failures.
  • Retrieval efficiency: RePair achieves stronger Recall@1 than all baselines at every Flickr30K synthesis budget from 20k to 100k samples.Error-agnostic methods saturate or regress as sample count grows, whereas RePair remains consistently ahead.

K.3 Post-Edit Verification by Human Annotators

Human verification finds that most retained edited pairs are valid hard contrasts while identifying direction- and dataset-dependent weaknesses. The strongest shortfalls concern scaffold preservation and minimality, especially for image editing.

  • Verification outcome: 92.5% of 800 retained post-edit pairs form valid hard contrasts.The evaluation covers 200 records for each dataset-by-direction split and tests false-positive validity, query matching, scaffold preservation, minimality, and contrast quality.
  • Direction effects: The I2T–T2I quality gap is 6.0 percentage points, concentrated in scaffold preservation and minimality.These criteria are particularly sensitive to how much of an image a diffusion editor rewrites.
  • Dataset effects: COCO30K exceeds Flickr30K by 6.0 points for I2T and 5.0 points for T2I in valid hard-contrast rates.The reported pattern is consistent with Flickr30K’s greater scene and caption diversity.
  • Selection validation: The 93.3% minimality rate corroborates the embedding-proximity criterion used for post-edit selection.The 99.5% first-criterion rate is measured on retained edited records, whereas QC precision covers all QC-accepted triplets.

L.1 Error Type Distribution

RePair’s error analysis shows that retrieval failures differ substantially by direction and are dominated by object-level mismatches. Editing remains broadly controlled, but image-editing failures and editor quality constrain performance.

  • Error distribution: Hallucination occurs in 36.9% of I2T failures versus 0.5% of T2I failures, a roughly 74× directional difference.The analysis attributes this asymmetry to captions being able to fabricate details, unlike real photographs.
  • Error distribution: I2T failures average approximately 1.8 error types per sample, compared with approximately 1.1 for T2I.The multi-label densities are 179.6% for I2T and 114.3% for T2I.
  • Error distribution: Object Error is most prevalent in both directions: 44.0% for I2T and 48.3% for T2I.Count and Relation Errors are less frequent, ranging from 12.4%–18.8% and 12.1%–15.8%, respectively.
  • Editing quality: Every COCO30K T2I error category reaches mean cosine similarity of at least 0.86 and median similarity of at least 0.89 after editing.Attribute edits are most precise, with mean similarity 0.912 and standard deviation 0.063.
  • Editing failure modes: Scaffold drift occurs exclusively in T2I, accounting for all 21 scaffold-drift cases among the 60 shortfall records.Semantic mismatch is the largest failure mode with 34 of 60 cases, while visible artifacts occur once among 800 records.
  • Editor sensitivity: A weak image editor can reduce T2I below Vanilla, with InstructPix2Pix reaching 43.86 versus Vanilla’s 44.15.The reported mechanism is scaffold drift, which violates the minimality property required by image editing.
  • Iterative synthesis: Multi-round synthesis yields monotonic Recall@1 gains, but incremental improvements and newly synthesized samples shrink across iterations.This establishes diminishing marginal returns as dominant failures are corrected.
Loading 2608.29604v1…