PaperlayerView exact revision on arXiv

Source-linked AI summary

INTENT: Invariance and Discrimination-aware Noise Mitigation for Robust Composed Image Retrieval

Zhiwei Chen, Yupeng Hu, Zhiheng Fu, Zixu Li, Jiale Huang, Qinlei Huang, Yinwei Wei

arXiv:2604.18051v1cs.CV

TL;DR

CIR can contain noisy triplet correspondences from both cross-modal mismatches and modality-inherent interference, while existing methods often overlook the latter and use rigid decision boundaries. INTENT addresses these issues with invariant visual composition and loyalty-aware discriminative learning, and experiments on two benchmarks report superior accuracy and robustness.

  • Problem

    CIR datasets contain noisy triplet correspondences from cross-modal mismatches and modality-inherent interference, but modality-inherent noise is often overlooked and existing NTC methods commonly use hard decision boundaries.

  • Method

    INTENT combines FFT-based causal intervention for visual-invariant composition with positive- and negative-sample optimization that constructs a loyalty-aware scalable decision boundary.

  • Results

    INTENT significantly outperforms most methods in accuracy and robustness across multiple benchmarks, while robust methods generally outperform ordinary approaches as noise increases.

  • Takeaways & Limitations

    The paper supports treating modality-inherent noise and correspondence discrimination as separate aspects of robust CIR learning.

Abstract

from arXiv · show

Composed Image Retrieval (CIR) is a challenging image retrieval paradigm that enables to retrieve target images based on multimodal queries consisting of reference images and modification texts. Although substantial progress has been made in recent years, existing methods assume that all samples are correctly matched. However, in real-world scenarios, due to high triplet annotation costs, CIR datasets inevitably contain annotation errors, resulting in incorrectly matched triplets. To address this issue, the problem of Noisy Triplet Correspondence (NTC) has attracted growing attention. We argue that noise in CIR can be categorized into two types: cross-modal correspondence noise and modality-inherent noise. The former arises from mismatches across modalities, whereas the latter originates from intra-modal background interference or visual factors irrelevant to the coarse-grained modification annotations. However, modality-inherent noise is often overlooked, and research on cross-modal correspondence noise remains nascent. To tackle above issues, we propose the Invariance and discrimiNaTion-awarE Noise neTwork (INTENT), comprising two components: Visual Invariant Composition and Bi-Objective Discriminative Learning, specifically designed to handle the two-aspect noise. The former applies causal intervention on the visual side via Fast Fourier Transform (FFT) to generate intervened composed features, enforcing visual invariance and enabling the model to ignore modality-inherent noise during composition. The latter adopts collaborative optimization with both positive and negative samples, and constructs a scalable decision boundary that dynamically adjusts decisions based on the loyalty degree, enabling robust correspondence discrimination. Extensive experiments on two widely used benchmark datasets demonstrate the superiority and robustness of INTENT.

1 Introduction

The introduction identifies two overlooked sources of noisy triplet correspondence in CIR and presents INTENT to address them through invariant composition and adaptive discrimination. It frames these components as complementary responses to modality-inherent noise and hard decision boundaries.

  • CIR and noisy correspondence: CIR retrieves target images from multimodal queries combining a reference image and modification text.The paradigm supports flexible retrieval across information processing, intelligent systems, and multimodal learning.
  • Sources of noise: Noisy Triplet Correspondence arises from modality-inherent noise and cross-modal correspondence noise in incorrectly matched triplets.Modality-inherent noise includes irrelevant content within the reference image, while cross-modal noise concerns mismatches across modalities.
  • Research gaps: Existing approaches largely overlook modality-inherent noise, whose interference can distort composition and undermine noisy-correspondence identification.Methods addressing cross-modal noise often rely on direct multimodal fusion, while methods suppressing intra-modal noise generally assume correct cross-modal matching.
  • Research gaps: Current NTC approaches commonly use hard decision boundaries, which struggle with semantic gaps caused by brief modification texts.The introduction motivates more flexible discrimination for correspondence decisions.
  • Proposed framework: INTENT combines Visual Invariant Composition and Bi-Objective Discriminative Learning to address the two noise aspects.VIC uses causal intervention via FFT to promote visual invariance, while BiODL uses positive and negative samples to construct a scalable boundary based on matching loyalty.
  • Evaluation: The paper reports that extensive benchmark experiments show INTENT significantly outperforms most methods in accuracy and robustness.The contribution statement attributes this evaluation to multiple benchmark datasets.

2 Related Work

The related-work context positions CIR as a multimodal retrieval task built from reference images and modification text, while connecting the paper to causal intervention for robust feature composition. It also notes that this intervention-oriented direction remains underexplored in CIR.

  • Composed Image Retrieval: CIR retrieves target images using reference images combined with modification texts.Prior CIR work includes separate visual and textual feature extraction followed by fusion, as well as joint learning with pretrained vision-language models.
  • Composed Image Retrieval: Recent CIR advances use pretrained vision-language models such as CLIP for joint learning and streamlined alignment.The related-work passage contrasts these models with earlier ResNet- and LSTM-based architectures.
  • Causal intervention: CIR remains underexplored as a setting for intervention-inspired strategies that encourage robust, generalizable, and causally meaningful image-text composition.The paper motivates tailoring such strategies to CIR’s compounding confounders.

3 The Proposed INTENT

INTENT addresses modality-inherent noise and cross-modal correspondence noise in CIR through visual invariance learning and adaptive correspondence discrimination. Its framework combines frequency-domain counterfactual composition with bi-objective learning based on query-target similarities and loyalty estimates.

  • Visual Invariant Composition: INTENT uses Visual Invariant Composition to reduce modality-inherent noise by comparing compositions from factual and counterfactual reference images.The counterfactual image is generated by altering amplitude spectra with FFT while preserving the original phase spectrum and key semantics.
  • Visual Invariant Composition: FFT-based intervention mixes cropped amplitude-spectrum regions from a reference image and an irrelevant image, then reconstructs a counterfactual image with altered noise patterns.Amplitude captures low-level style and texture, while phase retains high-level semantic structures.
  • Visual Invariant Composition: VIC enforces consistency between factual and intervened composed features so the model identifies visual invariants and becomes less sensitive to modality-inherent noise.The method uses a CKA-based causal consistency constraint over composed-feature Gram matrices.
  • Bi-Objective Discriminative Learning: Bi-Objective Discriminative Learning jointly considers negative-dominated and correspondence-dominated perspectives to improve discrimination under noisy triplet correspondence.The framework constructs a loyalty degree matrix from query-target similarities and uses it for a scalable soft decision boundary.
  • Bi-Objective Discriminative Learning: The method applies negative and positive weight rewards according to batch-level similarity patterns before estimating loyalty and optimizing the soft discriminative loss.Negative rewards address cases where correct correspondences may occur among negatives, while positive rewards apply when positives appear relatively clean.

4 Experiments

Experiments evaluate INTENT on FashionIQ and CIRR under noisy triplet correspondence, using benchmark comparisons, ablations, and qualitative cases. INTENT shows stronger robustness than competing methods and benefits from both its modules.

  • Experimental Setup: Experiments use FashionIQ and CIRR, with ablation and parameter-sensitivity settings applying a 20% noise ratio.The evaluation reports Recall@K metrics, including overall and subset recall for CIRR and category-level recall for FashionIQ.
  • Performance Comparison: INTENT surpasses TME on FashionIQ by 0.39%, 1.06%, and 1.44% in Avg at noise ratios of 20%, 50%, and 80%, respectively.Similar robustness trends are reported on CIRR.
  • Ablation Study: Removing VIC or its intervention causes noticeable performance drops, demonstrating the module’s contribution to visual invariance and reduced modality-inherent noise.The ablation study also reports degradation when components of the scalable decision boundary are removed, with removing both rewards producing the largest drop.
  • Ablation Study: Removing both reward components produces the largest ablation drop, supporting scalable decision boundaries over raw similarity scores.The comparison also finds that removing the negative-weight reward slightly outperforms removing the positive-weight reward under 20% noise.
  • Case Study: In qualitative cases, INTENT retrieves targets at top-1 while TME fails to retrieve them within the top five on FashionIQ and CIRR examples.The cases are presented as comparisons of top-5 retrieval results, with target images marked in colored boxes.

5 Conclusion

The conclusion identifies modality-inherent noise and hard decision boundaries as neglected NTC challenges and presents INTENT as a two-component response. Experiments on two benchmarks report superiority and robustness.

  • Conclusion: INTENT addresses modality-inherent noise with Visual Invariant Composition and ambiguous correspondence decisions with Bi-Objective Discriminative Learning.VIC uses FFT-based causal intervention, while BiODL uses positive and negative samples to construct a loyalty-degree-dependent scalable boundary.
  • Conclusion: Experiments on two benchmarks reveal INTENT’s superiority and robustness for noisy triplet correspondence.

A Module Validity Analysis

The VIC analysis compares intervention operations by their visual preservation and retrieval performance under noisy correspondence. FFT-based MixUp preserves semantic structure while achieving the strongest reported average.

  • Visual Analysis: FFT-based MixUp and Style Transfer preserve global semantic structure and style while introducing subtle appearance changes.Random Mask and Patch Shuffle create fragmented samples, whereas Gaussian Blur removes high-frequency details and can lose semantic cues.
  • Quantitative Comparison: 71.75 average (R@5, Rsubset@1) is reached by the weaker intervention group, whose excessive or unsupervised corruption harms model training.The group includes Random Mask, Patch Shuffle, and Gaussian Blur.
  • Interpretation: FFT-based MixUp is attributed an advantage because it perturbs high-frequency noise globally and semantically without destroying essential visual structures.The analysis contrasts this with operations that disrupt local continuity or blur key semantics.

A.2 Visual Analysis of the Scalable Decision

The BiODL analysis compares candidate rankings under similarity-based and loyalty-degree-based decisions. Full INTENT concentrates genuine correspondences at high loyalty ranks and pushes noisy candidates toward low confidence.

  • Ranking Visualization: Figure 6 compares similarity and loyalty-degree rankings across queries, with candidate targets ordered horizontally by loyalty degree.Smaller horizontal indices denote higher loyalty degrees, and green dots represent real correspondences.
  • Ranking Visualization: Full INTENT clusters real correspondences near the highest loyalty ranks, unlike the w/o Weight Reward variant.The figure uses gray dots for noisy or irrelevant candidates.
  • Decision Boundary: The scalable boundary assigns high confidence to genuine matches and pushes noisy or irrelevant candidates toward the lower-confidence region.The authors describe this boundary as more robust and aligned with the true correspondence distribution.
  • Decision Boundary: Weight Reward dynamically adjusts the decision boundary to the loyalty-degree distribution, supporting more stable correspondence identification across semantic complexity or noise ratios.

B Datasets

INTENT is evaluated on FashionIQ and CIRR, two benchmark datasets covering fashion retrieval and complex natural-scene retrieval. CIRR further tests robustness through fine-grained contrastive subsets and incomplete labels.

  • Two widely used benchmark datasets, FashionIQ and CIRR, are used to evaluate INTENT for composed image retrieval.
  • FashionIQ Dataset: FashionIQ contains 77,684 online images and 30,134 annotated triplets across dresses, shirts, and T-shirts.
  • CIRR Dataset: CIRR comprises 36,554 annotated triplets and 21,552 images derived from real-world scenes in NLVR2.
  • CIRR Dataset: Compared with FashionIQ, CIRR emphasizes multi-object scene interactions and uses challenging subsets and incomplete labels to reduce domain overfitting and frequent hard negatives.

C Efficiency Evaluation

INTENT combines strong retrieval performance with lower computational cost than TME. Its VIC ablation is faster to train but loses retrieval performance, indicating a performance–efficiency trade-off.

  • Efficiency evaluation reports FLOPs, inference time, training time, GPU memory, parameter count, and average recall across CIRR and FashionIQ.
  • INTENT reduces inference time to 0.010 seconds per sample, roughly one-twelfth of TME, despite slightly higher FLOPs.
  • The reported efficiency comparison covers SPRC, TME, INTENT, and INTENT ablation variants.
  • INTENT requires 3.195 seconds per training iteration versus TME’s 7.858 seconds, yielding a reported 59% training-efficiency increase.
  • Ablation: Removing VIC preserves similar inference speed and accelerates training to 2.564 seconds per iteration, but lowers average recall below both INTENT and TME.

D Algorithm of INTENT’s Training Procedure

INTENT’s training procedure combines frequency-domain intervention, multimodal composition, and robust optimization. Its BiODL module produces clearer diagonal correspondence structure than TME in similarity matrices.

  • Training begins by perturbing reference-image amplitudes in the FFT frequency domain while preserving semantic structure, producing a counterfactual view.
  • Original and counterfactual images are composed with modification text, integrating visual intervention with multimodal feature learning.
  • INTENT’s similarity matrix has a more pronounced continuous diagonal and clearer positive–negative contrast than TME on CIRR validation data.
  • Figure 7 visualizes query–target pairwise similarity matrices, where brighter diagonal elements indicate stronger matched correspondence.

E.2 More Case Study

Qualitative studies compare INTENT with TME and its w/o VIC variant across FashionIQ and CIRR. INTENT often retrieves correct or semantically aligned results under complex modifications, while failure cases expose possible false negatives in the datasets.

  • The case studies report top-5 retrievals for INTENT, w/o VIC, and TME on FashionIQ and CIRR, with colored borders marking ground-truth targets.
  • FashionIQ: INTENT ranks the correct FashionIQ target within the top five in all three reported examples.
  • Training Procedure: The training procedure takes reference, target, and modification inputs, with soft-discriminative and causal-consistency loss weights as parameters.
  • FashionIQ: For a multicolored blue-and-white garment query, INTENT retrieves the target at rank one, whereas TME misses key characteristics and w/o VIC generally ranks lower.
  • CIRR: On complex CIRR scenarios involving animals, indoor scenes, and crowds, INTENT consistently retrieves the correct target at top-1 and matches modification requirements closely.
  • The qualitative cases report accuracy improvements over TME and w/o VIC and support the proposed causal-consistency and loyalty-degree-based discriminative strategies.
  • Failure Cases: In observed failure cases, INTENT misses top-1 but includes the ground-truth target among top-ranked candidates whose other images match query semantics.
  • Failure Cases: The authors attribute these apparent failures to potential false-negative samples in the datasets.
Loading 2604.18051v1…