PaperlayerView exact revision on arXiv

Source-linked AI summary

ConeSep: Cone-based Robust Noise-Unlearning Compositional Network for Composed Image Retrieval

Zixu Li, Yupeng Hu, Zhiwei Chen, Mingyu Zhang, Zhiheng Fu, Liqiang Nie

arXiv:2604.20358v1cs.CV

TL;DR

CIR depends on costly, error-prone triplet annotations, and hard NTC can evade existing methods because visual similarity masks incorrect modification text. ConeSep addresses this with geometric boundary estimation, structured negative learning, and optimal-transport-based targeted unlearning, significantly outperforming current state-of-the-art techniques across FashionIQ and CIRR under varied noise ratios.

  • Problem

    NTC, especially hard noise, challenges existing noise-correspondence methods because strong reference-target similarity can violate the small-loss hypothesis.

  • Method

    ConeSep uses Geometric Fidelity Quantization, Negative Boundary Learning, and Boundary-based Targeted Unlearning to detect noise, construct diagonal negative combinations, and avoid unlearning backlash.

  • Results

    ConeSep significantly outperforms current state-of-the-art techniques across FashionIQ and CIRR under various noise ratios.

  • Takeaways & Limitations

    ConeSep provides a closed-loop approach for precise noise perception, structured repulsion, and targeted unlearning in noisy CIR.

Abstract

from arXiv · show

The Composed Image Retrieval (CIR) task provides a flexible retrieval paradigm via a reference image and modification text, but it heavily relies on expensive and error-prone triplet annotations. This paper systematically investigates the Noisy Triplet Correspondence (NTC) problem introduced by annotations. We find that NTC noise, particularly ``hard noise'' (i.e., the reference and target images are highly similar but the modification text is incorrect), poses a unique challenge to existing Noise Correspondence Learning (NCL) methods because it breaks the traditional ``small loss hypothesis''. We identify and elucidate three key, yet overlooked, challenges in the NTC task, namely (C1) Modality Suppression, (C2) Negative Anchor Deficiency, and (C3) Unlearning Backlash. To address these challenges, we propose a Cone-based robuSt noisE-unlearning comPositional network (ConeSep). Specifically, we first propose Geometric Fidelity Quantization, theoretically establishing and practically estimating a noise boundary to precisely locate noisy correspondence. Next, we introduce Negative Boundary Learning, which learns a ``diagonal negative combination'' for each query as its explicit semantic opposite-anchor in the embedding space. Finally, we design Boundary-based Targeted Unlearning, which models the noisy correction process as an optimal transport problem, elegantly avoiding Unlearning Backlash. Extensive experiments on benchmark datasets (FashionIQ and CIRR) demonstrate that ConeSep significantly outperforms current state-of-the-art methods, which fully demonstrates the effectiveness and robustness of our method.

1. Introduction

CIR depends on costly, error-prone triplet annotations, creating Noisy Triplet Correspondence (NTC) with especially difficult hard noise. ConeSep addresses three NTC challenges through geometric noise detection, structured negative anchors, and targeted unlearning.

  • Motivation: CIR retrieves a target image from a reference image and modification text, but its performance depends heavily on costly, error-prone triplet annotations.Manual subjectivity and LVLM hallucinations can create semantic inconsistency between modification text and image pairs.
  • NTC Challenges: NTC includes partial matching and hard noise, where strong reference-target similarity can make incorrect triplets appear clean under the small-loss hypothesis.This composite noise structure distinguishes NTC from simpler image-text or video-text mismatches.
  • NTC Challenges: Modality suppression lets dense visual similarity dominate sparse modification-text mismatch, producing coarse metrics that misclassify hard noise.Such samples can escape filtering and compromise learning.
  • NTC Challenges: Negative anchor deficiency makes pseudo-label correction questionable for identified hard noise because pulling noisy modifications toward a presumed target can cause semantic confusion.The paper motivates targeted unlearning as an alternative.
  • NTC Challenges: Unlearning backlash occurs when pushing noisy samples away creates a ripple effect that damages nearby clean representations in a crowded metric space.Robust NTC learning must therefore separate precise unlearning from collateral damage.
  • ConeSep: ConeSep combines Geometric Fidelity Quantization, Negative Boundary Learning, and Boundary-based Targeted Unlearning to address the three challenges, and it significantly outperforms current SOTA methods under varied noise ratios.The modules estimate noise boundaries, learn diagonal negative combinations, and formulate correction as optimal transport.

2. Related Work

Prior CIR research generally assumes accurate triplet correspondence, whereas real datasets contain noisy triplets from collection and annotation errors. Existing NTC methods address sample selection or realignment but do not handle noisy samples already learned by the model.

  • CIR with Noisy Correspondence: Most CIR studies assume accurate triplet correspondence, although large-scale datasets contain Noisy Triplet Correspondence from collection and annotation errors.NTC is described as more complex than traditional noisy-label problems and harmful to robustness.
  • CIR with Noisy Correspondence: Recent NTC methods use sample selection or realignment, but they focus on robustness during new knowledge acquisition rather than removing influence from already learned noisy samples.This leaves a gap between identifying noisy data and unlearning its model influence.

3. Methodology

ConeSep addresses noisy triplet correspondence through three modules that identify fidelity boundaries, construct explicit negative anchors, and target noisy samples without harming clean features.

  • ConeSep comprises Geometric Fidelity Quantization, Negative Boundary Learning, and Boundary-based Targeted Unlearning.These modules respectively address noise determination, negative-anchor construction, and targeted correction.
  • Geometric Fidelity Quantization (GFQ): GFQ estimates a similarity boundary from Gaussian-sampled features and uses fidelity scores to separate clean and noisy triplets.The fidelity function compares composed and target features, while threshold ω partitions Tclean and Tnoisy.
  • Negative Boundary Learning (NBL): NBL combines positive alignment with a negative path that constructs a Diagonal Negative Composition for each composed query.The negative feature is generated with learnable negative prompts and optimized against target and composed features.
  • Negative Boundary Learning (NBL): Target-oriented learning moves each negative composition away from its matching target and toward non-matching targets.A binary target matrix assigns opposite objectives to matching and non-matching pairs.
  • Negative Boundary Learning (NBL): Query-oriented learning constrains negative-composition similarity with the original composed feature to an interval around zero.The interval [α1, α2] encourages an approximately orthogonal relationship between the two features.
  • Boundary-based Targeted Unlearning (BTU): BTU uses the learned negative composition as an anchor and applies optimal transport to unlearn noise while preventing backlash on clean features.The module is designed to avoid the ripple effect caused by pushing noisy samples away.

4. Experiments

Experiments evaluate ConeSep on FashionIQ and CIRR under noisy triplet correspondence, comparing robustness, component contributions, sensitivity, and qualitative retrieval behavior. ConeSep consistently outperforms robust baselines, while its modules and hyperparameters show identifiable performance effects.

  • Experimental setup: ConeSep is evaluated on FashionIQ and CIRR using ablations and parameter-sensitivity analyses under a noise rate of σ = 0.2.The evaluation reports Recall-based retrieval metrics for both benchmarks.
  • Performance comparison: ConeSep achieves gains over HABIT of 0.92% at 20% noise, 1.54% at 50% noise, and 1.41% on FashionIQ’s AVG metric.The performance advantage widens as the noise rate increases, and a consistent pattern is observed on CIRR.
  • Ablation study: Removing GFQ components produces a consistent downward trend across all performance metrics, indicating that each component supports penetration of modality suppression.The ablations remove random sampling strategies, boundary participation, or the fidelity measure.
  • Ablation study: All NBL ablations underperform the full model, showing the value of Target-oriented and Query-oriented Learning for constructing structured negative boundaries.The affected variants are D#6–D#10.
  • Hyper-parameter sensitivity: Performance peaks at ω = 0.5 and κ = 0.5, while excessive values respectively misclassify clean samples or cause over-correction and Unlearning Backlash.Low values also cause incorrect clean/noise partitioning or insufficient correction for hard noise.
  • Case study: ConeSep retrieves fine-grained textual attributes and complex spatial relationships in case studies where TME relies excessively on reference-image features.The comparison covers FashionIQ and CIRR examples.

5. Conclusion

The paper frames NTC as a central challenge in CIR and proposes ConeSep as a three-stage solution combining geometric boundaries, negative composition, and optimal-transport unlearning. Experiments on FashionIQ and CIRR show strong performance across noise ratios.

  • Conclusion: NTC introduces three challenges—Modality Suppression, Negative Anchor Deficiency, and Unlearning Backlash—that limit coarse-metric noise handling.These challenges arise from the structure of noisy triplet correspondence in CIR.
  • Conclusion: ConeSep forms a closed-loop system using geometric boundaries, diagonal negative composition, and optimal transport for targeted noise unlearning.The design aims to achieve precise noise perception while avoiding unlearning backlash.
  • Conclusion: Experiments on two mainstream CIR benchmarks show that ConeSep significantly outperforms current state-of-the-art techniques across various noise ratios.The reported conclusion covers FashionIQ and CIRR.

A. Sinkhorn-Knopp Algorithm for Targeted Unlearning

The appendix derives Sinkhorn-Knopp updates for the entropy-regularized optimal-transport formulation used in targeted unlearning. The solution factorizes through a Gibbs kernel and is iteratively scaled to satisfy marginal constraints.

  • Problem definition: The targeted-unlearning problem minimizes masked transport cost while maximizing entropy over plans satisfying source and target marginal constraints.The feasible transport plans lie in Π(u, v), with u and v typically uniform marginal distributions.
  • Lagrangian and Gibbs kernel: The Lagrangian introduces multipliers α and β for the marginal constraints, and differentiation with respect to Pij yields the optimality condition.Setting the derivative to zero produces the exponential form of the optimal plan.
  • Lagrangian and Gibbs kernel: The optimal plan factorizes as P* = diag(a) K diag(b), where Kij = exp(−[Cmasked]ij/ϵ).The scaling vectors encode the Lagrange multipliers through ai = exp(−αi/ϵ) and bj = exp(−βj/ϵ).
  • Iterative scaling updates: Sinkhorn-Knopp alternates updates of a and b so that the factorized plan satisfies the prescribed marginal constraints.The updates use element-wise division by Kb and K^T a.
  • Iterative scaling updates: The algorithm typically converges within 10–20 iterations, and its final smooth transport plan supplies the soft target for targeted unlearning.The resulting P* guides the unlearning objective through Eq. 11.

B. Datasets

ConeSep is evaluated on two CIR benchmarks spanning fashion-focused and open-domain retrieval. FashionIQ emphasizes fine-grained attribute modifications in annotated image-text triplets.

  • Datasets: The evaluation uses FashionIQ, a fashion-domain benchmark, and CIRR, an open-domain benchmark.Both datasets are used to assess ConeSep’s performance.
  • FashionIQ: FashionIQ contains 77,684 high-resolution fashion images and 30,134 annotated triplets divided into Dresses, Shirts, and Tops&Tees.Its triplets are organized into three category-specific subsets.
  • FashionIQ: FashionIQ modification texts describe fine-grained visual attribute changes such as V-neck style or stripe density.These instructions require capturing subtle semantic differences.

C. Additional Quantitative Analysis

ConeSep combines robust retrieval performance with efficient inference, while its negative-anchor analysis shows strong geometric separation from composed features.

  • Computational complexity: ConeSep uses 915.69M parameters and 411.51G FLOPs, comparable to ordinary and robust baselines.
  • Efficiency: ConeSep achieves 0.0091 seconds per sample, faster than SPRC at 0.011 seconds and approximately 13.6× faster than TME at 0.124 seconds.Its modules constrain training but do not add inference calculations.
  • Retrieval performance: ConeSep achieves the best retrieval accuracy on FashionIQ and CIRR, with FashionIQ-Avg=64.97% and CIRR-Avg=80.43%.
  • Negative-anchor analysis: The NBL analysis evaluates the cosine similarity distribution between composed features Fc and diagonal negative compositions Fneg.
  • Negative-anchor analysis: The Fc–Fneg cosine similarity distribution is highly concentrated around 0, indicating an orthogonal relationship.

C.3. Comparison of Sample Set Partition Purity

ConeSep improves the purity of the high-fidelity sample set, but residual hard noise remains because visual similarity can prevent perfect filtering.

  • Sample-set partitioning: GFQ separates high-fidelity clean samples from low-fidelity noisy samples by estimating geometric noise boundaries.
  • Purity comparison: ConeSep’s filtered set has higher purity than TME’s, especially under substantial hard noise.At σ = 0.8, TME’s wrongly retained hard-noise region approaches its correctly retained clean region on FashionIQ.
  • Residual hard noise: The remaining pink samples show that some hard noise is difficult to eliminate completely because reference and target images can be extremely visually similar.
  • Soft correction: BTU therefore treats Tclean as imperfect rather than absolute ground truth and applies optimal-transport-based soft correction.This design tolerates residual impurities while using clean-sample information.

C.4. Additional Hyperparameter Analysis

Hyperparameter analyses identify moderate settings as preferable: ζ and ν peak at 0.5, while γ improves performance through 0.7 before declining.

  • Intra-modal weighting: Model performance peaks at ζ=0.5 for the intra-modal loss weight.ζ controls orthogonality between Fc and Fneg; overly low values poorly shape the negative composition.
  • Inter-modal weighting: Model performance is optimal at ν=0.5 for the inter-modal loss weight.ν regulates pushing Fneg away from the target image through Target-oriented Learning.
  • Inter-modal weighting: Excessively high ν overemphasizes negative-boundary construction and distracts optimization from pulling positive pairs closer.
  • OT-label balancing: Performance improves as γ increases from 0.1 to 0.7, then declines.γ balances the optimal-transport plan against the original hard labels when constructing soft labels.
  • OT-label balancing: Low γ underuses OT structure, whereas overly high γ weakens targeted-unlearning supervision for noisy samples.

D.1. GFQ Sampling Ablation Studies

GFQ’s Gaussian boundary-sampling strategy performs best across both benchmarks, while multiple samples improve estimation until performance saturates beyond K=4.

  • Sampling-strategy comparison: ConeSep with Gaussian sampling achieves 65.31% Avg on FashionIQ and 80.43% Avg on CIRR, outperforming the alternative sampling strategies.
  • Sampling-strategy comparison: Empirical sampling degrades performance because high-variance mini-batch statistics destabilize boundary estimation.
  • Sampling-strategy comparison: Laplace sampling generally outperforms Uniform sampling but remains below Gaussian sampling.Laplace better approximates the central tendency of high-dimensional features than Uniform sampling.
  • Number of random samples: Retrieval accuracy improves as K increases from 1 to 4, indicating that K=1 is insufficient for robust geometric-boundary estimation.
  • Number of random samples: Performance saturates and slightly declines beyond K=4, suggesting K=4 sufficiently captures the boundary distribution.

E. Algorithm of Training Procedure

ConeSep’s training procedure integrates fidelity-based noise separation, diagonal negative composition, and targeted unlearning into a unified optimization pipeline. It uses geometric boundaries and masked optimal transport to update representations while preserving clean correspondence.

  • Training pipeline: ConeSep integrates Geometric Fidelity Quantization, Negative Boundary Learning, and Boundary-based Targeted Unlearning into a unified training pipeline.The procedure forms a closed-loop optimization system combining all three modules.
  • Fidelity quantization: The algorithm computes composed and target features, estimates fidelity, and partitions each batch into clean samples with F ≥ω and noisy samples with F < ω.This separation follows similarity and fidelity quantization during feature extraction.
  • Targeted unlearning: Boundary-based Targeted Unlearning uses feature-derived cost and mask matrices to solve an entropy-regularized optimal transport problem and construct smooth soft labels.The soft labels support targeted unlearning while blocking noisy positive paths and preserving clean negative paths.
  • Negative Boundary Learning: Negative Boundary Learning computes the diagonal negative composition and jointly optimizes robust contrastive, query-oriented, and target-oriented losses.The NBL objective is LNBL = Lrobust + ζLintra + νLinter.
  • Semantic negative anchors: The learned negative direction represents semantic states that retain reference-image semantics while violating the modification instruction, such as black rather than red dresses or fewer rather than more monkeys.These oppositional retrievals provide the negative-anchor direction used by targeted unlearning.

F.2. NTC Identification Analysis

ConeSep’s qualitative analyses show that it separates clean and noisy triplets, handles fine-grained and large semantic changes, and can produce human-aligned alternatives even when the annotated target is not top-ranked. The analyses also identify false negatives in the current evaluation system as a boundary of these cases.

  • NTC identification: ConeSep assigns high fidelity to clean triplets and low fidelity to noisy correspondences, including hard noise with strong visual similarity but textual misalignment.Examples include fidelity scores of 0.249 for a clean change and 0.011 for a hard-noise case.
  • CIRR case studies: On CIRR, ConeSep bridges large semantic and action changes such as muffin-to-vegetable, sofa-to-bed, and holding-fish-to-walking-out-of-water transitions.The comparison contrasts ConeSep’s semantic shifts with TME’s tendency to remain visually close to the reference.
  • Diagonal negative composition: ConeSep’s composed feature follows modification semantics, while its diagonal negative composition retrieves visually related images that violate the instruction.FashionIQ examples retain black dress semantics instead of changing to red, while CIRR examples reverse the requested monkey-count change.
  • Failure-case analysis: Some apparent retrieval failures reflect false negatives: ConeSep may miss the annotated target at Top-1 while retrieving alternatives that better match the textual description.The paper connects these cases to limitations in current CIR evaluation and annotation.
  • FashionIQ case studies: On FashionIQ, ConeSep follows fine-grained changes in patterns, sleeve lengths, color, and garment attributes more accurately than TME.The qualitative comparisons emphasize reduced visual inertia from the reference image.
Loading 2604.20358v1…