HABIT: Chrono-Synergia Robust Progressive Learning Framework for Composed Image Retrieval

Zixu Li; Yupeng Hu; Zhiwei Chen; Shiqi Zhang; Qinlei Huang; Zhiheng Fu; Yinwei Wei

Source-linked AI summary

HABIT: Chrono-Synergia Robust Progressive Learning Framework for Composed Image Retrieval

Zixu Li, Yupeng Hu, Zhiwei Chen, Shiqi Zhang, Qinlei Huang, Zhiheng Fu, Yinwei Wei

arXiv:2604.18037v1Apr 20, 2026cs.CV

TL;DR

CIR suffers from Noise Triplet Correspondence caused by costly, subjective triplet annotation and complex semantic discrepancies. HABIT combines mutual-knowledge estimation with dual-consistency progressive learning, and experiments report superior robustness and retrieval performance under varying noise levels.

Problem
Noise Triplet Correspondence arises in CIR because costly, subjective triplet annotation produces annotation errors and inaccurate semantic alignment, while CIR’s semantic modification setting complicates robust matching.
Method
HABIT estimates sample cleanliness using the Transition Rate of mutual knowledge and progressively learns through collaboration between historical and current models.
Results
HABIT outperforms most methods across varying noise levels on two standard CIR benchmarks.
Takeaways & Limitations
The framework provides robust CIR learning under noisy triplet conditions by combining noise-aware label assignment with progressive adaptation.
Takeaways & Limitations
Unmentioned visual discrepancies vary across samples, limiting similarity-based noise assessment and potentially causing noisy correspondences to be misclassified.

Abstract

from arXiv · show

Composed Image Retrieval (CIR) is a flexible image retrieval paradigm that enables users to accurately locate the target image through a multimodal query composed of a reference image and modification text. Although this task has demonstrated promising applications in personalized search and recommendation systems, it encounters a severe challenge in practical scenarios known as the Noise Triplet Correspondence (NTC) problem. This issue primarily arises from the high cost and subjectivity involved in annotating triplet data. To address this problem, we identify two central challenges: the precise estimation of composed semantic discrepancy and the insufficient progressive adaptation to modification discrepancy. To tackle these challenges, we propose a cHrono-synergiA roBust progressIve learning framework for composed image reTrieval (HABIT), which consists of two core modules. First, the Mutual Knowledge Estimation Module quantifies sample cleanliness by calculating the Transition Rate of mutual information between the composed feature and the target image, thereby effectively identifying clean samples that align with the intended modification semantics. Second, the Dual-consistency Progressive Learning Module introduces a collaborative mechanism between the historical and current models, simulating human habit formation to retain good habits and calibrate bad habits, ultimately enabling robust learning under the presence of NTC. Extensive experiments conducted on two standard CIR datasets demonstrate that HABIT significantly outperforms most methods under various noise ratios, exhibiting superior robustness and retrieval performance. Codes are available at https://github.com/Lee-zixu/HABIT

1 Introduction

CIR retrieves a target image from a reference image and modification text, but noisy triplet correspondence arises from costly, subjective annotation and unmentioned visual discrepancies. HABIT addresses these challenges with mutual-knowledge estimation and dual-consistency progressive learning, and experiments show improved robustness under varying noise levels.

CIR uses a reference image and modification text to retrieve a semantically consistent target image.
Annotation errors and inaccurate semantic alignment arise because triplet labeling is costly and subjective, creating the Noise Triplet Correspondence problem.
Unmentioned visual discrepancies create sample-dependent semantic gaps that similarity-based noise assessment may misclassify.
Existing methods face two challenges: precisely estimating composed semantic discrepancy and progressively adapting to modification discrepancy.
HABIT estimates sample cleanliness through mutual-knowledge transition rates and uses historical-current model collaboration to retain good habits and calibrate bad ones.
Experiments on two standard CIR benchmarks show that HABIT outperforms most methods across varying noise levels.

2 Related Work

CIR methods use traditional or vision-language-pretrained architectures to compose reference-image and text features. Robust learning research increasingly addresses noisy cross-modal correspondence, but existing approaches often overlook CIR’s more complex triplet noise.

CIR retrieves a target image from a multimodal query consisting of a reference image and modification text.
Traditional CIR approaches extract image and text features with architectures such as ResNet and LSTM before multimodal composition.
Recent robust-learning work studies noisy correspondence across modalities, which can induce overfitting and performance degradation beyond simple label errors.
Existing visual-language and cross-modal methods mainly model two-modality noise, overlooking the challenges of real-world Composed Image Retrieval.

3 HABIT

HABIT addresses noisy triplet correspondence in composed image retrieval through Mutual Knowledge Estimation and Dual-consistency Progressive Learning. The framework estimates sample cleanliness from mutual knowledge transition rates and uses historical-current model collaboration to filter noise and preserve semantic consistency during training.

HABIT comprises Mutual Knowledge Estimation and Dual-consistency Progressive Learning modules for robust CIR under noisy triplet correspondence.The framework is designed to quantify sample cleanliness and progressively adapt to modification discrepancies.
Problem Formulation: CIR maps a reference image and modification text toward a corresponding target image in a shared metric space, despite possible noisy triplets.Noisy triplets may be partial matches or full mismatches between the modification text and image pair.
Mutual Knowledge Estimation: MKE extracts composed and target features, then uses mutual knowledge and transition rates to estimate semantic matching and sample cleanliness.The transition-rate comparison uses a lowest-loss standard sample and considers discrepancies from both composed and target features.
Mutual Knowledge Estimation: A small composed-target transition rate, together with standard-like feature transition rates, indicates a more reliable clean correspondence.The criterion compares each sample with the standard sample to estimate noisy correspondence.
Dual-consistency Progressive Learning: DPL combines historical and current cleanliness estimates to identify samples that remain outliers across iterations and continuously filter stable noisy triplets.Current and historical estimation sequences are clustered, merged into a noisy mask, and used for discrimination.
Dual-consistency Progressive Learning: Time-Flux Knowledge Updating preserves good habits through consecutive-iteration similarity consistency while calibrating residual bad habits with soft-margin and robust contrastive losses.Knowledge Consistency minimizes KL divergence between current and previous similarity distributions, while Lsoft and Lrank improve robustness to noisy negatives.

4 Experiments

Experiments evaluate HABIT on FashionIQ and CIRR under simulated NTC noise, comparing it with ordinary and robust baselines and analyzing its components and retrieval cases. HABIT achieves stronger robustness as noise increases and benefits from its mutual-knowledge and progressive-learning design.

Experimental settings: HABIT is evaluated on FashionIQ and CIRR using Recall@K metrics under NTC noise, with ablations conducted at σ = 0.2.The study compares performance across σ = 0.0, 0.2, 0.5, and 0.8 during training, while the ablation section uses σ = 0.2.
Performance comparison: 1.16% and 1.28% average improvements over TME are reported on CIRR at σ = 0.5 and 0.8, respectively.On FashionIQ, the corresponding gains are 0.94% and 1.31%, and the gap widens as noise increases.
Performance comparison: At σ = 0.8, SPRC’s Avg score is 15.12% lower than HABIT, whereas at σ = 0.2 it is only 2.63% lower.The results indicate greater sensitivity to noise for ordinary methods than for robust methods.
Ablation study: Removing the robust contrastive loss produces the worst ablation performance, while removing the Transition Rate causes the largest MKE-related drop.The ablation findings support the roles of robust contrastive learning and Transition Rate-based mutual-knowledge estimation.
Case study: HABIT retrieves the top-ranked CIRR image satisfying both the diver and sea-turtle requirements, while TME retrieves manta-ray images instead.The FashionIQ case likewise reports a top-1 retrieval matching the stated clothing attributes.

5 Conclusion

The conclusion presents HABIT as a robust solution to Noise Triplet Correspondence in composed image retrieval, combining transition-rate noise estimation with historical-current model collaboration. Experiments show superior performance and robustness across noise levels.

Conclusion: HABIT addresses precise composed semantic discrepancy estimation and progressive adaptation to modification discrepancy through MKE and DPL modules.MKE uses the transition rate of variational mutual information for noise-aware label assignment, while DPL collaborates between historical and current models.
Conclusion: HABIT outperforms most methods under various noise levels, demonstrating superior robustness and retrieval performance.The conclusion attributes this to progressively improved understanding of complex triplet relations and reduced misdetermination.

A Datasets

The experiments use two established composed image retrieval benchmarks: FashionIQ for fashion-domain retrieval and CIRR for open-domain retrieval. FashionIQ contains annotated triplets across three clothing categories.

CIRR: CIRR is the open-domain benchmark used alongside FashionIQ for composed image retrieval evaluation.The paper describes CIRR as targeting open-domain scenarios, complementing FashionIQ’s fashion-specific setting.
FashionIQ: FashionIQ is a fashion-focused benchmark with 77,684 web-crawled photos and 30,134 annotated triplets.Its triplets span dresses, shirts, and tops&tees, supporting evaluation of visual-textual alignment for fashion modifications.

B Comprehensive Performance Comparison on CIRR and FashionIQ

The comprehensive comparison examines model performance across CIRR and FashionIQ under varying noise ratios. HABIT consistently ranks first, while competing methods show greater fluctuation or lower recall in noisy settings.

Performance overview: Figure 4 compares models across noise ratios σ, using CIRR Avg, CIRR Rs.Avg, and FIQ(Avg) to summarize retrieval performance.Rows correspond to σ = 0.2, 0.5, and 0.8; colors identify models and lines show performance variation.
Training procedure: HABIT’s training procedure extracts composed and target features, estimates mutual knowledge, discriminates noisy samples, updates knowledge over time, and optimizes L_KL + L_soft + L_rank.The procedure uses Q-Former feature extraction, DBSCAN-based chronological noise discrimination, similarity computation, and the three-part HABIT loss.
Performance overview: HABIT consistently achieves the best performance across all datasets and noise conditions.The reported robustness is attributed to Transition Rate-based noise-aware estimation and the model’s progressive learning design.
Baseline behavior: RDE and SPRC fluctuate considerably across noise ratios and datasets, indicating limited stability under noisy and complex scenarios.TME remains relatively stable but consistently ranks second with lower recall than HABIT.

C Efficiency Evaluation

HABIT maintains strong retrieval performance while reducing computational cost, memory use, and inference latency relative to TME and SPRC. Its progressive learning adds negligible inference overhead and only a small training-time increase.

Efficiency Comparison: HABIT achieves 393.08G FLOPs, reducing computation by approximately 2.99% versus TME and 4.91% versus SPRC.All models use 915.69M parameters, so the improvement is not attributed to a larger model.
Efficiency Comparison: HABIT uses 11.37GiB memory, reducing consumption by about 8.38% versus TME and approximately 53.55% versus SPRC.
Speed: HABIT tests in 0.0089 seconds per sample, providing approximately 13.93 times acceleration over TME and around 1.24 times speedup over SPRC.Training takes 2.94 seconds per iteration, 62.60% less than TME but approximately 12.21% more than SPRC.
Accuracy and Efficiency: HABIT records FIQ-Avg = 64.38 and CIRR-Avg = 79.61 while balancing retrieval precision with efficiency overhead.
Ablation and Architecture: The w/o History variant has identical FLOPs, memory consumption, and testing time, showing that DPL operates during training without inference overhead.HABIT increases training time by only approximately 3.16% versus w/o History while yielding performance gains on CIRR and FIQ.
Ablation and Architecture: MKE and DPL add lightweight training computation without inference branches or re-ranking steps, reducing cost, memory use, and latency relative to TME and SPRC.The efficiency evaluation reports that the minor training overhead is accompanied by stable performance gains.

D HABIT Training Procedure and Dynamic Margin Calculation

The training procedure jointly optimizes HABIT’s MKE and DPL components. Its soft estimation margin uses a dynamic margin based on each sample’s cleanliness estimation, with the margin parameter empirically set to 0.2.

Training Workflow: Algorithm 1 presents the complete HABIT training workflow and shows how MKE and DPL are collaboratively optimized.
Dynamic Margin: The dynamic margin in the soft estimation margin loss depends on the estimation value e(I) for each sample.
Dynamic Margin: The margin parameter m is empirically set to 0.2 following prior work.

E.1 Failure Cases Analysis

HABIT remains highly relevant in challenging CIR failure cases, even when it does not retrieve the annotated ground truth. These cases also expose annotation limitations and possible false negatives in benchmark evaluation.

Failure Cases: HABIT faces challenging failure cases in real-world CIR involving complex semantic transformations.The analysis considers examples from both CIRR and FashionIQ.
CIRR: In the CIRR example, HABIT’s top-1 result matches the requested color and structure despite not being annotated as the ground truth.
CIRR: HABIT’s top candidates can remain highly relevant when the annotated target is not retrieved, whereas the comparison model shows greater divergence from fine-grained semantics.
FashionIQ: In the FashionIQ example, all HABIT top-5 results are highly similar in style, color, and type despite the ground truth not ranking first.
Evaluation Limitations: Some apparent failures reflect false negatives because retrieved images are visually and semantically correct but unmarked as targets due to annotation limitations.
Evaluation Limitations: The failure analysis reports consistently high Top-k relevance while revealing continuing challenges for annotation and evaluation in CIR benchmarks.

E.2 NTC Recognition Situation

HABIT’s MKE distinguishes clean, noisy, and ambiguous triplets using cleanliness estimates, while its similarity matrix shows stronger positive-pair separation than TME. These behaviors support noise-aware training and improved retrieval discrimination.

Cleanliness Estimation: MKE distinguishes clean and noisy CIR triplets under NTC conditions and produces cleanliness estimates for each correspondence.
Cleanliness Estimation: Clean examples receive high confidence, including scores of 0.96 for single-attribute and complex multi-object semantic changes.
Cleanliness Estimation: Noisy cross-category and detailed attribute mismatches receive low cleanliness estimates, including a score of 0.49.
Cleanliness Estimation: Visually ambiguous boundary cases receive intermediate confidence, such as 0.58 for a difficult dog-and-wolves composition.
Training Implications: The confidence-based discrimination supports robust downstream training and dynamic label correction by reducing the impact of noisy samples.
Similarity Matrix: In the CIRR similarity matrix, HABIT has brighter diagonal entries and darker off-diagonal regions than TME, indicating stronger positives and lower negative similarities.
Similarity Matrix: MKE and DPL are described as producing these differences by down-weighting noisy correspondence and hard negatives while calibrating the model through consistency and margin-based learning.