PaperlayerView exact revision on arXiv

Source-linked AI summary

Air-Know: Arbiter-Calibrated Knowledge-Internalizing Robust Network for Composed Image Retrieval

Zhiheng Fu, Yupeng Hu, Qianyun Yang, Shiqi Zhang, Zhiwei Chen, Zixu Li

arXiv:2604.19386v2cs.CV

TL;DR

CIR is limited by semantically ambiguous Noisy Triplet Correspondence, which makes small-loss noise identification unreliable and can cause representation pollution. Air-Know decouples arbitration from learning through offline MLLM arbitration, proxy knowledge internalization, and confidence-guided stream diversion, and experiments report superior performance across multiple CIR benchmarks.

  • Problem

    Semantic ambiguity from partial matches makes the small loss hypothesis unreliable for identifying noisy CIR triplets.

  • Method

    Air-Know uses offline MLLM arbitration to build an anchor dataset, internalizes expert knowledge in a lightweight proxy, and diverts training through confidence-guided clean and feedback streams.

  • Results

    Air-Know demonstrates superior performance and outperforms all state-of-the-art methods across multiple CIR benchmark datasets.

  • Takeaways & Limitations

    The three-stage framework decouples the arbiter from the learner while calibrating the base representation model to prevent representation pollution in NTC.

  • Takeaways & Limitations

    The paper defines partial matches specifically as cases where only one multimodal-query component aligns with the target.

Abstract

from arXiv · show

Composed Image Retrieval (CIR) has attracted significant attention due to its flexible multimodal query method, yet its development is severely constrained by the Noisy Triplet Correspondence (NTC) problem. Most existing robust learning methods rely on the "small loss hypothesis", but the unique semantic ambiguity in NTC, such as "partial matching", invalidates this assumption, leading to unreliable noise identification. This entraps the model in a self dependent vicious cycle where the learner is intertwined with the arbiter, ultimately causing catastrophic "representation pollution". To address this critical challenge, we propose a novel "Expert-Proxy-Diversion" decoupling paradigm, named Air-Know (ArbIteR calibrated Knowledge iNternalizing rObust netWork). Air-Know incorporates three core modules: (1) External Prior Arbitration (EPA), which utilizes Multimodal Large Language Models (MLLMs) as an offline expert to construct a high precision anchor dataset; (2) Expert Knowledge Internalization (EKI), which efficiently guides a lightweight proxy "arbiter" to internalize the expert's discriminative logic; (3) Dual Stream Reconciliation (DSR), which leverages the EKI's matching confidence to divert the training data, achieving a clean alignment stream and a representation feedback reconciliation stream. Extensive experiments on multiple CIR benchmark datasets demonstrate that Air-Know significantly outperforms existing SOTA methods under the NTC setting, while also showing strong competitiveness in traditional CIR.

1. Introduction

CIR is hindered by Noisy Triplet Correspondence, whose semantically ambiguous partial matches undermine small-loss noise identification and can pollute representations. Air-Know decouples arbitration from learning using offline MLLM expertise, proxy internalization, and confidence-based stream diversion, achieving superior NTC performance.

  • Problem: Noisy Triplet Correspondence constrains CIR because training annotations contain erroneous or semantically ambiguous matches.The problem arises from costly, subjective triplet annotation and hallucinations in large-model annotations.
  • Problem: Partial matches violate the small loss hypothesis because they are neither fully clean nor purely noisy.For example, a shirt-to-T-shirt triplet shares attributes while still mismatching the requested change.
  • Problem: Unreliable self-arbitration creates a vicious cycle in which pseudo-clean samples are trusted, representations collapse, and noise estimation deteriorates.The resulting representation pollution can spread through the representation space and decrease accuracy.
  • Motivation: MLLMs offer expert semantic arbitration, but their high inference cost prevents online use throughout training.This motivates an approach that preserves expert-level judgment without repeatedly invoking the MLLM.
  • Method: Air-Know uses EPA to build an offline anchor dataset, EKI to train a lightweight proxy arbiter, and DSR to split training into clean alignment and feedback reconciliation streams.The EKI proxy supplies matching confidence for dynamic gating, while the feedback stream corrects the base representation model.
  • Results: Extensive experiments show that Air-Know significantly outperforms existing state-of-the-art methods under noisy correspondence while remaining competitive in traditional CIR.This result is reported across multiple CIR benchmark settings.

2. Related Work

CIR retrieves a target image from a reference image and modification text, but noisy triplet correspondence remains more complex than conventional mismatched-pair noise. Existing methods such as TME address NTC while still relying primarily on the small loss hypothesis.

  • CIR with noisy correspondence: CIR retrieves a target image using a multimodal query consisting of a reference image and modification text.Recent approaches commonly use pretrained models such as CLIP and BLIP-2 for feature alignment and composition.
  • CIR with noisy correspondence: NTC involves semantic inconsistency within triplets, making it more complex than conventional false-positive or completely mismatched-pair noise.Its ambiguity distinguishes it from traditional noisy correspondence learning settings.
  • CIR with noisy correspondence: TME introduced specialized matching mechanisms for NTC but still primarily derives identification from the small loss hypothesis.The related-work passage frames this as an unresolved limitation of existing NTC handling.
  • Related methodology: Air-Know models a robust noisy geometric discrimination boundary using a variational-inference framework because the anchor dataset is sparse.The framework models the posterior distribution of network parameters.

3. Methodology

Air-Know decouples arbitration from representation learning through three modules: EPA obtains expert labels, EKI internalizes them in a lightweight Bayesian proxy, and DSR uses confidence to route training samples.

  • External Prior Arbitration: EPA uses an offline multimodal expert to annotate sampled triplets and construct a high-precision anchor dataset for noise discrimination.The expert produces binary Clean or Noisy labels after cross-validation and diagnosis of correspondence issues.
  • Expert-Knowledge Internalization: EKI transfers expert knowledge into a lightweight proxy that learns matching confidence from multimodal and geometric features.The proxy combines Q-Former features, pooled query-target representations, and the Geometric Deconstruction Vector.
  • Expert-Knowledge Internalization: The Geometric Deconstruction Vector combines query and target features, their difference, and their elementwise product to expose matching evidence.These components capture complementary global discrepancy and fine-grained commonality signals.
  • Expert-Knowledge Internalization: EKI maps reliable and noisy samples to separate regions, producing confidence above 0.5 for clean samples and at most 0.5 for noisy samples.This parameterization internalizes the expert’s arbitration logic for subsequent representation learning.
  • Expert-Knowledge Internalization: Because the anchor dataset is sparse and ill-posed, EKI models a posterior over proxy parameters instead of relying on point estimation.Variational inference approximates the posterior through an ELBO with reconstruction and prior-matching terms, while dropout supports stochastic prediction.
  • Dual-Stream Reconciliation: DSR uses EKI confidence as a dynamic gate to route samples into a clean alignment stream or a feedback reconciliation stream.The reconciliation stream targets hard noisy samples with low confidence but high query-target similarity to reduce their representation similarity.

4. Experiments

Experiments evaluate Air-Know on FashionIQ and CIRR under noisy correspondence, ablate its modules, and examine hyperparameter sensitivity and qualitative retrieval behavior. Air-Know remains robust across noise settings, while ablations support the roles of EPA, EKI, and DSR.

  • Performance Comparison: Air-Know is evaluated on FashionIQ and CIRR using noise ratios of 20%, 50%, and 80%.The experiments compare Air-Know with ordinary and robust baselines.
  • Performance Comparison: Air-Know achieves superior robustness across all noise settings, with its advantage over baselines increasing as noise rises.The authors attribute this to EKI’s credible confidence and DSR’s mitigation of representation pollution.
  • Ablation Study: Removing EPA or its multi-step verification significantly reduces performance, supporting external arbitration for complex NTC samples.The ablation study evaluates EPA removal and removal of cross-verification.
  • Ablation Study: EKI performance depends on combining basic semantics, geometric discrepancy, local and global proxy signals, and MC Dropout uncertainty estimation.The ablations indicate that geometric features and uncertainty estimation are jointly important.
  • Sensitivity Analysis: The MC Dropout rate p peaks at p = 0.1 on both FashionIQ and CIRR, with performance decreasing when p is too low or too high.Low p yields deterministic point estimation, whereas high p introduces destabilizing noise.
  • Case Study: Air-Know retrieves the target at Top-1 in all illustrated CIRR and FashionIQ cases, whereas TME fails in the shown comparisons.The case study attributes Air-Know’s behavior to EKI-guided geometric separation and confidence-based gating.

5. Conclusion

The paper identifies semantic ambiguity in NTC as a limitation of small-loss-based robust learning and proposes Air-Know to decouple expert arbitration from learner training. Its three-stage framework uses MLLM expertise offline and stream separation online, achieving superior performance on multiple CIR benchmarks.

  • Conclusion: Small-loss assumptions fail under NTC’s semantically ambiguous partial matches, causing self-dependent arbitration and representation pollution.The conclusion frames these issues as the central problem addressed by Air-Know.
  • Conclusion: Air-Know uses a three-stage framework to leverage MLLM expert judgment while avoiding high online inference overhead.The framework also calibrates the base representation model through stream separation.
  • Conclusion: Experiments on multiple CIR benchmark datasets show that Air-Know outperforms all SOTA methods.The conclusion reports superior performance without specifying a single metric.

A. Proof

The appendix establishes the variational-inference relationship between the ELBO and posterior KL divergence for the EKI module. Because the log-evidence is constant with respect to model parameters, maximizing the ELBO is equivalent to minimizing posterior divergence.

  • Proof: The log-evidence decomposes into the ELBO and the KL divergence between the variational approximation and the true posterior.The appendix identifies the grouped regularization term as KL divergence.
  • Proof: The proof starts from the KL divergence between the variational distribution qθ(W) and the true posterior p(W|Danchor).The derivation expands the logarithmic ratio and uses Bayes’ theorem.
  • Proof: The log-evidence log p(Danchor) is independent of W and therefore acts as a constant under the expectation.This constancy enables the ELBO decomposition.
  • Proof: Maximizing LELBO(θ) is strictly equivalent to minimizing DKL[qθ ∥ p(·|D)] because the evidence is fixed with respect to θ.Nonnegativity of KL divergence also establishes that the ELBO is a lower bound on log-evidence.

B. Datasets

The appendix describes FashionIQ as a standard fashion-domain CIR benchmark used to evaluate visual alignment with modification text. It contains 77,684 images and 30,134 annotated triplets across three major categories.

  • FashionIQ: FashionIQ contains 77,684 web-crawled fashion images and 30,134 annotated triplets.The dataset supports evaluation of composed image retrieval in the fashion domain.
  • FashionIQ: FashionIQ is divided primarily into dresses, shirts, and toptees categories.These categories define the dataset’s main fashion scenarios.

C. Cross-Validation of EPA

EPA uses a structured MLLM cross-validation process to analyze triplets, reason about semantic consistency and NTC causes, and produce binary Clean/Noisy labels for a high-precision anchor dataset.

  • EPA uses an offline MLLM expert to assign high-precision binary Clean or Noisy labels to a small anchor dataset for supervising the proxy arbiter.The resulting Danchor transfers expert judgment to EKI while decoupling the arbiter from the learner.
  • Step 1: Deconstruct Inputs: The first stage independently analyzes the reference image, target image, and modification text to establish structured factual descriptions.This deconstruction covers visual content, modification intent, and the relationships among triplet components.
  • Step 2: Compare & Reason: The second stage compares inferred visual changes with the modification text and applies NTC principles to diagnose semantic mismatches and partial matches.The reasoning chain documents factual inference, cross-validation, principle application, and final diagnosis.
  • Step 3: Judge & Conclude: The final stage compiles the reasoning chain into a standardized output containing the MLLM’s analysis workflow and final verdict.This format supports consistent dataset construction from the preceding reasoning process.

D. Training Details

This section states that the paper describes Air-Know’s architecture, optimization procedure, and two-stage progressive training strategy for reproducibility.

  • The section covers Air-Know’s network architecture, optimization procedure, and Two-Stage Progressive Training Strategy.

D.1. Architecture of the Lightweight Proxy

EKI uses a lightweight MLP proxy and a progressive training strategy to internalize expert knowledge, generate matching confidence, and guide two training streams.

  • Architecture: The EKI proxy is a three-layer MLP mapping 1024-dimensional GDVs through 512 and 256 hidden units to one confidence score.It uses ReLU and dropout after the first two linear layers and a sigmoid output.
  • Progressive Training: Stage 1 freezes the backbone and optimizes EKI using the anchor dataset to establish a reliable arbiter before joint training.
  • Progressive Training: During subsequent training, frozen EKI confidence scores divert samples into the clean alignment or feedback reconciliation stream while the backbone is optimized with the total objective.

E.1. Efficiency Evaluation

Air-Know combines low computational overhead with faster training and inference than the compared robust baseline, while its robustness depends on balancing feedback-stream tolerance and expert-labeling cost.

  • Efficiency: 402.51G FLOPs makes Air-Know lower-overhead than TME at 405.20G and SPRC at 413.38G, while all models use approximately 915M parameters.The reported reductions are approximately 0.66% versus TME and 2.63% versus SPRC.
  • Efficiency: Air-Know uses 16590 MiB GPU memory versus 12405 MiB for TME because parallel proxy and dual-stream computations require additional memory.
  • Efficiency: 2.805 s/iter training is slower than SPRC’s 2.624 s/iter but faster than TME’s 7.858 s/iter, while 0.010 s/sample inference is 12.4× faster than TME’s 0.124 s/sample.
  • Hyperparameter Analysis: The feedback-stream tolerance margin α controls penalties on high-similarity samples with low matching confidence, and its sensitivity is evaluated on FashionIQ and CIRR.
  • Hyperparameter Analysis: Both datasets peak at α = 0.7; lower values over-suppress gradients, whereas higher values allow constraints to become ineffective.FashionIQ falls to approximately 64.3 at α = 0.5, and excessive α can produce sparse or vanishing reconciliation gradients.
  • MLLM Evaluation: GPT-4o maintains 91.41% accuracy at 80% noise, slightly below GPT-5’s 94.43% while exceeding other lightweight models.The result supports using GPT-4o to provide clean supervision for EKI with higher efficiency than GPT-5.
  • MLLM Evaluation: EPA uses a three-stage process that deconstructs inputs, compares inferred visual changes with text, and outputs Clean or Noisy determinations.

F.2.1. Quantitative Results

The ablations show that structured cross-validation is necessary for reliable NTC arbitration: independent deconstruction counters misleading text, while reasoning preserves valid partial matches. Removing both stages performs worst across noise settings.

  • EPA ablations: 91.41% accuracy at σ = 0.8 fell to 89.25% (∆= −2.16%) without Step 1, showing the cost of removing objective input anchors.Without independent deconstruction, misleading text can induce hallucinated visual changes.
  • EPA ablations: 85.16% accuracy at σ = 0.2 dropped to 76.56% without Step 2, indicating that removing reasoning causes valid partial matches to be rejected.The omitted reasoning infers the actual visual difference and cross-validates it against the modification text.
  • EPA ablations: The end-to-end variant without Steps 1 and 2 performed worst across all noise settings.Unstructured prompting cannot simultaneously resist misleading text and tolerate partial matches.

G. More Qualitative Results

Qualitative results show that EKI assigns high confidence to semantically consistent triplets and low confidence to NTC cases with semantic conflicts. Its prompt architecture separates factual observation, reasoning, and final classification.

  • EKI confidence estimation: EKI maps semantically consistent samples to high-confidence regions and suppresses mismatched or text-image-inconsistent triplets with extremely low scores.These confidence estimates act as dynamic gating signals for DSR.
  • Clean samples: A subtle positional and attribute transformation received a confidence score of 0.96, demonstrating recognition of fine-grained semantic consistency.The example concerns the modification “Orange pouch is in front of blue pouch.”
  • Prompt architecture: The three-stage prompt deconstructs each input, reasons over semantic consistency and NTC type, then produces a determination from the evidence chain.The workflow first establishes objective factual anchors before comparison and judgment.
  • Noisy samples: A reference-target pair involving two pugs and shelves received a score of 0.17 because of a severe semantic disconnection.The model identified the case as a semantic mismatch.
  • Noisy samples: A triplet with the instruction “Show shorn dog” and images of bottles and vases received a low score of 0.19.The score reflects inconsistency between the modification text and the target image.
Loading 2604.19386v2…