PaperlayerView exact revision on arXiv

Source-linked AI summary

AutoConcept: Training-Free Concept-Guided Reranking for Metadata-Available Composed Image Retrieval

Tianyu Wang, Tianjiao Wu

arXiv:2609.01456v1cs.IRcs.CL

TL;DR

Metadata-available composed image retrieval reranking asks whether explicit concept evidence can improve a fixed retriever’s candidate pool beyond direct query- or attribute-to-metadata matching. AutoConcept uses a training-free concept memory to rerank candidates, with FashionIQ gains reported for both WeiMoCIR and LinCIR pools.

  • Problem

    The paper asks whether explicit concept evidence improves a strong CIR candidate pool beyond direct query-to-metadata or attribute-to-metadata matching.

    Source: §1 · p. 2

  • Method

    AutoConcept filters controlled concept evidence into an interpretable memory, activates query-relevant constraints, and combines base retrieval scores with metadata-based concept-candidate alignment without training.

    Source: §1 · p. 2

  • Results

    FashionIQ Recall@10 rises from 0.1125 to 0.1379 for WeiMoCIR and from 0.2605 to 0.3009 for LinCIR candidates under the same protocol.

    Source: §1 · p. 2

  • Takeaways & Limitations

    AutoConcept provides an inspectable concept-memory layer that improves both WeiMoCIR and LinCIR candidate pools and can consume participant-provided concept evidence.

    Source: §6 · p. 11

    The method is scoped to metadata-available product-style galleries and is most suitable when metadata is reasonably aligned with item appearance.

    Source: §5 · p. 10

Abstract

from arXiv · show

Composed image retrieval (CIR) retrieves a target image from a reference image and a text modification. This paper studies metadata-available CIR reranking, where a fixed CIR model first returns a candidate pool and gallery metadata is then used for second-stage concept-guided scoring. We introduce AutoConcept, a training-free reranker that converts concept evidence into an interpretable memory. AutoConcept filters noisy concepts, activates query-relevant positive constraints with an auxiliary negative penalty, and combines base retrieval scores with metadata-based concept-candidate alignment through inference-time calibration. On FashionIQ, AutoConcept yields significant early-rank improvements over WeiMoCIR and consistent plug-in gains on LinCIR candidate pools. Metadata-aware controls show that structured concept memory adds signal beyond direct query-text and extracted-attribute matching, while a query-only variant further supports the effectiveness of concept-level reranking. A supplementary real-human concept-label study indicates that the same memory interface can consume participant-provided evidence. These results position AutoConcept as an interpretable concept-memory reranker for product-style CIR galleries with available metadata.

1 Introduction

The introduction frames metadata-available CIR reranking as a way to apply inspectable concept constraints after a fixed retriever forms a candidate pool. AutoConcept builds concept memory to guide that reranking without training, and the FashionIQ experiments report gains on WeiMoCIR and LinCIR pools.

1 Introduction

  • With the same composed query and first-stage pool, concept memory promotes candidates matching query-relevant positive constraints and penalizes candidates matching negative constraints.The framework overview describes controlled evidence becoming explicit memory, followed by concept activation and top-K reranking.Source: Figure 1 · p. 2, Figure 2 · p. 3
  • The work asks whether explicit concept evidence improves a strong CIR candidate pool beyond direct query-to-metadata or attribute-to-metadata matching.CIR retrieves a target image from a reference image and a natural-language modification; fashion and e-commerce galleries commonly include item metadata.Source: §1 · pp. 1–2
  • AutoConcept filters controlled concept evidence, activates query-relevant positive constraints with an auxiliary negative penalty, and combines base scores with metadata-based concept-candidate alignment.It operates without training on the fixed first-stage model’s top-K pool, using inference-time score calibration.Source: §1 · pp. 2–3, Figure 2 · p. 3
  • FashionIQ Recall@10 rises from 0.1125 to 0.1379 for WeiMoCIR and from 0.2605 to 0.3009 for LinCIR candidates under the same metadata-available reranking protocol.The experiments use five controlled concept-evidence seeds.Source: §1 · p. 2
  • Metadata-aware controls compare concept memory with query-only, query-to-metadata, composed-text, and extracted-attribute matching, while aligned-versus-shuffled analysis tests query-evidence correspondence.A real-human concept-label study tests whether the memory interface can consume participant-provided item-level evidence.Source: §1 · pp. 2–3

2 Related Work

Related work situates AutoConcept alongside CIR backbones, plug-in reranking, and concept-based interpretability. Its distinguishing framing is an inspectable concept memory used to adjust candidate ordering after retrieval.

2.1 Composed Image Retrieval

  • CIR systems range from learned composition and attention-based fusion to zero-shot and training-free methods built on pretrained vision-language embeddings; AutoConcept adds memory over existing candidate pools.The cited approaches include mappings, textual inversion, and candidate verification [1] [29].Source: §2.1 · p. 3

2.2 Training-Free and Plug-in Reranking

  • Training-free CIR reduces training cost through fusion, score recombination, textual inversion, language-only training, or candidate verification; AutoConcept adjusts a supplied pool without per-query MLLM calls or reranker training [1] [29].It uses WeiMoCIR as its main training-free baseline and contrasts with constraint-based SoFT [13].Source: §2.2 · pp. 3–4

2.3 Concept Evidence and Controllable Reranking

  • Unlike related concept-interpretability and memory-based systems, AutoConcept’s concepts carry names, polarity, strength, grounding scores, and evidence items for inspection and diagnosis.The related-work framing connects inspectable control to concept activation, bottleneck, prototype, and whitening methods.Source: §2.3 · p. 4

3 Method

AutoConcept reranks a fixed first-stage candidate pool using controlled concept evidence and gallery metadata available under a target-blind protocol. It filters and activates concepts, aligns them with candidate metadata, and combines that evidence with base scores through inference-time calibration.

3.1 Task Setup

  • AutoConcept reranks only the candidate pool returned by a fixed base retrieval model, using controlled concept evidence to form positive constraints and a small auxiliary set of negative constraints.In the main FashionIQ experiments, the pool contains 100 candidates.Source: §3.1 · p. 4

3.2 Information Source Protocol

  • Concept evidence is constructed before reranking without target identity, target image or metadata, candidate ranks, or post-retrieval relevance labels; candidate metadata scores only items already retrieved.The main controlled evidence is sampled from same-category training-side item texts; a query-only variant extracts concepts from the modification text.Source: §3.2 · p. 4

3.3 Concept Memory Construction

  • The memory extracts and clusters attribute phrases, scores positive or negative evidence, and filters generic, conflicting, poorly grounded, or weak concepts.The filter retains frequent, meaningful apparel attributes such as sleeves, collar, buttons, stripes, and V-neck.Source: §3.3 · p. 5

3.4 Query-Relevant Concept Activation

  • A query-relevance gate activates concepts according to their relevance to the query, with the clipping range fixed before evaluation and shared across runs.Appendix A.1 reports sensitivity around the operating region.Source: §3.4 · p. 5

3.5 Candidate-Concept Alignment

  • Active concepts are aligned with candidate metadata, and positive alignment and the disliked-concept penalty are aggregated by max pooling.Candidate metadata text is used for scoring; query-specific weights depend on concept evidence and base-retriever confidence, while the disliked-concept term remains a conservative penalty.Source: §3.5 · pp. 5–6

3.6 Inference-Time Score Interpolation

  • Inference-time interpolation sets a query-specific coefficient within bounds fixed before evaluation, so weak concept activation does not override the base retriever.The interpolation bounds and base value are set using the base margin.Source: §3.6 · p. 7

4 Experiments

Experiments test AutoConcept on fixed candidate pools using metadata-aware controls, concept-evidence alignment analyses, participant-provided labels, and scoring-component ablations. Gains are reported on both candidate generators, with human-label results and ablations clarifying the approach's scope and components.

4.1 Dataset, Protocol, and Baselines

  • Table 3 reports consistent gains on both candidate generators, concentrated in early-rank metrics.The main evaluation uses FashionIQ, five controlled concept-evidence seeds, and a fixed top-100 pool; candidate metadata is introduced only in second-stage reranking.Source: §4.1 · pp. 7–8, Table 3 · p. 8

4.2 Significance Testing

  • Query-level paired bootstrap tests compare AutoConcept with WeiMoCIR using query resampling, percentile 95% confidence intervals, and two-sided p-values.Reported p-values are descriptive and unadjusted for multiple metrics; seed-level robustness is reported through Table 3 standard deviations.Source: §4.2 · p. 8

4.3 Metadata-Aware Reranking Controls

  • On the same top-100 WeiMoCIR pool, query-only AutoConcept improves over query-text, extracted-attribute, and resource-matched constraint reranking on early-rank metrics.The reference-text-enriched row is a strong text-only upper reference using richer textual signal, not a same-evidence concept-memory baseline.Source: §4.3 · p. 9, Table 5 · p. 9

4.4 Concept-Evidence Alignment

  • Aligned concept evidence is substantially stronger than shuffled evidence, supporting the role of coherent query-evidence alignment.The shuffled control preserves domain and category-level concept distributions while breaking query-evidence alignment.Source: §4.4 · p. 9, Table 6 · p. 9

4.5 Real-Human Concept Labels

  • Despite sparse participant histories not collected for benchmark targets, all four target-label metrics improve with human concept labels.The authors attribute the smaller gain relative to controlled evidence to limited item coverage, incomplete modifier coverage, and memories not constructed for each query's benchmark target.Source: §4.5 · pp. 9–10, Table 7 · p. 10

4.6 Ablation Analysis

  • On the FashionIQ seed-1 top-100 pool, max pooling matters because concept matches are sparse, while removing the negative penalty changes little.The ablation indicates gains are driven mainly by positive concept activation.Source: §4.6 · p. 10, Table 8 · p. 10

5 Scope and Extensions

AutoConcept is scoped to metadata-available product-style galleries, with missing or inconsistent metadata a practical boundary. The paper identifies broader domains, richer visual grounding, and other extensions, while reporting gate stability around its operating range.

5 Scope and Extensions

  • AutoConcept is a second-stage reranker for metadata-available product-style galleries, and missing or inconsistent metadata remains a practical boundary.The authors identify broader domains, richer visual grounding, larger human-preference studies, and stronger metadata denoising as extensions.Source: §5 · pp. 10–11
  • Gate sensitivity on FashionIQ is stable around the operating range used for inference-time calibration.Source: Figure 4 · p. 11

6 Conclusion

The conclusion presents AutoConcept as an explicit concept-memory layer for metadata-available composed image retrieval, with gains across two candidate pools and an interface that consumes participant-provided evidence.

6 Conclusion

  • AutoConcept adds an explicit concept-memory layer to metadata-available composed image retrieval, complementing retrieval-augmented and concept-controlled multimodal systems.Source: §6 · p. 11
  • AutoConcept improves both WeiMoCIR and LinCIR candidate pools and compares favorably with direct query and attribute metadata matching.Source: §6 · p. 11
  • The real-human concept-label study shows that the same interface can consume participant-provided evidence.Source: §6 · p. 11

A.1 Gate Sensitivity

The appendix reports gate and candidate-pool sensitivities, a Fashion200K check, and the reranking overhead. AutoConcept performs best with the default top-100 pool, while a broad gate range yields similar gains.

A.1 Gate Sensitivity

  • Gates from 0.30 to 0.40 yield similar gains, and every tested value remains above WeiMoCIR.The analysis changes only the query-concept relevance gate; τq can be inferred per query from the memory relevance distribution.Source: §A.1 · p. 11, Table 9 · p. 12
  • Positive concepts drive most of the observed gain in the current FashionIQ setting; disliked concepts remain a lightweight, inspectable penalty channel.The gate-sensitivity analysis holds the top-100 pool, controlled evidence setting, concept memory, and calibration bounds fixed.Source: §A.1 · p. 11

A.2 Candidate Pool Size

  • The default top-100 pool gives the strongest result; top-50 limits coverage, while larger pools add metadata noise for the same scorer.The sensitivity analysis uses FashionIQ seed 1.Source: §A.2 · p. 12, Table 10 · p. 12

A.3 Fashion200K Secondary Check

  • AutoConcept improves all four reported metrics over the same WeiMoCIR candidate generator on the Fashion200K secondary check.The check uses item text and sampled candidate pools.Source: §A.3 · p. 12, Table 11 · p. 12

A.4 Runtime and Latency

  • Cached AutoConcept adds 0.3199 ms/query on FashionIQ and 0.1928 ms/query on Fashion200K, or 5.58% and 6.18% of first-stage latency, respectively.The higher Fashion200K full cost reflects uncached text-encoder setup and metadata encoding over a smaller query set; those embeddings are precomputable.Source: §A.4 · p. 12, Table 12 · p. 13

B Implementation Details

Implementation fixes the first-stage top-100 candidate pool and shared evaluation settings, then applies metadata and concept scoring to rerank candidates. The relevance gate keeps the active concept set small.

B Implementation Details

  • The main FashionIQ experiment reranks the base retriever’s top-100 candidates, using the same pool for WeiMoCIR, AutoConcept, and metadata-aware controls.No query labels, target ranks, or relevance labels are used for tuning.Source: §B.1 · p. 13
  • Concept names, query text, and metadata text use all-MiniLM-L6-v2; CLIP ViT-B/32 supplies image embeddings and CLIP-based grounding.Quality-control rules and calibration bounds are fixed before evaluation and shared across seeds, candidate generators, and metadata-aware controls.Source: §B.1 · p. 13
  • The reranking loop scales as O(K(|A+_q|)d), with the relevance gate keeping the active concept set small.K denotes candidates, d the embedding dimension, and A+_q the active concept sets, as identified in the supplied passage fragments.Source: §B.1 · p. 13

C Dataset and Concept-Evidence Construction Details

The paper describes its FashionIQ and Fashion200K data settings, how concept evidence and shuffled controls are constructed, and how human labels enter the memory interface. It also details filtering, inspection, and deployment safeguards.

C Dataset and Concept-Evidence Construction Details

  • FashionIQ queries pair a reference image and two modification captions with a target image; controlled evidence comes from training-side item texts, while the query-only control uses modification text alone.Fashion200K is a secondary robustness check with composed queries from item text and sampled candidate pools.Source: C Dataset and Concept-Evidence… · p. 13
  • The aligned analysis uses attribute families and excludes target images, target metadata, candidate ranks, and target identity; its shuffled control permutes query-evidence mappings within category.The real-human study collects optional fixed-vocabulary tags on FashionIQ items, deduplicates labels, and converts them into the same concept-memory schema in a separate split.Source: C Dataset and Concept-Evidence… · pp. 13–14

D Concept Memory and Filtering Details

  • The filter removes generic or weak concepts while retaining visually meaningful apparel attributes such as sleeve length, neckline, buttons, stripes, and V-neck.Table 14 is identified as covering the main concept filter rules, but no rule-level contents are supplied.Source: D Concept Memory and Filtering Details · p. 14, Table 14 · p. 14

E Explanation Diagnostics and Ethics

  • AutoConcept exposes active concepts, evidence items, and candidate-side matches; its explicit named concepts support inspection, deletion, and editing.The human study uses voluntary item-level labels without requesting sensitive user attributes; larger deployments should avoid protected-attribute inference and expose memory-retention controls.Source: E Explanation Diagnostics and Ethics · p. 14
Loading 2609.01456v1…