Source-linked AI summary
Dialog-based Interactive Image Retrieval
Xiaoxiao Guo, Hui Wu, Yu Cheng, Steven Rennie, Gerald Tesauro, Rogerio Schmidt Feris
TL;DR
Interactive image retrieval is limited by restricted feedback formats that constrain how users express search intent. The paper introduces natural-language dialog retrieval trained with reinforcement learning and a user simulator, and reports better retrieval performance and more natural communication in footwear retrieval. Its main scope boundaries are a single-turn, history-independent user simulator and a policy simplified to ignore the number of remaining turns.
Problem
Existing interactive retrieval systems rely on binary relevance or fixed relative-attribute feedback, restricting the information users can convey about search intent.
Method
The paper trains an end-to-end dialog manager with reinforcement learning to optimize target-image rank, using a relative-captioning user simulator trained on human-written descriptions.
Results
In interactive footwear retrieval with real and simulated users, the framework outperforms existing techniques, while natural-language feedback is more effective than predefined relative attributes.
Takeaways & Limitations
Natural-language feedback supports more natural communication and significantly outperforms traditional predefined relative-attribute feedback in the demonstrated footwear-retrieval application.
Takeaways & Limitations
The user simulator is trained on single-turn data without dialog history, and the policy is simplified to ignore the number of remaining dialog turns.
Abstract
from arXiv · showhide
Existing methods for interactive image retrieval have demonstrated the merit of integrating user feedback, improving retrieval results. However, most current systems rely on restricted forms of user feedback, such as binary relevance responses, or feedback based on a fixed set of relative attributes, which limits their impact. In this paper, we introduce a new approach to interactive image search that enables users to provide feedback via natural language, allowing for more natural and effective interaction. We formulate the task of dialog-based interactive image retrieval as a reinforcement learning problem, and reward the dialog system for improving the rank of the target image during each dialog turn. To mitigate the cumbersome and costly process of collecting human-machine conversations as the dialog system learns, we train our system with a user simulator, which is itself trained to describe the differences between target and candidate images. The efficacy of our approach is demonstrated in a footwear retrieval application. Experiments on both simulated and real-world data show that 1) our proposed learning framework achieves better accuracy than other supervised and reinforcement learning baselines and 2) user feedback based on natural language rather than pre-specified attributes leads to more effective retrieval results, and a more natural and expressive communication interface.
1 Introduction
The paper introduces dialog-based interactive image retrieval, allowing natural-language feedback while optimizing target-image rank through reinforcement learning. It also develops a relative-captioning-based user simulator and evaluates the framework in footwear retrieval.
- Existing interactive retrieval systems constrain users to relevance or fixed relative-attribute feedback, limiting the information conveyed to the system.
- The dialog manager uses reinforcement learning to directly optimize the target object’s rank, a non-differentiable objective.
- A user simulator trained on human-written relative descriptions generates feedback about differences between candidate and desired images, reducing the need for annotated human-machine dialogs.
- Experiments with real and simulated users in interactive footwear retrieval report better performance than existing techniques, with natural-language feedback outperforming predefined relative attributes.
- The proposed task lets a dialog agent learn across multiple turns while users describe desired visual differences in natural language.
- The paper additionally introduces relative image captioning and a dataset supporting research on descriptions of salient visual differences between images.
2 Related Work
Prior work improves image retrieval through relevance feedback, natural-language queries, and visually grounded dialog, but these approaches generally rely on predefined interaction or query structures.
- Interactive Image Retrieval: Interactive retrieval has used binary or graded relevance feedback to indicate how well retrieved images match the desired result.
- Interactive Image Retrieval: Relative-attribute methods provide richer comparisons, but still depend on a fixed, predefined set of attributes.
- Image Retrieval with Natural Language Queries: Natural-language image and video retrieval methods connect visual content with language queries, forming a related research line.
- Visual Dialog: Visual dialog research has progressed from rule-based and slot-filling systems toward end-to-end encoder-decoder and sequence-to-sequence approaches.
3 Method
The dialog manager conducts iterative image retrieval by encoding candidate images and natural-language feedback, tracking dialog history, and selecting subsequent candidates. Training combines supervised triplet-loss pre-training with model-based reinforcement-learning policy improvement, supported by a relative-captioning user simulator.
- Dialog Manager: At each turn, the dialog manager presents a candidate image, receives feedback describing differences from the desired image, and selects another candidate using the dialog history.The process continues until the target is selected or the maximum number of turns is reached.
- Model Architecture: The Response Encoder fuses candidate-image and feedback representations, the GRU-based State Tracker aggregates history, and the Candidate Generator selects the next image.Candidate selection uses distances between the state representation and database image features.
- Policy Learning: The system optimizes target-image ranking percentile as a discounted reinforcement-learning reward because the ranking objective is non-differentiable.The reward at each turn represents the target image’s ranking percentile.
- User Simulation: A relative captioner serves as a user simulator by generating sentences describing prominent visual differences between target and candidate images.The simulator enables efficient exploration without collecting extensive human-machine dialogs, although it does not use dialog history.
- Policy Learning: Supervised triplet-loss pre-training improves initial target proximity, after which model-based policy improvement optimizes expected future reward.The model-based gradients are reported to have lower variance and faster convergence than traditional policy-gradient methods.
4 Dataset: Relative Captioning
The relative-captioning dataset was collected from shopping-chat scenarios to capture flexible natural-language descriptions of visual differences. Because users sometimes describe the target directly when images differ substantially, the training data combines relative and discriminative captions.
- Data Collection: Annotators role-played customers in shopping conversations and described desired products using natural expressions.The collection procedure was designed to support flexible language about visual differences.
- Caption Types: When target and reference images were sufficiently different, users often described the target’s visual appearance directly instead of using relative expressions.This behavior motivated augmentation with discriminative captions paired with additional dissimilar images.
- Caption Types: The captioner and dialog-based retriever were trained on both discriminative and relative captions to better represent and respond to real-user behavior.The dataset included 3600 discriminative captions paired with additional dissimilar images.
5 Experimental Results
Experiments evaluate policy learning, simulated user feedback, natural-language feedback, and real-user footwear retrieval. The proposed framework improves ranking over supervised and reinforcement-learning baselines, while unrestricted language feedback outperforms predefined attributes and supports natural interaction.
- Experimental setup: Experiments use simulated and real users to evaluate free-form dialog feedback and the dialog manager's pipeline components.All experiments use the Shoes dataset, with 10,000 training images and 4,658 testing images; performance is measured by average target-image rank percentile.
- 5.1 Analysis of the Learning Framework: 98% average ranking percentile is achieved by Ours after two dialog turns, consistently exceeding RL-SCST and supervised learning.Both reinforcement-learning methods outperform supervised pre-training, while Ours benefits from model-based policy improvement.
- 5.1 Analysis of the Learning Framework: Generated relative captions often differ from human annotations but can still capture the main visual differences between target and reference images.The user simulator approximates real-user feedback at low annotation cost.
- 5.2 Effectiveness of Natural Language Feedback: Natural-language feedback produces significantly higher target-image average ranking percentile than attribute-based methods across different dialog-turn counts.The comparison includes one, three, and ten predefined attribute phrases, with deep attribute estimates improving but not eliminating the performance gap.
- 5.3 User Study of Dialog-based Image Retrieval: 89.9% versus 70.3% average ranking percentile is obtained by the dialog manager and WhittleSearch, respectively, in five-turn real-user footwear dialogs.Users also reported that dialog-based feedback was more natural than selecting relevant attributes from a predefined list.
- 5.3 User Study of Dialog-based Image Retrieval: Users typically begin with coarse color or category descriptions and later provide fine-grained differences such as spatial, structural, and relative phrases.Examples include “leopard print on straps,” “thinner,” and “higher heel.”
6 Conclusions
The paper presents dialog-based interactive image retrieval as a computer-vision and language-understanding task. Its footwear application shows that natural-language feedback improves retrieval over predefined relative attributes while enabling more natural communication.
- Dialog-based interactive image retrieval is introduced as a task at the intersection of computer vision and language understanding.
- The proposed approach significantly outperforms traditional methods using a predefined relative-attribute vocabulary in interactive fashion footwear retrieval.
- Future work includes using product-item textual side information and user models conditioned on dialog histories for more realistic interactions.
A Data Collection
The relative captioning dataset is collected to support unconstrained but concise feedback about visual differences in footwear retrieval. Its collection procedure uses a customer-assistant scenario and also augments relative expressions with single-image captions.
- Data Collection: The dataset targets free-form comparative descriptions of visual differences that are difficult to express with predefined “more” or “less” attributes.
- Data Collection: Figure 7 presents the caption-length distribution and collected relative-caption examples, omitting the fixed leading phrase for brevity.
- Data Collection: Annotators complete a customer response in a shopping-assistant scenario, using a provided sentence prefix to encourage regular, specific feedback.
- Data Collection: Single-image captions are collected for 3,600 images and paired with multiple visually distinct reference images.
B Dataset analysis
The dataset contains concise but linguistically rich feedback, often combining multiple visual features with spatial or structural information. Examples show that users use fine-grained composed descriptions when predefined attributes are insufficient.
- Dataset analysis: Most collected captions contain 4 to 8 words while exhibiting a rich vocabulary.
- Dataset analysis: Users often compose multiple phrases rather than describing only one attribute, including fine-grained visual differences between otherwise similar shoes.
- Dataset analysis: Around two thirds of expressions combine more than one visual-feature type, and 40% contain spatial or structural propositional phrases.
- Dataset analysis: Human raters score relative captions from 1 for unrelated to 4 for captions with no errors.
- Dataset analysis: Some relative captions are concise, whereas others use more complicated phrases to identify key visual differences.
C Human Evaluation of Relative Captioning Results
The study compares relative captioning models using standard metrics and human evaluation, finding similar model performance but a persistent gap from human descriptions.
- Model comparison: Four relative captioning architectures were evaluated, including RC-FC, RC-FCA, RC-LNA, and RC-CNA.The models differed in feature fusion and attention mechanisms while using ResNet101 features.
- Evaluation: Common image-captioning metrics were considered unreliable for judging relative captions because visual differences can be described flexibly and annotations lack comprehensive variation.This motivated direct human evaluation of generated captions.
- Human evaluation: Human evaluation on 1,000 image pairs found similar performance across relative captioning models, with RC-CNA marginally better.Ratings covered captions generated by each model and human-provided descriptions.
- Human evaluation: Human-provided descriptions outperformed all automatically generated captions, and some generated captions contained incorrect or nonsensical attribute descriptions.These errors illustrate the complexity of relative image captioning and indicate room to improve the user simulator.
D Experimental Configurations
The experiments use separate Shoes training and held-out testing images, with the user simulator trained on non-overlapping training image pairs; interaction examples illustrate the system’s feedback format.
- Data split: The Shoes dataset was split into 10,000 training images and 4,658 held-out testing images.The split was selected randomly because no official training and testing split was reported.
- Data split: The user simulator used the same training and testing split as the dialog manager, with training image pairs excluded from the testing images.This prevents overlap between simulator training examples and test images.
- Interaction examples: The interaction examples display user feedback below corresponding images, omitting the repeated phrase “Unlike the provided image, the ones I want” for brevity.The figure presents examples of users interacting with the proposed dialog manager.
E Discussions on the Dialog Manager
The discussion highlights natural-language dialog’s retrieval flexibility while identifying limitations in caption quality, vocabulary coverage, dialog-memory retention, and objective alignment.
- Dialog-based User Interaction: Human-user dialogs reached the target image within the top 100 images, about the 97% ranking percentile, within five turns in all shown examples.Successful feedback combined direct references to target attributes with relative comparisons to candidate images.
- Dialog-based User Interaction: Natural-language feedback can express varied visual differences more flexibly than selecting from a fixed attribute vocabulary.A predefined vocabulary may require hundreds of words, making manual selection impractical.
- Limitations: The system is less responsive to some low-frequency simulator-generated words, such as “slouchy.”The discussion links this limitation to the simulator’s fixed post-training vocabulary.
- Limitations: The simulator lacks dialog-history memory, and the authors propose fine-tuning the dialog manager on real users to adapt to new vocabularies.The current simulator also has a fixed vocabulary after training.
- Dialog Manager Learning Framework: Triplet-loss pretraining improves target proximity but deviates from the ranking objective because similar triplet losses can yield very different target-image ranks.The RL framework instead optimizes the non-differentiable target rank end-to-end.
- Limitations: The dialog manager sometimes forgets information from past turns, such as a “yellow accents” requirement disappearing later in the dialog.The authors suggest architectures that incorporate dialog history more effectively could alleviate this issue.