PaperlayerView exact revision on arXiv

Source-linked AI summary

Sequential Attention GAN for Interactive Image Editing

Yu Cheng, Zhe Gan, Yitong Li, Jingjing Liu, Jianfeng Gao

arXiv:1812.08352v4cs.CVcs.AIstat.ML

TL;DR

Image editing relies heavily on manual effort and is time-consuming, while interactive design could follow users’ commands. SeqAttnGAN uses context from prior turns to generate edits from iterative feedback and outperforms baseline methods across two datasets on visual quality, image-text relevance, and sequence consistency.

  • Problem

    Image editing relies heavily on manual effort and is time-consuming, motivating an interactive system that follows users’ commands.

    Source: §1 · p. 1

  • Method

    SeqAttnGAN is a conditional GAN that uses context history to generate images conforming to users’ iterative feedback.

    Source: §1 · p. 2

  • Results

    SeqAttnGAN outperforms baseline methods on two datasets across visual quality, image-text relevance, and image sequence consistency.

    Source: §7 · p. 8

  • Takeaways & Limitations

    Zap-Seq and DeepFashion-Seq provide benchmarks comprising image sequences paired with free-form descriptions.

    Source: §1 · p. 2

Abstract

from arXiv · show

Most existing text-to-image synthesis tasks are static single-turn generation, based on pre-defined textual descriptions of images. To explore more practical and interactive real-life applications, we introduce a new task - Interactive Image Editing, where users can guide an agent to edit images via multi-turn textual commands on-the-fly. In each session, the agent takes a natural language description from the user as the input and modifies the image generated in the previous turn to a new design, following the user description. The main challenges in this sequential and interactive image generation task are two-fold: 1) contextual consistency between a generated image and the provided textual description; 2) step-by-step region-level modification to maintain visual consistency across the generated image sequence in each session. To address these challenges, we propose a novel Sequential Attention Generative Adversarial Net-work (SeqAttnGAN), which applies a neural state tracker to encode the previous image and the textual description in each turn of the sequence, and uses a GAN framework to generate a modified version of the image that is consistent with the preceding images and coherent with the description. To achieve better region-specific refinement, we also introduce a sequential attention mechanism into the model. To benchmark on the new task, we introduce two new datasets, Zap-Seq and DeepFashion-Seq, which contain multi-turn sessions with image-description sequences in the fashion domain. Experiments on both datasets show that the proposed SeqAttnGANmodel outperforms state-of-the-art approaches on the interactive image editing task across all evaluation metrics including visual quality, image sequence coherence, and text-image consistency.

1 INTRODUCTION

Interactive image editing frames image generation as a multi-turn exchange in which users guide revisions with natural-language commands. SeqAttnGAN and two fashion datasets are introduced for this task, and experiments report better performance than state-of-the-art techniques.

1 INTRODUCTION

  • Interactive image editing lets users iteratively refine generated images through multi-turn natural-language interactions.Users can respond to intermediate results, and the system further refines the images.Source: §1 · pp. 1–2
  • Prior approaches were restricted to keyword inputs or single-turn generation, limiting the information users could convey and the degree of interaction.The introduction also motivates interactive design tools as a way to reduce the manual effort involved in image editing.Source: §1 · pp. 1–2
  • SeqAttnGAN uses a neural state tracker and conditional GAN to incorporate prior images and descriptions, with attention and multimodal regularization for region-specific refinement.The model is trained end-to-end on full sequence sessions.Source: §1 · p. 2
  • Zap-Seq and DeepFashion-Seq provide crowdsourced image-description sequences as benchmarks for interactive editing.They contain 8,734 and 4,820 sessions, respectively.Source: §1 · p. 2
  • Experiments on Zap-Seq and DeepFashion-Seq show SeqAttnGAN outperforming state-of-the-art techniques, with reported gains in contextual relevance and consistency.Qualitative analysis and a user study also favor natural-language feedback over keywords or visual attributes.Source: §1 · p. 2

2 RELATED WORK

Prior image-generation and editing work spans varied input modalities and interactive vision-language tasks, but many editing approaches restrict user input or interaction to a single turn. This work situates multi-turn image editing among sequential vision tasks that use visual information in distinct ways.

2.1 Image Generation and Editing

  • Language-based image editing is framed as reducing labor while helping users create visual data, and related work includes systems for segmentation, colorization, and natural-language image modification.The cited systems include phrase-based segmentation and intent classification for simple language edits.Source: §2.1 · p. 2
  • AttnGAN, FashionGAN, and text-adaptive GANs support fine-grained generation or editing, but prior paradigms often restrict inputs to predefined attributes or a single turn.Earlier conditional GANs also used images, sketches, scene graphs, object layouts, or text as inputs.Source: §2.1 · p. 2

2.2 Sequential Vision Tasks

  • Sequential vision tasks include visual dialog, GuessWhat?!, and interactive image generation, but several dialog settings rely mainly on text while vision plays a complementary role.The paper distinguishes its approach by relying heavily on text to guide editing throughout each session.Source: §2.2 · p. 2
  • The paper contrasts its multi-turn image-generation datasets derived from real fashion images with ChatPainter’s continuous editing and CoDraw’s cartoon-image setting.CoDraw is based on the Abstract Scenes dataset.Source: §2.2 · p. 3

3 BENCHMARKS: ZAP-SEQ AND DEEPFASHION-SEQ

The benchmarks model editing as an iterative sequence of image revisions guided by textual feedback. Zap-Seq and DeepFashion-Seq are built from image sequences paired with crowdsourced descriptions of changes between consecutive images.

3 BENCHMARKS: ZAP-SEQ AND DEEPFASHION-SEQ

  • At each turn, users describe a desired change to the generated image, and the system modifies the prior image until the user is satisfied or the turn limit is reached.This defines the task as sequential editing rather than independent image generation.Source: §3 · p. 3
  • Existing image-generation datasets are mostly single-turn, so the authors introduce Zap-Seq and DeepFashion-Seq as sequential-editing benchmarks collected through crowdsourcing.The source image pools are UT-Zap50K and DeepFashion.Source: §3 · p. 3
  • The datasets use 3–5-image sequences with slight attribute differences, selected from UT-Zap50K and DeepFashion to approximate editing sessions.The selection emphasizes significant changes such as colors and styles before finer details such as patterns and accessories.Source: §3 · p. 3
  • Crowdsourced annotators provide free-form sentences describing differences between image pairs, with sentence prefixes available to encourage specific descriptions.The images and descriptions are assembled into interactive sequences.Source: §3 · p. 3, Figure 2 · p. 3
  • After quality filtering, Zap-Seq contains 18,497 descriptions and DeepFashion-Seq contains 12,765, with concise descriptions and diverse vocabularies.Most descriptions are 4–8 words; the datasets have 943 and 687 unique words, respectively.Source: §3 · p. 3

4 SEQUENTIAL ATTENTION GAN

SeqAttnGAN generates an image sequence from an initial image and per-turn natural-language descriptions, carrying prior context forward. Its regional attention and image-text matching regularizer support turn-by-turn refinement.

4 SEQUENTIAL ATTENTION GAN

  • SeqAttnGAN encodes an initial image and each turn’s description, then combines the current words with prior context to generate the next image.The model uses a neural state tracker and attention module to carry context and refine image subregions.Source: §4 · pp. 3–4, Figure 3 · p. 4

4.1 Neural State Tracker

  • The neural state tracker uses a GRU hidden state to absorb information from preceding turns, initialized from the input-image feature.The hidden state is the context vector passed through the sequence.Source: §4.1 · p. 4

4.2 Attention Module

  • At each turn, attention aligns current words with prior-state image regions, and the shared generator produces the image; GAN losses sum across sequence turns.All turns share the same image generator and discriminator.Source: §4.2 · pp. 4–5, §4 · p. 4

4.3 Deep Multimodal Similarity Regularizer

  • The DAMSM regularizer matches synthesized images with user sentences using word-level attention scores, supporting region-specific alignment and visual diversity.The matching loss is defined symmetrically for image-to-text and text-to-image matching.Source: §4.3 · p. 5

5 EXPERIMENTS

Experiments compare SeqAttnGAN with three baselines on Zap-Seq and DeepFashion-Seq using quantitative, qualitative, and human assessments. Results favor SeqAttnGAN on image consistency and human rankings, with a dataset-specific exception in the IS/FID comparison.

5.1 Datasets and Baselines

  • Experiments use an 85/5/10 train/validation/test split on both datasets and compare SeqAttnGAN with StackGAN, AttnGAN, and TAGAN.The evaluation also includes quantitative, qualitative, and human assessments.Source: §5.1 · p. 5, §5 · p. 5

5.2 Quantitative Evaluation

  • SeqAttnGAN slightly trails TAGAN on Zap-Seq but performs best on DeepFashion-Seq; its SSIM results also show greater consistency with ground-truth images than all baselines.The quantitative evaluation reports IS and FID for each dataset and SSIM for generated images against ground truth.Source: §5.2 · p. 5, Table 3 · p. 6
  • Examples show better regional and global edits matching ground truth and descriptions, while AttnGAN and TAGAN are less context-relevant and StackGAN fares worse on quality and sequence consistency.The examples compare generated images with ground truth and reference descriptions.Source: §5.2 · p. 6, Figure 4 · p. 6

5.3 Human Evaluation

  • SeqAttnGAN achieves the best human-study ranks for both visual quality and context relevance among the compared methods.The study asked annotators to rank generated sequences on these two dimensions independently; lower ranks indicate better results.Source: Table 4 · p. 6, §5.3 · p. 6
  • Real-user dialogues often progress from high-level attributes to fine-grained or comparative feedback, and SeqAttnGAN captures both global and subtle changes.Reported fine-grained examples include “white shoelace,” while comparative descriptions include “thinner” and “more open.”Source: §5.3 · pp. 6–7

5.4 Ablation Study

  • Removing attention or DAMSM reduces performance and produces images that diverge from previous images and do not reflect descriptions well.Figure 6 compares SeqAttnGAN with variants lacking attention or DAMSM.Source: §5.4 · p. 7, Table 5 · p. 7, Figure 6 · p. 7
  • Single feedback phrases are easiest to handle, while SeqAttnGAN also achieves good scores on composed and propositional phrases in Zap-Seq.The phrase types are grouped by text structure and evaluated with IS and FID.Source: §5.4 · p. 7, Table 6 · p. 7

6 DATA COLLECTION AND ANALYSIS

The section describes how feedback descriptions were collected and analyzes their length and structure. Most captions are concise, and many combine multiple visual-feature phrases.

6 DATA COLLECTION AND ANALYSIS

  • The section includes examples of user-system dialogue sessions and visualizations of attended image regions during editing.The dialogue examples are arranged with one interactive exchange per row; the attention figure illustrates regions attended at each step.Source: Figure 7 · p. 8, Figure 8 · p. 8
  • The collection interface elicited feedback descriptions, yielding 18,497 for Zap-Seq and 12,765 for DeepFashion-Seq after manual cleaning.The interface aimed to balance lexical flexibility with avoiding irrelevant phrases.Source: §6 · p. 8
  • Most captions contain 3–8 words, and more than half combine feedback about multiple visual-feature types.Composite captions can include spatial or structural details.Source: §6 · p. 8, Table 7 · p. 8

7 CONCLUSION

The conclusion presents interactive image editing, two sequential image-description datasets, and SeqAttnGAN. Experiments on both datasets favor the model across visual quality, image-text relevance, and image-sequence consistency.

7 CONCLUSION

  • The paper introduces interactive image editing and the Zap-Seq and DeepFashion-Seq datasets, which pair image sequences with textual descriptions.The conclusion places the task at the intersection of computer vision and language.Source: §7 · p. 8
  • SeqAttnGAN outperforms baseline methods on both datasets across visual quality, image-text relevance, and image-sequence consistency.Source: §7 · p. 8
  • Future work includes applying the model to other image types and improving consistency and natural-language interaction, including intent understanding and co-reference resolution.Source: §7 · p. 8
Loading 1812.08352v4…