Source-linked AI summary
Text-to-Image Generation Grounded by Fine-Grained User Attention
Jing Yu Koh, Jason Baldridge, Honglak Lee, Yinfei Yang
TL;DR
The paper addresses grounded image generation from detailed narratives paired with mouse traces. TRECS predicts labels, retrieves and composes trace-positioned segmentation masks, then synthesizes an image from the complete segmentation. It outperforms direct text-to-image generation in reported human evaluations and, on LN-COCO, automatic metrics.
Problem
Detailed narratives paired with mouse traces require image generation that captures both language and spatial intent.
Method
TRECS predicts object labels, retrieves semantically relevant masks, composes them according to trace order, and applies segmentation-to-image translation.
Results
TRECS is preferred over AttnGAN for realism and language alignment in human evaluations, and beats it on LN-COCO automatic metrics.
Takeaways & Limitations
TRECS demonstrates the viability of grounded text-to-image synthesis for long narratives using spatial mouse traces.
Takeaways & Limitations
The retrieved-and-composed segmentation masks differ from the gold-standard masks used to train the mask-to-image models.
Abstract
from arXiv · showhide
Localized Narratives is a dataset with detailed natural language descriptions of images paired with mouse traces that provide a sparse, fine-grained visual grounding for phrases. We propose TReCS, a sequential model that exploits this grounding to generate images. TReCS uses descriptions to retrieve segmentation masks and predict object labels aligned with mouse traces. These alignments are used to select and position masks to generate a fully covered segmentation canvas; the final image is produced by a segmentation-to-image generator using this canvas. This multi-step, retrieval-based approach outperforms existing direct text-to-image generation models on both automatic metrics and human evaluations: overall, its generated images are more photo-realistic and better match descriptions.
1. Introduction
Localized Narratives pair detailed descriptions with time-aligned mouse traces, enabling grounded text-to-image synthesis. TRECS uses these signals in a sequential retrieval-and-composition pipeline and outperforms direct generation models in reported evaluations.
- The system predicts object labels for narrative words with a BERT tagger trained from noisy narrative-to-image alignments.
- A dual encoder retrieves contextually relevant segmentation masks, selecting one per trace sequence to maximize spatial overlap.
- Selected masks are composed in trace order into separate background and foreground canvases, producing a full scene segmentation for image synthesis.
- Localized Narratives average 41.8 words, four times the length of MS-COCO captions, making detailed descriptions challenging for direct text-to-image models.
- TRECS is preferred over AttnGAN by people for realism and description alignment on LN-COCO, with 77% realism preference and 52.4% alignment preference.
- TRECS generates high-quality images aligned with language and spatial mouse traces using a sequential model.
2. The TRECS System
TRECS converts detailed narratives and mouse traces into a complete scene segmentation before synthesizing the image, addressing weaknesses in direct text-to-image generation. Its sequential pipeline refines word-level labels, retrieves spatially aligned masks, composes foreground and background regions, and translates the result into a realistic image.
- Motivation: TRECS addresses direct-model weaknesses by using detailed narratives and fine-grained traces to capture object identity, placement, scale, and composition.Existing end-to-end models were observed to capture visual gist but often lacked well-defined objects and coherent composition.
- 2.1. Sequence Labeling with Pixel Semantics: TRECS first refines noisy word-trace assignments into image-label tags using tf-idf, IBM Model 1, an HMM, and BERT auto-supervision.Convex-hull regions reduce trace noise, while constrained HMM decoding and BERT improve semantic tagging without gold per-word annotations.
- 2.2. Semantically Aligned Mask Retrieval: A dual encoder retrieves narrative-relevant training images, then selects one class-matching mask per traced instance by maximizing mIOU with its trace hull.This retrieval step supplies contextually appropriate masks while preserving the spatial placement indicated by mouse traces.
- 2.3. Segmentation Composition: Retrieved masks are composed in trace order with separate foreground and background canvases, placing thing masks over stuff masks to form a complete segmentation.The ordering addresses the tendency of larger stuff regions to occlude identifiable object regions.
- 2.4. Mask-to-Image Translation: The first three stages create a description- and trace-consistent scene segmentation, which a mask-to-image model converts into a photo-realistic image.Segmentation-to-image generation is more constrained than free-form text-to-image generation and can produce high-fidelity images from full masks.
3. Evaluation Metrics
The evaluation combines human pairwise judgments with automated distributional and image-language metrics. Human protocols separately assess photo-realism and alignment to the conditioning narrative, while IS and FID quantify image quality and similarity to real images.
- Image Quality: Human image-quality evaluation presents two model outputs for the same narrative and asks five annotators to choose the more photo-realistic image.Randomized presentation and majority voting reduce variation in pairwise judgments.
- Image Quality: Inception Score rewards generated images that contain clear objects and exhibit prediction diversity, with higher values indicating better performance on these properties.The metric uses ImageNet-pretrained Inception predictions over generated images.
- Image Quality: Fréchet Inception Distance compares fitted Inception-feature distributions for real and generated images, with lower scores indicating greater similarity.FID improves on IS by directly comparing real and generated examples.
- Image-Text Alignment: Image-language alignment can be measured indirectly through captioning-model BLEU, METEOR, and CIDEr scores, but this approach relies on two generative models and automated text similarity.The paper therefore also uses direct human comparisons conditioned on the original narrative.
4. Experimental Results
TRECS is evaluated against fine-tuned AttnGAN on LN-COCO and LN-OpenImages using automatic metrics, human judgments, and ablations. It generally produces more realistic and better-aligned images, although automatic image-quality metrics favor different models across datasets.
- 4.1. Main Results: 77.0% of LN-COCO human comparisons preferred TRECS images for quality, with the same preference trend on LN-OpenImages; unanimous decisions favored TRECS 88.3% to 11.7% and 84.1% to 15.9%, respectively.The evaluations compared models fine-tuned on LN-COCO training data.
- 4.1. Main Results: TRECS composes scene masks from traced object placements before mask-to-image synthesis, explicitly imposing object shapes and using strong segmentation-to-image models.This staged design is intended to handle longer, detailed narratives and improve object coherence compared with direct generation.
- 4.1. Main Results: Automatic image-quality metrics reversed leadership across datasets: TRECS was 3.1 FID points better on LN-COCO but 5.3 FID points worse on LN-OpenImages.Fine-tuning AttnGAN on LN-COCO improved its FID from 59.4 to 51.8.
- 4.1. Main Results: 52.4% of LN-COCO alignment judgments preferred TRECS to 32.0% for AttnGAN, while LN-OpenImages results were closer at 45.8% versus 40.5%.Among unanimous decisions, TRECS was selected 71.0% versus 27.8% on LN-COCO and 61.4% versus 37.9% on LN-OpenImages.
- 4.2. Ablations: Auto-supervised BERT training improved image quality from 20.0/49.9 or 20.7/49.0 to 21.3/48.7, while k=5 retrieved masks produced the best LN-COCO images.CC-FPSE generally outperformed SPADE on IS/FID and was used for the primary results; fine-tuning CC-FPSE on noisy retrieved masks produced no notable improvement.
5. Conclusion
TRECS produces higher-quality images that better align with descriptions in human evaluations, supporting more natural user control of scene size and placement.
- Human evaluations find TRECS produces higher-quality images that better align with descriptions than AttnGAN.