Source-linked AI summary
Visual Conceptual Blending with Large-scale Language and Vision Models
Songwei Ge, Devi Parikh
TL;DR
This paper asks whether large-scale language and image-generation models can blend visual concepts, a capacity associated with human creativity. It uses prompted language models to describe a blend from an input object and text-based image models to depict it. The evaluations find that language models outperform knowledge bases for concept association and large-scale image-generation models outperform prior methods for visual depiction.
Problem
The paper asks to what extent recent large-scale language and image-generation models can perform visual conceptual blending, a capacity associated with creativity and innovation.
Source: Introduction · p. 1Method
Prompted language models identify a related object and produce a blend description from one input object, then text-based image-generation models render the description.
Source: Introduction · p. 1, Approach · pp. 2–3Results
Large-scale models significantly outperform traditional knowledge-base and image-generation baselines in the paper's concept-association and visual-depiction evaluations.
Source: Introduction · p. 1, Evaluation · pp. 3–4Takeaways & Limitations
The experiments support using an appropriate composition of recent large-scale models for visual conceptual blending.
Source: Conclusion · p. 4
Abstract
from arXiv · showhide
We ask the question: to what extent can recent large-scale language and image generation models blend visual concepts? Given an arbitrary object, we identify a relevant object and generate a single-sentence description of the blend of the two using a language model. We then generate a visual depiction of the blend using a text-based image generation model. Quantitative and qualitative evaluations demonstrate the superiority of language models over classical methods for conceptual blending, and of recent large-scale image generation models over prior models for the visual depiction.
Introduction
Conceptual blending is a human creative capacity associated with innovation, and computational systems for it have been studied for decades. This paper tests whether large-scale language and image-generation models can identify a related concept, describe a blend, and depict it visually.
Introduction
- Concept blending is presented as a capacity linked to innovation in tools, art, and science, and as a potential way for systems to help people explore combinations of concepts.The paper places this work in a research area spanning theoretical studies and creativity-support tools.Source: Introduction · p. 1
- The framework uses a language model to identify a relevant object and describe its blend with the input, then generates an image from that description.For example, it can describe “the moon sliced like an orange.”Source: Introduction · p. 1
- Large-scale models significantly outperform traditional baselines for concept association and visual blend generation.The evaluations compare language models with knowledge bases on a simile dataset and image-generation models with an existing GAN approach.Source: Introduction · p. 1
Related Work
Prior visual-blending systems include human-support workflows, sketch-based methods, description-based blending, and image-generation techniques. This paper also builds on prompt-based language-model reasoning and text-conditioned image generation.
Visual Conceptual Blending
- Earlier visual-blending approaches include user-assisted image combination, sketch concept shifts, annotated sketch blending, style transfer, deep dream, and search-and-compose.The literature also includes a roadmap and a questionnaire study of 15 hybrid animal images for assessing blend quality.Source: Related Work · pp. 1–2
Analogical Reasoning with Language Models
- Prompt-based language models can use masked-word prediction as task-specific reasoning, an approach this paper adapts to identify blend concepts and properties.Prior analogical-reasoning work also uses knowledge bases, which the authors describe as incomplete and rigid.Source: Related Work · p. 2
Deep Generative Models for Images
- Text-based image generation is relevant to visual blending because it can use a textual blend description to guide image generation.The paper discusses GAN- and VAE-based methods and describes DALL·E as modeling image vectors jointly with text embeddings.Source: Related Work · p. 2
Approach
The approach separates visual conceptual blending into reasoning and image generation. Prompted language models identify a related object and a property for a blend description, which text-guided image models then render.
Reasoning Phase
- The reasoning phase starts with one input object and identifies a related object to form a blend description, a more general setting than receiving both concepts as input.The paper illustrates this with the input moon.Source: Approach · p. 2
- Simile-inducing prompts predict a related object and then a property, which are combined into a sentence describing the blend.For moon, the example retrieves ghost and dead to produce “a moon that is dead like a ghost.”Source: Approach · p. 2
- Shape-guided prompts first identify an input object's shape, then retrieve another object with that shape and use a property to describe the blend.For moon, the example retrieves shell and smooth; other connecting properties can include speed or reflection.Source: Approach · pp. 2–3
Generation Phase
- BigSleep and DeepDaze generate images from the blend description using CLIP similarity guidance, with BigGAN and SIREN respectively.BigSleep iteratively updates a random Gaussian vector to increase the generated image's similarity to the text.Source: Approach · p. 3
Evaluation
The evaluation tests language-model concept association against a knowledge base and compares generated blend images through human judgments. Language models outperform ConceptNet on the reasoning task, while CLIP-based image models outperform DF-GAN in human preference comparisons.
Reasoning Phase
- The reasoning evaluation uses 66,442 extracted property–object pairs because the simile dataset has inconsistent language, including missing subjects and pronoun subjects.The task is narrowed to predicting a property from an object.Source: Evaluation · p. 3
- The language-model comparison includes ELMO, BERTBase, BERTLarge, and GPT, while ConceptNet relations provide knowledge-base candidates.For ConceptNet queries, qualified object phrases are reduced to their root objects; 96.34% of evaluation objects are found in the knowledge base.Source: Evaluation · p. 3
- ConceptNet has significantly lower precision than language models on the simile-dataset property-prediction evaluation.The evaluation reports whether the target property appears among the top 10, 100, or 1,000 candidates; the table covers language models and a knowledge base.Source: Evaluation · p. 3, Table 3 · p. 3
Generation Phase
- CLIP-based BigSleep and DeepDaze significantly outperform DF-GAN in human evaluations of visual blends, and BigSleep is preferred over DeepDaze.The judgments cover object recognition, blend quality, description fit, interest, and aesthetics.Source: Evaluation · p. 4, Figure 3 · p. 4
Conclusion
Large-scale language and image generation models produce simple visual blends that outperform previous methods. The conclusion points to richer prompting and blending strategies as future work, illustrated by a houseboat example.
- Experiments show that large-scale language and image generation models generate simple visual blends that are significantly better than previous methods.Source: Conclusion · p. 4
- Future work includes novel prompts to connect concepts and more complex blending strategies given the identified concepts.Source: Conclusion · p. 4
- The houseboat example combines structural relationships with two properties: accommodation from a house and being on water from a boat.Source: Conclusion · p. 4