PaperlayerView exact revision on arXiv

Source-linked AI summary

Inspiration Seeds: Learning Non-Literal Visual Combinations for Generative Exploration

Kfir Goldberg, Elad Richardson, Yael Vinker

arXiv:2602.08615v3cs.CV

TL;DR

Generative image models often execute detailed textual ideas rather than support the vague, visual exploration that precedes ideation. Inspiration Seeds generates diverse combinations from two images using CLIP-SAE-based visual decomposition, and produces non-trivial, coherent relationships that outperform literal-composition baselines.

  • Problem

    Text-driven image generators offer limited support for the early creative phase when ideas are vague, intuitive, and primarily visual.

    Source: §1 · pp. 1–2

  • Method

    Inspiration Seeds uses CLIP Sparse Autoencoders to decompose images into visual aspects, then trains an image-conditioned model on synthetic triplets to recombine two inputs without text.

    Source: §1 · p. 2, §4.2 · p. 5

  • Results

    The method generates varied, visually coherent non-trivial combinations that reveal deeper relationships and outperform models favoring literal composition.

    Source: §1 · p. 2, §5.1 · p. 5, §5.3 · p. 10

  • Takeaways & Limitations

    The framework supports visual ideation by surfacing alternative connections between references before users know or articulate what they want to create.

    Source: §1 · p. 2

    The method supports only two inputs, offers limited control, can produce trivial or dominated combinations, and takes approximately 30 seconds per image.

    Source: §6 · p. 10

Abstract

from arXiv · show

While generative models have become powerful tools for image synthesis, they are typically optimized for executing carefully crafted textual prompts, offering limited support for the open-ended visual exploration that often precedes idea formation. In contrast, designers frequently draw inspiration from loosely connected visual references, seeking emergent connections that spark new ideas. We propose Inspiration Seeds, a generative framework that shifts image generation from final execution to exploratory ideation. Given two input images, our model produces diverse, visually coherent compositions that reveal latent relationships between inputs, without relying on user-specified text prompts. Our approach is feed-forward, trained on synthetic triplets of decomposed visual aspects derived entirely through visual means: we use CLIP Sparse Autoencoders to extract editing directions in CLIP latent space and isolate concept pairs. By removing the reliance on language and enabling fast, intuitive recombination, our method supports visual ideation at the early and ambiguous stages of creative work.

1 Introduction

Inspiration Seeds reframes generative image synthesis as visual exploration for vague, early-stage ideas. It combines two images into diverse, non-trivial compositions using visually derived decompositions rather than textual instructions.

  • Designers often discover ideas by noticing unexpected relationships among visual references, but recognizing hidden qualities can require substantial experience.Source: §1 · p. 1
  • Existing text-to-image and editing models mainly execute articulated prompts, leaving limited support for intuitive, primarily visual exploration.Source: §1 · pp. 1–2
  • Inspiration Seeds takes two images and generates multiple combinations that surface difficult-to-verbalize relationships between their visual qualities.Source: §1 · p. 2
  • Unlike trivial object insertion, the method transfers deeper qualities such as a leaf’s decay pattern, green tones, and aged appearance onto another subject.Source: §1 · p. 2, Figure 3 · p. 2
  • CLIP Sparse Autoencoders extract visual factors, cluster them into complementary aspects, and provide a fast, text-free basis for synthetic training data.Source: §1 · p. 2
  • The trained model produces varied, visually coherent non-trivial combinations across image pairs and outperforms models favoring literal composition.Source: §1 · p. 2

2 Related Work

Related work spans tools for organizing visual references, controllable image generation, concept decomposition, and visually inspired generation. These approaches generally rely on predefined operations, textual supervision, or costly optimization rather than open-ended visual recombination.

Design and Modeling Inspiration

  • Ideation tools help users organize and compare curated visual material, but primarily navigate existing examples rather than generate new combinations.Source: §2 · p. 3

Image Generation and Personalization

  • Modern text-to-image and personalization methods provide detailed control or reproduce known concepts, but assume ideas can be articulated textually and are not designed for open-ended recombination.Source: §2 · p. 3

Concept Decomposition

  • Image decomposition methods typically follow fixed axes such as objects, attributes, or style-content separation, limiting discovery of new visual factors.Source: §2 · p. 3
  • InspirationTree discovers unexpected attributes but requires multiple concept images, textual inversion, and hours of per-concept optimization, making it inefficient and unstable at scale.Source: §2 · p. 3
  • CLIP Sparse Autoencoders offer a related alternative by decomposing representations into sparse visual factors in a single forward pass without per-image optimization.Source: §2 · p. 3

Visually Inspired Generation

  • Prior visually inspired generation methods support exploration through selection, interpretation, or embedding-based operations, yet typically depend on predefined composition rules.Source: §2 · p. 3

3 Preliminaries

The paper uses Sparse Autoencoders to expose interpretable factors in CLIP representations and a pretrained image-generation model for multimodal synthesis and editing. The decomposition pipeline turns sparse CLIP features into opposing visual directions and decoded image aspects.

Sparse Autoencoders for CLIP

  • Sparse Autoencoders address CLIP representation polysemanticity by learning overcomplete, sparse factors whose decoder columns represent learned feature directions.Source: §3 · pp. 3–4
  • The decomposition pipeline encodes an image with CLIP, retains top-k SAE activations, decodes them, clusters the resulting vectors, and computes an editing direction between cluster centroids.Source: Figure 4 · p. 4
  • Moving the CLIP embedding in opposite directions along this axis and decoding with Kandinsky yields two images emphasizing distinct aspects of the source.Source: Figure 4 · p. 4, §3 · p. 4

Sparse Autoencoders for CLIP

  • In CLIP space, the SAE maps 1280-dimensional embeddings into a sparse 163k-dimensional representation whose features correspond to visual concepts.Source: §3 · p. 4

FLUX.1 Kontext

  • FLUX.1 Kontext is a rectified-flow model that unifies image generation and editing by processing concatenated image and text token sequences.Source: §3 · p. 4

4 Method

Inspiration Seeds learns to generate visual combinations by reversing image decomposition: rich images are split into aspects, then used to supervise a two-image composition model. The pipeline uses CLIP SAE directions for decomposition and fine-tunes Flux.1 Kontext to produce varied combinations from image pairs.

4 Method

  • The method trains fθ(IA, IB) → Icomb by decomposing visually rich images, using each original image as ground-truth supervision for recombination.This inversion avoids manually constructing large collections of non-obvious concept-pair triplets.Source: §4 · p. 4

4.1 Image Pool Construction

  • The image pool combines multi-attribute prompts with vague prompts expanded into diverse interpretations, producing images containing separable yet non-trivial visual aspects.The two strategies respectively improve reliable decomposability and visual diversity.Source: §4.1 · p. 4

4.2 Image Decomposition via CLIP SAEs

  • CLIP SAE decomposition clusters decoded directions from the top 32 activated features into two groups, retaining the 50% nearest to each centroid for coherence.The resulting editing direction is the difference between cluster centroids, and opposite movements of the original embedding produce two edited embeddings.Source: §4.2 · p. 5
  • The edited CLIP embeddings are converted into aspect images with Kandinsky, using λ=1 for final dataset construction because the aspects are well-separated while maintaining image quality.Interpolation along the editing direction controls the decomposition strength.Source: Figure 5 · p. 4, §4.2 · p. 5

4.3 Training

  • The final pool contains 2085 decomposed triplets, and Flux.1 Kontext is fine-tuned with LoRA to generate combinations from two images placed on a shared canvas.At inference, varying the random seed produces multiple combinations without additional user guidance.Source: §4.3 · p. 5, Figure 6 · p. 5

5 Experiments

Inspiration Seeds supports visual exploration through varied, iterative combinations of image pairs. Across qualitative and quantitative evaluations, it produces non-trivial combinations and decompositions that preserve relationships to inputs while separating visual aspects.

5.1 Visual Exploration Results

  • Different references and random seeds produce distinct transformations, while iterative recombination accumulates visual qualities from multiple sources.Examples include crystalline, fabric-like, organic, and architectural cupcake transformations, plus portraits combining botanical, bioluminescent, or fungal qualities.Source: §5.1 · pp. 5, 8, Figure 7 · p. 6
  • The exploration canvas organizes pairings and seed variations into browsable branches rather than converging on one correct output.This workflow lets users compare alternatives and continue from promising combinations.Source: Figure 8 · p. 7, §5.1 · p. 8

5.2 Evaluating Visual Combinations

  • Our method integrates visual cues from both inputs into coherent forms, whereas baselines commonly copy layouts, insert objects, or produce other trivial combinations.Examples include beetle patterns transferred from mushrooms, bubble-like sponge-jellyfish forms, and shared materiality between portrait skin and sculpture fabric.Source: §5.2 · pp. 8–9, Figure 9 · p. 8
  • Our method elicits longer reconstruction descriptions than baselines, and description length increases from duplication or insertion toward texture transfer and other relationships.The evaluation uses VLM-generated word counts as a proxy for combination complexity; Table 1 also reports trivial-pattern percentages.Source: §5.2 · p. 9, Table 1 · p. 9, Figure 10 · p. 9
  • A 35-participant study supports description length as a proxy for visual-combination complexity, with more complex relationships requiring longer explanations.Participants classified outputs by relationship type, including duplication, insertion, texture or structure transfer, other relationships, and unrelated images.Source: §5.2 · p. 9, Figure 11 · p. 9

5.3 Decomposition Results

  • Our decomposition achieves the highest harmonic mean on 915 images while keeping both components related to the input and distinct from each other.Compared with feed-forward T2I and I2I baselines, it produces meaningful decompositions similar to InspirationTree without its costly optimization and multiple-image requirement.Source: §5.3 · pp. 9–10, Figure 12 · p. 10

6 Limitations

The method has limitations in combination reliability, input scale, user control, and generation speed.

6 Limitations

  • Combinations can fail when one input dominates, one spatial structure persists across seeds, or the result remains a trivial insertion.These failure modes constrain the reliability of non-trivial visual recombination.Source: §6 · p. 10
  • The current system supports only two input images and offers limited control over how much each input contributes.The authors identify multi-image composition and continuous contribution control as directions for richer interaction.Source: §6 · p. 10
  • Generation takes approximately 30 seconds per image, limiting the immediacy of the interactive experience.The authors identify near-real-time generation as a practical improvement target.Source: §6 · p. 10

7 Discussion and Conclusions

Inspiration Seeds reframes generative models as tools for exploratory visual ideation, generating non-trivial combinations from image pairs without predefined relationship categories. Its description-complexity evaluation indicates richer integration of visual aspects than other methods.

  • Inspiration Seeds generates non-trivial visual combinations from pairs of images for exploratory visual work.Source: §7 · p. 10
  • CLIP SAE-based decomposition automatically generates training data without predefined relationship categories, supporting combinations beyond fixed transformations such as style transfer or object insertion.Source: §7 · p. 10
  • The method’s description-complexity evaluation finds combinations requiring richer descriptions than those generated by other methods, indicating deeper integration of visual aspects.Source: §7 · p. 10

Supplementary Material

The supplementary material presents an interactive exploration canvas in which users select image pairs, inspect generated combinations, and continue branching from promising results. The workflow supports organizing references and outputs into an evolving visual exploration space.

A Visual Exploration Interface

B.1 Training Details

The implementation conditions generation on two images arranged on a fixed canvas and uses a fixed prompt to remove textual bias. Training uses LoRA fine-tuning, while decomposition retains top activation features and cluster representatives.

B.1 Training Details

  • Two 512 × 512 input images are placed in opposite corners of a 1024 × 1024 canvas and combined with a fixed prompt during generation.Source: §B.1 · p. 13
  • LoRA fine-tuning uses rank 32 for linear layers, rank 16 for convolutional layers, learning rate 10−4, batch size 1, and 15,000 steps.Source: §B.1 · p. 13
  • Training takes approximately 24 hours on one NVIDIA L40s GPU, while generation takes approximately 34 seconds per image on the same hardware.Source: §B.1 · p. 14

B.2 Decomposition Details

  • The decomposition pipeline retains the top 32 SAE features and keeps the 50% of vectors in each cluster closest to its centroid.Source: §B.2 · p. 14, Figure 14 · p. 14

C Dataset Construction

The dataset combines templated and LLM-expanded prompts to create images containing multiple visual properties and varied conceptual content for decomposition. The supplementary comparison material reports all four seeds per method for each input pair.

C Dataset Construction

C.1 Templated Prompts

  • Ontology-based prompts sample object categories, motif families, materials, colors, styles, and environments to create images with multiple structured visual properties.Source: §C.1 · p. 14
  • Product-design prompts combine one of 16 product subjects with two inspirations from distinct categories, interleaving identity, form, purpose, palette, and inspiration phrases.Source: §C.1 · p. 14

C.2 LLM-Expanded Prompts

  • LLM-expanded prompts begin with vague concepts, then use Gemini to produce longer prompts with three concrete stylistic interpretations for image generation.Source: §C.2 · p. 14

D.1 Comparison with Baselines

  • The complete baseline comparison shows all four seeds generated per method for each input pair, beyond the single outputs shown in the main paper.Source: §D.1 · p. 14

D.2 Decomposition Results

The section compares decomposition and composition approaches, including an SAE-based decomposition pipeline and CLIP-space interpolation. CLIP interpolation tends to produce blended or incoherent results.

D.2 Decomposition Results

  • Additional decomposition examples compare the SAE-based approach with T2I and I2I baselines.Source: §D.2 · p. 15

E Comparison with CLIP Space Interpolation

F Description Complexity Evaluation

The evaluation measures how difficult outputs are to describe from their two inputs and documents a user study for classifying input-output relationships. Figures compare methods across description complexity, generation, and decomposition results.

F Description Complexity Evaluation

G User Study Details

  • The study recruited 35 participants who completed 25 trials, classifying each output alongside its two input images using five relationship options, including near-duplicates, insertions, texture transfers, and other relationships.Source: G User Study Details · p. 16
  • The study sampled 25 outputs across description lengths, including 11 from the method and 7 each from Nano Banana and Qwen-Image-2511.Source: G User Study Details · p. 16
  • The comparison includes CLIP interpolation outputs from four random seeds and decomposition results contrasting the proposed approach with T2I and I2I baselines.Source: Figure 22 · p. 21, Figure 30 · p. 29

F Description Complexity Evaluation

Related papers

Loading 2602.08615v3…