Source-linked AI summary
Generative Art Using Neural Visual Grammars and Dual Encoders
Chrisantha Fernando, S. M. Ali Eslami, Jean-Baptiste Alayrac, Piotr Mirowski, Dylan Banarse, Simon Osindero
TL;DR
The paper explores how parts of artistic production can be automated while leaving other aspects to artists. It evolves hierarchical visual grammars from text prompts and scores images with a pretrained image-text dual encoder; the system produces varied structured images, but reuses existing representational conventions.
Problem
Artistic production is only partly formalizable, leaving open which parts of the process can be automated and which remain the artist’s responsibility.
Source: §1 · p. 3Method
The system evolves hierarchical visual grammars from linguistic prompts and uses a pretrained multimodal image-text critic to score generated images.
Source: §1 · pp. 4, 6, §2.3 · p. 11Results
The hierarchical image-generation system evolved with an image-text dual encoder critic produces a wide variety of structured images.
Source: §1 · p. 6Takeaways & Limitations
The system automates image production from titles while leaving users to choose generative processes and prompts and to decide what to do with the resulting images.
Source: §1 · p. 5, §4 · p. 17The system can only rehash existing representational conventions, and its multimodal critic may reflect internet-image biases that make some image classes harder to score.
Source: §4 · p. 20
Abstract
from arXiv · showhide
Whilst there are perhaps only a few scientific methods, there seem to be almost as many artistic methods as there are artists. Artistic processes appear to inhabit the highest order of open-endedness. To begin to understand some of the processes of art making it is helpful to try to automate them even partially. In this paper, a novel algorithm for producing generative art is described which allows a user to input a text string, and which in a creative response to this string, outputs an image which interprets that string. It does so by evolving images using a hierarchical neural Lindenmeyer system, and evaluating these images along the way using an image text dual encoder trained on billions of images and their associated text from the internet. In doing so we have access to and control over an instance of an artistic process, allowing analysis of which aspects of the artistic process become the task of the algorithm, and which elements remain the responsibility of the artist.
1 Introduction
Generative art uses a specifiable formal process, while human artists often retain critical evaluation; this paper shifts some representational choices to an evolutionary image generator and its text-image critic. The system evolves visual grammars from linguistic prompts, using an image-text dual encoder to score outputs.
1 Introduction
- Generative art involves a specifiable formal system, but production need not be wholly formal or digital; human artists often remain the critical evaluators.The paper frames artistic production as partly algorithmic, with motivation and judgments of artistic worth still difficult to formalize.Source: §1 · pp. 1, 3
- Unlike GAN workflows centered on curated datasets, the approach uses linguistic prompts and an image-text critic; artists can steer aesthetics by choosing algorithms and constraints.The authors distinguish evolutionary search from backpropagation as allowing greater human control over output aesthetics.Source: §1 · p. 4
- The critic and generator make some representational decisions that artists may not know or understand, including cultural references, and the critic does not explain its choices.The paper presents this outsourcing of representation as novel and notes concerns about decisions an artist might find objectionable.Source: §1 · p. 5
- The system evolves visual grammars rather than images directly, using stochastic search instead of gradient backpropagation and a pretrained multimodal critic to score image-text resemblance.The resulting image reflects both what the generator can readily produce and what the critic scores highly.Source: §1 · pp. 4, 6
- A hierarchical image-generation system evolved with an image-text dual encoder critic produces a wide variety of structured images.The authors describe the system as a neural Lindenmayer system and report examples evolved from scratch within 100,000 binary tournaments.Source: §1 · p. 6
2 Methods
The method evolves genotypes through text-conditioned binary tournaments, using a dual encoder to score generated images against prompts. Its hierarchical visual grammar builds images from strokes and objects, with separate levels able to specify what is drawn and where it appears.
2.1 The overall evolutionary framework
- Each of 100,000 binary tournaments scores two generated images against the text prompt, then mutates the fitter genotype and replaces the lower-fitness one.Genotypes encode both an input string and recurrent-network parameters.Source: §2.1 · p. 6, Figure 4 · p. 9
2.2 Generative and mutational mechanisms
- The system tests several image representations, finding smooth closed Bézier shapes with flood fills more effective than geometric primitives, Tangram-like grids, or individual pixels.Pixel-based evolution could yield adversarial images that scored highly but did not resemble the prompt to human observers; random projective transformations were applied during evaluations.Source: §2.2 · p. 7, Figure 2 · p. 7
- The single-layer generator feeds evolved input vectors to LSTMs that produce strokes, whose lines encode position, color, thickness, and angle.Vectors specify stroke origin, length, and LSTM choice; the generator is ballistic and does not refer to the image while producing it.Source: §2.2 · pp. 7–10
2.3 Deep Neural Image Grammar
- The hierarchical grammar lets top-level vectors specify objects while intermediate vectors specify strokes, with placement controlled independently at either level.Variable-length strings and mutations can add or remove objects, strokes, and lines; lower-level outputs can include Bézier curves.Source: §2.3 · pp. 11–12, Figure 6 · p. 12
2.4 The Image Text Dual Encoder Critic
- A dual encoder embeds images and text separately, then scores their cosine similarity to evaluate how closely an image matches its prompt.The authors used an ALIGN-trained model and note that they did not systematically compare pretrained dual encoders.Source: §2.4 · pp. 12–13
3 Results
Evolution can raise an image’s fitness while its resemblance to a prompt becomes less apparent, and the same prompt can lead to different visual attractors. The hierarchical visual grammar produces clearer, structured images and can repeat objects across locations.
3.1 General principles in Image Evolution
- For “scream,” fitness continued to improve even as the image began to resemble the title less to a human observer.The authors suggest this may reflect pathological overfitting; random projective transformations were then applied during genotype evaluation.Source: §3.1 · pp. 13–14, Figure 7 · p. 14
- Rerunning evolution for the same prompt produced different visual attractors, including varied depictions of “tiger in a jungle.”In a single run for a dog-and-parrot prompt, depictions competing for prominence focused on different aspects of the title.Source: §3.1 · pp. 14–15, Figure 8 · p. 15, Figure 9 · p. 16
- Small changes to a generative procedure altered image style and feel; the authors found pleasing images easier to search for with simpler programs.Figure 10 varies the number of LSTMs in the single-level system; added complexity could obscure ordered visual structure.Source: §3.1 · p. 15, Figure 10 · p. 17
- Pruning can identify marks that contribute to fitness, including the apple’s stem and red top, as well as two purple marks not obviously related to the apple.The authors generally did not use pruning during evolution because it is costly; asynchronous evolution can reduce genomes when smaller ones are useful.Source: §3.1 · pp. 15–16
3.2 Evolution of Visual Grammars
- Hierarchical visual grammar images became clearer and more structured, and the system could repeat similar objects at different locations.Only 224×224 versions were evaluated by the dual encoder, although the generative process could produce high-resolution images.Source: §3.2 · pp. 16–17
4 Discussion
The discussion presents the visual grammar and multimodal transformer as a way to evolve title-conditioned images, while emphasizing the system’s representational limits, biases, and the artist’s continuing role.
4 Discussion
- The system couples an evolvable visual grammar with a multimodal transformer to produce simple abstracted representations from image titles, and is particularly effective at color.The authors identify sensitivity to the image being produced as an obvious next step for the generator.Source: §4 · p. 17
- Unlike CPPNs, the recurrent system is less efficient but appears better at rapidly evolving images toward a specified target, with hierarchical object priors proposed as an explanation.The authors also report that evolving background color can trap search in poor local optima, while initial selection for high fitness can reduce diversity through trivial solutions such as curves over lines.Source: §4 · pp. 17–19
- The authors characterize the system as “painting from memory”: its strokes and color mixing evoke painting, while its image associations reflect internet pairings of images and words.They contrast GANs’ finer-grained interrogation of a discriminator with “photography from memory.”Source: §4 · p. 19
- The system reuses existing language-to-image conventions rather than inventing new representational conventions; the authors reserve that possibility for reinforcement learning with behaving agents.They allow that stylistic novelty, such as new ways of shading or depicting a face, could arise, while calling for further investigation of its limits.Source: §4 · p. 20
- The authors suggest the process could aid artists with mark making and task briefs, and could be extended to evolve existing images or generate images from rendered 3D models.These are proposed possibilities, not reported evaluations of those uses.Source: §4 · p. 20
- The multimodal transformer can reproduce internet biases: self-portraits tend toward white men, and rarer classes may score worse and produce worse images on average [35].The authors note that this may make designers less likely to attempt rare classes.Source: §4 · p. 20
- The artist describes prompt experiments that shifted from seeking human-like images to accepting unexpected outputs, while selection and later arrangement remained personal decisions.For “Jungle in the Tiger,” the artist reports taking 16 consecutive independent evolutionary runs without censorship, then later reconsidering the resulting work.Source: §4 · pp. 21–22