Source-linked AI summary
Modern Evolution Strategies for Creativity: Fitting Concrete Images and Abstract Concepts
Yingtao Tian, David Ha
TL;DR
Traditional genetic algorithms and gradient-based methods have been used to make procedural drawings, motivating a revisit of evolutionary strategies for computational creativity. The paper evolves triangle placements for image and language targets, finding improvements over traditional genetic algorithms and results comparable to gradient-based methods, while producing diverse language-aligned abstractions.
Problem
The paper revisits evolutionary strategies as an alternative to gradient-based methods for computational creativity and compares them with traditional genetic algorithms and gradient-based approaches.
Source: §1 · p. 2Method
Results
Modern ES improves quality and efficiency over traditional genetic algorithms and is comparable to gradient-based methods; fitness reaches 99.62% versus 97.23% for the basic algorithm.
Source: §4 · p. 6, §6 · p. 11Takeaways & Limitations
ES produces diverse geometric abstractions aligned with human interpretations of language, and the paper suggests it could potentially transfer across creative domains with less adaptation effort.
Source: §6 · p. 11At 200 triangles, some prompts pose challenges: “a picture of Tokyo” yields a poor result, possibly because overly shadowed triangles are difficult to compose.
Source: §5 · p. 8
Abstract
from arXiv · showhide
Evolutionary algorithms have been used in the digital art scene since the 1970s. A popular application of genetic algorithms is to optimize the procedural placement of vector graphic primitives to resemble a given painting. In recent years, deep learning-based approaches have also been proposed to generate procedural drawings, which can be optimized using gradient descent. In this work, we revisit the use of evolutionary algorithms for computational creativity. We find that modern evolution strategies (ES) algorithms, when tasked with the placement of shapes, offer large improvements in both quality and efficiency compared to traditional genetic algorithms, and even comparable to gradient-based methods. We demonstrate that ES is also well suited at optimizing the placement of shapes to fit the CLIP model, and can produce diverse, distinct geometric abstractions that are aligned with human interpretation of language. Videos and demo: https://es-clip.github.io/
1 Introduction
Geometric abstraction and algorithmic art provide the backdrop for using simple shapes in computer-generated art. The paper revisits evolutionary strategies for fitting images and language prompts with triangles.
1 Introduction
- Geometric abstraction uses composed primitives, a style with roots in Cubism and connections to abstract expressionism and minimalism.The introduction situates this approach within art that abandons traditional perspective or strips forms to essential qualities.Source: §1 · pp. 1–2
- The method places transparent triangles and uses fitness to guide evolution toward a target image or a text-described concept.For text prompts, fitness is the cosine distance between CLIP embeddings of the rendered canvas and text.Source: Figure 1 · p. 1, Figure 2 · p. 2
- Algorithmic art includes genetic algorithms that iteratively apply artist-designed rules, with semi-transparent overlapping triangles used to approximate target images.These iterative methods can capture the image-creation process.Source: §1 · p. 2
- The Mona Lisa example fits a target image with PGPE and ClipUp using 50 triangles over 10,000 steps.The figure presents the target, evolved result, and evolution process.Source: Figure 3 · p. 2
- Modern evolution strategies improve quality and efficiency over traditional genetic algorithms and are comparable to differentiable rendering methods for fitting images.The method also optimizes triangle placement to fit CLIP and produces diverse geometric abstractions aligned with human interpretations of language.Source: §1 · p. 2
2 Background
Prior work uses gradients, generative models, and evolutionary algorithms for procedural drawing and language-guided image generation. The paper distinguishes its simpler triangle representation and modern evolution strategy from related approaches.
2 Background
- Procedural drawing research includes differentiable renderers optimized with gradient descent, generative models sampling from latent spaces, and language-guided drawing methods.CLIP and CLIPDraw are presented as larger-scale and procedural-sketch approaches to combining language and image generation.Source: §2 · p. 3
- The closest cited approach uses a CLIP-like model and evolutionary optimization, but parameterizes art with a neural Lindenmayer system rather than individually placed triangles.The paper characterizes the triangle representation as simpler and complementary in style to that approach [10].Source: §2 · p. 3
- PGPE estimates gradients through black-box fitness evaluations, while CMA-ES is described as more performant in practice but quadratic in parameter count per iteration.ClipUp is presented as an optimizer tailored to PGPE; its linear iteration cost supports larger parameter counts.Source: §2 · p. 3
- CLIP maps text and images into a shared embedding space used to guide GANs and differentiable renderers through gradients.Related work also hypothesizes that the generator or renderer's properties shape the resulting artistic style.Source: §2 · p. 3
3 Modern Evolution Strategies for Creativity
The method evolves configurations of transparent triangles, evaluating them against either an image target or a text prompt. It combines modern PGPE with ClipUp and uses design choices intended to stabilize fitness evaluation and preserve visible layers.
3 Modern Evolution Strategies for Creativity
- The pipeline renders candidate triangle configurations, scores fitness against an image or text target, and uses that score to guide evolution.The method uses PGPE with ClipUp to evolve better candidate configurations.Source: §3 · pp. 3–4
- Each of N triangles has 10 parameters for three vertices and RGBA color, giving 10N parameters while treating N as an upper bound on active triangles.Unwanted triangles can be made transparent, allowing the method to use fewer than N visible shapes.Source: §3 · p. 4
- Figure 4 compares fits across several targets using 10, 25, 50, or 200 triangles, each run for 10,000 steps.Its caption defines fitness as 100%−L2 Loss.Source: Figure 4 · p. 4
- Image fitting uses pixel-wise L2 loss, while concept fitting uses cosine distance between CLIP embeddings of the rendered canvas and text prompt.Because evolution is black-box optimization, neither rendering nor fitness computation must be differentiable.Source: §3 · p. 5
- Random-crop fitness averaging is used to prevent overfitting and improve stability, while noisy backgrounds model uncovered canvas regions with a max-entropy assumption.Triangle alpha is capped at 0.1 to prevent front triangles from overly shadowing those behind them.Source: §3 · p. 5
4 Fitting Concrete Target Image
For concrete images, the method fits targets with triangles; modern ES outperforms a basic evolutionary algorithm and achieves slightly higher fitness than gradient-based optimization, with a different visual style.
Fitting Concrete Target Image
- With 50 triangles and 10,000 steps, the method fits Mona Lisa with shapes capturing fine textures and large backgrounds, adjusting positions and colors coarse-to-fine.The result is described as a distinctive art style.Source: §4 · p. 5
Choice of ES Algorithm
- 99.62% vs. 97.23% fitness: PGPE with ClipUp beats mutation with simulated annealing at the same parameter budget, and 56 times more iterations do not let the basic algorithm catch up.Figure 5 compares the final results after 10,000 iterations and the basic algorithm after 56 times more iterations.Source: Figure 5 · p. 5, §4 · p. 6
Number of triangles and parameters
- PGPE scales linearly with parameters and fits 10, 25, 50, and 200 triangles (100, 250, 500, and 2000 parameters), treating triangle count as an abstractness-detail budget.The authors report that extra triangles can be used to gain fitness and let artists balance abstractness against detail.Source: §4 · p. 6
Comparison with Gradient-based Optimization
- ES achieves similar yet slightly higher fitness than the differentiable renderer, while allocating large triangles to backgrounds and small ones to detailed textures.The renderer tends to introduce textures absent from the target, especially in the background; the authors attribute the stylistic difference to placement versus transparent-color compositing.Source: Figure 6 · p. 6, §4 · p. 6
5 Fitting Abstract Concept with CLIP
With CLIP cosine-distance fitness, the method searches triangle placements for text prompts; it produces language-aligned abstractions, with triangle count serving as a detail–abstraction budget.
5 Fitting Abstract Concept with CLIP
- With 50 triangles and 2,000 steps, ES converges on abstract results aligned with human interpretations across prompts from single words to long sentences.The results do not depict a particular image, and the evolution process adjusts triangle placement.Source: §5 · p. 7
- Using 10, 25, 50, or 200 triangles gives a fitting budget that balances detail against abstraction across prompts ranging from single words to long sentences.These settings correspond to 100, 250, 500, and 2,000 parameters, respectively.Source: §5 · pp. 7–8, Figure 8 · p. 8
- The model could comfortably handle at least up to 50 triangles, but 200 sometimes posed challenges: a corporate-headquarters prompt yielded a better result, whereas a Tokyo-picture prompt yielded a poor one.The authors suggest difficulty composing overly shadowed triangles as a possible explanation and leave it for future study.Source: §5 · p. 8
- Four runs with 50 triangles produce diverse abstractions for the same prompts, offering multiple candidates for human creators to choose from.Each run uses 2,000 iterations.Source: §5 · p. 8, Figure 9 · p. 9
- The gradient baseline renders triangles, scores CLIP cosine distance, and backpropagates; with 200 triangles, both methods align prompts, while ES yields clearer shape boundaries.The comparison reports clearer abstraction and boundaries for ES than for the differentiable renderer.Source: §5 · pp. 9–10, Figure 10 · p. 10
6 Discussion and Conclusion
The paper applies modern evolution strategies to triangle-based computational art, reporting advantages over genetic algorithms and comparability to gradient methods. It presents ES as a renderer-agnostic alternative while leaving broader comparisons and art forms for future work.
6 Discussion and Conclusion
- Modern ES with triangle drawing primitives improves quality and efficiency over traditional genetic algorithms and is comparable to gradient-based methods.The paper also reports diverse geometric abstractions aligned with human interpretations of language and images.Source: §6 · p. 11
- The authors leave general comparisons between ES and gradient methods open and identify art forms beyond the explored minimalism as future work.Source: §6 · p. 11
- Unlike domain-specific gradient methods, ES is described as agnostic to how the renderer works, with potential to unify domains with less adaptation effort.Source: §6 · p. 11