Combining LLMs and Genetic Search for ARC-AGI-2

1 Introduction

• ARC-AGI-2 tests abstract reasoning through visual grid puzzles whose rules must be inferred from examples and applied to a test input.The benchmark is normally easy for humans but extremely difficult for machines because its tasks are open-ended.Source: §1 · p. 1
• LLMs have limited attempts to generate and debug programs, while random GA seeds lack information about relevant operations or parameters.The proposed approach uses the LLM to suggest starting programs and the GA to search around them.Source: §1 · p. 1
• The LLM writes DSL programs to seed the GA, then leaves the process while the GA and Machine execute and score candidates across generations.The loop continues until it finds a solution or times out.Source: §1 · pp. 1–2, Figure 1 · p. 2
• The Machine uses fixed grid channels, registers, objects, and arguments; mutation and crossover may make bad programs but cannot create broken syntax or runtime errors.The Machine is a small virtual machine for running the programs.Source: §1 · pp. 1–2
• 2/60 tasks are solved by the initial LLM programs, and genetic search solves 4 additional tasks, bringing the total to 6/60; unseeded search solves none.The evaluation covers 60 public tasks.Source: §1 · p. 2

2 Related Work

• SOAR uses repeated evolutionary rounds with LLM involvement, while HYSYNTH, ConceptSearch, and ABPR use other LLM-guided search or debugging approaches [3] [4] [5] [6].SOAR also fine-tunes its model between rounds.Source: §2 · p. 2
• Here, the LLM generates initial programs once and leaves; the GA evolves fixed-width DSL programs that cannot break their structure or cause runtime errors.The approach combines the LLM’s ability to produce good but imperfect solutions with the GA’s search over similar candidates.Source: §2 · p. 2
• LLM conversations and genetic algorithms can both be run in parallel, with the latter reusing the same GPUs.Source: §2 · p. 2

3.1 Overview

• Each task is parsed, seeded with LLM-generated programs, evolved against training examples, and tested using the first program that matches every training example.The search never sees the correct test output; a task counts as solved only if the same program also produces that output.Source: §3.1 · p. 3

3.2 The Machine and program format

• The Machine stores grid channels, up to eight objects, and 176 registers; its 20-gene DSL restricts expressivity but keeps mutations executable.Fixed-range fields and operation-specific handling prevent syntax errors and runtime errors, while the restricted language expresses fewer ideas than Python.Source: §3.2 · p. 3

3.3 Preparing the ARC-2 task for the LLM

• Deterministic perception converts each grid into color masks and connected objects, then fills the object list and registers used by the LLM and Machine.The LLM receives the same object and register names used by the Machine, so generated programs can access precalculated values such as an object's bounding box.Source: §3.3 · p. 4

3.4 LLM proposal stage

• The LLM is guided through planning, operation selection, parameter generation, and complete DSL programs, with valid unique proposals and joined fragments retained.The final chromosome request is made twice; neighboring fragments may be joined in order up to the 20-gene limit.Source: §3.4 · p. 4

3.5 Genetic search

• Two independent islands of 4,000 programs each use 40% LLM seeds; tournament selection, crossover, mutation, and elitism evolve candidates by training-pixel fitness.Search stops at the first exact training fit, 200 generations, or 600 seconds; score 1 means every training output is reproduced.Source: §3.5 · pp. 4–5

3.6 ARC-2 contest pipeline and compute

• The contest pipeline caps LLM generation at 60% of the 12-hour runtime, then runs seeded tasks on available GPUs and reserves time for genetic search.Development used one 4-GB consumer GPU, while Kaggle tests used two T4 GPUs.Source: §3.6 · p. 5

4.1 Protocol

• The evaluation used the first 60 tasks by task ID, searched using training examples, and counted a task as solved only when the test grid was entirely correct.The random-seed comparison was also run, but its configuration changed during development.Source: §4.1 · pp. 5–6

4.2 Component outcomes

• The solve count rose from two original LLM solutions to six after genetic search, adding four correct programs.Table 3 compares the original LLM pool with results after search on the same 60 tasks.Source: Table 3 · p. 6, §4.2 · p. 6
• Nine programs exactly fit the training samples, but only six also produced correct test outputs and counted as solved.Table 4 lists the exact training fits; a task counts as solved only when the test output is correct.Source: §4.2 · p. 6, Table 4 · p. 6

4.3 Random seeded control

• Random-seeded controls found no solutions, but the control configuration changed during development and was not a clean controlled experiment.The three Table 5 runs were illustrative, used core operations, and were not the complete control dataset; a stronger future test would hold all 41 operations and other settings fixed across all 60 tasks.Source: §4.3 · pp. 6–7, §4.1 · p. 6, Table 5 · p. 7

4.4 What evolution does

• The four additional correct programs appeared after 22, 35, 50, and 80 generations; some tasks improved in jumps after flat periods, while others improved gradually.Resource-limited runs may stop while scores are still rising; low flat scores may reflect limits in what the Machine can express, while high flat scores may reflect a local optimum.Source: §4.4 · p. 7

5.1 Why the LLM+GA combination works

• Fitness curves for three solved and three unsolved tasks illustrate different convergence behaviors.Source: Figure 3 · p. 8
• The six solved tasks show that an LLM can provide useful starting programs that genetic search turns into additional correct solutions, but do not establish an optimal design.Source: §5.1 · p. 8

5.2 Future work and research questions

• The 41-operation DSL is one design choice; future work asks whether a better set can match LLM reasoning while remaining small and regular enough for genetic search.Source: §5.2 · p. 8
• Because Qwen3.5-4B was not trained or tuned for the Machine yet sometimes produced one-try solutions, the authors suggest designing a Machine that is easier for an LLM to use correctly.Source: §5.2 · p. 8
• The current GPU executor handles 8,000 programs in parallel, with 20 genes per program and fixed 30×30 grids; more efficient kernels or memory layouts could support larger states, longer programs, or more operations.Source: §5.2 · p. 8

5.3 Limitations

• Evaluation used the first 60 tasks by ID, with the first 20 used during development; incomplete random-seed control also prevents exact attribution of improvement to LLM seeding.Source: §5.3 · p. 9

• The LLM provides a useful starting point, while the Machine keeps programs valid and runnable so the GA can search many candidates.Source: §6 · p. 9
• The GA can test about 1.6 million program instances for one ARC-2 task without another LLM call.Source: §6 · p. 9
• The authors describe the result as preliminary and hope the technique will be useful to researchers exploring program solution spaces.Source: §6 · p. 9

• Table 6 records the LLM inference configuration used to generate the program pool.Source: A LLM Inference Configuration · p. 10, Table 6 · p. 10

• The DSL has 41 operations: 27 core operations and 14 LLM-gated operations.Source: Table 7 · p. 11