PaperlayerView exact revision on arXiv

Source-linked AI summary

EvoMaestro: Toward Interpretable and Steerable LLM-Driven Program Evolution

Feng Liang, Sizhe Cheng, Yikai Li, Ruijie He, Xiaolin Wen, Yong Wang

arXiv:2610.03721v1cs.HCcs.MA

TL;DR

LLM-driven program-evolution systems leave experts with limited support for understanding growing populations or steering search. EvoMaestro combines a steerable framework with visual analytics, informed by a formative study. A 12-participant study found improved understanding, reduced cognitive workload, and support for expert steering.

  • Problem

    Automated program-evolution systems provide limited support for experts to understand growing populations and steer the search.

    Source: §1 · p. 2

  • Method

    The authors derived six design requirements from a formative study with 8 experts, then built a steerable framework and visual analytics interface for multi-level inspection and intervention.

    Source: §1 · p. 2, §5 · p. 5

  • Results

    In a within-subjects study with 12 participants, EvoMaestro improved understanding, reduced cognitive workload, and supported expert steering.

    Source: §1 · p. 2, §9 · p. 11

  • Takeaways & Limitations

    Preserving expert agency in open-ended evolution requires both informed judgment and a way for that judgment to shape subsequent automation.

    Source: §8.1 · p. 9

    LLM-generated explanations may be plausible but incorrect, and experts may miss errors, especially outside their areas of expertise.

    Source: §8.2 · p. 10

Abstract

from arXiv · show

Recent program evolution systems use large language models (LLMs) to generate and iteratively improve populations of programs, producing striking advances in mathematics, algorithm design, and scientific computing. Yet these systems largely operate as fully automated black boxes. As populations grow, domain experts must make sense of the scores, code changes, reasoning, and algorithmic ideas across many programs, while current interfaces provide limited support for understanding or redirecting the evolution. We characterize this need to understand and steer populations of evolving algorithmic ideas as semantic oversight. A formative study with 8 domain experts yields six design requirements for this emerging human-computer interaction problem. We then propose a steerable program evolution framework that lets expert judgments shape subsequent evolution. Built on this framework, EvoMaestro is an interactive visual analytics system that organizes evolution information from population overview to source code. It helps experts locate and compare noteworthy programs, guide evolution with natural language, combine promising ideas, and prune unproductive directions. A seven-day system demonstration illustrates how an expert applied these capabilities throughout a long-running evolution process. A within-subjects study with 12 participants shows that EvoMaestro improves users' understanding of evolution processes, reduces cognitive workload, and supports expert steering. These findings suggest that preserving expert agency in open-ended LLM-driven search requires support for both informed judgment and the ability to shape subsequent automation.

1 Introduction

LLM-driven program evolution can yield advances across research fields, but its automated interfaces leave experts struggling to understand growing populations and redirect search. EvoMaestro combines a steerable framework with visual analytics, informed by a formative study and evaluated with users.

1 Introduction

  • LLM-driven evolution uses populations of programs that are iteratively modified and scored, producing advances in mathematics, algorithms, and scientific computing.The LLM reasons about selected programs and proposes modifications; evaluation scores feed back into the population.Source: §1 · p. 2
  • Experts face two linked challenges: understanding a growing tree of variants and steering evolution through interventions such as redirecting, combining, or pruning lineages.Existing interfaces do not effectively surface rich metadata, and current systems lack effective channels for human input during a run.Source: §1 · p. 2
  • A formative study with 8 domain experts yielded six design requirements for an interactive system addressing program-evolution oversight.The paper frames the need to understand and steer evolving algorithmic ideas as an emerging HCI problem.Source: §1 · p. 2
  • EvoMaestro combines a steerable evolution framework with coordinated visual views that disclose information from population overviews to line-level code annotations.The framework permits expert interventions and exposes population structure, performance, and strategies for interpretation.Source: §1 · p. 2
  • In a within-subjects study with 12 participants, EvoMaestro improved understanding, reduced cognitive workload, and supported expert steering.The paper also demonstrates its use on a real-world optimization task.Source: §1 · p. 2

2 Related Work

Prior work establishes LLM-driven program evolution and visualizes evolutionary processes, but interfaces for these systems provide limited real-time steering. Human oversight of open-ended algorithmic ideas requires experts to interpret and act on generated strategies.

2.1 LLM-Driven Program Evolution

  • LLM-driven evolution systems use LLMs to generate code variants, with examples reporting mathematical, scientific, and algorithmic discovery; open-source tools have broadened access.The approach applies where candidate solutions are executable programs and can be evaluated automatically with a sufficiently reliable computable metric.Source: §2.1 · pp. 2–3

2.2 Visualization for Evolutionary Algorithms

  • Prior evolutionary-visualization work covers population dynamics and fitness landscapes, while existing LLM-evolution interfaces are largely post-hoc dashboards rather than real-time steering tools.LLM-generated edits carry reasoning and summaries and have structurally complex code diffs, unlike the solution spaces in many traditional evolutionary systems.Source: §2.2 · p. 3

2.3 Human Oversight of Iterative Program Generation

  • Human-AI programming research often centers on one assistant or artifact, whereas program evolution requires experts to curate persistent alternatives and compare strategies across lineages.Open-ended generated ideas can fall outside predefined categories, so experts may need to describe suggestions and directions in natural language.Source: §2.3 · p. 3
  • Semantic oversight combines understanding generated strategies with checking them against code and evaluation results, then deciding which lineages to preserve, combine, prune, or redirect.The paper positions this oversight as population-level attention routing, multi-level inspection, and intervention that shapes future generations.Source: §2.3 · p. 3

3 Background: LLM-Driven Program Evolution

LLM-driven program evolution repeats a four-stage loop over a population of candidate programs until a termination condition is met. The process is fully automatic in existing systems.

3 Background: LLM-Driven Program Evolution

  • A population database stores candidate programs as evolution-tree nodes with source code and metadata, and some systems partition the population into islands.Islands evolve independently and may exchange nodes through migration.Source: §3 · p. 3
  • Parent selection probabilistically chooses one or more population nodes, typically favoring higher-scoring candidates for further refinement.Selected nodes serve as starting points for the next generation.Source: §3 · p. 3
  • Node generation prompts an LLM to modify selected parents through mutation or crossover, producing a patch that creates a new program.Mutation uses one parent; crossover combines ideas from multiple parents, and patches may include a natural-language intent description.Source: §3 · p. 3
  • Node evaluation executes each new program, assigns user-defined domain-specific scores, and returns the evaluated node and metadata to the population.The loop continues until a time budget, target score, or maximum node count is reached.Source: §3 · p. 3

4 Formative Study

The formative study examined how experts inspect and steer LLM-driven optimization, using ShinkaEvolve as a baseline and interviews with domain experts. Findings yielded six requirements spanning information overload and human intervention.

4 Formative Study

  • The formative study examined experts’ approaches to iterative algorithm improvement, information needs, and desired interventions, using ShinkaEvolve’s web UI as a baseline.The baseline was selected as one of the advanced open-source LLM-driven program-evolution tools.Source: §4 · p. 4

4.1 Participants and Procedures

  • Eight participants with algorithm-design or optimization experience explored a 50-node ShinkaEvolve run in think-aloud interviews lasting about 50–60 minutes.Participants came from universities and companies; prior evolutionary-algorithm experience was not required.Source: §4.1 · p. 4

4.2 Findings and Design Requirements

  • Thematic analysis distilled six design requirements into two groups: information overload (DR1–DR4) and lack of human-in-the-loop interaction (DR5–DR6).Participants felt overwhelmed by interface information yet lacked what they needed to understand strategies, promising nodes, and search progress.Source: §4.2 · p. 4, §4.2.1 · p. 4

4.2.1 Problem 1: Information Overload.

  • DR1 calls for immediately visible, scannable explanations of each node’s algorithmic idea rather than code and scores alone.Participants described patch descriptions as useful when found, but often buried; one expert copied code into ChatGPT to understand its idea.Source: §4.2.1 · p. 4
  • DR2 calls for comparison across multiple levels, including islands, side-by-side code diffs, and strategy-level clustering.Participants found code comparisons tedious and parent-child comparisons insufficient when scores were similar.Source: §4.2.1 · p. 4
  • DR3 calls for population-level summaries, while DR4 calls for filtering and prioritizing decision-relevant nodes.Suggested cues included performance thresholds, novelty metrics, and salient performance jumps.Source: §4.2.1 · p. 4

4.2.2 Problem 2: Lack of Human-in-the-Loop Interaction.

  • DR5 calls for targeted interventions such as guiding mutations, combining ideas, and choosing which lineages continue evolving.Participants also wanted to track nodes of interest and prevent selected nodes from evolving to save resources.Source: §4.2.2 · pp. 4–5
  • DR6 calls for flexible pacing between passive monitoring, throttled observation, and step-by-step control, with shifts as situations demand.Participants rejected a binary choice between full automation and manual control.Source: §4.2.2 · p. 5

5 EvoMaestro

EvoMaestro adds expert interventions to the automatic evolution loop and organizes population-to-code views so experts can understand and steer evolving programs. Its interface supports inspecting strategies, comparing nodes, and acting on promising or unproductive directions.

5.1 Steerable Program Evolution

  • Experts can guide mutations with Suggest, combine nodes with Merge, ban unproductive parents, and control the evolution pace.Suggest and Merge create child nodes; Ban removes a node from future parent selection, while pacing can be automatic, paused between generations, or one node at a time.Source: §5.1 · p. 5
  • Reasoning-based embeddings compare algorithmic intent rather than code syntax, supporting clustering and dissimilarity views; experts can toggle to code-based embeddings.The embeddings use each node’s patch description and reasoning chain.Source: §5.1 · p. 5
  • A fifth review-prioritization stage flags noteworthy nodes, by default those improving at least 15% over their parent, without changing selection or scores.Experts can adjust the threshold or use other signals, such as embedding dissimilarity.Source: §5.1 · p. 5

5.2 EvoMaestro Interface

  • The interface follows a progressive workflow from population overview to node inspection and comparison, then to evolution steering.Population-level orientation precedes inspecting specific nodes and acting on those insights.Source: §5.2 · p. 5

5.2.1 Evolution Understanding (DR3, DR4).

  • Population views encode generation, island, performance, lineage, and strategy similarity, while clustering and score-distribution views reveal population patterns.The radial tree highlights strong nodes; clusters use reasoning-based embeddings, and score thresholds can dim low-scoring nodes.Source: §5.2.1 · pp. 5–6, Figure 2 · p. 6
  • Node inspection progresses from hover triage and lineage tracing to strategy explanations, line-level code annotations, and side-by-side comparison.Experts can inspect scores, patch descriptions, and metadata, then compare performance and strategy dissimilarity across selected nodes.Source: §5.2.3 · p. 6, §5.2.1 · p. 6

5.2.3 Evolution Steering (DR5, DR6).

  • A unified node menu lets experts guide or merge nodes, ban unproductive lineages, and control the evolution pace.Suggest and Merge create child nodes in real time; Ban removes a node from future parent selection.Source: §5.2.3 · pp. 6–7

6 System Demonstration

In a seven-day demonstration, one domain expert used EvoMaestro to understand, inspect, and steer a long-running program evolution. He reported shifting from code review toward active human-AI collaboration while steering without writing code.

System Demonstration

  • The expert repeated a cycle of understanding the population, inspecting nodes, intervening with domain knowledge, and resuming automatic evolution.Before resuming, he typically banned unproductive nodes and set a node budget, then reviewed results on return.Source: §6 · p. 7
  • After 92 automatic nodes, the expert identified two top strategies and found earlier time-bounded search loops had largely failed.The Evolution Overview indicated that successful nodes used adaptive attempt budgets.Source: §6 · p. 7
  • Inspection found the best node, N-91, had a longest runtime of 20s, far below the limit and leaving significant computation time unused; its search logic used precomputation, pruning, and steal mechanisms.Although N-91 had similar scores to three other nodes, it was distinct from them.Source: §6 · p. 7
  • The expert suggested using time up to the timeout, producing N-94 at 44.5k, then merged complementary approaches to produce N-99 at 45.0k.The merge combined deterministic waterfilling in N-65 with N-91.Source: §6 · p. 7
  • Over seven days, evolution produced more than 200 nodes, with best score 45.8k; the expert steered entirely through EvoMaestro without writing code.The expert used the system for approximately 16 hours and described a shift from tedious code review toward active human-AI collaboration.Source: §6 · p. 7

7 User Study

In a within-subjects comparison with 12 participants, EvoMaestro scored higher than the baseline on all eight shared questionnaire items and reduced cognitive workload. Participants also described progressive disclosure and steering as useful, while noting that formulating effective guidance remained challenging.

7.1 Study Setup

  • The 12 participants completed a counterbalanced within-subjects comparison of EvoMaestro and ShinkaEvolve on two optimization tasks.The study assessed understanding, workload, and steering support rather than final program quality.Source: §7.1 · p. 8, §7 · p. 8

7.2.1 Closed-ended Questions.

  • EvoMaestro significantly outperformed the baseline on all eight shared Likert items, including understandability and usability.Understandability medians were 6.0 versus 5.0, and usability medians were 7.0 versus 5.0.Source: §7.2.1 · p. 9
  • Maestro Chat received a median rating of 7.0, while intervention-item medians ranged from 6.0 to 6.5.Identifying intervention needs scored slightly higher than conducting interventions, indicating that formulating effective guidance remained more challenging.Source: §7.2.1 · p. 9
  • Overall NASA-TLX workload was lower with EvoMaestro than the baseline (median 3.08 vs. 4.25; p=0.004, δ=−0.417).Frustration showed the largest subscale improvement (p=0.004, δ=−0.535); performance and effort did not differ.Source: §7.2.1 · p. 9

7.2.2 Qualitative Findings.

  • Participants said progressive disclosure eased overload by revealing population-level orientation before node-level details on demand.They contrasted this with the baseline’s presentation of all data at once, which some found overwhelming.Source: §7.2.2 · p. 9
  • Participants valued steering and used recommendations as starting points and learning aids, while also requesting richer automation support.Requested capabilities included global recommended actions and guidance for groups of related nodes.Source: §7.2.2 · p. 9
  • Participants with different backgrounds found distinct paths through the system: code explanation via Maestro Chat or hands-on learning through steering.One participant found Chat verbose and did not use it, but still found the system accessible through Merge.Source: §7.2.2 · p. 9

8 Discussion

The discussion argues that expert agency in open-ended LLM search requires both informed judgment and a way to shape subsequent automation. It also identifies limits in the demonstration, generated explanations, scalability, and transfer to other settings.

8.1 Lessons Learned

  • Experts need to route attention, compare strategies, and inspect code, diffs, logs, and results; scores alone do not explain whether a candidate’s strategy makes sense.The discussion describes these steps as semantic oversight and notes that progressive disclosure alone is insufficient.Source: §8.1 · p. 9
  • In the evaluation, information organization, abnormal-node location, and node comparison were among EvoMaestro’s strongest improvements, while overall cognitive workload decreased.The authors suggest that LLM-generated explanations can reduce the cognitive cost of oversight.Source: §8.1 · p. 10
  • Visible intervention effects let experts inspect population responses, supporting trust and letting domain knowledge shape later generations without constant human control.Participants linked intervention capability with trust in the system’s output.Source: §8.1 · p. 10
  • Task-level alignment requires ongoing inspection, correction, and monitoring because initial prompts and evaluators may not anticipate every concern that emerges during search.Understanding helps experts verify whether strategies remain consistent with intent; steering lets them redirect divergent strategies.Source: §8.1 · p. 10

8.2 Limitations and Future Directions

  • The seven-day system demonstration illustrates sustained steering, but its co-author participant’s prior involvement and motivation may limit how well the usage patterns generalize.The authors call for case studies with independent domain experts.Source: §8.2 · p. 10
  • Generated explanations may be plausible yet incorrect, and experts may miss errors, especially outside their expertise.The authors suggest stronger provenance and cross-model checking and validation.Source: §8.2 · p. 10
  • Evaluation covered 50-node trees and a demonstration of approximately 200 nodes, so support for thousands of nodes remains unestablished.Filtering helped at the tested scales; larger populations may require hierarchical aggregation, collapsed lineages, or focus-and-context techniques.Source: §8.2 · p. 10
  • Future work includes group steering and pacing controls, evaluation in other fields, and transfer to iterative multi-agent systems; other fields may require different representations and interventions.The authors also identify connecting domain concepts, algorithms, code, and evaluation results as an HCI challenge.Source: §8.2 · pp. 10–11

9 Conclusion

The conclusion presents EvoMaestro as a system for expert understanding and steering of LLM-driven program evolution. The included study and design details describe its formative process, radial overview, and merge recommendations.

9 Conclusion

  • In a within-subjects study with 12 participants, EvoMaestro received significantly higher ratings on all shared understandability and usability items and reduced cognitive workload.A seven-day demonstration illustrated sustained expert steering across more than 200 nodes.Source: §9 · p. 11

A.1 Participants

  • All formative-study participants used LLM-based programming tools daily; none had implemented or published EA-related work.EA familiarity was self-reported, with “None” indicating no prior exposure and “Basic” indicating familiarity without implementation experience.Source: Table 1 · p. 12

A.2 Procedure

  • The formative sessions used a seven-stage protocol, including exploration of a roughly 50-node ShinkaEvolve run, follow-up questions, and two design probes.The probes presented overload-reduction visualizations and steering mechanisms.Source: §A.2 · p. 12

B.1 Radial Layout Design

  • The radial overview maps generation to radius, islands to angular sectors, and lineage to edges; a top-down alternative became difficult to read above approximately 50 nodes.In the radial view, exact angular positions are layout-driven and carry no data meaning.Source: §B.1 · p. 13

B.2 Merge Recommendation

  • Merge recommendations rank remaining candidates using evaluation quality and dissimilarity, with adjustable weighting; experts inspect candidates and choose a partner.The system presents the five highest-ranked candidates, but the ranking does not determine whether two strategies are semantically complementary.Source: §B.2 · p. 13

C Task Details: OXC Routing Optimization

The appendix section provides an overview and full mathematical formulation of the OXC routing optimization task used in Section 6.

C Task Details: OXC Routing Optimization

C.1 Task Overview

The task evolves programs for joint OXC topology configuration and traffic routing in a large AI computing cluster. Its formulation specifies a partitioned multi-plane network and balances congestion against reconfiguration cost.

C.1 Task Overview

  • Programs jointly optimize OXC topology configuration and traffic routing in a large AI computing cluster, with each evaluation limited to 50 seconds.The task balances peak link congestion and physical reconfiguration cost across a complex search space.Source: §C.1 · p. 13

C.2 Mathematical Formulation

  • The network has N groups, M OXCs, S spines and L leaves per group, and is partitioned into P independent planes with K parallel spine–OXC links.Cross-plane links are prohibited, and the total number of ports per OXC is denoted R.Source: §C.2 · pp. 13–14
  • Valid real-time routes symmetrically traverse LeafA, SpineA, an OXC, SpineB, and LeafB.A specific OXC port is mapped using its group, spine, and link identifiers.Source: §C.2 · p. 14
  • The formulation minimizes both peak active flows on any physical link and the edit distance between successive OXC port-connection states.The second objective represents the time penalty of physically rotating micro-mirrors.Source: §C.2 · p. 14

D.1 Participants

  • Table 2 profiles evaluation-study participants by self-reported LLM-evolution experience; all were daily users of LLM-based programming tools.The experience categories range from none to expert.Source: Table 2 · p. 14

D.2 Tasks

The evaluation used two multi-objective optimization tasks and counterbalanced interface and task assignments. Participants completed questionnaires, workload ratings, open-ended questions, and structured sensemaking and intervention activities.

D.2 Tasks

  • The drone task balances route length, noise near schools, and signal degradation, while the graph-routing task balances end-to-end latency and energy use.Both tasks involve non-trivial trade-offs, domain reasoning, and diverse algorithmic strategies; each began with about 50 nodes across six islands.Source: §D.2 · p. 14

D.3 Counterbalancing

  • A 2 × 2 Latin square counterbalanced the assignment of conditions and tasks.Table 3 presents the counterbalanced assignments.Source: §D.3 · p. 14, Table 3 · p. 14

D.4.1 Custom Likert Questionnaire (7-point scale).

  • The 7-point questionnaire shared understandability and usability items across conditions, with additional Maestro Chat and intervenability items for EvoMaestro.Understandability covered population evolution, strategies, locating notable nodes, comparing solutions, and identifying node algorithms.Source: §D.4.1 · pp. 14–15

D.4.2 NASA-TLX.

  • NASA-TLX workload ratings were collected after each condition across mental, physical, and temporal demand, performance, effort, and frustration.Ratings used a 7-point scale, with higher values indicating greater workload except for the performance item, where higher meant poorer perceived success.Source: §D.4.2 · p. 15

D.4.3 Open-ended Questions.

  • Open-ended prompts and a post-study interview asked about node selection, visual cues, algorithm understanding, intervention rationale, tool preferences, and perceived control.The interview also gathered feature-level feedback and reflections on how each interface shaped understanding.Source: §D.4.3 · p. 15, §D.4.4 · p. 15

D.5 Procedure

  • Each approximately 80-minute session included tutorials, training, 25-minute task periods, questionnaires, a break, a second condition, and a post-study interview.Task activities included identifying promising nodes and strategies, and, in EvoMaestro, issuing natural-language suggestions and multi-parent merges.Source: §D.5 · p. 15

D.6 Detailed Results

The detailed-results materials describe paired comparisons of shared Likert items and NASA-TLX scores, including scale direction and effect-size thresholds.

D.6 Detailed Results

  • Table 4 reports paired comparisons of shared 7-point Likert items, with higher scores better and all differences significant at p < 0.05.Effect sizes are classified as large at |δ| ≥ 0.474.Source: Table 4 · p. 16
  • The supplied legend labels “Medium effect” and “Large effect” as distinct effect-size categories, but does not specify a numeric threshold between them.Source: §D.6 · p. 16
  • Table 5 reports NASA-TLX results on a 7-point scale, where lower scores are better and significant differences are marked with an asterisk.Its Performance subscale reflects participants’ perceived success on the study task, not generated-program performance.Source: Table 5 · p. 16
Loading 2610.03721v1…