Source-linked AI summary
Understanding the Challenges in Iterative Generative Optimization with LLMs
Allen Nie, Xavier Daull, Zhiyi Kuang, Abhinav Akkiraju, Anish Chaudhuri, Max Piasevoli, Ryan Rong, YuCheng Yuan, Prerit Choudhary, Shannon Xiao, Rasool Fakoor, Adith Swaminathan, Ching-An Cheng
TL;DR
Generative optimization remains difficult to adopt because engineers must make hidden choices when configuring learning loops. The paper studies starting artifacts, credit horizons, and experience batching across ML-agent pipelines, Atari, and prompt optimization, finding that no universal setup works across tasks. These results motivate practical, task-sensitive guidance while leaving other optimizer degrees of freedom outside the ablation.
Problem
Only 9% of surveyed agentic systems used automated design, motivating investigation of the hidden learning-loop choices that make generative optimization difficult to productionize.
Method
The paper studies starting artifacts, credit horizons, and experience batching through case studies in ML-agent pipelines, Atari game-playing, and prompt optimization.
Results
Across three domains, no single universal learning-loop recipe worked across all tasks; starting artifacts also produced substantially different ML-pipeline outcomes.
Takeaways & Limitations
Learning-loop design choices should be treated as explicit, task-dependent engineering decisions rather than implementation details.
Takeaways & Limitations
The study does not ablate other degrees of freedom, including the feedback oracle, optimizer LLM, or specific optimization procedures.
Abstract
from arXiv · showhide
Generative optimization uses large language models (LLMs) to iteratively improve artifacts (such as code, workflows or prompts) using execution feedback. It is a promising approach to building self-improving agents, yet in practice remains brittle: despite active research, only 9% of surveyed agents used any automated optimization. We argue that this brittleness arises because, to set up a learning loop, an engineer must make ``hidden'' design choices: What can the optimizer edit and what is the "right" learning evidence to provide at each update? We investigate three factors that affect most applications: the starting artifact, the credit horizon for execution traces, and batching trials and errors into learning evidence. Through case studies in MLAgentBench, Atari, and BigBench Extra Hard, we find that these design decisions can determine whether generative optimization succeeds, yet they are rarely made explicit in prior work. Different starting artifacts determine which solutions are reachable in MLAgentBench, truncated traces can still improve Atari agents, and larger minibatches do not monotonically improve generalization on BBEH. We conclude that the lack of a simple, universal way to set up learning loops across domains is a major hurdle for productionization and adoption. We provide practical guidance for making these choices.
1 Introduction
LLM-based generative optimization can iteratively improve programs and LLM systems from feedback, but remains rarely adopted in production. The paper argues that hidden learning-loop design choices, including initialization and learning evidence, help explain this gap.
- LLMs are used to repeatedly modify programs or other LLM systems against objectives such as accuracy, latency, or success rate.
- Only 9% of surveyed agentic systems employed any automated design, including simple LLM-assisted prompt tuning.
- Existing agentic libraries provide optimization mechanisms, so low adoption is not primarily attributed to missing infrastructure or software abstractions.
- The paper hypothesizes that adoption is limited by the hidden difficulty and engineering burden of configuring effective learning loops.
- The analysis focuses on starting artifacts, credit horizons, and experience batching as learning-loop decisions that parallel established machine-learning design problems.
2 Building a Learning Loop
A generative-optimization learning loop requires an initial system, an optimizer, execution feedback, and a designed learning context. The paper isolates three choices—editable starting artifacts, trace horizon, and experience batching—and finds that effective settings are task-dependent.
- A learning loop begins with an initial system that maps inputs to outputs and an oracle that supplies optimization feedback.
- The learning context can include inputs, outputs, feedback, the current system, task background, and prior successes or failures.
- Starting Artifact: Starting artifacts include code, prompts, files, documentation, and design sketches, while engineers also choose which system components the optimizer may change.
- Credit Horizon: The credit horizon determines how many execution steps enter the optimizer’s context, especially when systems receive both short-term and long-term feedback.
- Experience Batching: The optimal number of independent experience examples varies by task, and the best configurations for all three problems are task-dependent.
3 Related Work
Prior work studies agent loops, context engineering, memory, and optimization libraries, but generally emphasizes successful applications rather than the design choices that make learning loops unstable or difficult.
- Learning Loops: Agent loops support self-debugging, self-correction, and self-refinement within individual task executions, unlike the paper’s focus on optimizing systems across iterations.
- Context Engineering: Context-engineering work often manages or compresses information, while this paper examines learning-context trace horizons and the number of independent traces.
- Agentic Libraries: Optimization libraries implement learning loops through candidate selection methods such as cross-validation and Pareto optimization.
- Agentic Libraries: These libraries primarily showcase successful applications rather than investigating the design choices and instabilities that complicate implementation.
4 ML Agent Case Study for the Starting Artifact Problem
The ML-agent case study shows that initialization and parameter constraints materially affect the systems reachable through generative optimization. Functionally equivalent documentation can yield different average and best-case pipeline performance across tasks.
- Setup: The study compares single-function and many-function initializations for generating automated ML training pipelines.
- Setup: The two initializations provide equivalent documentation information but differ in pipeline modularization.
- Experiment: The optimizer uses validation performance as its objective, selects the best checkpoint after 20 optimization steps, and evaluates predictions on Kaggle’s hidden test set.
- Results: 11.5%-22.4% was the average gap between the learned ML pipeline and ResearchAgent across the two tasks, while the best learned model surpassed 86.6% of human submissions.
- Results: On Spaceship Titanic, the many-function initialization surpassed 86.6% of submissions versus 72.7% for one-function; on Housing Price, the ordering reversed at 75.6% versus 54.6%.
- Takeaways: Different initial systems produced measurably different average and best-case performance across five runs.
5 Atari Game Case Study for the Credit Horizon Problem
The Atari case study tests whether optimizers should learn from immediate rewards or longer execution traces. Across games, the better credit horizon depends on the task and its feedback structure.
- Task and setup: Atari provides a controlled multi-step setting for comparing immediate-reward updates with full-episode credit horizons.The task uses object-centric state representations and asks an LLM optimizer to revise a Python game-playing program.
- Task and setup: The study compares one-step traces containing one observation, action, and immediate reward with multi-step full-rollout traces.Both conditions use the same starting artifacts, five trials per game, and 30 optimization iterations.
- Results: Multi-step optimization outperforms one-step optimization in 4 of 8 games, while one-step optimization performs better in the other 4.Multi-step wins on Pong, Breakout, Space Invaders, and Asterix; one-step wins on Freeway, Enduro, Q*bert, and Seaquest.
- Interpretation: Longer traces help when effective play depends on delayed consequences, whereas frequent short-horizon updates can suit tasks with more immediate feedback.Space Invaders benefits from coordinating shooting and movement, while Freeway can benefit from frequent updates.
- Efficiency: Generative optimization achieves competitive Atari scores with substantially less wall-clock time than traditional deep RL despite using one environment instance instead of ten.The efficiency comparison uses normalized scores and contrasts LLM optimization with PPO and DQN baselines.
6 BigBench Extra Hard Case Study for the Experience Batching Problem
The BBEH case study examines how many independent execution experiences should be presented to an LLM optimizer per update. Batch size affects learning dynamics and generalization in task-specific ways rather than yielding one universally best setting.
- Problem and setup: The study treats the number of execution traces per update as an experience-batching decision analogous to minibatching in stochastic gradient descent.Each trace contains an input, output, and feedback signal for optimizing prompted LLM systems.
- Learning dynamics: Larger batches often smooth or accelerate early learning but can plateau earlier and do not reliably produce the best final test performance.Smaller batches are noisier yet sometimes continue improving for longer.
- Problem and setup: The experiment compares batch sizes of 1, 3, and 5 examples per optimizer update on BBEH prompted LLM systems.The optimizer jointly updates the prompt and answer-extraction code and evaluates on held-out test examples.
- Results: The optimal batch size is task-dependent, and larger batches do not monotonically improve generalization.Batch size 1 is best for Disambiguation QA and Movie Recommendation; batch size 3 for Geometric Shapes, Linguini, and Boolean Expressions; batch size 5 for Dyck Languages and Causal Understanding.
- Cross-framework check: The best batch-size choice varies across frameworks as well as tasks, indicating that the trade-off is not merely specific to one optimizer implementation.LangGraph favors batch size 5 on all three shared tasks, while DSPy favors different batch sizes across tasks.
7 Conclusion and Discussion
The paper identifies starting artifacts, credit horizons, and experience batching as learning-loop choices that materially influence generative optimization outcomes across three domains. Because other factors were not ablated, the study isolates these design choices rather than establishing a universal recipe.
- Three learning-loop decisions—starting artifact, credit horizon, and experience batching—critically influence optimization outcomes across ML pipelines, Atari programs, and BBEH prompt optimization.The paper reports no single universal recipe that works across all tasks.
- The experiments span ML agent pipelines, Atari game-playing programs, and prompt optimization on BBEH, but find no universal setup across these domains.
- The study does not ablate feedback-oracle design, optimizer-LLM choice, or specific optimization procedures.It instead isolates learning-loop design choices often treated as implementation details.
- The authors conjecture that sustained research may yield robust defaults for starting artifacts and learning-context structure across tasks and agent designs.This is presented as a future possibility rather than an established result.
A.1 Experimental Setup
The MLAgentBench setup isolates how the editable starting artifact affects optimization by comparing monolithic and modular workflows under the same outer protocol. Validation-based, staged natural-language feedback guides repeated pipeline revisions while held-out test submissions remain reserved for final evaluation.
- Two initialization schemes expose either one full pipeline function or separate preprocessing, feature-selection, training, and prediction components to the optimizer.Both receive the same task description and high-level implementation hints; their decomposition differs.
- The many-function design gives the optimizer a structured hypothesis class and makes editable pipeline components explicit.Components include preprocessing, feature selection, ensemble construction, training, and prediction.
- Staged task-specific feedback provides more direction than a bare validation score while retaining room for nontrivial revisions.Feedback templates are defined separately for Spaceship Titanic and Housing Price.
- Validation metrics drive optimization, while Kaggle test submissions are reserved for final external evaluation.The train-validation split is created outside the agent.
- The outer protocol is identical across initializations: OptoPrime runs 20 steps with memory size 5, computes validation metrics, and converts them into natural-language feedback.The principal difference is the editable program exposed to the optimizer.
- Meta-overfitting occurs when successive workflow revisions improve the immediate validation-driven objective while producing brittle pipelines with weaker generalization.Each step trains a new model from scratch; the changing behavior is in the optimizer’s code revisions.
B.1 LLM Optimizer Experimental Setup
The Atari experiments use object-centric observations and game-specific program decompositions to study credit horizons under staged feedback. One-step traces update after immediate rewards, whereas longer rollouts trade delayed-consequence information against context use and update frequency.
- The Atari setup uses frame skipping, sticky-action probability 0.0, optimizer memory size 5, and Claude Sonnet-3.5 through OptoPrime.
- The learned Atari agent receives structured object dictionaries from OCAtari rather than raw pixels across eight games.This representation exposes semantically meaningful state variables to the optimizer.
- The initial workflows decompose control differently: Pong predicts trajectories, Breakout combines trajectory prediction with paddle targeting, and Space Invaders separates movement from firing.These decompositions reflect distinct game-control demands.
- Staged game-specific feedback guides the optimizer toward qualitatively better behavior rather than reporting score alone.The design resembles reward shaping without requiring constant human per-step feedback.
- One-step credit uses a single action and immediate reward, whereas multi-step credit supplies a longer rollout before proposing a revision.Representative rollout lengths are 400 steps for Pong, 300 for Breakout, and 25 for Space Invaders.
- Longer traces reveal delayed consequences but consume more context and reduce update frequency.Pong and Breakout offer more informative local geometric rewards, while Space Invaders requires longer-term coordination.
- Deep-RL comparison reports the time until each run first reaches its highest score rather than total 10M-step training time.This accounts for regressions in unstable deep-RL performance.
B.6 Representative Learned Artifact Examples
The learned Atari artifacts contain compact, game-specific strategies rather than merely local code cleanups. Representative revisions become more anticipatory in Pong, target higher-value structures in Breakout, and coordinate movement, firing, and threat response in Space Invaders.
- The examples compare representative initial code, learned revisions, runtime and score statistics, and higher-level strategies across the Atari artifacts.
- Pong: Pong revisions add distance- and momentum-aware interception beyond local action thresholds.
- Breakout: Breakout revisions target tunnels and higher-value bricks, encoding where the return should send the ball next.
- Space Invaders: Space Invaders revisions combine action constraints with threat-aware positioning and coordinate firing with movement.
C.1 Experimental Setup
The BBEH study evaluates iterative optimization of a compact two-part agent across eight language-understanding and reasoning tasks. It varies the number of independent execution traces shown to the optimizer while holding the total update budget fixed.
- Benchmark and tasks: The study covers eight BBEH tasks spanning logical, spatial, language, recommendation, rule-based, and causal reasoning.The task set includes Dyck Languages, Boolean Expressions, Geometric Shapes, Linguini, Disambiguation QA, Movie Recommendation, Boardgame QA, and Causal Understanding.
- Batching procedure: At each update, the optimizer receives concatenated traces from k randomly sampled training examples, with k ∈ {1, 3, 5}.Each trace contains the question, prompt, raw response, extracted answer, and correctness feedback.
- Agent configuration: The agent has two trainable components: a prompt template and an answer-extraction function.The prompt is revised across updates, while answer extraction converts the raw response into the evaluator’s expected exact format.
- Feedback and evaluation: Feedback combines a binary correctness score with a short message that either confirms success or reveals the expected answer and requests revision.The evaluator uses option-token matching for multiple-choice tasks and exact string matching otherwise.
- Experimental budget: The total update budget is fixed at 15 across batch sizes, requiring 1, 3, or 5 epochs for batch sizes 1, 3, or 5.Thus, batch-size comparisons change both the number of examples per context and the number of passes over the training set.
C.5 Task-Specific Observations
The batching effect is strongly task-dependent: larger batches can smooth early learning, but they do not consistently produce the best final test performance. The broader learning-loop analysis frames this result as a consequence of how optimizer-facing contexts are constructed.
- Task-Specific Observations: Larger batches often produce smoother early learning, but they do not consistently yield the best final test performance.Smaller batches are noisier yet sometimes continue improving for longer.
- Task-Specific Observations: The study’s batch-aware optimization setup builds feedback contexts from minibatch executions rather than exposing the optimizer to a single example.This makes batching part of the learning-context design rather than merely an implementation detail.
- Task-Specific Observations: The compared implementations optimize trainable instructions or graph functions while using the same answer-matching rule as the main experiments.Multiple-choice answers are matched by option token, and other answers by exact match.
- Task-Specific Observations: The framework connects these formal objects to engineering choices about modularization, optimizable components, and feedback supplied to the optimizer.It focuses on optimization-step learning rather than explicit memory systems or test-time parameter training.
- Task-Specific Observations: The learning-loop formalism separates a parameterized system, its execution trace, and the feedback returned by a trace oracle.In OPTO, the trace oracle returns a computational graph involving the parameters together with feedback attached to the output node.
D.2 Workflow Graphs and Learning Graphs
A workflow graph represents one concrete execution, while a learning graph combines one or more such experiences through a task-appropriate template. The template determines the optimizer’s context and must match whether examples are independent, interactive, or temporally linked.
- Workflow Graphs: A workflow graph records one execution of Wθ, including its input, intermediate computation, output, feedback, and parameter dependencies.The parameterized workflow Wθ is distinct from gi, the concrete graph induced by running it on xi.
- Workflow Graphs: Workflow graphs can represent modular systems such as retrieval followed by an LLM response, with separate parameters controlling each component.This makes graph structure and parameter interfaces part of the editable workflow.
- Learning Graphs: A learning graph Glearn is constructed from one or more workflow graphs through a fixed template before being shown to the optimizer.The template specifies how individual experiences are combined into the optimizer-facing object.
- Learning Templates: Interactive learning uses one workflow graph per update, whereas batch learning aggregates independent examples into one learning graph.The batch template attaches aggregated feedback to the combined output, with concatenation represented by ⊕.
- Learning Templates: Episodic learning links workflow graphs through environment transitions because one execution influences the next observation.This differs from batch learning, where examples are independent and merely aggregated.
- Design Implications: Choosing the wrong learning template can create order sensitivity or objective mismatch when optimization should generalize across data or account for delayed consequences.The credit horizon is the number of steps included when a template captures long-term workflow effects.
- Design Implications: The starting artifact includes workflow structure and specifications, which determine the search space available to the optimizer.Changing one monolithic function differs from changing coordinated components with explicit interfaces.
- Design Implications: The paper focuses on specifying an initial artifact, executing it, collecting feedback, and presenting the resulting evidence to a generative optimizer.Feedback design is acknowledged as important but is not treated comprehensively in this paper.