PaperlayerView exact revision on arXiv

Source-linked AI summary

optimize_anything: A Universal API for Optimizing any Text Parameter

Lakshya A Agrawal, Donghyun Lee, Shangyin Tan, Wenjie Ma, Karim Elmaaroufi, Rohit Sandadi, Sanjit A. Seshia, Koushik Sen, Dan Klein, Ion Stoica, Joseph E. Gonzalez, Omar Khattab, Alexandros G. Dimakis, Matei Zaharia

arXiv:2605.19633v1cs.CLcs.AIcs.LGcs.NEcs.SE

TL;DR

Existing LLM optimizers are specialized by artifact type or optimization mode. optimize_anything unifies text-based optimization across diverse domains, achieving state-of-the-art results across six tasks, including 32.5% →89.5% ARC-AGI accuracy.

  • Problem

    Existing LLM optimization systems are limited to specific artifact types or optimization modes, leaving their generality across fundamentally different domains unestablished.

  • Method

    optimize_anything provides a declarative API combining evaluator feedback, reflection, candidate search, Pareto-based scoring, and single-task, multi-task, and generalization modes.

  • Results

    Across six diverse domains, optimize_anything matches or surpasses specialized tools, including 32.5% →89.5% ARC-AGI accuracy.

  • Takeaways & Limitations

    LLM-based text optimization functions as a general-purpose problem-solving paradigm rather than a method limited to code or prompts.

  • Takeaways & Limitations

    The system assumes artifacts are representable as text, while multi-task benefits depend on problems sharing transferable structure.

Abstract

from arXiv · show

Can a single LLM-based optimization system match specialized tools across fundamentally different domains? We show that when optimization problems are formulated as improving a text artifact evaluated by a scoring function, a single AI-based optimization system-supporting single-task search, multi-task search with cross-problem transfer, and generalization to unseen inputs-achieves state-of-the-art results across six diverse tasks. Our system discovers agent architectures that nearly triple Gemini Flash's ARC-AGI accuracy (32.5% to 89.5%), finds scheduling algorithms that cut cloud costs by 40%, generates CUDA kernels where 87% match or beat PyTorch, and outperforms AlphaEvolve's reported circle packing solution (n=26). Ablations across three domains reveal that actionable side information yields faster convergence and substantially higher final scores than score-only feedback, and that multi-task search outperforms independent optimization given equivalent per-problem budget through cross-task transfer, with benefits scaling with the number of related tasks. Together, we show for the first time that text optimization with LLM-based search is a general-purpose problem-solving paradigm, unifying tasks traditionally requiring domain-specific algorithms under a single framework. We open-source optimize\_anything with support for multiple backends as part of the GEPA project at https://github.com/gepa-ai/gepa .

1 Introduction

optimize_anything presents a declarative, uniform API for optimizing arbitrary text artifacts with LLM proposals and evaluator feedback. It unifies single-task search, multi-task search, and generalization across diverse domains, achieving strong results while showing that diagnostic feedback accelerates optimization.

  • Motivation: A wide range of artifacts—including CUDA kernels, scheduling policies, agent architectures, SVGs, and prompts—can be serialized, evaluated, and improved through one interface.The common structure is a text artifact, an evaluator, and LLM-proposed improvements based on diagnostic feedback.
  • System: The declarative API accepts a seed artifact or natural-language objective, an evaluator with optional diagnostics, and optionally a dataset, while handling search mechanics.The system manages prompt construction, reflection, candidate selection, and search strategy.
  • Contributions: Across six domains, one system matches or surpasses specialized tools, including ARC-AGI accuracy rising from 32.5% to 89.5%, cloud costs cut by 40%, and 87% of CUDA kernels matching or beating PyTorch.These results span code, prompts, agent architectures, numerical configurations, and images.
  • Optimization modes: The framework unifies single-task, multi-task, and generalization modes under one interface, including the first multi-task mode.Prior systems supported only one mode, with generalization limited to prompt optimization in the cited comparison.
  • Feedback: Actionable diagnostic feedback yields 4-6× faster convergence and substantially higher final performance than score-only feedback across prompt optimization, circle packing, and CUDA kernels.The API provides a uniform channel for diagnostics such as stack traces, profiler data, rendered images, and structured error reports.

2 Related Work

optimize_anything builds on LLM-based program evolution, prompt optimization, self-improvement, and agent architecture search, extending these approaches into a unified framework for optimizing arbitrary text artifacts. Its mechanisms generalize reflective feedback and architecture search while supporting established evolutionary search methods.

  • LLM-based program evolution: LLM-based program evolution uses evolutionary search to discover algorithms, with AlphaEvolve, OpenEvolve, ShinkaEvolve, and FunSearch representing successive extensions.AlphaEvolve uses Gemini models with island-based MAP-Elites; OpenEvolve adds model-agnostic open-source support, while ShinkaEvolve adds novelty-based rejection sampling and adaptive ensemble selection.
  • Prompt optimization: Prompt optimization includes reflective, Pareto-based, evolutionary, and gradient-like methods, while optimize_anything extends GEPA beyond prompts to arbitrary text artifacts.GEPA outperforms MIPROv2 and GRPO, and optimize_anything supports GEPA’s evolutionary search as one optimization backend.
  • LLM self-improvement and reflection: optimize_anything’s SI mechanism generalizes verbal reinforcement, iterative self-feedback, and LLM mutation by treating diagnostic feedback as a declarative evaluator contract.This connects optimize_anything to Reflexion, Self-Refine, and Evolution through Large Models without hardcoding self-critique.
  • Agent architecture search: Its generalization mode subsumes agent architecture search by evolving agent code and prompts jointly while evaluating the resulting agents on tasks.This frames ADAS and AFlow as special cases in which the optimized artifact is agent code.

3 The optimize_anything API

optimize_anything exposes a minimal text-optimization API built around a seed artifact and evaluator, with optional diagnostic side information. Its behavior supports single-task search, multi-task transfer across related problems, and generalization to unseen examples, selected by dataset and validation-set inputs.

  • Core API: The API requires a seed artifact and evaluator, whose score can be accompanied by optional diagnostic Side Information for proposer reflection.Side Information may contain open-ended text, structured data, multiple sub-scores, or images for vision-capable LLMs.
  • Core API: Users declare the artifact, evaluator, and domain knowledge without specifying mutation prompts, task-specific templates, island configurations, or EVOLVE-BLOCK markers.Optimization backends handle execution after the user specifies the problem components.
  • Core API: Seedless mode lets the LLM bootstrap a candidate from a natural-language objective when providing a starting artifact is difficult or requires domain expertise.The paper demonstrates this mode on a 3D modeling task.
  • Optimization modes: Multi-task search uses a related-task dataset to transfer optimization insights, sharing a Pareto frontier across tasks during proposal.The paper reports faster convergence and more solved problems than single-task runs in CUDA kernel generation, while noting no prior LLM-evolution framework supports this mode.
  • Optimization modes: Generalization uses both dataset and validation set so one optimized artifact performs well on unseen examples, unlike multi-task search, which yields specialized artifacts.In agent architecture discovery, the artifact is an entire agent evaluated on unseen ARC-AGI puzzles.

4 Method

optimize_anything provides a backend-agnostic API that unifies text-artifact optimization across single-task, multi-task, and generalization settings. Its core method combines actionable side information with Pareto-frontier search and focused reflection to preserve complementary strengths and enable targeted mutations.

  • Unified optimization formulation: The backend-agnostic interface supports single-task search, multi-task search across related problems, and generalization from training examples to unseen validation examples.Artifacts can be prompts, agents, policies, or other text strings evaluated by scores and optional actionable side information.
  • Actionable side information: The evaluator returns both a score and diagnostic side_info that may include compiler errors, runtime exceptions, profiler summaries, per-test results, execution traces, critiques, or images.This typed primitive is designed to support domain-portable proposer logic and multimodal feedback.
  • Pareto-frontier search: The system tracks per-task or per-metric scores and maintains a Pareto frontier, preserving candidates that are best on any objective rather than selecting only the highest average.The frontier accumulates complementary strengths and supports transfer between related problems.
  • Reflection and mutation: Each reflection step exposes the proposer to a minibatch of 2–3 examples, allowing focused improvements instead of requiring simultaneous optimization across all examples.The system executes the selected candidate, collects scores and side information, and asks the proposer LLM to diagnose failures and produce an updated artifact.
  • Candidate selection: Candidate mutation in the default GEPA backend is sampled in proportion to Pareto-front appearance, focusing exploration on solutions effective across more objectives.A candidate receives probability proportional to the number of objectives on which it achieves the best score.

5 Experiments

Across six diverse domains and three optimization modes, optimize_anything achieves strong generalization and task-specific gains, including major improvements in coding-agent skills, infrastructure algorithms, ARC-AGI, AIME, CUDA kernels, circle packing, and image generation. Ablations show that side information and multi-task transfer improve convergence, final performance, and scaling efficiency.

  • Infrastructure optimization: 40.2% cost savings over Dijkstra routing and 7.8% cost savings are achieved for CloudCast and Can’t Be Late, respectively.The optimized methods evolve toward provider-aware routing and adaptive scheduling using spot-availability patterns, switching-cost analysis, and slack-based thresholds.
  • Agent-system optimization: 57 percentage point gain raises ARC-AGI test accuracy from 32.5% to 89.5% as a 10-line agent becomes a 300+ line, four-component system.The evolved architecture uses rule induction, code generation with exec()-based verification, and fallback mechanisms.
  • Prompt optimization: 13.3pp gain raises GPT-4.1-mini from 46.67% to 60.00% on AIME 2025, outperforming MIPROv2’s 51.33% on the same benchmark.Only the system prompt changes, evolving from one generic sentence into a structured six-rule reasoning framework.
  • Multi-task optimization: 87% of generated CUDA kernels match or beat PyTorch baseline performance, while 48% achieve 10%+ speedups and 25% achieve 20%+ speedups.Multi-task optimization transfers kernel-generation insights across 31 operations through a shared Pareto front.

6 Why the Framework Works: Optimization Trajectory Analysis

Trajectory analysis on circle packing identifies targeted algorithmic shifts and multi-module Pareto leapfrogging as mechanisms underlying optimize_anything’s effectiveness. Side information directs which failure mode to address, while both the code artifact and refiner prompt are tracked on the Pareto front.

  • Trajectory mechanisms: Side information enables targeted algorithmic shifts by revealing which failure mode to address next.Examples include switching from collapsed radii to linear programming and from poor centers to sequential linear programming.
  • Trajectory mechanisms: Failure-mode diagnosis enables directed rather than blind mutations.The trajectory analysis contrasts targeted responses with unguided mutation.
  • Trajectory mechanisms: Multi-module Pareto leapfrogging tracks both the code artifact and refiner prompt on the Pareto front.The passage identifies this joint Pareto-front tracking as a mechanism driving effectiveness.

7 Discussion

The discussion finds that multi-task search transfers effectively when tasks share underlying optimization patterns, but can hurt when tasks are fundamentally independent. It also explains how side information provides actionable diagnostics and enables qualitatively novel optimized artifacts across domains.

  • When does multi-task search help?: Multi-task search helps most when problems share underlying optimization patterns despite differing specifics, as in CUDA kernels with reusable memory and warp-level strategies.Multi-task mode discovers shared strategies once and transfers them across operations, whereas single-task mode must rediscover them independently.
  • When does multi-task search hurt?: Multi-task search can degrade performance when tasks lack shared transferable structure, because jointly optimizing independent tasks introduces noise instead of useful cross-transfer.Circle packing across different N exemplifies this limitation: optimal configurations change unpredictably and provide no transferable structure.
  • The role of SI across domains: Side information converts scalar scores into actionable diagnostics, with domain-specific feedback exposing failures in code, agents, and cloud scheduling.Compiler errors and runtime diagnostics support code tasks; per-puzzle traces reveal failing agent components; temporal decision structure is exposed in scheduling.
  • Artifacts optimized by optimize_anything: Optimized artifacts span structured prompts, agent architectures, bilevel algorithms, and cloud-scheduling methods, demonstrating qualitatively novel strategies from LLM reasoning and diagnostic feedback.Examples include multi-stage ARC-AGI pipelines, provider-aware Steiner trees for CloudCast, and break-even cost analysis for Can’t Be Late.

8 Limitations

optimize_anything inherits limitations from LLM-based optimization: proposal quality depends on the proposer model, while evaluation can be expensive despite sample efficiency. The passage cites weaker candidates from weaker models and a $144 ARC-AGI evaluation cost.

  • Model dependence: Proposal quality depends on the proposer LLM’s capabilities, with weaker models producing weaker candidates.This limitation is confirmed by the proposer sensitivity analysis in Table 8.
  • Evaluation cost: $144 for ARC-AGI illustrates that evaluation cost can be high when evaluators involve expensive operations.The passage identifies this figure in Table 9.
  • Evaluation cost: LLM-based optimization is highly sample efficient and therefore calls evaluators less often.Sample efficiency partially offsets the cost of expensive evaluation operations.

9 Conclusion … D Detailed Algorithm

The paper concludes that optimize_anything provides a unified, backend-agnostic interface for diverse text-artifact optimization tasks, while also evaluating solver-code optimization, seedless generation, and its core Pareto-based algorithm. Across these sections, the system matches or improves on specialized baselines, bootstraps candidates from objectives, and supports single-task, multi-task, and generalization settings.

  • 9 Conclusion: A declarative interface consisting of a seed artifact, evaluator, and optional dataset can match or outperform purpose-built tools across diverse domains.The framework unifies three optimization modes, treats Side Information as a first-class evaluator contract, and uses Pareto-based search across metrics and examples.
  • A Use of Generative AI: The authors used ChatGPT, Gemini, Claude, and Cursor to generate parts of the work, while stating that experiment design and details were original.
  • B Blackbox Mathematical Optimization: At 2,000 evaluations, optimize_anything finds better solutions than Optuna on 7 out of 10 selected problems where Optuna struggles with lower budgets.The passage attributes Optuna’s failures to structural limitations including TPE’s trap-basin convergence and CMA-ES’s smooth-unimodal-landscape assumption.
  • C Seedless Mode: 3D Unicorn: In seedless mode, an objective alone lets the LLM bootstrap a Python artifact that generates a 3D unicorn, with rendered multi-view images returned as Side Information.The evaluator uses build123d and pyrender, renders PNGs, and a VLM scores the views while iterative refinement changes geometry, proportions, and anatomy.
  • D Detailed Algorithm: Algorithm 1 initializes and evaluates a seed candidate, repeatedly selects a Pareto-frontier candidate, evaluates it on a minibatch, and reflects on scores and Side Information to propose fixes.Improving candidates are evaluated on the full dataset, nondominated candidates are retained, dominated candidates are pruned, and the highest-average-score candidate is returned.
  • D Detailed Algorithm: The same loop supports single-task, multi-task, and generalization modes by varying dataset interpretation and evaluation, with held-out validation measuring generalization.Single-task datasets are singletons, multi-task elements are independent problems, and ParetoSelect samples nondominated candidates proportionally to frontier frequency in GEPA’s default backend.

E Multi-Task Scaling Tables

The section evaluates multi-task scaling on KernelBench, covering both 10-problem scaling and a 20-problem comparison of single-task search with MT20. Table 6 defines f1.x as the fraction of kernels achieving at least x% speedup over the PyTorch baseline.

  • Table 6 measures multi-task scaling across 10 KernelBench problems.
  • f1.x denotes the fraction of kernels achieving ≥x% speedup over the PyTorch baseline.
  • Table 7 compares single-task search with MT20 on 20 randomly sampled KernelBench problems.

F Optimization Trajectory Analysis: Full Details … J.1 Coding Agent Skills: Bleve Repository

The paper analyzes how optimize_anything improves solutions through targeted algorithmic shifts, module-level leapfrogging, and Pareto diversity, while documenting optimizer sensitivity, evaluator-driven multimodal search, and discovered domain-specific artifacts. The excerpts also show optimized prompts and coding skills emphasizing rigorous reasoning, early testing, minimal changes, and continuous verification.

  • F Optimization Trajectory Analysis: Full Details: Side information enables targeted algorithmic shifts by revealing which failure mode to address next, unlike score-only feedback that encourages undirected mutations.In circle packing, reflection moves from collapsed radii to LP, poor center placement to SLP, and local saturation to bilevel L-BFGS.
  • F Optimization Trajectory Analysis: Full Details: The code artifact and refiner prompt leapfrog across a shared Pareto front as each module discovers and absorbs stronger algorithms.The refiner moves from LP to SLP while the code module catches up, with reported intermediate scores of 0.98, 1.93, 2.61, and 2.63.
  • F Optimization Trajectory Analysis: Full Details: Pareto diversity prevents premature convergence by retaining greedy, LP, SLP, bilevel L-BFGS, and CMA-ES candidates across multiple quality dimensions.This preserves structurally diverse parents for proposing new candidates instead of refining only one approach.
  • G Proposer Sensitivity and Optimization Cost: GPT-5-nano reduces optimization cost significantly but underperforms GPT-5.1 on final achieved performance, while both improve substantially over the seed.The proposer-sensitivity table compares model cost and final performance; the supplied passage does not provide the underlying values.
  • G Proposer Sensitivity and Optimization Cost: Total optimization spend is dominated by the evaluator, with reflection cost described as minimal.This conclusion is summarized in the total-cost table for each experiment.
  • H Image Generation Details: For visual tasks, VLM evaluation rates one image property per call, making multi-task search a Pareto-frontier search over visual goals.SVG images are rendered for evaluation, while CAD evaluation uses three equidistant screenshots of the 3D object.
  • I Optimized AIME Prompt: The optimized AIME prompt requires careful, structured solutions that define notation and constraints, justify reasoning, handle dead ends explicitly, and isolate the final answer.It also favors exact or structural arguments over unnecessary approximation, trial-and-error, and random guessing.
  • J.1 Coding Agent Skills: Bleve Repository: The discovered Bleve coding skills emphasize running tests early, using failures and stack traces diagnostically, making minimal reviewable changes, and verifying continuously.They recommend narrowing tests quickly, adding focused table-driven unit tests including accented or Unicode edge cases, and avoiding scratch main.go files in the repository root.

J.2 ARC-AGI Agent Architecture … L Artifact Availability

optimize_anything evolves simple seeds into substantially more elaborate artifacts across ARC-AGI, routing, scheduling, CUDA, and circle packing, with demonstrations showing strong task-specific outcomes. The paper also provides public source and reproduction artifacts, with documented hardware requirements.

  • J.2 ARC-AGI Agent Architecture: The ARC-AGI agent grows from a 10-line seed into a 300+ line, 4-stage pipeline with rule induction, verified code generation, debugging, and fallback.The pipeline allows up to 2 fix attempts and falls back from code-first execution to direct LLM prediction.
  • J.3 CloudCast Routing Algorithm: The 178-line CloudCast algorithm replaces a Dijkstra seed with provider-aware Steiner tree routing that optimizes egress costs.Its implementation includes provider-aware weighting, Pareto-frontier candidate selection, diverse Steiner strategies, and bandwidth-aware greedy allocation.
  • J.4 Can’t Be Late Scheduling Policy: The 110-line scheduling policy adds break-even switching analysis, persistent spot-unavailability tracking, and increasingly aggressive slack-ratio thresholds as deadlines approach.These behaviors are absent from the simple deadline-checking seed.
  • J.5 CUDA Kernel: LayerNorm: 3.32× speedup over the PyTorch baseline is achieved by the optimized LayerNorm kernel using float4 vectorization, two-pass execution, and warp-shuffle reductions.Float4 vectorization cuts memory overhead by ∼4× per the passage.
  • J.6 Circle Packing Algorithm: For n=26 circles, the evolved circle-packing algorithm combines LP radius optimization, L-BFGS-B center optimization, SLP boosts, CMA-ES exploration, and relocation heuristics.The 480+ line bilevel optimizer starts from a simple greedy packing seed.
  • K Demonstration: 89.5% accuracy on ARC-AGI test puzzles is reached by the evolved agent using Gemini 3 Flash as both proposer and agent model.The demonstration begins from a naive 10-line single-call agent and iteratively adds sub-agents, code generation, debugging, and fallback logic.
  • K Demonstration: The circle-packing demonstration evolves a greedy seed into a 480+ line bilevel optimizer that outperforms AlphaEvolve’s reported solution for n=26 circles.The system discovers LP-derived gradients, CMA-ES exploration, and other algorithmic components absent from the seed.
  • L Artifact Availability: The source code and complete reproduction artifact are publicly available through the GEPA project and optimize-anything-artifact repository.Most domains run on a single CPU host with API access, while KernelBench requires an NVIDIA V100 32GB GPU with CUDA 12.1+.
Loading 2605.19633v1…