From Hand-Crafted to LLM-Based Variation Operators in Metaheuristics: A Tutorial

1 Introduction

• LLMs generate or edit candidate solutions, heuristics, or programs directly at the variation step, changing the role of machine learning from assisting search to participating in it.The paper calls this shift a semantic turn: variation becomes a model call conditioned on information available when the operator executes.Source: §1 · p. 1
• Practitioners lack a shared operational framework for choosing prompt content, despite choices that can shift validation, evaluation, and inference costs.The motivating example contrasts heuristic scores, executable code, and natural-language reflections in a TSP search procedure.Source: §1 · p. 1
• The framework classifies operators by dominant conditioning channel—Numeric, Symbolic, or Linguistic—and by whether the emitted artifact is Transient, Amortized, or Transfer.These descriptors organize how the prompt conditions variation and how the resulting artifact is used in the search.Source: §1 · p. 2
• The tutorial operationalizes the framework with a reusable build template, a synthesis of representative methods, and a cost-aware decision guide.The build template includes parsing, validation, and bounded repair; the lens is positioned as complementary to surveys organized by model role, interaction direction, integration stage, or review metadata.Source: §1 · p. 2, Table 1 · p. 2
• The lens is descriptive, not predictive: it makes no universal claim that more explicit prompt representations improve optimization performance.Whether added structure helps is presented as an empirical, task-dependent question; the tutorial focuses on text-prompt-conditioned variation.Source: §1 · p. 2

Metaheuristic search loop

• A metaheuristic repeatedly proposes, evaluates, and selects candidates; an LLM call can replace the hand-coded variation step, with parsing and bounded repair before acceptance.The model call is costly in latency, token use, and API or deployment cost relative to classical moves, so call budgets affect when it is invoked.Source: §2 · p. 3

LLM as a conditional sampler

• An LLM samples a completion token by token from a prompt-conditioned distribution, while temperature, top-k, and top-p reshape the sampling distribution.Structured-output schemas and tool or function calling constrain completion form as generation-interface choices and validation aids.Source: §2 · pp. 3–4, Figure 1 · p. 4

Prompt and search history

• Prompt phrasing, ordering, and delimiters can change likely continuations, while finite context windows require accumulated feedback to be summarized or pruned, potentially losing information.LLM calls are stateless at the API level unless relevant adaptation is written into successive prompts.Source: §2 · p. 4

Proposal distribution

• The operator is an artifact-level proposal distribution q(x′ | r): prompt representation r shapes candidate proposals, while parsing, validation, and bounded repair reject outputs that remain malformed or infeasible.The induced distribution covers the variation module, not only raw token sampling; changing r can affect validity, repair effort, locality, and diversity.Source: §2 · pp. 4–5, Figure 2 · p. 5

Related optimization perspectives

• LLM variation resembles distribution-based optimization and expensive black-box search, but these analogies do not transfer their assumptions or formal properties to LLM operators.The paper contrasts implicit prompt-induced proposals with EDAs’ explicit candidate distributions and Bayesian optimization’s surrogate and acquisition rule.Source: §2 · pp. 5–6

Boundaries with related literature

• The framework focuses on prompt-conditioned models that generate variation themselves, distinguishing them from assistive ML that informs a conventional metaheuristic and from weight-only or non-text conditioning.Parameter tuning within a fixed algorithm structure and selection among predefined operators are boundary cases distinct from synthesizing new heuristics or programs.Source: §2 · pp. 5–6

The organizing lens

• The framework pairs dominant conditioning channel with artifact persistence to classify LLM variation operators and guide method selection.The descriptors are independent, though often coupled in current methods.Source: §3 · pp. 6, 9

Conditioning channels

• Numeric carries evaluative information, Symbolic exposes machine-interpretable behavior, and Linguistic supplies operative natural-language rationale or strategy.Methods may use several channels; classification names the dominant signal, with Numeric recorded as a supporting base channel when appropriate.Source: §3 · pp. 6–7

Classification rule

• Classify by identifying operative search-state content, filtering boilerplate, and auditing which channel most changes proposals while preserving a valid scaffold.If two non-base channels both matter without a clear dominant signal, report a hybrid rather than forcing one label.Source: §3 · pp. 7–8

Classification caveat

• The channel audit is a placement and information-signal diagnostic, not a clean causal test; prompt sensitivity and channel interactions can alter its result.It classifies intended design signals rather than inferring which tokens the model internally weights.Source: §3 · p. 8

Examples and agreement

• The ReEvo-style example is Linguistic-dominant with Symbolic support because its operative reflection steers code changes; nine of twelve methods received unanimous coder classifications (Gwet’s AC1 [39] = 0.73).The example demonstrates the classification procedure, not that reflection is causally necessary for performance across tasks.Source: §3 · p. 8

Artifact persistence

• Transient artifacts are used inside the loop, Amortized artifacts are emitted offline for near-zero-runtime reuse, and Transfer reuses source-domain artifacts or principles in a new domain.Persistence combines call location and artifact portability, which usually align but can diverge.Source: §3 · p. 9, Figure 4 · p. 9

Operator anatomy

• A runnable operator combines prompt assembly, sampling, parsing and validation with bounded repair, and integration into the search loop.The full copyable prompt template and Python reference loop are provided in Appendix A.Source: §4 · pp. 9–10, Figure 5 · p. 10

Reference loop

• Each search step builds a prompt from the current state and history, then allows up to K+1 samples as validation failures trigger diagnostic-guided retries.Exhausted retries consume a step but skip candidate acceptance; each step counts against the outer budget regardless of acceptance or repair attempts.Source: §4 · pp. 10–11

Validation

• The validator checks envelope and representation, parses the payload, then tests feasibility, returning layer-specific diagnostics for targeted repair.Cheap structural checks precede feasibility checks; AST parsing and import allow-listing are not a security sandbox.Source: §4 · pp. 11–12

Worked solution trace

• In the illustrative TSP trace, one invalid tour is repaired; the valid 8.000-length tour improves on the 9.236 incumbent and is accepted after two samples.The example is illustrative, not a benchmark result.Source: §4 · p. 14

Reusable-program trace

• In the illustrative bin-packing trace, an emitted best-fit heuristic reaches the 19-bin lower bound and runs without further LLM calls.The 19-bin result is for the illustrative instance set; the reusable heuristic is Amortized.Source: §4 · pp. 14–15

Design choices

• Transient operators adapt to current search state but pay per-move inference cost; offline Amortized artifacts front-load model cost and avoid runtime calls.The template also exposes trade-offs in model choice, feedback, prompt layout, output level, and selection, alongside risks such as invalid outputs and context overflow.Source: §4 · p. 16

5 Classification of representative methods according to the lens

• The survey selectively maps primary-source-checked representatives chosen for lens coverage or distinctive mechanisms, rather than offering a systematic census.It excludes near-equivalents, out-of-scope systems that do not vary candidates inside a loop, and methods that could not be checked against a primary source.Source: §5 · pp. 16–17

5.1 Numeric-conditioned methods

• Numeric methods mostly vary encoded candidate solutions inside the search loop; their prompts need not include executable machinery or verbal rationale, leaving structure to encodings, move sets, or validators.OPRO and EvoPrompt are boundary cases: numeric scores steer searches over text prompts rather than numeric solution encodings.Source: §5.1 · pp. 18–19

5.2 Symbolic-conditioned methods

• Symbolic methods typically condition on formal artifacts and emit executable code for offline reuse, but MOLLEO, LLEMA, and GI-LLM illustrate transient, in-loop formal-artifact variation.Code emission alone does not establish Symbolic dominance: EoH is classified as Linguistic because natural-language ideas steer its code edits.Source: §5.2 · pp. 19–20

5.3 Linguistic-conditioned methods

• Linguistic methods range from transient language-guided search through amortized idea-and-code generation to transferable principles, with rationale offering re-bindability but less mechanical checkability.The final emitted code in offline methods is Amortized even when reflections used during meta-search are Transient.Source: §5.3 · p. 20

5.4 Boundary and hybrids

• For hybrids and boundary cases, classify the prompt signal that proposes the varied artifact, while separately annotating controller decisions, local-search post-processing, and tool orchestration.Source: §5.4 · p. 21

5.5 Classification overview

• Representative methods cluster on or below a Numeric/Transient–Symbolic/Amortized–Linguistic/Transfer diagonal, a structural tendency rather than a rule or quality ranking.Empty pairings indicate current research gaps, not impossibilities; Numeric/Amortized and Numeric/Transfer methods are undeveloped, while classical non-LLM work shows score-based transfer is possible.Source: §5.5 · p. 21

• Method choice routes on evaluator readiness, budget, call locus, and artifact persistence, while solution quality and reproducibility serve as verification filters.The five dimensions also include LLM cost and latency; no shared performance matrix yet supports statistical ranking across methods.Source: §6 · p. 22
• 0.89% BPP gap: MCTS-AHD reports the smallest matched online-bin-packing gap; EoH's 20-generation, 2,000-query run matches or improves FunSearch in its evaluation.Source: §6 · p. 22
• The evidence table is not a leaderboard: methods use different benchmarks, and even the narrow matched comparisons do not isolate conditioning-channel effects from search mechanisms.MCTS-AHD, FunSearch, EoH, and ReEvo share an online-bin-packing evaluator; HSEvo provides another matched comparison across BPO, TSP, and OP.Source: §6 · pp. 22, 24
• Add an LLM only to fill a capability gap; otherwise prefer classical methods when inference cost, evaluation cost, or lack of automatic output checking cannot be justified.The guide recommends a matched no-LLM baseline under the same evaluation budget as both a design gate and an evaluation control.Source: §6 · p. 24
• The guide is a routing aid, not a universal selector; actual deployment also depends on validation, evaluator readiness, budget, wall-clock constraints, and reproducibility.Its binary branches simplify choices, and final method selection still requires target-task validation, budget parity, and drop-channel ablation.Source: §6 · p. 25

• Prompt wording, examples, ordering, schemas, and context placement can change proposals within a conditioning channel, so studies should archive templates and test meaning-preserving variants.Drop-channel ablation identifies load-bearing signals but does not measure sensitivity to prompt variants within a channel.Source: §7 · p. 25
• LLM proposals can already lack diversity before selection begins, so diversity measures and mitigations should address inputs, populations, and decoding as well as downstream search.The paper distinguishes generation-time similarity from diversity loss caused by evolutionary selection or local improvement.Source: §7 · pp. 26–27
• Report evaluator calls, model samples, tokens, repair attempts, accepted candidates, and wall-clock separately; use identical evaluators, strong baselines, controls, repeated runs, and statistical tests.Equal budgets alone are insufficient when scoring sources differ, including when generator and judge share a model family.Source: §7 · p. 27
• Batching or asynchronous calls can hide latency only when generated artifacts pass automatically through parsing, feasibility checks, sandboxing, bounded repair, and diagnostics.API use can add rate limits, latency variance, data-retention constraints, and moderation filters; self-hosting shifts bottlenecks to hardware and serving configuration.Source: §7 · p. 27
• Model and endpoint changes can shift proposal distributions, making pinned versions, decoding settings, repair policies, and released artifacts important for reproducibility.Offline-Amortized methods use the model during design, whereas Transient methods remain model-dependent at each run.Source: §7 · p. 27

Theoretical foundations

• A theory of LLM-induced search remains premature; proposed work should begin with finite-budget results for validated proposal families or measurable conditions on valid-proposal mass and quality.The relevant process includes proposal, validation, repair, and acceptance; its effective proposal kernel may depend on prompt history, decoding, and validation or repair state.Source: §8 · p. 28

Benchmarking infrastructure

• Benchmark suites should align evaluators and resource accounting, use fresh instances and classical baselines, and test contamination, robustness, and cross-domain transfer.Classical benchmarks may fail to isolate an LLM operator when evaluator differences, token or repair budgets, contamination, or validation harnesses vary.Source: §8 · p. 28

The cost-quality frontier

• The cost-quality frontier is unresolved, with evidence suggesting task-dependent effects rather than a general benefit from more explicit conditioning.Matched comparisons should vary conditioning, persistence, and model scale while reporting quality per token, validity, repairs, diversity, latency, and energy or accelerator use.Source: §8 · pp. 28–29

Auditability and interpretability

• Prompt and artifact audits can test whether conditioning changes outputs and artifact families, but they do not establish how the model internally computes those changes.Drop-channel and content-level ablations test prompt sensitivity; edit distance, AST similarity, clustering, and behavioral comparisons can examine generated artifacts.Source: §8 · p. 29
• Interpretability remains a separate open problem: stated reasoning is not direct access to internal computation, and the relevance of circuit analysis or causal editing to optimization artifacts is unknown.Progress would pair content ablations with analyses of proposal performance and structural or behavioral families.Source: §8 · p. 29

Transfer and artifact portability

• Transfer requires inspectable, sufficiently instance- or domain-invariant artifacts that can be rebound without rediscovery; current coverage mostly demonstrates reuse within task distributions.Executable artifacts must decouple from their source evaluator, while principle-level artifacts may travel more easily but cannot be mechanically verified in the target domain; source-target tests should compare design effort with rediscovery.Source: §8 · pp. 29–30

Agentic and multi-agent metaheuristics

• Agentic and multi-agent systems raise open questions about role composition, diversity, auditability, and cost; controlled comparisons should separate role-structure gains from extra calls and selection pressure.Role messages can propagate erroneous diagnoses, and additional roles can multiply calls and repairs; studies should log call graphs, role-specific token costs, and accepted-artifact lineage.Source: §8 · p. 30

Beyond prompt channels

• Weight-internalized and multimodal conditioning extend beyond text prompts, raising open questions about cross-domain generalization and whether visual inputs add information beyond equivalent text.Controlled comparisons should remove or replace visual or weight-internalized signals while holding loops and budgets identical.Source: §8 · p. 30

Conclusion

• The framework classifies operators by dominant prompt conditioning—Numeric, Symbolic, or Linguistic—and by whether artifacts are transient, amortized, or transferred.Conditioning concerns what information is load-bearing at variation time; persistence concerns what survives the model call and how it is used.Source: §9 · p. 30
• The framework supports method placement, operator construction, and family selection, with placement based on load-bearing prompt content rather than the emitted artifact alone.A complete variation loop includes prompt structure, output format, parsing, validation, bounded repair, evaluation, and feedback into search.Source: §9 · pp. 30–31
• Credible evidence requires validated artifacts, bounded repair, sandboxing for executable code, matched comparisons, and released prompts, artifacts, logs, and audits.Open gaps include finite-budget theory, contamination-resistant benchmarks, cost-quality evidence across conditioning and model scale, and transfer beyond within-distribution reuse.Source: §9 · p. 31

Resources

• The appendix identifies a repository with prompt templates, reference code, and supplementary resources for adoption and reproducibility.The scaffold instantiates the prompt anatomy and build loop described earlier.Source: A Reusable templates · pp. 37–38

Prompt template

• The prompt template has context, conditioning, instruction, and format slots, with problem-specific fields supplying task details, representation, constraints, search state, and output structure.The representation and constraints stay fixed within conditioning, while incumbent, scores, and diagnostics provide dynamic search-state information.Source: A Reusable templates · p. 37
• The template requests one candidate in a stable CANDIDATE envelope, enabling a parser to remain unchanged across problems.The envelope names the artifact type and representation and contains the candidate payload.Source: A Reusable templates · p. 37

Reference loop

• The reference loop samples a completion, parses and checks feasibility, repairs invalid output up to a retry limit, evaluates valid candidates, applies an acceptance rule, and records feedback.The spec binds render, parse, feasible, and repair_prompt; the loop returns the best candidate found within its search-step budget.Source: A Reusable templates · pp. 37–38

Concrete binding

• Rebinding parse, feasibility, and evaluation can retarget the same control flow from TSP to bin packing, job-shop scheduling, or program search.In the TSP example, feasibility checks that each city appears exactly once, and acceptance keeps a candidate only when tour length decreases.Source: A Reusable templates · p. 38

Validation of the placement rule

• Three independent coders agreed on nine of twelve method placements, with Gwet’s AC1 = 0.73; disagreements clustered where channel descriptions underdetermined the representation.Table 3 lists channels present and the source-described operative channel, audited where possible by removing or neutralizing that signal.Source: B Placing every method: a drop-channel… · p. 38, Table 3 · p. 39
• On online bin packing, both EoH-style conditions reached the optimum, with mean gap 0% and 6/6 candidates optimal, so the ablation did not demonstrate performance necessity for the natural-language channel on that task.The parent code and scores were held fixed; the comparison removed the natural-language idea.Source: B Placing every method: a drop-channel… · p. 39
• The audit reports intended steering signals rather than universal performance effects; solution-as-text placements can depend on whether varied text is treated as encoding or operative language.EoH remains classified as Linguistic because the idea is the source-described design-intent signal distinguishing it from code-only automatic heuristic design.Source: B Placing every method: a drop-channel… · p. 39

C Extended coverage map

• The broader method set is intended to confirm that the body’s selection is representative of the field’s breadth.Entries are read against their primary sources.Source: C Extended coverage map · p. 40
• Entries marked adjacent are foils because they translate a specification rather than vary a candidate; the table is not a systematic census.The systematic census is provided by [25].Source: C Extended coverage map · p. 40, Table 4 · p. 41
• Table 4 maps methods by dominant conditioning channel and persistence, with assignments following Section 3.It also reports the channels each method draws on.Source: C Extended coverage map · p. 40, Table 4 · p. 41