Source-linked AI summary
Dynamic Adaptation of the LLM Context for Generating Routines with Coupled Semantics
Gnaneswar Villuri, Hashmath Shaik, Alex Doboli
TL;DR
LLM code generation struggles when component correctness depends on runtime coupling that textual descriptions cannot resolve. The paper proposes a validation-generation framework combining structured execution feedback, knowledge-graph constraints, multiple candidates, and simulated annealing. It outperforms iterative baselines on seven of eight problems at 300 and 600 evaluations and leads at 1000 evaluations on cross-coupled optimization.
Problem
LLMs struggle with static binding, where component meaning depends on another component’s runtime behavior rather than its textual description.
Method
Dynamic context adaptation uses a validation agent, structured execution feedback, a knowledge graph, multi-candidate generation, and simulated-annealing selection.
Results
The method outperforms iterative baselines on seven of eight problems at both 300 and 600 evaluations, and achieves the highest 1000-evaluation score on cross-coupled optimization.
Takeaways & Limitations
Structured execution feedback is the primary contributor, while simulated annealing and knowledge-graph grounding provide additional support.
Takeaways & Limitations
The validation agent lacks counterfactual reasoning, feedback is bounded by the test suite, and a single incumbent limits diversity on multi-basin landscapes.
Abstract
from arXiv · showhide
LLM-based code generation fails when correctness depends on execution-dependent coupling: the meaning of one routine is defined by the runtime behavior of another, a relationship that cannot be resolved from textual descriptions alone. This limitation, which we call static binding, is not confined to explicitly coupled problems; it appears to varying degrees whenever correctness depends on joint execution behavior across components, from explicit cross-coupled optimizers to subtler joint constraints in packing, routing, and symbolic search. This paper proposes dynamic context adaptation, a sample-efficient validation-generation loop designed for this setting. A validation agent extracts structured diagnostic information from execution traces, providing gradient-like guidance to a generation agent that proposes multiple candidates per iteration. A knowledge graph derived from the problem description supplies semantic constraints to the generation agent. Simulated annealing selects among candidates to avoid greedy collapse. Our method outperforms zero-shot, Reflexion, and OpenEvolve on seven of eight problems at both 300 and 600 evaluations (p < 0.01), a regime where population-based search has not yet accumulated sufficient diversity to compete. Notably, on the primary motivating problem (cross-coupled optimization), our method also achieves the best score at 1000 evaluations, consistent with the hypothesis that structured execution feedback is most beneficial when correctness depends on runtime coupling. Ablation results confirm that structured execution feedback is the primary driver.
1 Introduction
The paper identifies static binding as a limitation of LLM code generation when component meaning depends on runtime behavior, and proposes dynamic context adaptation to address it. The method combines structured execution feedback, knowledge-graph constraints, multi-candidate generation, and simulated annealing, with strong early-budget results.
- Problem: LLMs struggle when one component’s correctness depends on another component’s runtime behavior rather than its textual description.The paper calls this limitation static binding.
- Approach: The proposed method combines a validation-generation loop, a knowledge-graph layer, multi-candidate generation, and simulated annealing.Structured execution feedback replaces free-form reflection, while simulated annealing balances exploration and exploitation.
- Results: At 300 and 600 evaluations, the method outperforms iterative baselines on seven of eight problems with p < 0.01.The paper frames this as sample-efficient improvement before population-based diversity accumulates.
- Contributions: The paper characterizes its contribution as a formal taxonomy of meaning dependencies and a KG-based two-agent architecture for code synthesis.These contributions accompany the empirical sample-efficiency result.
2 Motivation
The motivation is that component meaning cannot always be inferred statically: runtime behavior may determine dependencies among routines. The paper illustrates this limitation through maze navigation, cross-coupled optimization, and a four-case taxonomy of meaning dependencies.
- Example 2: Cross-coupled optimization: In cross-coupled optimization, the optimization algorithms must be coupled because each procedure’s decisions are linked to the others.The LLM generated the data-management routines correctly but handled the optimization algorithms independently.
- Runtime-dependent meaning: Meaning sometimes can only be determined after executing a component and observing its behavior.This motivates finding context layers that account for static linking and inter-component relations.
- Example 1: Maze navigation: In maze navigation, the generator’s meaning and actions depend on the robot’s actions, making the relationship sender-related.The scanner routines are presented as independent because their meanings do not depend on another routine.
- Meaning-dependency taxonomy: Static linking assumes component meanings are independent, but the paper distinguishes independent, sender-related, receiver-related, and cross-correlated cases.The taxonomy specifies how meaning dependencies vary across components.
- Dynamic adaptation: The proposed context-finding loop transforms independent routine meanings into meanings related to other routines using validation agents and test vectors.This operationalizes the treatment of dependency cases 2–4.
4 System Architecture
The system isolates an optimization-sensitive component and iteratively refines it through execution-based validation, structured generation, knowledge-graph constraints, and simulated-annealing selection. The architecture is designed to preserve exploration while adapting candidates to observed runtime behavior.
- Validation-generation loop: Each iteration validates the incumbent through execution logs, extracts structured diagnostics, and has a generation agent propose k new candidates.Diagnostics summarize failure modes, missed opportunities, and constraint violations.
- Validation-generation loop: The generation prompt combines a task summary, KG-derived constraints, structured feedback, and the current incumbent candidate.The loop operates on an isolated optimization-sensitive component within an LLM-generated codebase.
- Knowledge graph: The knowledge graph extracts typed nodes and edges from the problem description, then inserts a dependency-traced subgraph into agent prompts as structured JSON.The transformation is described as Mtext → MKG → Mcode.
- Simulated annealing: Simulated annealing accepts better candidates unconditionally and sometimes accepts worse candidates using a Metropolis probability with geometric cooling.The stated parameters are T0 = 2.5, α = 0.85, and Tmin = 0.01.
- Simulated annealing: The cooling schedule front-loads exploration before structured feedback has accumulated sufficient signal.The paper links this choice to avoiding collapse into locally adequate policies and enabling bold structural changes.
5 Problems and Cost Functions
The evaluation contains eight problems: two motivating tasks with explicit coupled-semantics dependencies and six standard benchmarks with normalized quality scores. The problems test maze constraints, cross-coupled resource allocation, and latent execution-dependent coupling in broader optimization and search settings.
- Benchmark scope: The benchmark suite contains eight problems, including two original coupled-semantics tasks and six standard benchmarks.The standard tasks are included for direct comparison with prior work.
- P1: Maze navigation: Maze navigation awards up to 30 total points across three instances subject to key-precedence constraints and a strict 17-instruction ISA.Validation inspects execution logs for loops, wasted stack operations, and invalid token usage.
- P2: Cross-coupled optimization: Cross-coupled optimization jointly allocates power to surgical teams and dispatches repair crews for grid repair.The validation agent flags coupling violations such as surgeries during active outages.
- P3–P8: Standard benchmarks: The six standard benchmarks use normalized scores in [0, 1] and maximize quality relative to an optimum or reference solution.They include circle packing, function minimization, TSP, filter design, online judge programming, and symbolic regression.
- P3–P8: Standard benchmarks: Although not explicitly framed as coupled-semantics tasks, the standard benchmarks contain latent execution-dependent coupling.The supplied description identifies joint constraints and related dependencies across these tasks.
6 Experiments
The experiments compare the method with zero-shot, Reflexion, OpenEvolve, and a quality-ceiling reference across evaluation budgets, then isolate contributions through ablation. The method has an early-budget advantage across most problems, while structured feedback is the largest contributor.
- Setup: Experiments use Qwen/Qwen2.5-72B-Instruct, 10 random seeds, and budgets extending to approximately 1000 evaluations per problem.The setup uses 333 iterations × 3 candidates and reports mean ± std.
- Main results: At 300 and 600 evaluations, the method outperforms OpenEvolve on seven of eight problems, with all seven comparisons significant at p < 0.01.Function minimization is the exception at both budgets.
- Main results: At 1000 evaluations, OpenEvolve outperforms the method on seven of eight problems, while the method leads cross-coupled optimization at 0.694 versus 0.681.The cross-coupled result is significant at p = 0.019, d = 0.52.
- Convergence: The running-maximum curves favor the method through approximately 650 evaluations for circle packing and 710 for maze navigation before OpenEvolve surpasses it at 1000 evaluations.Figure 4 reports mean ±1 std over 10 seeds.
- Ablation study: Removing structured feedback causes the largest ablation drop: −28% on maze, −31% on circle packing, −12% on function minimization, and −11% on cross-coupled optimization.All reported drops have p < 0.001 and d > 1.1.
- Ablation study: Replacing simulated annealing with greedy selection reduces maze navigation by −16% and circle packing by −9%, while increasing standard deviation across conditions.The simulated-annealing effect is smaller but significant on function minimization at p = 0.018, d = 0.44.
7 Discussion
The discussion characterizes structured execution feedback as directional guidance for coupled behavior while identifying limits from surface-level diagnosis, discrete-action variance, bounded tests, and single-incumbent search.
- Discussion: Structured diagnostics identify both the location and cause of execution failures, guiding a shift from local edits toward globally coupled solutions.In circle packing, this meant treating all 26 circles as a jointly constrained system.
- Limitations: The validation agent lacks counterfactual reasoning, while discrete action spaces make single-token changes behaviorally high-variance.Feedback is also bounded by the test suite, and a single incumbent limits diversity on multi-basin landscapes.
8 Conclusion
The paper presents dynamic context adaptation for execution-grounded code generation and reports strong sample efficiency across most tested problems. It identifies future extensions for broader dependency structures, diversity recovery, and formal verification.
- Dynamic context adaptation grounds code generation in execution feedback through a validation-generation loop with KG grounding and SA selection.
- The method outperforms all iterative baselines on seven of eight problems at both 300 and 600 evaluations.
- Cross-coupled optimization receives the best full-budget score, where runtime coupling most directly favors structured feedback.
- The crossover with population-based search occurs at 600–800 evaluations, indicating a complementary tradeoff.
- Future work targets multi-hop dependency chains, incumbent populations for multi-basin landscapes, and formal verification oracles.