Source-linked AI summary
The Role of Agentic Instruction Categories for Code Generation: A Study of Five Python Libraries
Warren Gerald Polandra, Yoshiki Higo, Raula Gaikovina Kula
TL;DR
The effects of agent-instruction categories on agent-file-generation cost and subsequent code quality remain unclear. This study tests all 31 non-empty combinations across two LLMs and five Python libraries, finding that instruction count alone does not predict token cost and compilation does not ensure full test success.
Problem
It remains unclear how agent-instruction categories, alone or combined, affect agent-file-generation cost and the quality of code generated from those files.
Source: §I · p. 1Method
The study evaluates all 31 non-empty combinations of five instruction categories using two LLMs across five Python libraries.
Source: §I · p. 2Results
Instruction count alone did not predict token cost, and compilation did not ensure full correctness; 12.99% of GPT-5.4 variants passed all human-written tests.
Source: §VI · p. 10Takeaways & Limitations
Agent files may benefit from being tailored to specific code-generation objectives rather than relying on one comprehensive file.
Source: A. Implications · p. 8Findings are preliminary and may not generalize beyond five trivial Python libraries, two LLMs, one harness, one reconstruction task, and one run per condition.
Source: B. Threats to Validity · p. 9
Abstract
from arXiv · showhide
Agentic coding systems increasingly rely on repository-level agent files, such as AGENTS.md, to communicate project knowledge and developer intent. However, it remains unclear which categories of agent instructions enable efficient agent-file generation and accurate code generation. We evaluate five instruction categories: Repository Overview, Function Description, Implementation Detail, Testing, and Reconstruction Steps, across all 31 non-empty combinations using two Large Language Models (LLMs), Gemma 4 and GPT-5.4, and five human-written Python libraries. Our study addresses two research questions, comprising five sub-questions, on agent file generation and code generation. Results show that the number of instructions alone does not determine token usage for agent-file generation, with substantial variation across models and libraries. Although most AI-generated libraries compiled successfully regardless of the instruction combination, functional correctness remained limited. No Gemma 4 variant passed all tests, whereas 12.99% of GPT-5.4 variants did. Function Description achieved Gemma 4's highest partial-pass rate, while Reconstruction Steps achieved GPT-5.4's highest full-pass rate. For structural similarity to human-written code, Reconstruction Steps was most frequently associated with libraries closest to the originals in Lines of File and total Cyclomatic Complexity, while Testing and Reconstruction Steps were jointly most frequent for File Count and Function Count.
I. INTRODUCTION
Repository-level agent files communicate project-specific knowledge and developer intent, but the effects of instruction categories on file-generation cost and subsequent code quality remain unclear. The study evaluates five instruction categories across 31 combinations, two LLMs, and five Python libraries.
I. INTRODUCTION
- Repository-level agent files include AGENTS.md, copilot-instructions.md, and CLAUDE.md, and provide project-specific information such as purpose, conventions, commands, and constraints.The instructions communicate what an agent should know and how it should act within a repository.Source: §I · p. 1
- The study examines how instruction categories affect both agent-file-generation cost and the quality of code generated from the resulting files.The gap matters because low-quality generated code can undermine generative AI productivity gains.Source: §I · p. 1
- The evaluation covers all 31 non-empty combinations of five instruction categories, using two LLMs and five Python libraries.The categories include Repository Overview, Function Description, Implementation Detail, Testing, and Reconstruction Steps.Source: §I · pp. 1–2
- The study measures code generation through compilability, functional correctness, and structural similarity to human-written code.Its research questions also examine agent-file-generation token usage and which instruction combinations produce the highest or lowest usage.Source: §I · p. 2
- More instructions neither increased agent-file-generation cost nor guaranteed higher-quality generated code.The study provides empirical evidence using real-world open-source projects and a replication package.Source: §I · p. 2
II. METHODOLOGY
The study selects five trivial Python libraries as a controlled case study and evaluates their generated counterparts. Selection applies size or complexity criteria and requires human-written tests.
A. Data Source
- The case study uses five trivial Python libraries to examine agentic instructions while limiting confounding factors from larger or more complex projects.The authors sampled from a prior PyPI library dataset and limited the set to support consistent validation and evaluation.Source: A. Data Source · p. 3
- Libraries qualified through total Lines of File ≤35 or maximum single-function Cyclomatic Complexity ≤10, and each selected project included human-written test cases.All five libraries had maximum single-function CC ≤10; their LoF, file count, and function count varied.Source: A. Data Source · p. 3
- The selected libraries cover email verification, audio comparison, link conversion, bit-array manipulation, and a Python actor-library implementation.The set comprises areyoustillthere, audiodiff, autolink, bitmap, and compactor.Source: A. Data Source · p. 3
B. Agent-file Generation Process (RQ1)
The agent-file generation process creates each instruction combination as an AGENTS.md file through a Plan-and-Build workflow. Token usage is then analyzed for combinations with the highest and lowest usage.
B. Agent-file Generation Process (RQ1)
- The experiment generated 31 agent files per library, for 310 files across five libraries and two LLMs.The combinations included one to five of the five instruction categories; OpenCode supplied the Plan and Build agent modes.Source: B. Agent-file Generation Process (RQ1) · p. 3
- The Plan Agent inspected the library and produced a structured agent-file plan, while the Build Agent executed that plan and wrote one AGENTS.md file.The Plan Agent had read-only access; the Build Agent could edit workspace files.Source: B. Agent-file Generation Process (RQ1) · pp. 3–4
- The generation prompt supplied the chosen instruction combination as context, alongside a role, directive, and constraints for the documentation task.The Plan Agent first created a plan for the Build Agent to execute.Source: B. Agent-file Generation Process (RQ1) · pp. 3–4
- Token usage was recorded from Plan and Build interactions, then analyzed to identify the highest- and lowest-usage combinations by category and model.The analysis also identified the highest- and lowest-token combinations across all categories, libraries, and models.Source: B. Agent-file Generation Process (RQ1) · p. 4
C. Code Generation Process (RQ2)
The code-generation process reconstructs each library from its agent file, then evaluates compilation, test performance, and structural similarity to the human-written original. Structural metrics are not treated as direct measures of overall code quality.
C. Code Generation Process (RQ2)
- Each reconstruction used a single prompt and one AGENTS.md file, with the Plan Agent creating a full development plan and the Build Agent executing it.The working directory contained only the agent file; a five-minute time limit was set, though agents often ignored it.Source: C. Code Generation Process (RQ2) · p. 4
- Compilation success required generated code to pass Python bytecode compilation with python -m compileall without errors.This measures syntactic validity and basic module-level cohesion.Source: C. Code Generation Process (RQ2) · p. 4
- Functional correctness was evaluated by running the original, unmodified human-written test suites against generated libraries.Results were categorized as Passed, Partial, or Failed; Partial includes any result passing at least one but not all tests, masking differences in pass rates.Source: C. Code Generation Process (RQ2) · p. 4
- Structural similarity compares generated and original libraries using Lines of File, File Count, total Cyclomatic Complexity, and Function Count.Total CC was used because maximum function-level CC often tied, and these structural metrics do not directly establish maintainability, readability, architectural quality, or overall code quality.Source: C. Code Generation Process (RQ2) · pp. 4–5
III. RESULTS
Across the evaluated libraries, instruction combinations alone did not determine agent-file token usage, and successful compilation often did not translate into functional correctness. Instruction categories showed different associations with correctness and structural similarity.
A. Agent-file Generation Cost (RQ1)
- Token usage did not correlate with instruction count; the four-category combination of Repository Overview, Function Description, Implementation Details, and Reconstruction Steps using GPT-5.4 had the highest usage.The lowest-usage combinations varied across libraries and models, and the lowest observed combination was Gemma 4 on bitmap with Function Description and Implementation Detail.Source: A. Agent-file Generation Cost (RQ1) · p. 5, Table IV · p. 6
- Token usage varied more with the LLM and target-library characteristics than with the instruction combination.All five highest-token-usage combinations were for compactor; the authors note its higher file count as a possible explanation, not one established by these results.Source: A. Agent-file Generation Cost (RQ1) · p. 5
B. Instruction Categories for Generated-Code Effectiveness
- Gemma 4 compiled 88.39% of variants and GPT-5.4 compiled 98.06%, with compilation largely insensitive to instruction categories.The generated libraries compiled successfully across nearly all instruction combinations.Source: B. Instruction Categories for… · pp. 5–6, Table V · p. 6
- No Gemma 4 variant passed all human-written tests, while 12.99% of GPT-5.4 variants passed all tests; Function Description led Gemma 4 partial passes and Reconstruction Steps led GPT-5.4 full passes.The category advantages over the next-best category were only 1.25% for Gemma 4 and 3.75% for GPT-5.4, and the highest test success rates used one-, two-, or three-category combinations.Source: Table VI · p. 6, B. Instruction Categories for… · pp. 6–7
- Reconstruction Steps appeared most often among closest Lines of File and total Cyclomatic Complexity variants, while Testing and Reconstruction Steps jointly appeared most often for File Count and Function Count.Reconstruction Steps appeared in 6 of 10 closest Lines of File cases and 7 of 11 closest total Cyclomatic Complexity cases; Testing and Reconstruction Steps each appeared in 11 of 16 closest File Count cases, and jointly appeared in 12 of 19 closest Function Count cases.Source: B. Instruction Categories for… · p. 7
- Function Description had Gemma 4’s highest partial-pass rate, while Reconstruction Steps had GPT-5.4’s highest full-pass rate, but neither category was optimal across outcomes.The findings also associate Reconstruction Steps with closest Lines of File and total Cyclomatic Complexity, and its combination with Testing with closest File Count and Function Count.Source: B. Instruction Categories for… · p. 7
IV. DISCUSSION
The discussion section introduces implications, future-work potential, and threats to validity.
IV. DISCUSSION
- The section frames its discussion around implications, future work, and threats to validity.The supplied passage names these topics but gives no further detail.Source: §IV · p. 7
A. Implications
The implications emphasize that instruction-category value varies by code-generation objective and that instruction count alone is not a reliable guide to token cost or code quality. The authors propose configurable, task-aware instruction selection as a direction for future work.
A. Implications
- Instruction categories affect different code-quality aspects, and more categories do not guarantee higher-quality code or lower agent-file token cost.The authors suggest tools that identify, rank, or compose relevant instructions for a reconstruction or code-generation objective.Source: A. Implications · pp. 7–8
- The value of an agent file depends more on instruction relevance than on the number of categories it contains.The authors also report that no single category was optimal for all evaluated outcomes.Source: A. Implications · pp. 7–8
- The authors propose treating agent files as configurable resources, with task-aware selection of instructions relevant to the repository, model, and objective.Future work could examine category selection for missing-function implementation or unit-test repair, and benchmarks could evaluate individual instructions rather than only complete agent files.Source: A. Implications · pp. 7–8
B. Threats to Validity
The study’s validity is constrained by its evaluation measures and experimental setting, including limited coverage of software quality and a single reconstruction run per condition.
B. Threats to Validity
- Token usage measures agent-file generation cost, not agent-file quality; compilation, tests, and structural metrics also omit security, performance, reliability, maintainability, readability, and architecture.Human-written tests may miss intended behaviors because they focus on important functionality rather than the libraries’ entire behavior.Source: B. Threats to Validity · p. 9
- LLM nondeterminism and prompt wording can vary results, while independently generated files may contain different amounts of information across conditions.Source: B. Threats to Validity · p. 9
- Instructions can include content from other categories, blurring condition boundaries and affecting the observed results.Each condition was also evaluated with only one reconstruction run; repeated runs could produce different outcomes and a more robust evaluation.Source: B. Threats to Validity · p. 9
- The study evaluates only library reconstruction from scratch; category effects may differ for bug fixing or refactoring, and results may not generalize to other harnesses.Source: B. Threats to Validity · p. 9
- Findings are preliminary and may not generalize beyond five trivial Python libraries, two LLMs, one agent harness, one reconstruction task, and one run per condition.The authors call for larger and more diverse systems, repeated conditions, and additional agent-file formats and harnesses.Source: B. Threats to Validity · p. 9
V. RELATED WORKS
Prior work leaves unclear which agent-file instruction categories contribute to code-generation outcomes; this study examines those categories across cost and generated-code measures.
V. RELATED WORKS
- Existing studies report that agents struggle to reconstruct complete libraries and that repository-level agent files can sometimes hinder performance.Source: §V · p. 9
- Which instruction categories contribute to these outcomes remains unclear.Source: §V · p. 9
- The study investigates category-level effects on token consumption, functional correctness, and generated-library structure.Source: §V · p. 9
A. AGENTS.md Impact on Tokens
Prior work measured agent-file costs and effects without isolating instruction categories; this study examines categories’ roles in cost and generated-code outcomes.
A. AGENTS.md Impact on Tokens
- Lulla et al. found that AGENTS.md reduced generation costs by approximately 20%, but treated it as one artifact and did not test category-specific effects on cost or correctness.Source: A. AGENTS.md Impact on Tokens · p. 9
- This study analyzes individual instruction categories’ contributions to agent-file generation cost and generated-code correctness.Source: A. AGENTS.md Impact on Tokens · p. 10
B. Repository-level Agentic Context Files
- Gloaguen et al. reported that agent files generally reduced task success rates while increasing inference costs by more than 20%; they attributed this to broader exploration, testing, and file traversal.They recommended keeping repository-level instructions to the minimum necessary for development tasks.Source: B. Repository-level Agentic Context… · p. 10
- The study evaluates how each agent instruction category relates to generation cost, functional correctness, and generated-library structure.Source: B. Repository-level Agentic Context… · p. 10
C. Library Generation from Scratch
The study situates its library-reconstruction evaluation among prior work, then reports category-related outcomes across models and libraries, with scope limitations and a replication package.
C. Library Generation from Scratch
- Commit0 evaluates library reconstruction from natural-language requirements, but not instruction-content effects; this study examines category contributions to token use, correctness, and structural similarity.Source: C. Library Generation from Scratch · p. 10
VI. CONCLUSION
- The evaluation crossed 31 non-empty combinations of five instruction categories with two LLMs and five human-written Python libraries, measuring token use, compilation, tests, and four structural metrics.Source: §VI · p. 10
- Instruction quantity alone did not reliably indicate agent-file generation cost: highest and lowest token-use combinations varied across models and libraries.The highest usage was for Repository Overview, Function Description, Implementation Details, and Reconstruction Steps with GPT-5.4; both lowest-use combinations included Function Description and Implementation Details with Gemma 4.Source: §VI · p. 10
- No Gemma 4 variant passed all human-written tests, compared with 12.99% of GPT-5.4 variants; Reconstruction Steps most often aligned with closeness in Lines of File and total Cyclomatic Complexity.Compilation succeeded for 88.39% of Gemma 4 variants and 98.06% of GPT-5.4 variants; Function Description had Gemma 4’s highest partial-pass rate, while Reconstruction Steps had GPT-5.4’s highest full-pass rate.Source: §VI · p. 10
- Because the study uses five small libraries, two LLMs, one harness, one reconstruction task, and one run per condition, its findings challenge any assumption that more instructions necessarily improve outcomes.The authors call for broader systems and repeated experiments to support more reliable instruction-selection guidance.Source: §VI · p. 10
VII. REPLICATION PACKAGE
- The replication package contains datasets and experimental artifacts, including generated agent files, reconstruction metrics, evaluation metrics, analysis scripts, and results.Source: §VII · p. 10