L2MAC: Large Language Model Automatic Computer for Extensive Code Generation

1 INTRODUCTION

• Fixed context windows limit LLMs’ ability to generate extensive, logically coherent codebases because distant information is truncated or lost.The limitation undermines tasks requiring dynamic integration of multiple outputs and attention to distant information.Source: §1 · p. 1
• Existing memory-augmented methods mainly read external information, append new memories, or use specialized stores, limiting modification and cross-domain adaptation.These approaches lack in-place updates and consistency mechanisms needed for interdependent code structures.Source: §1 · pp. 1–2
• L2MAC combines task-oriented context management, precise memory read/write operations, and checks for errors and instruction completion.The framework uses a Control Unit to tailor context, update existing memory regions, and trigger corrections from syntactic errors and unit tests.Source: §1 · p. 2
• Code-L2MAC instantiates L2MAC for code generation, while the framework is designed as a general-purpose stored-program automatic computer for long outputs.The implementation uses a Control Unit and entire-memory file-store tools for code generation.Source: §1 · p. 2, Figure 1 · p. 2
• 90.2% Pass@1 on HumanEval is reported alongside state-of-the-art large-codebase results and successful entire-book generation.The paper also reports self-generated unit tests and tool-based verification for correcting errors in large interrelated code structures.Source: §1 · p. 3

2 BACKGROUND

• An LLM maps a fixed-length token context to a probability distribution over output tokens, so its context length is finite.The paper gives GPT-4 and GPT-3 context lengths of 8,192 and 4,097, respectively.Source: §2 · p. 3
• The paper motivates L2MAC as a parallel extension of LLMs toward stored-program computers, building on the relationship between finite-state and stored-program computational models.External read-and-write memory has been shown to let an LLM simulate a universal Turing machine, although L2MAC focuses on LLM-based automatic computers.Source: §2 · p. 3
• Stored-program computers combine a processor, Control Unit, and memory that stores instructions and data for automatically executing a specified task.The Control Unit extracts instructions and data, sets processor state, and overwrites memory with output.Source: §2 · p. 3

3 L2MAC FRAMEWORK

• L2MAC comprises an LLM processor, a memory file store, and a Control Unit that controls execution and provides read-and-write capabilities.The framework leaves some implementation choices open while fixing these core functionalities.Source: §3 · p. 3

3.1 LLM-BASED PROCESSOR

• The LLM processor generates task outputs and can request memory reads, writes, and output checks through tools assisted by the Control Unit.Its stochastic outputs and limited context require explicit context handling and periodic correctness checks.Source: §3.1 · pp. 3–4

3.2 MEMORY

• The instruction registry stores the prompt program’s sequential steps, while the file store holds information to read, write, evaluate, and ultimately return as output.Each instruction is executed by a separate LLM agent.Source: §3.2 · p. 4

3.3 CONTROL UNIT

• The Control Unit loads instructions, manages multi-turn context, executes tools, checks completion, and resets context using summaries when instructions finish.It can continue an instruction across context windows while retaining progress and file-selection guidance.Source: §3.3 · p. 4, §3.3.1 · pp. 4–5

3.3.1 TASK-ORIENTED CONTEXT MANAGEMENT (P1)

• To stay within the context limit, the Control Unit minimizes context use, stores relevant output in the file store, and carries forward summaries for subsequent iterations.In coding, summaries indicate which files should be read or modified and what progress has been made fixing errors.Source: §3.3.1 · p. 5

3.3.2 PRECISE READ/WRITE TOOLS FOR ENTIRE MEMORY (P2)

• Unlike append-only or highly structured memories, L2MAC lets agents scan, add, delete, and update any part of the file store.Editable memory supports downstream subtasks that require modifying earlier outputs because of imperfect planning or hallucination.Source: §3.3.2 · p. 5

3.3.3 CHECKING THE GENERATED OUTPUT (P3)

• L2MAC evaluates the file store after writes with domain-specific checkers or self-generated tests, returning errors to the LLM for iterative correction.Cycle prompting also checks whether the current instruction is complete, preventing discussion without a stored solution.Source: §3.3.3 · pp. 5–6

4 CODE-L2MAC

• Code-L2MAC applies L2MAC to large codebase generation while separating the general long-generation framework from coding-specific design choices.The implementation details its memory layout and read logic while leaving the core framework adaptable to other domains.Source: §4 · p. 6
• The file store contains files shorter than the residual context margin, with semantically meaningful paths that guide retrieval and prioritization.When context fills, summaries preserve which files to read or exclude so the LLM can scan memory systematically across iterations.Source: §4 · p. 6

Single and reflecting LLMs

• Single LLM and reflecting approaches remain confined by the context window even when reflection or evaluation improves smaller code-generation tasks.Code-L2MAC instead uses evaluation for both in-context learning and codebase consistency.Source: §5 · pp. 6–7

Comparison criteria

• Table 1 compares related methods on context management, precise read/write access for entire memory, and generated-output checking.These properties frame the distinctions relevant to extensive code-generation tasks.Source: Table 1 · p. 7

Memory-augmented LLMs

• Existing memory-augmented LLMs mainly target conversation, summarization, retrieval, or structured reasoning rather than general-purpose large codebase generation.Read-focused methods and specialized tuple, dictionary, or database memories do not provide a broadly applicable coding memory.Source: §5 · pp. 6–7

Autonomous agent LLMs with memory

• Autonomous agents can replan from high-level goals, but runtime reformulation may deviate from detailed user intentions.Code-L2MAC is designed to fulfill the detailed task without altering or forgetting parts of it.Source: §5 · p. 7
• AutoGPT summarizes the task into six sentences and compresses action history, risking forgotten files and content during iterative code generation.Its rolling summary can lose knowledge of previously created files and is compounded by continual replanning.Source: §5 · p. 7

6 EXPERIMENTS AND EVALUATION

• The evaluation compares Code-L2MAC with AutoGPT, GPT-4, and reflecting methods across three system-design tasks and HumanEval.All benchmark methods receive the same tools used by Code-L2MAC.Source: §6 · pp. 7–8
• Features % measures the proportion of user-specified features fully and functionally implemented, alongside # Errors, LOC, and Tests Passed.Features % is assessed by a separate GPT-4 call that examines the generated code; results include 95% confidence intervals.Source: §6 · pp. 7–8

6.1 MAIN RESULTS

• Code-L2MAC fully implements the highest percentage of user-specified features across system-design tasks, with minimal syntactical errors and many passing tests; it also reaches 90.2% Pass@1 on HumanEval.The evaluation further demonstrates applicability to extensive text generation by writing an entire book.Source: §6.1 · p. 8, Table 2 · p. 8

6.2 INSIGHT EXPERIMENTS

• Code-L2MAC retains more user-specified requirements in its task instructions, whereas AutoGPT compresses them into six sentences and loses crucial task information.This tests task-oriented context management during operation.Source: §6.2 · p. 8, Figure 3 · p. 9
• Figure 4 tracks file creation, reading, and writing across operation steps to examine precise access to the evolving codebase.The analysis asks whether agents can understand and update files created many instruction steps earlier.Source: §6.2 · p. 8, Figure 4 · p. 9
• Code-L2MAC detects and corrects syntactical errors by receiving evaluator output and modifying the codebase, while AutoGPT allows errors to compound.This evaluates output checking and error correction during generation.Source: §6.2 · p. 9
• Code-L2MAC uses self-generated unit tests to check functionality and repair regressions after updating existing files, unlike AutoGPT.The comparison connects test-based checking with maintaining compatibility among code files.Source: §6.2 · p. 9

7 CONCLUSION AND FUTURE WORK

• L2MAC is presented as a practical stored-program computer framework that augments LLMs with memory for long output generation.Code-L2MAC instantiates the framework for large codebases and is described as surpassing existing solutions.Source: §7 · p. 9
• L2MAC’s performance remains bounded by the underlying LLM across planning, tool use, and other tasks.This is identified as a limitation and an open direction for future work.Source: §7 · p. 9
• Listing all file names in the context window imposes an implicit constraint on codebase scale.The paper states that this design limitation could be improved.Source: §7 · p. 9
• The authors identify complex instruction flows, instruction reprogramming, and interpretable prompt programs as directions for future work.These directions are discussed alongside the framework’s current design constraints.Source: §7 · p. 9
• The tool may produce undesirable outputs when tasks are incorrectly specified and is prone to hallucinations, motivating content filtering of final outputs.The ethics statement recommends a content filter for the final output.Source: §7 · p. 10

Appendix

• The implementation and experimental materials are reported as available for reproducibility, including code and detailed appendices.The paper points to benchmark, method, implementation, and metric details in the appendices.Source: §7 · p. 10, Appendix · p. 16

A STORED-PROGRAM COMPUTERS (SPC)

• Stored-program computers combine a processor, Control Unit, and memory for instructions and data, enabling reprogrammable execution without manual intervention.Their reliable operation also includes checks and error-correction mechanisms for processor outputs.Source: A STORED-PROGRAM COMPUTERS (SPC) · p. 17

A.1 SPC IS A UNIVERSAL TURING MACHINE

• An SPC corresponds to a universal Turing machine because its memory stores both program instructions and data, allowing it to perform any computable task given suitable instructions.The program counter corresponds to the machine state, while memory instructions correspond to the transition function.Source: §A.1 · p. 17

A.2 WHAT IS AN AUTOMATIC COMPUTER?

• The paper defines an automatic computer as a system that executes predefined instructions without human intervention, unlike an autonomous agent that devises and adapts its own plans.L2MAC is intended as a general-purpose instruction-following computer that can execute different prompt programs without system redesign.Source: §A.2 · pp. 17–18
• The paper positions LLMs as finite-state machines that can be extended with stored-program and external-memory mechanisms toward LLM-based automatic computers.This framing does not aim to reproduce conventional CPU arithmetic and logic operations, but to explore LLM-based automatic computation.Source: §A.2 · p. 17

B CONTROL UNIT OPERATION

• The Control Unit manages the LLM context window, executes instructions, checks outputs, and enables read/write tool use.It operates as a finite-state machine coordinating the LLM and external memory.Source: B CONTROL UNIT OPERATION · p. 18
• Code-L2MAC decomposes a user task into sequential instructions, reads relevant files, and generates or updates files in the external store.This conditions each instruction on existing outputs without losing vital context.Source: Figure 5 · p. 18
• Automatic evaluation checks syntax and self-generated unit tests, enabling iterative correction while building a complete codebase in the file store.The checks target both code correctness and desired functionality.Source: Figure 5 · p. 18

C CODE-L2MAC HIGH LEVEL IMPLEMENTATION DETAILS

• Code-L2MAC generates a sequential prompt program from the user’s feature requirements, then executes its instructions through a control unit and separate LLM-driven steps.The control unit initializes the instruction registry, queries the LLM to define the program, and processes instructions sequentially.Source: C CODE-L2MAC HIGH LEVEL IMPLEMENTATION… · pp. 19–20
• The file store contains the generated code, partitioned into context-sized files whose paths and names provide semantic cues for locating related functionality.The design lets the LLM read or write a full file while avoiding the need to place the entire codebase in context.Source: C CODE-L2MAC HIGH LEVEL IMPLEMENTATION… · pp. 19–21
• The control unit maintains bounded context by carrying system and summary messages, iterating on each instruction, and summarizing progress when execution approaches the context limit.After an instruction completes, evaluator feedback is incorporated and the next cycle retains the system message, summary, and current instruction.Source: C CODE-L2MAC HIGH LEVEL IMPLEMENTATION… · pp. 20–21
• After writes, syntax checks and existing tests are run, and evaluator feedback can trigger completion or further changes before the next instruction executes.The process halts after all instructions complete and returns the file store.Source: C CODE-L2MAC HIGH LEVEL IMPLEMENTATION… · pp. 20–21

C.1 READ / WRITE IMPLEMENTATION DETAILS

• Code-L2MAC exposes file-listing, file-viewing, reading, writing, execution, and deletion tools so the LLM can inspect, extend, test, and revise the codebase.The control unit lists paths, returns selected file contents, and executes requested writes or code runs.Source: C CODE-L2MAC HIGH LEVEL IMPLEMENTATION… · pp. 19, 21
• File paths remain visible as a semantic prior for selecting files, while context summarization records relevant files when functionality spans multiple files.The LLM can continue reading files after an uninformative initial read, and the control unit summarizes relevant context when the window fills.Source: §C.1 · pp. 21–22

D.1 VANILLA LLMS

• Prior long-context and memory methods are limited by context scaling, weak long-context retrieval, non-updatable memories, read-only interaction, or domain-specific memory structures.The related-work discussion contrasts these limitations with Code-L2MAC’s file-based, writable memory and control-unit execution.Source: §D.1 · p. 22, §D.2 · p. 22, §D.3.1 · p. 23

D.3.3 AUTNOMOUS AGENT LLMS WITH MEMORY

• Unlike autonomous agents that lossy-compress goals and action history, Code-L2MAC preserves a detailed initial plan and tracks generated files as the evolving memory state.Unlike reflection methods restricted to recent context, it can refine earlier files outside the current context and manage total outputs larger than the context window.Source: §D.3.3 · p. 24, §D.5 · p. 24

D.6 TURING MACHINES, STORED PROGRAM COMPUTERS AND MACHINE LEARNING

• Prior LLM-based UTM and SPC instantiations are theoretically insightful but only simulate existing computational frameworks.The paper contrasts these simulations with its own focus on practical task-solving capability.Source: §D.6 · p. 25
• L2MAC instead exploits the SPC framework to help an LLM automatically solve potentially long tasks from a single user prompt.Its stated focus is the LLM's stochastic natural-language capability, not reproducing a UTM or SPC.Source: §D.6 · p. 25
• A related memory-augmented recurrent approach extended a Recursive Neural Network toward Turing-machine or von Neumann-style computation by supervised learning.That work targeted inference of simple algorithms after training, unlike L2MAC's long-task generation focus.Source: §D.6 · p. 25

E.1 CODEBASE GENERATION SYSTEM DESIGN TASK

• The benchmark converts conventional system-design exercises into implementation tasks requiring fully functional code that satisfies user-specified requirements.The tasks target architecture, modules, components, interfaces, and data structures through executable minimal systems rather than conceptual outlines.Source: §E.1 · p. 25
• The benchmark uses realistic prompts derived from system-design interview questions and evaluates each method on the same requirements and resulting codebase output.Evaluation methodology is detailed in Appendix G.Source: §E.1 · pp. 25, 27
• The task suite covers feature-rich applications including URL shortening, microblogging, real-time chat, recipe sharing, event planning, and financial tracking.Requirements include domain-specific capabilities such as analytics and expiration, social interactions, encrypted messaging and offline mode, recommendations, event logistics and budgeting, and personal finance tracking.Source: §E.1 · pp. 25–26
• The requirements cover URL shortening with active, legitimate validation, unique or custom links, original-URL redirection, click/date-time/geographic analytics, user accounts, administrative controls, and expiration dates/times.Source: §E.1 · p. 27
• The codebase-generation task requires a personal finance application covering accounts and security, transaction tracking, budgets, investments, reports, and alerts.Requirements include bank-account linking, multifactor authentication, expense and income visualization, budget-limit alerts, investment performance tracking, financial reports, and customizable alerts.Source: §E.1 · p. 29

E.2 HUMANEVAL BENCHMARK

• HumanEval, introduced by Chen et al. (2021), evaluates standalone Python functions generated from docstrings using held-out unit tests across 164 programming problems.The problems assess language comprehension, algorithms, and simple mathematics, resembling software interview questions.Source: §E.2 · pp. 29–30

F BENCHMARK METHOD IMPLEMENTATION DETAILS

• The benchmark includes AutoGPT, GPT4, CodeT, Self-Refine, and Reflexion, adapting each with file access, testing, sampling, refinement, or reflection procedures for codebase generation.AutoGPT failures that produced no code or entered infinite loops were excluded from its results.Source: F BENCHMARK METHOD IMPLEMENTATION… · pp. 30–31
• Code-L2MAC executes a generated instruction program through a control unit, using separate sub-agents, file tools, tests, and code checks to build the codebase incrementally.The control cycle selectively exposes existing files, permits file writes and execution, and asks agents to verify or correct each subtask.Source: F BENCHMARK METHOD IMPLEMENTATION… · p. 31, §F.1 · pp. 32–34
• Code-L2MAC retries an instruction up to rMax = 30, restarts with a progress summary when context is exceeded, and raises temperature to escape detected message loops.The implementation uses GPT4-0613 at temperature 0.01, with temperature increases capped at 1.0 during loop escape.Source: F BENCHMARK METHOD IMPLEMENTATION… · p. 31, §F.1 · p. 34

F.1 CODE-L2MAC LOW LEVEL IMPLEMENTATION DETAILS

• The low-level setup constrains agents to provided function calls, local in-memory database mocks, whole-file writes, and autonomous execution without human input.The system prompts require functional code, existing-file inspection, and test generation, while persistence between steps or write operations is unavailable.Source: §F.1 · pp. 32–33

G EVALUATION METRICS

• The metric is motivated by the absence of automated validation tools and the fact that large-scale code can fulfill the same high-level requirements through multiple implementation approaches.Thus, evaluation compares how completely different methods implement user-specified features rather than requiring one fixed implementation.Source: G EVALUATION METRICS · p. 34
• Features % measures the proportion of user-specified features that are fully and functionally implemented in the generated codebase.It is computed as implemented features divided by total specified features, reported as a percentage.Source: G EVALUATION METRICS · p. 34
• An iterative GPT-4 API call examines all generated files, counts fully implemented features, and a regex parser extracts the final count.Files are supplied as named strings containing filenames and contents; the resulting count is converted into Features %.Source: G EVALUATION METRICS · p. 34
• A human study found that Features % correlates with professional software engineers’ manual code-review counts of implemented features.The human comparison is reported in Appendix H.6.Source: G EVALUATION METRICS · p. 34
• The generated code implements 15 checked features spanning URL shortening, redirection, analytics, user accounts, administration, and expiration.The feature checklist includes 15 completed items, while custom short links and clicker geographical location remain unchecked.Source: G EVALUATION METRICS · pp. 35, 39
• A regex parser extracts the final “FEATURES_FUNCTIONAL” line, enabling checklist-based counting of fulfilled features in the generated codebase.The checklist helps the LLM track which requirements are fully implemented and supports counting against user-specified feature requirements.Source: G EVALUATION METRICS · p. 39
• The evaluation reports generated lines of code, PyLint-detected syntax errors, self-generated tests passed with PyTest, and executable-line coverage measured with Coverage.PyLint is configured to detect only errors, while coverage is computed by running the generated unit tests at evaluation time.Source: G EVALUATION METRICS · pp. 39–40
• Each benchmark method is evaluated over 10 random-seed runs per task environment, with every metric reported alongside its 95% confidence interval.This repeated-run procedure applies across task environments and benchmark methods.Source: G EVALUATION METRICS · p. 40

G.1 MOTIVATION FOR FEATURE %

• Feature % measures the proportion of user-specified functions or features that GPT4 judges to be fully implemented, providing a graded alternative to the sparse “Is it fully functional?” metric.The metric reflects degrees of correctness because codebases may contain bugs while remaining partially functional.Source: §G.1 · p. 40

H.1 HUMANEVAL BENCHMARK RESULTS

• 90.2% Pass@1 makes Code-L2MAC state of the art on HumanEval, outperforming all listed baselines and standalone GPT-4.HumanEval evaluates Python function completion on 164 problems using held-out unit tests, with pass@k measuring top-k functional accuracy.Source: §H.1 · pp. 40–41, Figure 7 · p. 40

H.2 L2MAC FOR WRITING AN ENTIRE BOOK

• L2MAC also generates long general-purpose text by producing the complete recipe book “Twirls & Tastes: A Journey Through Italian Pasta.”The demonstrated book spans regional history, cultural context, recipes, pasta-making techniques, and a glossary across Italian regions.Source: §H.2 · pp. 41–42
• For book generation, L2MAC replaces code-specific error checking with a general consistency check against previously generated output and uses GPT-4 1106-Preview.This preserves the same framework while adapting its checking mechanism from code correctness to textual coherence.Source: §H.2 · p. 41
• The system autonomously writes complete text files, views existing files before editing, and incorporates changes into the generated book without human input.Source: §H.2 · p. 46
• Each listed step is executed by a separate sub-ChatGPT model, with optional validation to verify successful completion.Source: §H.2 · p. 46
• Book construction begins with a detailed outline, proceeds through chapters in chronological order, and conditions later files on earlier outlines and chapters.Source: §H.2 · p. 46
• The initial planning prompt requires a step-by-step plan of at most 10 steps covering every feature requirement and including consistency checks during implementation.Source: §H.2 · p. 46
• L2MAC’s control-unit prompts were modified from code generation and checking to text generation and checking.Source: §H.2 · p. 47

H.3 ADDITIONAL MAIN RESULTS

• The comparison uses identical user-specified feature requirements for every method in the long code-generation task.The methods are evaluated under the same requirements described in Appendix E.Source: §H.3 · p. 47
• Code-L2MAC fully implements the highest percentage of user-specified features across all tasks, with minimal syntactical errors and many passing self-generated unit tests.Table 3 reports Features %, # Errors, LOC, Tests Failed, and Tests Passed, averaged over 10 random seeds with 95% confidence intervals.Source: Table 3 · p. 47
• The L2MAC URL shortener example includes application, configuration, service, and test files covering routing, accounts, administration, analytics, and error responses.The displayed tests check administrative operations, a successful home response, and a missing-URL 404 response.Source: §H.3 · pp. 47, 49
• The URL-shortener tests cover account creation, URL viewing, editing, deletion, analytics, administration, monitoring, custom aliases, and expiration handling.The tests assert successful responses and explicit errors for nonexistent users, URLs, unavailable aliases, invalid links, and expired links.Source: §H.3 · p. 50
• The component-level tests verify short-URL generation, original-URL recovery, custom aliases, click analytics with locations and timestamps, expiration errors, and user-account operations.User-account tests also check duplicate usernames, empty URL lists, successful edits and deletions, and nonexistent-user or nonexistent-URL errors.Source: §H.3 · p. 50
• The example implementations include URL shorteners with six-character random identifiers, redirection, click and geolocation tracking, and expiration-date handling.A Flask implementation returns status 201 when shortening a URL and redirects a valid, unexpired link with status 302.Source: §H.3 · pp. 51–52

H.4 CODE L2MAC IN ACTION.

• The L2MAC transcript begins with planning for an Online Chat App environment task and specifies fully functional code without placeholders.The instructions restrict code output to provided functions and the write_files operation.Source: §H.4 · p. 52
• L2MAC decomposed the Global Chat Service into up to 10 detailed steps covering setup, authentication, profiles, contacts, messaging, groups, stories, web access, offline mode, and final testing.Each planned step included implementation requirements and tests intended to verify completion.Source: §H.4 · pp. 53–54
• The coding setup required autonomous full-file generation, compatibility with existing code, prescribed function-call JSON, and an in-memory dictionary because no database or persistent disk storage was available.The instructions also required viewing files before writing and using tests to validate the generated implementation.Source: §H.4 · p. 54
• Separate sub-ChatGPT agents executed listed subtasks through controlled function calls, while the control unit checked completion and could request correction or mark a step complete.The workflow used functions such as write_files, pytest_files, and sub_task_step_complete, with the control unit conditioning new code on existing files.Source: §H.4 · pp. 54–55
• 1 test passed in 0.04 s after L2MAC implemented the initial Flask application and in-memory user database.The generated app.py and test_app.py created and retrieved user data, and pytest reported 1 passed.Source: §H.4 · p. 55
• The Code-L2MAC workflow incrementally builds applications, establishing the main application and Flask structure, mock database, and passing tests before loading the next instruction.The mock database uses an in-memory dictionary to store and retrieve user data.Source: §H.4 · p. 55

H.5 ADDITIONAL DIVERSE PROGRAMMING CODE GENERATION TASKS

• Code-L2MAC fully implements the highest percentage of requested features across recipe, event-planner, and financial-tracking applications, with minimal errors, many passing tests, and high Cov %.Table 4 averages results over 10 random seeds with 95% confidence intervals; Cov % measures unit-test code coverage.Source: §H.5 · p. 55, Table 4 · p. 56

H.6 HUMAN EXPERT VALIDATION OF FEATURES IMPLEMENTED PERCENTAGE METRIC

• Human expert review finds Code-L2MAC significantly outperforms baselines on Human Expert Features %, while Features % strongly correlates with expert judgments (ρ = 0.976).This supports Features % as a scalable, cost-effective proxy for ground truth, consistent with prior work on LLM evaluators (Chiang & Lee, 2023).Source: Table 5 · p. 56, §H.6 · p. 56

H.7 CHALLENGES AND THE EVALUATION OF HUMAN-WRITTEN TEST-CASES

• Hand-written tests constrain implementation choices and require expert-crafted, task-specific cases, limiting their scalability for open-ended codebases.Providing tests can also hint at how the implementation should be designed.Source: §H.7 · pp. 56–57
• ρ = 0.695 between HT % and Features % supports human-written-test performance as a correlated evaluation signal.The metric was computed over five random-seed runs after adapting tests to each generated codebase.Source: §H.7 · pp. 56–57
• ρ = 0.928 between HT % and Features % when tests were excluded from generation provides further evidence that HT % tracks the proposed evaluation metric.This experiment removed hand-written tests from every generation context while retaining the same overall setup.Source: §H.7 · p. 57

H.8 GENERATING 1,000+ LINES OF CODE WITH CODE-L2MAC

• 1,000+ LOCs were generated by Code-L2MAC-Large for an Online Chat application, reaching 5x the LOC of AutoGPT.This variant removed restrictions such as limiting the number of instructions to 10.Source: §H.8 · p. 57, Table 8 · p. 58

H.9 CODE-L2MAC ABLATION WITH NO INSTRUCTION SUMMARIZATION MESSAGE

• Removing the instruction summarization message did not affect output-code quality by much, because completed outputs remain available in the external file store.The summarization step was intended to help the LLM locate relevant files faster by exploiting expected similarity between contiguous instructions.Source: §H.9 · pp. 57–58, Table 9 · p. 58

H.10 ADDITIONAL TASKS ON IMPLEMENTING A NEW FEATURE IN AN EXISTING LARGE CODE BASE WITH CODE-L2MAC OF UP TO 165,000 LOCS

• Code-L2MAC implemented new features while operating on existing codebases of up to 165,000 LOCs.The tasks used three open-source applications and required conditioning new code generation on the existing codebase; results were averaged over 10 random seeds.Source: §H.10 · pp. 58–59, Table 10 · p. 59

I FUTURE WORK

• The current implementation limits file handling because reading outputs an entire file and writing rewrites it, while listing all file paths also consumes context.The authors propose selective file-part operations and hierarchical folder navigation as remedies.Source: I FUTURE WORK · p. 60

• L2MAC’s general-purpose stored-program design suggests it could extend beyond code generation to other task domains.Source: §I.1 · p. 61
• Effective evaluation tools are especially important for applying L2MAC to domains where output correctness must be checked.Source: §I.1 · p. 61
• Potential applications include AutoML, mathematical-proof generation, and long textual documents such as books.Source: §I.1 · p. 61
• Experiments on these additional applications remain beyond the scope of this paper.Source: §I.1 · p. 61