PaperlayerView exact revision on arXiv

Source-linked AI summary

LoLBench: Evaluating Coding Agents with Long-Horizon Proposals on Large Software Systems

Yun Peng, Zihan Wu, Zeyang Zhuang, Xin Zhou, Rui Shu, Xu Han, Chun Yong Chong, Yuan Wang, Jiakun Liu

arXiv:2609.37143v1cs.SEcs.CL

TL;DR

Many coding benchmarks evaluate implementation from detailed specifications, while practical modular development also requires grounding user intent and high-level design in an existing system. LoLBench evaluates this proposal-to-implementation process across large software systems. Agents resolve few tasks, and reference-derived implementation guidance improves resolved rates.

  • Problem

    Many coding benchmarks provide detailed specifications and evaluate implementation, leaving open whether agents can produce correct code directly from user intent and high-level design.

    Source: §1 · pp. 1–2

  • Method

    LOLBENCH is a multilingual benchmark of 100 modular development tasks across 29 large software systems, built from human-written enhancement proposals and evaluated with system-level tests.

    Source: §1 · pp. 2–3

  • Results

    Across 28 agents, the highest resolved rate is 14%; reference-derived implementation guidance improves resolved rates by 16–22 percentage points across three agents, reaching at most 34%.

    Source: §1 · p. 3, §5 · p. 10

  • Takeaways & Limitations

    The results identify perception and implementation as continuing challenges for coding agents in modular development on large software systems.

    Source: §5 · p. 10

    The evaluation covers 28 agents built from six models and five scaffolds, with one attempt per task and agent, so findings may not generalize to other agents or resource budgets.

    Source: G LIMITATIONS · p. 31

Abstract

from arXiv · show

Modern coding agents can deliver increasingly large repository-level changes, and recent benchmarks reflect this by emphasizing long-horizon tasks with large reference implementations. Many benchmarks evaluate coding agents' implementation capability to produce correct code edits from detailed specifications. However, practical modular development tasks also require the perception capability of grounding user intent and high-level design to derive a specification. We introduce LoLBench to evaluate both capabilities through the entire proposal-to-implementation process on large software systems. It is a multilingual benchmark of 100 tasks across 29 software systems in five domains. Each task provides a human-written enhancement proposal with user intent and high-level design. On average, proposals contain about 5,000 words, software systems contain 2.4 million source lines of code (LoC), and implementation pull requests (PRs) change approximately 5,500 LoC. Across 28 agents we evaluated, the best agent resolves only 14% of tasks and achieves a 52.7% Fail-to-Pass (F2P) pass rate. Failure analysis identifies incomplete code localization as a major bottleneck, while providing reference-derived file trees alongside API specifications improves resolved rates by 16--22 percentage points (2.4--17$\times$), reaching at most 34%. These results show that both perception and implementation remain central challenges for coding agents in practical modular development on large software systems. LoLBench is available at https://huggingface.co/datasets/lolbench26/LoLBench.

1 INTRODUCTION

Many coding benchmarks test implementation from detailed specifications, while practical modular development also requires grounding user intent and high-level design in an existing system. LoLBench evaluates that full process and finds substantial challenges in both perception and implementation.

1 INTRODUCTION

  • Many coding benchmarks provide detailed specifications or localized bug reports, leaving open whether agents can produce correct implementations directly from user intent and high-level design.Modular development combines the work of understanding an abstract proposal and implementing it while preserving compatibility with the rest of a large system.Source: §1 · pp. 1–2
  • Across 28 agents, the best resolved rate is 14%, while reference-derived file trees and API specifications improve three agents’ resolved rates by 16–22 percentage points, reaching at most 34%.Claude Code with Opus 5 also achieves the highest completed rate of 26.3% and an F2P pass rate of 52.7%.Source: §1 · p. 3
  • The paper distinguishes perception complexity—grounding intent and design into a specification—from implementation complexity—producing correct, regression-free changes in an existing system.The benchmark also uses system-level tests and maps F2P tests to implementable proposal sections for fine-grained evaluation.Source: §1 · p. 3
  • LOLBENCH evaluates 100 modular development tasks across 29 large software systems in five domains, using human-written enhancement proposals as task inputs.Agents must ground proposal intent and design in the existing system, then deliver regression-free changes; proposals average approximately 5,000 words, systems 2.4 million source LoC, and implementation PRs approximately 5,500 LoC.Source: §1 · pp. 2–3

2 RELATED WORK

Prior coding benchmarks largely assess implementation or isolate repository exploration; LoLBench is framed as evaluating both perception and implementation. The paper also situates its evaluation among coding agents that combine repository exploration, editing, testing, and repair.

2.1 CODING BENCHMARKS

  • Repository-level benchmarks primarily assess implementation correctness, while SWE-Explore (Zhang et al., 2026) isolates code-region ranking; LOLBENCH jointly evaluates grounding proposals and implementing changes.Its tasks use proposals written before their reference implementations and require cross-module changes in existing systems.Source: §2.1 · p. 3

2.2 CODING AGENTS

  • Software-engineering agents combine repository exploration, editing, testing, and repair, while general-purpose scaffolds and commercial systems provide varied agent environments.The evaluation selects representative models and scaffolds for the broader context required by modular development.Source: §2.2 · p. 4

3 BENCHMARK CONSTRUCTION

LoLBench is built by collecting qualifying enhancement-proposal and implementation pairs, aligning proposal requirements with code changes, packaging executable tasks, and augmenting tests. Its benchmark statistics describe large repositories and proposals with relatively little explicit implementation guidance.

3 BENCHMARK CONSTRUCTION

  • The construction process collects enhancement proposals and corresponding implementations, aligns proposal sections with changes, builds tasks, and augments behavioral tests.The paper presents this as a four-phase process.Source: §3 · p. 4

3.1 PROJECT COLLECTION

  • An enhancement proposal–implementation pair is retained only when the system, proposal, PR size, timing, uniqueness, and test criteria meet specified thresholds.The system must have at least 50k source LoC; the proposal must include an implementable section; and the PR must change at least 1k LoC across at least 10 files, with source code exceeding half of changed lines.Source: §3.1 · p. 4

3.2 EP-PR MATCHING

  • Proposal sections are classified as implementable or contextual, then changed files, classes, and functions are mapped to the implementable sections they directly realize.Mappings undergo structural and semantic validation and an LLM-judge review, with candidates retained at a weighted score of at least 8.0.Source: §3.2 · pp. 4–5

3.3 TASK CONSTRUCTION

  • Tasks give agents the pre-implementation proposal and codebase, hide the reference solution, and evaluate solutions with F2P behavior tests and P2P regression checks.Tasks are retained only when the reference solution applies, all F2P tests pass after it, and P2P tests pass both before and after it.Source: §3.3 · p. 5

3.4 TASK ENHANCEMENT

  • F2P suites are augmented with tests that distinguish incorrect solution mutants and, for tasks with under 50% original F2P line coverage, tests that improve coverage.F2P tests are also mapped to implementable proposal sections to assess partial progress.Source: §3.4 · p. 5

3.5 BENCHMARK STATISTICS

  • LOLBENCH spans 100 tasks, 29 systems, five domains, and eight programming languages; its repositories average 2.4 million source LoC and implementation PRs change 5,499 LoC.Only 7.3% of target class names appear in proposals, which the paper presents as evidence of limited implementation guidance.Source: §3.5 · pp. 5–6

4 EVALUATION

The evaluation finds low task resolution across 28 agents, with incomplete localization and missing cross-module context prominent in failure analyses. Reference-derived implementation guidance improves results, while model and scaffold choices affect effectiveness and efficiency.

4.1 EXPERIMENT SETUP

  • The evaluation paired six models with five scaffolds, leaving 28 agents after two pairings were unavailable; each task had one attempt except for provider or infrastructure retries.Agents ran in sandboxed tasks with public network access disabled and a six-hour wall-clock limit.Source: §4.1 · pp. 6–7

4.2 EFFECTIVENESS ANALYSIS

  • 14% was the highest resolved rate among 28 agents; Claude Code with Opus 5 also led completed rate at 26.3%.Across agents, mean and median resolved rates were 3.4% and 2.0%, respectively.Source: §4.2 · p. 7

4.3 EFFICIENCY ANALYSIS

  • With Opus 5, Claude Code and mini-SWE-agent resolved 14 tasks each, but mini-SWE-agent used 3.2× the time, 5.0× the generated tokens, and 2.8× the cost.Across models, relative variability was about 2.6 times that across scaffolds for Completed and F2P Pass rates.Source: §4.3 · p. 7, §4.2 · p. 7
  • Code localization was the largest phase for 26 of 28 agents during initial generation and 21 during repair, accounting for 60.4% and 44.2% of turns, respectively.Figure 3 separates initial solution generation from repair loops after failed verification.Source: §4.3 · pp. 7–8, Figure 3 · p. 7

4.4 FAILURE ANALYSIS

  • Code localization received the largest failure attribution at 30.9 units, nearly all of which came from missing cross-module context at 30.0 units.Code editing followed with 25.3 attribution units; the values are fractional weights aggregated across solution failures.Source: §4.4 · p. 8

4.5 PERCEPTION CAPABILITY ANALYSIS

  • Inspection recall ranged from 46.4–86.4% for files and 31.9–83.1% for functions, while inspection precision was 16.1–36.3% and 6.8–19.2%, respectively.Modification recall remained limited, and broad repository exploration did not ensure coverage of reference edit locations.Source: §4.5 · p. 9, Figure 4 · p. 9
  • Reference-derived file trees and API specifications increased resolved rates by 16–22 percentage points across three agents.For Claude Code with Opus 5, F2P pass rate rose from 52.7% to 72.7%.Source: §4.5 · p. 9, Table 4 · p. 9

5 CONCLUSION AND FUTURE WORK

LoLBench combines intent-to-implementation tasks with large software systems, and the evaluation identifies incomplete cross-module context as a major failure mode. The appendices describe the benchmark composition and a validation pipeline for matching enhancement proposals to implementation pull requests.

5 CONCLUSION AND FUTURE WORK

  • Across 28 agents, the highest resolved rate was 14%, while reference-derived implementation guidance improved resolved rates by 16–22 percentage points across three agents.The authors conclude that translating proposals into concrete changes remains challenging and identify perception capability as a future-work focus.Source: §5 · p. 10

A STATISTICS OF PROJECTS IN LOLBENCH

  • The 100 tasks span 29 projects in five domains and eight programming languages; all repositories exceed 50k source LoC, and 12 exceed one million.Mean repository size ranges from 52.5k source LoC for FastAPI to 7.7 million for OpenJDK.Source: A STATISTICS OF PROJECTS IN LOLBENCH · p. 12, Table 5 · p. 13

B DETAILS OF EP–PR MATCHING

  • EP–PR matching classifies proposal sections, maps changed files, classes, and functions to implementable sections, then validates mappings before quality ranking.The mapping artifact includes section summaries, PR-file summaries, mapped implementation blocks, and unmapped-section and file tables.Source: B DETAILS OF EP–PR MATCHING · p. 13

B.1 STATIC VALIDATION

  • Static validation checks mapping structure, completeness, arithmetic, and scope ownership, including changed class/function pairs recovered from pre- and post-PR syntax trees.The rules require each changed code entity to have a unique semantic owner.Source: §B.1 · pp. 13–14

B.2 SEMANTIC VALIDATION

  • Eight focused LLM audits assess semantic mapping criteria; candidates advance only after unresolved static and semantic findings are cleared.A curator reviews reported evidence, regenerates faulty mappings, and reruns both validation stages.Source: §B.2 · p. 15

B.3 QUALITY RANKING

The ranking procedure first enforces hard validation constraints, then scores valid EP-to-PR mappings for semantic quality. It weights section scores by importance and retains only candidates meeting the score threshold and expert review.

B.3 QUALITY RANKING

  • Valid mappings are ranked for semantic quality, while validation rules remain hard constraints.The judge evaluates EP-to-PR alignment rather than the engineering quality of the upstream implementation.Source: §B.3 · p. 15
  • Mappings are scored on requirement coverage, precision, scope correctness, granularity, and requirement specificity.Coverage receives weight 0.40, precision 0.25, scope 0.15, granularity 0.10, and specificity 0.10.Source: §B.3 · p. 16, Table 6 · p. 16
  • Section importance is assigned independently of mapped-file count, and candidates are retained only after passing both validation gates, scoring at least 8.0, and expert review.Broad or weakly justified mappings are penalized through precision, granularity, and requirement specificity.Source: §B.3 · p. 16, Table 7 · p. 17

C DETAILS OF TASK ENHANCEMENT

The task-enhancement process constructs requirement-violating solution mutants and augments F2P tests to expose missed behavior. Tests are mutation- and coverage-guided, audited for validity, and restricted to permitted public behavior.

C.1 INCORRECT SOLUTION MUTANT GENERATION

  • Three proposal-level mutations create requirement violations: reverting a section, mismatching an explicit requirement, or altering required structure or execution flow.Generated solution candidates are checked for semantic non-equivalence, required to identify the violated section and scenario, and screened on P2P tests before expert review.Source: §C.1 · p. 17

C.2 F2P TEST AUGMENTATION

  • Mutation-guided tests target mutants that survive original F2P tests, asserting enhancement-proposal outcomes through public system-level or end-to-end behavior.Candidates must fail before implementation, pass with the reference implementation, and fail on targeted mutants; internal-symbol tests are rejected.Source: §C.2 · pp. 18–19
  • Coverage-guided generation targets tasks with original F2P coverage below 50%, prioritizing uncovered reference-patch spans reachable through permitted public behavior.Retained tests must exercise previously uncovered lines, validate against both pre-implementation and reference behavior, and report remaining unreachable gaps.Source: §C.2 · pp. 18–20
  • 95.97% of 1,688 retained mutants are killed by the combined original and added tests, versus 61.08% by original tests; mean F2P line coverage rises from 60.07% to 78.50%.The audit rejects tests that depend on internal symbols or newly introduced symbols absent from the enhancement proposal.Source: §C.2 · p. 18

D EFFECTIVENESS ON DIFFERENT PROGRAMMING LANGUAGES

Effectiveness varies by programming language and scaffold, but modular-development tasks remain difficult across language groups. Python, C/C++, and Rust have higher average F2P pass rates than Java, Go, and TypeScript, and scaffold rankings reverse between Python and C/C++.

D EFFECTIVENESS ON DIFFERENT PROGRAMMING LANGUAGES

  • The median resolved rate is zero in six of seven language groups; the strongest agent resolves 32.0% of Python tasks, while no agent resolves a TypeScript task.Best resolved rates for Java, Go, Rust, and C/C++ range from 11.1% to 14.3%; mean completed rates range from 4.0% to 13.8% outside the two-task C# group.Source: D EFFECTIVENESS ON DIFFERENT… · p. 20, Figure 5 · p. 22, Figure 6 · p. 23
  • 48.5% for C/C++ and 46.3% for Rust exceed 31.7% for Python, 21.7% for TypeScript, 13.5% for Go, and 6.0% for Java in mean language-specific F2P pass rates.These rates average equally over all 28 agents.Source: D EFFECTIVENESS ON DIFFERENT… · p. 20, Figure 7 · p. 24
  • For Opus 5, Claude Code leads on Python, but OpenCode leads on C/C++ in both resolved and F2P pass rates.On Python, Claude Code reaches 32.0% resolved and 78.5% F2P pass, versus OpenCode's 8.0% and 51.7%; on C/C++, OpenCode reaches 14.3% and 65.4%, while Claude Code resolves none and passes 59.4% of F2P tests.Source: D EFFECTIVENESS ON DIFFERENT… · p. 21, Figure 5 · p. 22, Figure 7 · p. 24

E PHASE DISTRIBUTION ON TOKEN USAGE

Agents spend substantially more tokens on input than generation, with code localization the largest usage phase and verification another major input cost. Localization remains prominent during repair loops.

E PHASE DISTRIBUTION ON TOKEN USAGE

  • 11.0× as many non-cached input tokens as generated tokens are used per attempt on average; the median of agent-level ratios is 7.9×.Mean usage is 1.60 million non-cached input tokens and 145.8 thousand generated tokens per attempt.Source: E PHASE DISTRIBUTION ON TOKEN USAGE · p. 26
  • 52.4% of non-cached input tokens and 45.4% of generated tokens go to code localization, the largest macro-averaged shares.Localization is the largest phase for 26 of 28 agents' non-cached input and 19 agents' generated tokens; within repair loops it accounts for 39.5% of non-cached input.Source: E PHASE DISTRIBUTION ON TOKEN USAGE · p. 26
  • 26.4% of non-cached input tokens go to verification, exceeding code editing at 19.8%.Verification is the second-largest input phase overall, rising from 23.5% during initial generation to 36.6% during repair loops.Source: E PHASE DISTRIBUTION ON TOKEN USAGE · p. 26

F.1 METHOD

The failure analysis combines rule-based and semantic judgments to identify failure modes from agent trajectories, then assigns fractional attribution across failed tasks. Its procedure emphasizes evidence-linked judgments and conservative confidence handling.

F.1 METHOD

  • Seven failure phases are analyzed using predefined rules for 18 failure modes and an LLM judge for the six remaining modes.The predefined rules detect objective candidates; semantic judgments address cases requiring interpretation.Source: §F.1 · p. 28
  • Normalized event logs generate suspect candidates, such as unread reference files for localization misses and partially edited file sets for cross-module failures.These candidates require further analysis and do not alone establish a failure mode.Source: §F.1 · p. 28
  • Requirement-understanding and planning judgments use an outcome-blinded LLM judge that must cite visible trajectory steps and excludes low-confidence or unparseable judgments.The judge sees neither the reference solution nor task outcome; hidden reasoning and ambiguous evidence call for lower confidence.Source: §F.1 · pp. 28–29
  • Only high- and medium-confidence candidates enter attribution, with confidence scores of 1.0 and 0.6, respectively.The judge is instructed to distinguish temporary sequencing from deliberate scope reduction and to require evidence when treating a reduction as rescinded.Source: §F.1 · pp. 29–30
  • Candidate scores are normalized within each failed task, allocating exactly one attribution unit per task before phase and failure-mode aggregation.Reported values are fractional allocations rather than counts of tasks with a particular error flag.Source: §F.1 · p. 30

F.2 RESULTS

Across the failure phases, agents show recurring problems with requirement scope, dependency localization, implementation, verification, and repair. The reported attributions identify incomplete cross-module context as the largest localization failure mode for all 28 agents.

Requirement Understanding

  • For 27 agents, scope misjudgment is the largest or joint-largest requirement-understanding attribution; direct requirement misreading reaches at most 1.0 units per agent.For Claude Code with DeepSeek V4 Flash, scope misjudgment accounts for 8.9 of 11.2 phase attribution units.Source: §F.2 · p. 30

Task Planning

  • Nearly all planning attribution comes from flawed plans, while abandoned plans receive at most 0.8 units per agent.Flawed plans account for all 10.5 planning units for Claude Code with DeepSeek V4 Flash and all 10.1 units for Pi with GLM-5.2.Source: §F.2 · p. 30

Code Localization

  • Missing cross-module context is the largest localization failure mode for all 28 agents, accounting for 41.4 of 41.7 units for Codex with Kimi K3.Missed relevant files and wrong-file localization receive at most 2.1 and 0.5 units, respectively, across agents.Source: §F.2 · p. 30

Code Editing

  • Incorrect patches lead editing attribution for 22 agents, while omitted relevant changes lead for the other six.For Pi with GPT-5.6 Sol, incorrect patches and omitted changes contribute 19.4 and 20.5 units, together accounting for all 39.9 editing units.Source: §F.2 · pp. 30–31

Code Verification

  • Absence of recognized validation is the leading verification failure mode for all 28 agents.For mini-SWE-agent with DeepSeek V4 Flash, it accounts for 17.5 of 24.9 verification units, followed by 6.3 units for missing tests after the final edit.Source: §F.2 · p. 31

Self Repair and Tool Use

  • Unverified repairs and repeated ineffective attempts dominate repair attribution, while tool-use attribution ranges from 0.0 to 1.5 units per agent.Repeated ineffective attempts lead for all six agents using Pi; tool-use failures contribute relatively little overall.Source: §F.2 · p. 31

G LIMITATIONS

The paper limits what its findings establish: the benchmark evaluates functional correctness on a bounded set of systems and agent configurations, while task context and failure diagnoses also have constraints.

G LIMITATIONS

  • LOLBENCH evaluates functional correctness but does not separately assess readability, maintainability, or consistency with project conventions.Passing behavioral tests does not establish broader engineering quality or long-term maintainability.Source: G LIMITATIONS · p. 31
  • The benchmark covers a limited set of systems drawn from communities with established proposal and review processes, limiting generalization to other practices or less formal requirements documentation.The source communities constrain data-source diversity and the number of eligible proposals.Source: G LIMITATIONS · p. 31
  • Evaluation covers 28 agents built from six models and five scaffolds, with one attempt per task and agent except for infrastructure retries.Results may not generalize to other agents, resource budgets, or run-to-run variability.Source: G LIMITATIONS · p. 31
  • Curated proposals and repositories cannot exhaustively reproduce issue threads, design discussions, or review comments that may clarify expected behavior or constraints.Some task difficulty may therefore arise from incomplete specification context.Source: G LIMITATIONS · p. 31
  • Failure attributions are diagnostic estimates because LLM-judge conclusions may vary with the model, prompt, and interpretation of incomplete evidence.Different judges may assign co-occurring errors to different phases or failure modes.Source: G LIMITATIONS · p. 32

H A TASK EXAMPLE

This task example presents the proposal to replace CPython’s LL(1) parser with a PEG parser and explains the choice of PEG, its grammar and AST-construction features, and the parser’s staged migration. It also discusses packrat parsing’s memory tradeoff and reports benchmark performance close to or better than the current parser.

Task instance

  • The cpython 1 task reproduces PEP 617 but removes the test plan and validation sections to avoid disclosing hidden tests.Agents were instructed to work from the existing workspace and submit implementation changes in solution.patch; tests were optional and separate.Source: H A TASK EXAMPLE · pp. 32–33

Overview

  • PEP 617 proposes replacing CPython’s LL(1)-based parser with PEG to eliminate grammar workarounds, reduce maintenance costs, and lift the LL(1) restriction.The proposal says the maintenance benefits concern the grammar, parser, and AST generation.Source: H A TASK EXAMPLE · p. 33

Rationale

  • LL(1) restrictions make some natural language constructs difficult to express, while ambiguous alternatives may require equivalent rules that are harder for readers to understand.The proposal also notes that CPython’s grammar is technically LL(1) but includes rules that are not, with workarounds in the grammar and elsewhere in CPython.Source: H A TASK EXAMPLE · p. 37
  • The proposal identifies left-recursive rules as another grammar form precluded by LL(1), despite their usefulness for expressing language properties directly.It also says several CPython grammar rules are not actually LL(1) and are handled with workarounds.Source: H A TASK EXAMPLE · pp. 38, 40

Background on PEG parsers

  • PEG uses ordered choice to avoid ambiguity, while packrat parsing [1]_ memoizes rule matches to avoid exponential worst-case time at the cost of more memory than a simple LL(1) parser.Packrat parsing loads the entire program into memory and allows arbitrary backtracking.Source: H A TASK EXAMPLE · pp. 39–40

H A TASK EXAMPLE

  • The current grammar uses a broader named-expression rule and defers rejecting unwanted constructs to AST construction, burdening later stages and misrepresenting valid Python syntax.Similar workarounds occur in other rules, and some LL(1) grammar constraints can make desired syntax impossible.Source: H A TASK EXAMPLE · p. 40

Complicated AST parsing

  • AST generation is tightly coupled to parse-tree shape, leaving alternatives implicit and requiring code to infer them from child counts and implementation details.This makes AST-generation code difficult to follow and forces it to inspect and reason about incoming parse trees.Source: H A TASK EXAMPLE · p. 41

Lack of left recursion

  • LL(1) grammars cannot express left recursion, and rewriting recursive rules flattens parse trees that later stages must reshape, obscuring associativity and adding work.For a + b + c, the rewritten form yields a flat tree that must be post-processed into a left-recursive tree.Source: H A TASK EXAMPLE · p. 41

Intermediate parse tree

  • The current parser retains an intermediate concrete syntax tree that is otherwise little used, consumes considerable memory, and complicates AST generation and maintenance.The tree contains many single-child chains.Source: H A TASK EXAMPLE · p. 42

The new proposed PEG parser

  • The proposal introduces a PEG parser generator, a meta-grammar that generates its Python parser, and generated parsers that directly produce C and Python AST objects.The meta-grammar removes the need for manually written parsers.Source: H A TASK EXAMPLE · p. 42

Left recursion

  • The proposed PEG parser supports left recursion using memoization, including indirect and hidden left-recursive rules, following techniques described by Medeiros et al. [2]_ and Warth et al. [3]_.The approach uses a memoization cache rather than static variables.Source: H A TASK EXAMPLE · pp. 42–43

A task example

  • Grammar actions let PEG rules directly describe AST construction, with examples targeting both C-based Python AST objects and Python AST objects.The proposal says actions make AST composition clearer and more maintainable, with helper functions factoring out common AST manipulations.Source: H A TASK EXAMPLE · pp. 45, 47, 48

Migration plan

  • The migration plan enables the PEG parser alongside the old parser in Python 3.9 alpha 6, keeps the old parser through Python 3.10, then removes it.While both parsers remain, no grammar addition may require PEG, so the grammar stays LL(1); flags and an environment variable select the parser initially.Source: H A TASK EXAMPLE · pp. 48–49

Performance and validation

  • For this particular test file, parsing and compiling to bytecode took 1.28 seconds with the new parser (681 MiB maximum RSS), versus 1.44 seconds and 836 MiB maximum RSS with the current parser.Source: H A TASK EXAMPLE · p. 49
  • 3.290 s versus 3.367 s for compiling the stdlib to bytecode, with maximum RSS of 77 MiB versus 70 MiB for the new and current parsers, respectively.The authors characterize the new parser as slightly faster but using about 10% more memory on this payload, which they consider acceptable.Source: H A TASK EXAMPLE · p. 50

H A TASK EXAMPLE

  • The author chose a custom Python parser partly to support EBNF conveniences and syntax trees matching grammar structure, then selected PEG for its accessibility without considering other LR variants or LL.The earlier choice over LALR(1) was based primarily on intuition and prior experience with grammar-development effort and shift-reduce conflicts.Source: H A TASK EXAMPLE · p. 50

Related papers

Loading 2609.37143v1…