Acquiring and Verifying Repository Norms for Coding Agents

1 Introduction

• Repository norms include project requirements, recommendations, and supported conventions governing implementation, testing, and documentation, with applicable conditions and exceptions.Overlooking them can leave contributions incomplete or change established behavior despite passing functional tests.Source: §1 · p. 1
• Automated norm acquisition must interpret evidence distributed across repository sources and determine which changes a norm governs, including its conditions and exceptions.Explicit rules may be qualified by code and tests, while a norm’s evidence location need not define its scope.Source: §1 · pp. 1–2
• Retrieval and documentation systems provide repository context but may leave unclear which contributor obligations apply and which artifacts a change must include.Reliable guidance requires checking its content and applicability against repository evidence.Source: §1 · p. 2
• RepoNorm extracts explicit candidates, infers candidates from code practices, verifies support and applicability against repository evidence, and packages selected norms for agents.Bounded Git-history queries provide additional evidence when needed.Source: §1 · p. 2
• RepoNormBench contains 121 coding tasks with refined descriptions, strengthened functional tests, repository-grounded reference norms, and automated compliance checks.The evaluation measures norm compliance separately from requested behavior and assesses norm quality through precision and reference-norm coverage.Source: §1 · pp. 2–3
• RepoNorm raises Overall NCR by 7.42–10.77%, Contribution NCR by 31.64–45.44%, and Prompt-omitted NCR by 11.34–17.69% relative to Raw across three models.It also exceeds CodeWiki on all three measures for every model; observed functional-success gains over Raw are not statistically significant after Holm correction.Source: §1 · p. 2

2 Motivation

• Both example patches passed their functional checks yet missed a repository obligation: a Jinja contribution requirement or a Tiktoken behavioral contract.The examples illustrate the challenges of finding norms beyond task descriptions and establishing where they apply.Source: §2 · p. 3, Figure 1 · p. 3

2.1 Challenge 1: Discovering Norms Beyond the Task Description

• Jinja requires tests, relevant documentation and docstring updates, and a CHANGES.rst entry for user-facing filter changes, even when the task specifies only runtime behavior.The example patch passed all 171 functional checks but omitted the changelog entry, which the task did not mention.Source: §2.1 · pp. 3–4

2.2 Challenge 2: Establishing Applicability

• Tiktoken’s existing surrogate recovery must also apply to new text-encoding paths that call the Rust encoder; the chunking path omitted it despite passing all 18 functional checks.Verification must establish a candidate norm’s supported scope, conditions, and exceptions; RepoNorm uses explicit extraction, implicit inference, unified verification, and packaging.Source: §2.2 · p. 4

3 Approach

• RepoNorm discovers explicit and implicit norm candidates, verifies their content and applicability against repository evidence, and delivers selected norms in a static package.An existing coding agent selects which norms to consult while carrying out a task.Source: §3 · p. 4

3.1 Overview and Norm Representation

• Each norm represents governed behavior, artifact scope, an activation condition, and any exception; evidence provenance is tracked separately from scope.Unified verification records evidence, counterevidence, decision status, normative strength, and delivery policy.Source: §3.1 · p. 5

3.2 Explicit Norm Extraction

• Explicit extraction discovers repository instructions, documentation, configurations, and workflows, then extracts candidate rules with scopes, conditions, exceptions, and evidence references.Verification checks that evidence supports the rule and its applicability, distinguishing requirements, recommendations, and descriptive facts.Source: §3.2 · pp. 5–6

3.3 Implicit Norm Inference

• Implicit inference tests hypotheses from source observations against an explicit code population, using counterexamples to narrow scope with at most one local refinement.The Jinja example narrows empty-input behavior to filters that select an element, rather than all sequence-consuming filters.Source: §3.3 · pp. 6–7

3.4 Unified Norm Verification

• Unified verification assembles current evidence, seeks bounded Git history when needed, and evaluates candidates against support, counterevidence, conflicts, and unresolved issues.Historical evidence can resolve ambiguity but cannot independently establish acceptance or rejection.Source: §3.4 · pp. 8–9
• Verification permits one candidate refinement round; changes to scope or conditions require renewed evidence collection and checks, with further revisions or incomplete rechecking leading to review.Previously inspected observations are retained during rechecking.Source: §3.4 · p. 9

3.5 Norm Packaging and Delivery

• The static Markdown package organizes norms by applicability path and preserves each rule’s semantics and evidence; agents can inspect path pages or the catalog, while developers may optionally review entries.Coding experiments use the Accepted-Only View, not the developer-review interface.Source: §3.5 · pp. 9–10

4 Evaluation

• The evaluation measures norm compliance as its primary outcome and functional correctness as a secondary outcome, alongside norm quality and acquisition and guided-coding costs.It includes compliance with contribution requirements that functional tests do not directly assess.Source: §4 · p. 10

4.1 Experimental Setup

• RepoNormBench covers 121 coding tasks across six selected Python repositories and 14 repository versions, with an evaluation-owned test suite for functional success.The repositories were selected from CooperBench to keep computational and norm-annotation costs manageable.Source: §4.1 · p. 10
• The benchmark refines all task descriptions and strengthens functional tests, while adding repository-grounded norm checks to assess practices separately from functional correctness.Its reference includes 160 distinct norms and 1,024 task–norm associations; hidden tests, reference solutions, and reference norms are withheld from acquisition and task inputs.Source: §4.1 · pp. 10–11
• Each model is tested on all 121 tasks under Raw, CodeWiki, and RepoNorm conditions, using reusable task-independent resources prepared without task descriptions, reference solutions, or hidden tests.The evaluated models are DeepSeek V4 Flash, Qwen 3.5 Flash, and Gemini 3.1 Flash-Lite.Source: §4.1 · p. 11

To what extent do coding-agent solutions comply with repository-specific norms?

• Behavioral NCR changes are smaller and mixed, and all six paired 95% intervals include zero; intervals for Overall, Contribution, and Prompt-omitted NCR are positive in all 18 comparisons.Source: To what extent do coding-agent… · p. 12
• Prompt-omitted NCR improves 11.34–17.69% over Raw and 15.00–16.93% over CodeWiki across the three models.Source: To what extent do coding-agent… · pp. 12–13
• Across three models, RepoNorm raises Overall NCR 7.42–10.77% over Raw and 8.61–10.64% over CodeWiki; Contribution NCR rises 31.64–45.44% over Raw and 29.86–48.03% over CodeWiki.Source: To what extent do coding-agent… · p. 12, Table 2 · p. 13
• On 651 originally unmentioned task–norm pairs, RepoNorm NCR is 10.89–18.37% higher than Raw and 14.99–17.88% higher than CodeWiki across models.Source: To what extent do coding-agent… · p. 14
• On 107 tasks excluding confirmed reminder removals, Overall NCR remains 7.85–10.63% higher than Raw and 8.77–11.18% higher than CodeWiki.Source: To what extent do coding-agent… · p. 14
• All sampled norms were valid, with full evidence support for 41 and partial support for one; all sampled task–norm associations were applicable.Source: To what extent do coding-agent… · p. 14
• All 1,010 checks expected to pass did so; checks detected 302 of 341 expected violations, with all 39 misses involving misleading-documentation variants.Documentation checks use a change signal as a proxy for delivery, which does not guarantee content correctness.Source: To what extent do coding-agent… · p. 14
• After excluding 59 documentation task–norm pairs from every condition and model, Overall NCR remained 8.57–11.53% higher than Raw and 8.59–11.85% higher than CodeWiki.Contribution NCR remained 31.21–55.84% above Raw and 33.77–63.31% above CodeWiki.Source: To what extent do coding-agent… · p. 14

Does RepoNorm improve the functional correctness of coding-agent solutions?

• RepoNorm had the highest joint success for all three models (10, 6, 2 tasks versus Raw’s 4, 3, 1 and CodeWiki’s 5, 1, 1), but no corrected comparison cleared the threshold.Joint success remained uncommon across models and conditions.Source: Table 3 · p. 15, Does RepoNorm improve the functional… · p. 15
• Observed functional success exceeded Raw, but paired comparisons did not establish improvement after multiplicity correction.The section reports that joint success remained uncommon.Source: Does RepoNorm improve the functional… · p. 16

4.4 RQ3: Norm Quality

• 86.00% sampled precision in the Accepted-Only View accompanied 21.88% full and 39.38% full+partial coverage; review raised coverage to 52.50% and 78.13% while precision fell to 75.17%.Coverage was assessed against the same task-relevant reference norms for both views.Source: Table 4 · p. 16, §4.4 · p. 16
• Reviewers agreed on 126 of 140 entries (90.00%), and their shared judgments matched GPT-6-Astra’s assessments on 114 of those 126 entries (90.48%).The authors interpret this agreement as supporting reliability on the sampled entries.Source: §4.4 · p. 16
• The review-expanded view trades higher reference-norm coverage for lower sampled precision, and entries with any inaccurate field count as incorrect even if their guidance remains task-relevant.Some entries classified as incorrect may still contain task-relevant guidance.Source: §4.4 · p. 17

4.5 RQ4: Cost Analysis

• Acquisition averaged 9.18 minutes per reusable package (2.14 hours total), while RepoNorm coding used 33.65% more recorded tokens and 25.33% more cumulative execution time than Raw.The package acquisition time was 3.2% of cumulative coding-agent execution time with RepoNorm.Source: Table 5 · p. 17, §4.5 · p. 17

• Model-assisted reference norms may be incomplete and reflect author judgment, so coverage measures recovery only of this task-relevant reference set.Precision samples may miss rare errors or imperfectly represent the output population.Source: §5 · p. 17
• Tasks share repository versions, potentially understating uncertainty and weakening inference about new repositories; one execution per condition-task also leaves run-to-run variation unmeasured.Category-group analyses are exploratory, and NCR intervals are pointwise rather than simultaneous.Source: §5 · p. 18
• Evaluation covers six relatively small repositories, three coding models, one agent harness, and Python source analysis; other languages, larger repositories, and agent configurations remain untested.Source: §5 · p. 18
• Fixed analysis budgets may miss norms or counterexamples as repositories grow, packages can become stale, and incremental updating is not yet sufficiently optimized.Frequent or extensive repository changes require ongoing time and resources, and agents may overlook relevant norms among many entries.Source: §5 · p. 18

• Prior work mines conventions or constraints from source code, execution traces, and documentation, while RepoNorm checks norms governing implementation, tests, and documentation against evidence and counterexamples.The cited approaches include learning naming and formatting conventions and mining specification constraints.Source: §6 · p. 18
• Repository-context systems support code completion and modification during a task, whereas RepoNorm prepares norms before the task is specified and complements the coding loop.Source: §6 · p. 18
• Documentation and instruction systems provide reusable repository guidance, while RepoNorm checks norm content and applicability and attaches support, conditions, and exceptions.Prior instruction-file evaluations found general instruction-following without consistent task-resolution gains in the evaluated settings.Source: §6 · p. 18
• Compliance tools assess design constraints or task-specific instruction checklists; RepoNorm’s evaluation separates norm precision and reference coverage from downstream compliance and functional success.Source: §6 · p. 18

• Across three coding models and 121 tasks, RepoNorm improved Overall NCR by 7.42–10.77%, Contribution NCR by 31.64–45.44%, and Prompt-omitted NCR by 11.34–17.69% relative to Raw, and exceeded CodeWiki on all three measures for every model.Source: §7 · p. 18
• Observed functional success was 5.79–9.09 percentage points higher than Raw, but paired functional and joint-success comparisons did not establish significant improvement after correction; joint success remained uncommon.Source: §7 · p. 19
• RepoNorm’s default configuration achieved 86.00% sampled precision, and the authors conclude that contribution completeness remains distinct from functional test success.They support making repository obligations explicit and evaluating them alongside functional correctness.Source: §7 · p. 19