PaperlayerView exact revision on arXiv

Source-linked AI summary

Correct Code, Broken Contributions? SWE-CC: Benchmarking Repository Policy Compliance for Coding Agents

Hai Dang Truong, Rayner Goh, Thanh Le-Cong, Yintong Huo

arXiv:2610.06193v1cs.SEcs.AI

TL;DR

Functional test success does not establish whether coding agents follow repository contribution policies, leaving their compliance insufficiently evaluated. SWE-CC converts documentation into machine-checkable policies and audits agent trajectories and deliverables. Across the evaluated tasks, agents violate policies frequently, with half of violations occurring before final commit packaging.

  • Problem

    Existing benchmarks largely assess whether patches pass tests, leaving agent compliance with repository-specific contribution policies unevaluated.

    Source: §1 · pp. 1–2

  • Method

    SWE-CC compiles documentation from 12 repositories into 823 machine-checkable policies and audits coding-agent trajectories and final deliverables.

    Source: §1 · p. 2

  • Results

    50.3% of policy violations occur before final commit packaging, and agents initiate policy searches in only 28.9% of executions.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Current coding agents violate a large fraction of project guidelines, and roughly half of those violations occur during intermediate execution.

    Source: §5 · p. 9

Abstract

from arXiv · show

Autonomous coding agents now resolve a substantial share of real-world GitHub issues. However, passing functional tests differs fundamentally from producing a high-quality contribution acceptable for merging. Mature open-source projects publish repository-specific contribution policies, spanning style, git, testing workflows, to ensure code quality and long-term maintainability. Because existing benchmarks evaluate patches solely on unit tests, agent compliance with repository governance remains unknown. In this paper, we introduce SWE-CC, a benchmark evaluating code and process compliance in autonomous software engineering. We develop a semi-automated pipeline that converts developer documentation across 12 open-source repositories into 823 machine-checkable atomic policies. SWE-CC introduces two features: 1) lightweight, deterministic checker functions that represent each policy, 2) a comprehensive auditing mechanism that inspects both agent runtime behaviors and final deliverables. We evaluate the compliance of agent workflows in 500 end-to-end software contribution tasks extended from SWE-bench Verified. Our evaluation of four LLMs under two agent scaffolds shows that modern agents suffer from coding compliance issues: although agents produce functionally correct patches, they still violate 43.1 percent of applicable project policies, with nearly half of all violations occurring during intermediate execution steps. These results show that functional correctness does not guarantee real-world readiness, highlighting that future software engineering agents must reliably conform to repository governance to enable safe and trustworthy deployment.

1 INTRODUCTION

Functional test success does not ensure that an agent follows repository contribution policies. SWE-CC evaluates those policies across agent workflows and final deliverables, finding frequent violations among current coding agents.

Motivation

  • Repository contribution policies cover conventions, commits, and documentation, and are presented as important for code quality and long-term maintainability.The introduction also connects non-compliant AI contributions with maintainer review overhead and describes an agent acting during an explicit code freeze.Source: §1 · p. 1
  • A patch can pass the full test suite while violating a repository rule, as in a Django Boolean assertion that uses assertTrue instead of required assertIs.Because SWE-bench evaluates test execution alone, such policy violations can go undetected.Source: §1 · p. 2, Figure 1 · p. 2

Contribution

  • SWE-CC converts repository documentation into 823 machine-checkable policies across 12 repositories and audits both runtime behavior and final deliverables.Its checker functions apply policies across tasks in a repository and support reproducible evaluation and fine-grained violation diagnostics.Source: §1 · p. 2

Empirical findings

  • 50.3% of policy violations occur before final commit packaging, and agents initiate policy searches in only 28.9% of executions.Across 500 end-to-end tasks, agents gain an average 8.75% compliance rate when processed policies are provided in context.Source: §1 · p. 2

2 RELATED WORK

Prior coding-agent benchmarks primarily evaluate functional correctness, while newer work assesses selected non-functional properties and patch acceptance. SWE-CC addresses gaps in governance scope and policy retrieval by auditing workflow and requiring in-situ policy discovery.

Prior benchmarks

  • Most software engineering benchmarks evaluate functional correctness through existing tests, while newer benchmarks target efficiency, security, or general code quality.SWE-Gate and SWE-SHIELD mine implicit review criteria from pull-request discussions, while RepoComplianceBench focuses on AI-assistance disclosures.Source: §2 · p. 3
  • RepoComplianceBench narrows its scope to AI-assistance disclosures, whereas SWE-CC evaluates a broader range of repository governance.Other nearby benchmarks mine implicit patch-acceptance criteria from historical pull-request discussions.Source: §2 · p. 3

Limitations of prior work

  • Existing evaluations can omit intermediate agent behavior and policy discovery, even though real-world guidelines may govern workflows such as testing and PR metadata.SWE-CC audits the workflow against repository documentation and requires agents to navigate and reason over policies in the repository.Source: §2 · p. 3

3 SWE-CC

SWE-CC compiles repository contribution documentation into executable policy checkers and uses them to assess complete coding-agent contribution workflows. Its construction and evaluation span policy extraction, checker validation, benchmark characterization, workflow settings, and compliance metrics.

3 SWE-CC

  • SWE-CC formalizes repository governance as deterministic checkers that evaluate agent runtime behavior and final deliverables, with non-applicable policies left untriggered.The benchmark construction pipeline extracts atomic policies, compiles checkers, and applies them to agent trajectories and outputs.Source: §3 · pp. 3–4, Figure 2 · p. 4

3.1.1 POLICY EXTRACTION

  • Policy extraction starts from contributor documentation and repository files, then decomposes obligations into atomic rules with preconditions, required behavior, and source context.Sources include contributor pages and policy-bearing files such as pull-request templates and AGENTS.md.Source: §3.1.1 · p. 4

3.1.2 CHECKER CONSTRUCTION

  • Checker construction retains mandatory policies with inspectable evidence, objective pass/fail conditions, and actions within an autonomous contributor’s purview.Each checker pairs an applicability precondition with a post-condition that can mark targets satisfied, violated, or undetermined; synthetic tests and expert review validate checkers.Source: §3.1.2 · pp. 4–5

3.2 IMPLEMENTATION DETAILS AND RUNNING EXAMPLE

  • The pipeline extracts 1,759 atomic policies from 12 repositories, of which 823 pass quality and scope filters for executable checkers.In a sample of 150 checkers, annotators agreed on 94.0% of verdicts and accepted 87.2%.Source: §3.2 · p. 5
  • The SymPy example turns a documentation rule limiting commit summaries to 71 characters into a checker that measures each agent-produced commit.A 41-character summary satisfies the rule, while a 93-character summary violates it.Source: §3.2 · p. 5

3.3 BENCHMARK CHARACTERISTICS

  • The 823 policies vary from 2 to 167 per project and rarely overlap across projects; documentation and docstrings form the largest policy category at 27%.Language and framework style account for 20%, and test style for 19%; Django requires an issue number in commit messages while Flask prohibits one.Source: §3.3 · p. 6, Figure 3 · p. 6, Table 2 · p. 6

3.4 EVALUATION

  • The evaluation uses 500 end-to-end tasks extended from SWE-bench Verified and compares native policy discovery with a consolidated file of processed rules.Both settings audit intermediate workflow behavior and final PR deliverables; consolidated provision removes the search and retrieval bottleneck.Source: §3.4 · p. 6

3.5 METRICS

  • Triggering rate measures policies brought into scope, compliance rate measures success on triggered policies, and resolve rate measures test-based functional success.Applicability preconditions distinguish policies that apply to an agent’s workflow or deliverable from those that do not.Source: §3.5 · p. 7

4 EXPERIMENTS

The experiments evaluate policy compliance across coding agents, scaffolds, and policy-provision settings. Agents frequently violate repository policies, including during execution, while policy access improves compliance without materially changing functional resolution.

4.1 EXPERIMENTAL SETUP

  • The evaluation pairs two agent frameworks with four LLM backbones across 500 tasks and two policy settings, auditing 8,000 workflows.The scaffold configurations retain their original prompts and tools, with task prompts extended for commit and pull-request steps.Source: §4.1 · p. 7

4.2 RESULTS

  • 50.3% of violations among resolved runs occur in agent trajectories, so auditing final patches alone misses process violations.One example is running tests before committing, which can only be checked in the trajectory.Source: Figure 5 · p. 8, §4.2 · p. 8
  • 7.7 of 22.6 triggered policies were violated per execution on average (34.1%); consolidated policies reduce the mean to 6.6.Figure 4 reports the violation distribution over functionally resolved cases.Source: Figure 4 · p. 8, §4.2 · p. 8
  • Table 4 breaks down triggering and compliance by governance domain; AI-assisted contribution policies have the lowest native compliance, at 15%.These policies require AI disclosures, which agents often omit even when guidelines are provided.Source: Table 4 · p. 8, §4.2 · p. 9
  • Native runs often fail to retrieve policies: 71.1% do not attempt to fetch them, and more than half of fetch attempts retrieve no policy text.With consolidated policies, 91.6% of runs open the mounted file and 99.6% of its content reaches context, yet compliance remains low.Source: Table 5 · p. 9, §4.2 · p. 9
  • Consolidated policies raise compliance by an average of 8.75% while functional resolution rates fluctuate within 2%; the best configuration still leaves about 30% of policies violated.The largest reported gains are +15.8% for mini-SWE-agent and +12.2% for OpenHands under GPT-5.6 Luna.Source: §4.2 · p. 9

5 CONCLUSION

The conclusion presents SWE-CC as a benchmark for policy compliance across execution trajectories and final patches. Its evaluation finds frequent policy violations, with about half occurring during intermediate execution.

5 CONCLUSION

  • SWE-CC converts repository contributor guidelines into executable checkers and evaluates compliance in both agent trajectories and final patches.The evaluation found that models violate a large fraction of project guidelines, with roughly half of violations occurring during intermediate execution.Source: §5 · p. 9
  • The authors argue that agent evaluation should measure procedural compliance alongside final-code correctness and plan to expand SWE-CC across models, scaffolds, and repositories.They identify joint reporting of compliance and functional correctness as a path toward safer deployment and reduced human maintenance overhead.Source: §5 · p. 10

A BENCHMARK STATISTICS

This appendix section describes the benchmark statistics it contains.

A BENCHMARK STATISTICS

  • The appendix provides each project’s corpus and one example policy for each category.The section supports Section 3.3.Source: A BENCHMARK STATISTICS · p. 13

A.1 PER-PROJECT CORPORA

The per-project corpus statistics describe the benchmark’s policies and checker evidence.

A.1 PER-PROJECT CORPORA

  • Table 6 reports each project’s task count, policy counts across filtering stages, checker evidence types, and share of approximate checks.The evidence-type columns partition binding rules; approximate-check shares are described in Appendix B.5.3.Source: Table 6 · p. 13, §A.1 · p. 13

A.2 POLICY CATEGORIES

The policy taxonomy uses eight categories fixed before extraction so category meanings remain consistent across projects; Table 7 presents one extracted mandatory policy per category.

A.2 POLICY CATEGORIES

  • Eight categories form a closed list fixed before extraction, keeping each category consistent across projects.Table 7 provides one mandatory policy example per category.Source: §A.2 · p. 13

B CONSTRUCTION PIPELINE DETAILS

The construction appendix documents a staged pipeline for sourcing, extracting, classifying, and compiling repository policies. It uses independent repository agents with human review and validates checker behavior with authored cases.

B CONSTRUCTION PIPELINE DETAILS

  • Independent agents process each repository under a fixed procedure because diffuse, lengthy documentation across twelve projects cannot be assembled manually.A human decides which agent-drafted material is admitted.Source: B CONSTRUCTION PIPELINE DETAILS · p. 14

B.1 SETUP

  • Documentation is pinned to each project’s latest published build, with affected sources aligned to one commit when development aliases serve mixed versions.Moving-only aliases are resolved by hand and recorded.Source: §B.1 · p. 14

B.2 STEP 1: SOURCING POLICY DOCUMENTS

  • Sourcing applies a literal contribution-scope test, traverses documentation plus one hop to related pages, and separately bundles off-navigation files as raw text.Sources are classified as obligation-bearing, context-only, or out of scope; bug-report requirements are kept distinct because the setting cannot reach them.Source: §B.1 · p. 14, §B.2 · p. 15

B.3 STEP 2: EXTRACTING ATOMIC POLICIES

  • Extraction splits repository-specific contribution requirements into atomic rows that retain source quotations, conditions, requirement levels, and shared categories.Interface descriptions are converted into protocol-level conformance obligations rather than separate rules for each member.Source: §B.3 · pp. 15–16

B.4 STEP 3: FILTERING CHECKABLE POLICIES

  • Classification applies evidence type before scope, then assigns in-scope rules a must-or-should level using fixed rubrics and a centrally set conflict tiebreaker.Batches span the whole sheet to support duplicate detection across sources; unresolved rows remain labelled with low confidence rather than blank.Source: §B.4 · p. 16

B.5 STEP 4: COMPILING POLICIES INTO CHECKER FUNCTIONS

  • Step 4 compiles each mandatory in-scope policy into an executable checker, while policies labelled should receive no checker by design.All 823 mandatory in-scope rules were compiled; a precondition selecting no targets yields inapplicable, not pass.Source: §B.5 · p. 16, §B.5.1 · pp. 16–17

B.5.1 CONSTRUCTION

  • Checker construction uses the extracted rule row alone, including its source sentence, atomic restatement, evidence type, and trigger condition.Claude Opus 5 authored the checks following the released checker-authoring guide.Source: §B.5.1 · p. 16

B.5.2 AUTHORED TEST CASES

  • The released suite contains 2,603 authored cases, and 767 of 823 checkers (93%) cover satisfying, violating, and inapplicable outcomes.The remaining 56 lack all three case types by design because of human-in-the-loop requirements, unavailable evidence, or checks that select only a violating construct.Source: §B.5.2 · p. 17

B.5.3 EXACT AND APPROXIMATE CHECKS

Checks are labelled exact or approximate according to whether they decide a policy directly or encode an underspecified reading. The section reports compliance patterns for exact checks and explains why approximation is not a confidence score.

B.5.3 EXACT AND APPROXIMATE CHECKS

  • 294 checks are exact and 529 approximate out of 823, with approximate-check shares ranging from 24% in SymPy to 85% in astropy.Table 8 reports compliance rates over all 823 checks and the 294 exact checks, plus gains from consolidated policies.Source: §B.5.3 · p. 17, Table 8 · p. 17
  • Among the 294 exact checks, compliance rates change by at most 4.7 percentage points, while consolidated policies raise compliance for every model on both scaffolds.DeepSeek V4 Flash remains first in three of four columns; the other models reorder within a few points.Source: §B.5.3 · p. 17, Table 8 · p. 17
  • Exact versus approximate marks whether a check decides its policy directly or stands in for an underspecified reading, not the check’s confidence.Approximate checks can disagree with a reader applying the same rule, so review assesses rule encoding and label correctness separately.Source: §B.5.3 · p. 17

B.5.4 HOW AN APPROXIMATE CHECK DECIDES

Approximate checks use deterministic code to test visible traces of policy conditions, with authored cases and review addressing how those traces represent the rules. The companion section defines which agent-owned artifacts are graded and how verdicts remain reproducible.

B.5.4 HOW AN APPROXIMATE CHECK DECIDES

  • An approximate check tests a visible trace of a policy property, using the trace in its precondition or post-condition and documenting the direction of possible error.The check makes its decision with code rather than a model call.Source: §B.5.4 · p. 18
  • For Django’s past-tense commit-subject rule, the check tests the leading verb, accepting -ed endings or one of 118 listed irregular past forms.The period requirement is checked exactly, while the tense clause is approximate; the two clauses are graded separately.Source: §B.5.4 · p. 18, Figure 6 · p. 18
  • The check agrees with a reader on 3,647 of 3,661 Django commits (99.6%) after excluding 24 verbs spelled identically in both tenses.Thirteen disagreements accept the ticket-prefix verb while the description is imperative; one misses Doc’d, absent from the irregular-form list.Source: §B.5.4 · pp. 18–19
  • Approximate-check techniques share a limit: they can err when the trace and the underlying property come apart.The human audit covered 97 approximate checks; Table 10 presents four other techniques and one policy example each.Source: §B.5.4 · p. 19, Table 10 · p. 19

B.5.5 OWNERSHIP AND DETERMINISM

  • Checks grade only agent-responsible code: created artifacts, touched lines, or enclosing consistency targets, depending on the ownership mode.This excludes pre-existing repository content from grading.Source: §B.5.5 · p. 19
  • Ordinary Python checks run unchanged on stored runs, making verdicts reproducible; static tools are run on both base and head, and the difference is scored.The authors rejected model-as-judge scoring because repeated judgments could differ and one model’s compliance would be judged by another model.Source: §B.5.5 · p. 19

B.6.1 REVIEW PROCEDURE

The checker review combined automated screening, execution against authored cases, and a blind human audit of a stratified sample. Raters independently judged whether checks faithfully encoded their source rules and behaved correctly.

B.6.1 REVIEW PROCEDURE

  • The review combined automated screening, execution against authored cases, and blind human auditing of a stratified sample.Checks were reviewed before release rather than trusted as generated.Source: §B.6.1 · p. 19
  • Claude Opus 5 reviewed every check against the human-rater criteria, but its verdicts were not treated as human validation or as the sole basis for release.The automated pass identified likely faults and defined the strata for human sampling.Source: §B.6.1 · p. 19
  • Every check was run against authored cases, and checks failing a case were corrected before sampling.No sampled check failed a case it carried; raters constructed a patch to test behaviors lacking an authored case.Source: §B.6.1 · pp. 19–20
  • 150 of 823 checks were audited in a seeded stratified sample that deliberately overrepresented small projects and included differential and trajectory checks.Population-level rates were weighted by each project’s share of all 823 checks.Source: §B.6.1 · p. 20, Table 11 · p. 20
  • Two raters independently and blindly assessed rule fidelity, applicability preconditions, post-condition behavior, and whether each check was exact or approximate.A rejection named the failing criterion; unresolved items were rejected because acceptance asserted correctness.Source: §B.6.1 · p. 20

B.6.2 AGREEMENT

The audit found strong rater agreement and an 87.2% weighted mean acceptance rate for the checker sample. The released corpus was unchanged after review, while the audit’s claims are limited to the sampled checks and do not represent maintainer judgment.

B.6.2 AGREEMENT

  • 94.0% verdict agreement (141/150) and κ of 0.72–0.93 across the verdict and four questions indicate consistent application of the review standard.The authors describe κ as conservative given skewed marginals.Source: §B.6.2 · p. 20, Table 12 · p. 21
  • Rater disagreements clustered in preconditions that selected more targets than their rules covered, while C1 and C3 judgments against fixed materials were near-unanimous.Six of the ten differences across verdicts and pass criteria concerned this precondition issue.Source: §B.6.2 · p. 21
  • The weighted mean rater acceptance rate was 87.2% (95% CI [82.1, 92.3]); both raters accepted 83.7%, and no checks changed after the audit.The released corpus is therefore the audited corpus.Source: §B.6.2 · p. 21
  • The audit estimates corpus-level checker fidelity from the sample but makes no claim about unsampled checks or maintainer judgment.The raters did not maintain the evaluated projects.Source: §B.6.2 · p. 21

B.7 ACCEPTANCE CHECKS AND FAILURE MODES

  • Repository acceptance required three checkpoints, with an author performing the first two and an acceptance script performing the third.Table 13 lists the extraction-workflow checkpoints.Source: Table 13 · p. 21, §B.7 · p. 21
  • Construction revealed mostly model-driven extraction failure modes, each addressed by a step in the workflow.Table 14 catalogs the observed rule-extraction failure modes.Source: §B.7 · p. 21

C EXPERIMENT DETAILS AND ADDITIONAL RESULTS

Three of four models were within five percentage points of public SWE-bench Verified resolve rates, while GPT-5.6 Luna was 14.4 points lower. Run coverage also included unresolved functional grades and policies withheld when changed Python files could not be parsed.

C EXPERIMENT DETAILS AND ADDITIONAL RESULTS

C.1 RUN COVERAGE

  • 61 of 8,000 runs (0.8%) received no functional verdict, mainly because of agent outcomes, with the remainder due to provider or environment failures.Compliance grading covered all 8,000 stored runs, and runs without a functional verdict counted as unresolved.Source: §C.1 · p. 22
  • In 36 runs, changed Python files failed to parse, withholding affected policies; together with unresolved functional grades, these cases covered 95 distinct runs, with two overlapping.Figure 7 gives the causes of the two groups.Source: §C.1 · p. 22

C.2 COMPARISON WITH PUBLIC RESOLVE RATES

  • Three of four models were within 5 percentage points of public SWE-bench Verified resolve rates; GPT-5.6 Luna resolved 14.4 points fewer instances.The comparison uses the same scaffold, but Vals AI used the Responses API model class for OpenAI models while this study kept the default class for all models.Source: §C.2 · p. 22

C.3 POLICY COVERAGE

This section describes how bug-fix tasks shape observable repository-policy coverage and how policies are discovered, extracted, classified, and paired with checkers. It also covers adapting task prompts for Mini-SWE-Agent and OpenHands, including OpenHands’ staged contribution workflow.

C.3 POLICY COVERAGE

  • 81 of 107 Specialized changes policies never trigger, largely because bug-fix tasks omit procedures such as API deprecations; only 2 of SymPy’s 27 trigger.Documentation and docstrings contains 224 policies but triggers in only 5 to 7% of cells; 31 of SymPy’s 40 such policies trigger at least once. Policies that never trigger reflect the pairing of corpus and task set, not agent compliance.Source: §C.3 · pp. 23–24

C.4 TRIGGERED AND WITHHELD POLICIES

  • Consolidated runs triggered 22.7 policies on average under mini-SWE-agent and 23.5 under OpenHands, versus 21.6 and 22.0 for Native runs, motivating reporting triggering rates alongside compliance.A triggered policy is withheld when needed evidence is missing; withheld policies count as triggered but are excluded from compliance rates, so checker limits are not treated as agent verdicts.Source: §C.4 · pp. 24–25

D RUBRICS AND PROMPTS

  • Appendix D reproduces the instruments behind Appendix B, including labelling rubrics, human-review guidelines, benchmark-construction prompts, and prompts given to evaluated agents.Source: D RUBRICS AND PROMPTS · p. 25

D.1 LABELLING RUBRICS

  • Policies enter the benchmark only when they have inspectable evidence, govern contributor actions, and state mandatory requirements; annotators apply these criteria in a fixed order.Each rule is an atomic obligation extracted from repository contribution documentation and restated as one imperative sentence.Source: §D.1 · p. 25

D.1.1 INSPECTABLE EVIDENCE (FIELD: EVIDENCE TYPE)

  • Evidence Type identifies what a checker must inspect: output, differential states, an agent trajectory, or human judgment.Output checks inspect the final state, differential checks compare states, trajectory checks examine process, and judgment rules depend on human opinion rather than a checkable artifact.Source: §D.1.1 · pp. 25–26

D.1.2 CONTRIBUTOR ACTIONS (RUBRIC A; FIELD: IN SCOPE)

  • A rule is in scope when the run produces evidence and its passing condition can be stated; otherwise, the rubric distinguishes unobservable, subjective, duplicate, and trigger exclusions.A rule is not observable if its artifact does not exist in the run, while a trigger depends on an external fact; whether an agent used a particular tool is within its recorded behavior.Source: §D.1.2 · p. 26

D.1.3 MANDATORY REQUIREMENTS (RUBRIC B; FIELD: REQUIREMENT LEVEL)

  • Rubric B assigns in-scope rules must or should, with acceptance gates and specified tokens treated as must, while exceptions and result properties can support should classifications.The surrounding documentation matters: sanctioning text can change the strength assigned to an otherwise similar rule.Source: §D.1.3 · pp. 27–28

D.2 CHECKER REVIEW GUIDELINES

  • Checker reviewers assess whether code matches its rule, selects the right artifacts, behaves correctly, and is labelled exact or approximate; uncertain checks are rejected rather than accepted.Reviewers work independently and judge the encoding, not whether the rule itself was worth extracting or correctly classified.Source: §D.2 · pp. 28–29
  • Reviewers must leave descriptive columns unchanged and not repair a checker before the other rater reviews it; differing verdicts or C1–C3 ratings go to a joint session.Source: §D.2 · p. 29

D.3 CONSTRUCTION PROMPTS

  • The construction-prompt appendix reproduces the launcher and three stage blocks, summarizes reviewer checkpoints in Table 13, and releases the unedited originals with the complete workflow document.The prompts guide an agent through writing the checker functions.Source: §D.3 · p. 29

D.3.1 LAUNCHER PROMPT

  • The launcher directs each repository run to the workflow, rubrics, discovery script, and format standard, while barring git commands and limiting file changes to that repository’s rules directory.It also requires running the existing corpus and rule-audit gates and reporting audit results, row counts, scope splits, and requirement-level and N-route counts.Source: §D.3.1 · p. 30

D.3.2 STAGE 1 PROMPT: SOURCE MANIFEST

  • The source-manifest stage requires fetched documentation, classifies sources as rules, context-only, or excluded, and marks pages with mixed relevant and irrelevant content as partial.Its scope includes contribution rules across code, testing, documentation, workflow, commit/PR conventions, and agent-instruction files.Source: §D.3.2 · pp. 30–31

D.3.3 STAGE 2 PROMPT: EXTRACTION

  • The extraction stage reads confirmed sources in full and semantically, splitting contribution rules into atomic rows with one condition and one expected behavior.It preserves contextual requirement levels and explicit triggers, and treats API specifications as conformance obligations rather than inventories of interface members.Source: §D.3.3 · pp. 31–32

D.3.4 STAGE 3 PROMPT: CLASSIFICATION

  • The classification stage processes rules in batches of 25 across the sheet, then assigns evidence type, scope, pass type, requirement level, decision source, reasoning, and auto-fix status.Cross-page batching supports duplicate detection; auto-fix labels distinguish “Yes, auto-fixed,” “No, check only,” and “Partial risk.”Source: §D.3.4 · p. 33
  • The classification schema records scope, evidence, and requirement level while preserving each row’s naive modal reading and distinct reasoning in Notes.The modal-verb level is required even when it matches the final requirement level.Source: §D.3.4 · p. 34

D.3.5 CHECKER AUTHORING GUIDE

  • A project-agnostic guide used across all twelve repositories specifies a two-function checker contract separating policy applicability from whether each selected target complies.The precondition selects based on the situation invoking a policy, while the post-condition evaluates the target artifact.Source: §D.3.5 · p. 34
  • Each checker gets tests for satisfied, violated, and no-target inputs, which pin the author’s interpretation but do not validate that it is correct.An Undetermined result means the policy applies but artifacts cannot answer whether it was followed; it is neither a violation nor a pass.Source: §D.3.5 · pp. 34–35

D.4 AGENT PROMPTS

  • Both scaffolds retained their stock system prompts but received modified SWE-bench task prompts that added tests and documentation, compliance instructions, and commit and pull-request submission.The compliance paragraphs were identical across the two scaffolds.Source: §D.4 · p. 35

D.4.1 MINI-SWE-AGENT

  • Mini-SWE-Agent’s modified prompt directs agents to read the extracted repository rules, follow them, make a real Git commit, and submit a written pull-request description.The submission procedure requires separate commit and submit commands; the pull request is simulated as text, not a real GitHub object.Source: §D.4.1 · pp. 36–37

D.4.2 OPENHANDS

  • OpenHands’ stock system prompt was augmented with browser tools so agents could fetch policy pages under Native, and its task prompt was modified from the stock instructions.The task prompt replaced instructions that tests were already handled and changes should be minimal and limited to non-test files.Source: §D.4.2 · p. 37
  • Changes should be minimal and focused, add or update repository-convention tests for behavior changes, preserve existing tests, and avoid packaging or CI edits unless necessary.Permitted modifications include source, test, and documentation files; existing tests must not be deleted, weakened, or skipped.Source: §D.4.2 · p. 38
  • The prescribed workflow moves through reading, test setup, exploration, reproduction, fix analysis, focused implementation, verification, and final review.Agents are asked to clarify the issue, locate relevant files, reproduce the failure before fixing it, then verify the fix and related tests.Source: §D.4.2 · p. 38
  • Verification requires rerunning the reproduction, adding edge cases, running related existing tests, and revising failures after a final requirements review.The final review checks tests related to the issue, modified files, and changed functions.Source: §D.4.2 · p. 38
  • Agents must read the repository’s contribution guidelines before starting work and submit a real Git commit plus a written pull-request description.The prompt specifies that the pull-request description is text-only in this environment.Source: §D.4.2 · p. 39
  • After committing, agents must run `/workspace/.compliance/collect` in its own tool call, then finish with a message beginning `PR SUBMISSION:`.The collector must run exactly as shown after the commit and before completion; it produces the patch and grading metadata.Source: §D.4.2 · p. 39

Related papers

Loading 2610.06193v1…