PaperlayerView exact revision on arXiv

Source-linked AI summary

Precedent or Intent? What Coding Agents Follow When Evolving Existing Code

Haichao Zhu, Geyang Xu, Zhaorui Yang, Jiyuan Wang, Qian Zhang

arXiv:2610.38244v1cs.SEcs.AI

TL;DR

When multiple implementations satisfy functional requirements, coding agents must decide whether to follow repository precedent or new task instructions, although existing code rarely records its rationale. Controlled experiments and open-source validation show that agents follow precedent over shared defaults and that written priority rules can redirect them more reliably than examples alone.

  • Problem

    How coding agents prioritize defaults, repository patterns, and explicit task guidance remains poorly understood when multiple implementations satisfy functional requirements.

    Source: §1 · p. 2

  • Method

    The study varies implementation precedents and task guidance in controlled experiments, comparing conditions bidirectionally and examining validity boundaries in real repositories.

    Source: §3.1 · p. 4, §3.3 · p. 5, §5 · pp. 12–13

  • Results

    359/360 trials followed repository precedent over shared defaults; local precedents yielded 96% adoption (347/360), versus 4% (13/360) for sibling-resource precedents.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Design-decision traceability should preserve implementation choices along with the rationale and priorities guiding future changes.

    Source: §7 · p. 14

    The controlled experiments cover one Flask API task family, while open-source validation covers two implementation choices across four repositories and explicit guidance in one choice.

    Source: §5 · p. 13

Abstract

from arXiv · show

Existing code reveals past implementation decisions but rarely explains the reasoning behind them. When coding agents implement new features or modify existing functionality, past decisions may conflict with new task instructions. This raises a fundamental question for agentic software evolution: when multiple implementations satisfy the same functional requirements, how do coding agents decide whether to follow existing code or new task instructions? We investigate this question through bidirectional, controlled experiments in purpose-built repositories, followed by validation in real-world open-source repositories. Our approach isolates three competing sources of guidance: (1) agents' default preferences, (2) repository precedents, and (3) explicit task instructions. We conduct 5,625 trials across four coding agents and six implementation pairs, with validation in four open-source repositories spanning three programming languages. We find that agents share the same default implementation choices, yet overwhelmingly follow repository precedent when it conflicts with those defaults (359/360 trials). Crucially, not all precedents carry equal influence: three supporting local files yield 96% adoption, compared with only 4% for sibling-resource files, and changing file placement alone can shift adoption from 0% to 100% at a fixed example count. When new task intent conflicts with existing code, written priority rules achieve substantially higher success than code examples alone (59/60 versus 38/60 joint successes for two agents on two implementation pairs). The effectiveness of explicit rules also extends to open-source repositories, where a harness-loaded rule raises joint success from 0/120 to 117/120. These findings motivate design-decision traceability that preserves not only implementation choices but also the rationale and priorities that should guide future changes.

1 Introduction

Coding agents must reconcile past implementation decisions with new task intent, but the rationale behind existing code is often unavailable. This study tests how defaults, repository precedents, and explicit task guidance shape those choices, finding that precedent and written rules can redirect agents toward different implementations.

1 Introduction

  • Existing code reveals past implementation decisions but rarely explains why they were chosen or whether they reflect constraints that should persist.The paper frames this as a challenge when requirements evolve and task instructions conflict with established patterns.Source: §1 · p. 2
  • The study isolates agents’ defaults, repository precedent, and explicit task guidance through controlled experiments and open-source validation.It varies precedent location and support, then tests guidance conditions including written priority rules and code examples.Source: §1 · p. 2
  • 359/360 trials followed repository precedent when it conflicted with agents’ shared default choices.Without relevant precedents, all four agents favored the same implementation in each of six pairs.Source: §1 · p. 2
  • With three supporting files, local precedents yielded 96% adoption (347/360), versus 4% (13/360) for sibling-resource precedents.At a fixed example count, file placement alone could shift adoption from 0% to 100%.Source: §1 · pp. 2–3
  • Written priority rules overrode conflicting precedent more reliably than code examples alone, with joint success in 59/60 versus 38/60 trials.In open-source repositories, a harness-loaded rule raised joint success from 0/120 to 117/120.Source: §1 · p. 3
  • The authors motivate design-decision traceability to preserve implementation choices and the rationale and priorities guiding future changes.Their contributions establish this prioritization question as a software-evolution research problem and introduce bidirectional controlled experiments.Source: §1 · p. 3

2 Motivating Example

The motivating example separates an endpoint’s required behavior from an open implementation choice: how to serialize its task identifier. It illustrates the conflict between matching existing code and following new task intent.

2 Motivating Example

  • The new endpoint can serialize the task identifier as a JSON number or a string while satisfying the stated behavioral specification.Both implementations retrieve the same task using an identical query and differ only in identifier serialization.Source: §2 · p. 3
  • An existing task handler serializes identifiers as numbers, establishing a precedent the new endpoint could carry forward.Other implementation patterns may coexist elsewhere in the repository, so historical code need not present one consistent pattern.Source: §2 · p. 3
  • A task instruction to use string identifiers and prioritize that instruction conflicts with preserving the existing numeric representation.The research problem is how agents choose among defaults, repository precedent, and explicit instructions when requirements allow multiple implementations.Source: §2 · pp. 3–4

3 Approach

The study separates agents’ defaults, repository precedent, and task guidance while keeping the functional task and hidden tests unchanged. It combines controlled experiments with validation in open-source repositories.

3.1 Overview

  • Each implementation pair contains two alternatives accepted by the same hidden tests, and the controlled study uses six pairs spanning naming, representation, and code structure.The purpose-built Flask repository supplies the study setting for the three-stage design.Source: §3.1 · p. 4, Table 1 · p. 5

3.2 Stage 1: Establishing Default Preferences

  • Stage 1 removes the six task and project endpoint files, then defines an agent’s default as the option chosen in more than half its sessions.In the ID-serialization case, all four models default to numeric IDs.Source: §3.2 · p. 5

3.3 Stage 2: Manipulating Repository Precedents

  • Stage 2 varies alternative-use support in local and sibling-resource files while holding the task description’s implementation choice unspecified.The bidirectional design varies one group while the other stays at baseline, enabling comparisons of their influence and of agreement between groups.Source: §3.3 · p. 5
  • At fixed support, Stage 2 changes which endpoint file carries the alternative to separate file placement from example count.The placement interventions include alternative use in the create, update, or list endpoint at 1/3 support.Source: §3.3 · p. 6, Figure 4 · p. 6

3.4 Stage 3: Introducing Explicit Task Guidance

  • Stage 3 pits a specified target against endpoints using the opposite implementation, then compares no guidance, a written priority rule, a code example, and both together.The study also tests delivering the same rule through the task description or the harness’s native instruction file.Source: §3.4 · p. 6

3.5 Measuring Outcomes

  • Functional success is measured with hidden tests, while implementation choice is classified separately; Stage 3 joint success requires both.The experiments comprise 5,625 trials in 375 cells, with 15 independent sessions per cell on fresh repository instances.Source: §3.5 · pp. 6–7, Table 2 · p. 7

3.6 Open-Source Repository Validation

  • Open-source validation varies local and sibling-resource patterns across four REST API repositories in Python, Go, and TypeScript.Because repositories fix handler number and identity, this validation tests precedent group, not support level or file placement.Source: §3.6 · p. 7

4 Evaluation

The evaluation uses repeated controlled trials to separate agents’ default choices, repository precedent, and explicit task guidance, then tests key effects in open-source repositories. Across these conditions, local precedent strongly shapes implementation choices, while written priority rules can redirect agents when task intent conflicts with existing code.

4 Evaluation

  • The evaluation asks how defaults, local and sibling-resource precedents, and explicit guidance influence implementation choices, using controlled experiments and open-source validation.The controlled repository addresses RQ1–RQ4; the open-source study tests the local-versus-sibling contrast and explicit rules.Source: §4 · p. 7

4.1 Experimental Setup

  • Each trial is an independent agent session on a fresh repository instance, and the study comprises 5,625 trials.Saved outputs are checked against hidden functional tests and pair-specific classifiers.Source: §4.1 · p. 8

4.2 RQ1: Default Implementation Preferences

  • Without relevant precedent, all four agents favor the same implementation in each of six pairs; when all endpoints use the opposite implementation, agents follow precedent in 359/360 trials.The zero-precedent defaults are nearly deterministic, with 359/360 runs choosing the default.Source: §4.2 · pp. 8–9, Figure 5 · p. 8

4.3 RQ2: Influence of Local Precedent

  • Local support raises alternative adoption from 0% without support to 96% with three supporting local files, and file placement can shift adoption from 0% to 100% at fixed support.The placement effect was detected in three of four tested settings.Source: §4.3 · pp. 9–10, Figure 6 · p. 10

4.4 RQ3: Influence of Sibling-Resource Precedent

  • Conflicting sibling-resource patterns produce only 13/1,079 adoptions (1.2%), compared with 347/360 when three supporting files are local.Matching sibling patterns can reinforce local patterns, raising response-envelope adoption from 8/15 to 15/15 in one tested setting.Source: §4.4 · p. 10

4.5 RQ4: Redirecting Choices with Explicit Guidance

  • 59/60 trials achieve joint functional success and target adoption with a written priority rule, versus 38/60 with code examples alone and 0/60 without guidance.Rules in task descriptions and harness-loaded native instruction files each achieve 60/60 in the delivery experiment; the controlled RQ4 comparison is summarized in Figure 7.Source: §4.5 · p. 11, Figure 7 · p. 11

4.6 Generalization to Open-Source Repositories

  • In four open-source repositories, native-file rules achieve 117/120 joint successes versus 0/120 without guidance, while local-pattern adherence varies by implementation choice.Agents followed local precedent in all 945 classifiable successful error-handling implementations and 355/417 (85%) successful response-envelope implementations; sibling-resource patterns had no detectable effect on either choice.Source: §4.6 · p. 12

5 Threats to Validity

The study's validity is bounded by its controlled settings, selected implementation-pattern measures, and limited task and repository coverage.

  • The experiments measure selected implementation choices, not maintainability or overall code quality.Source: §5 · p. 12
  • The zero-precedent condition measures task-specific choices without relevant endpoint examples, not context-free preferences; native-file tests cover harness-loaded rules, not autonomous discovery.Source: §5 · p. 12
  • Support-level experiments vary example count and file assignment together, while a separate analysis shows sensitivity to file identity.Source: §5 · p. 12
  • Open-source error-handling analysis counts task success over 960 trials and local adoption over 945 classifiable successful implementations; guidance and envelope tables report 15-run cells.Source: Table 4 · p. 12, Table 5 · p. 12, Table 6 · p. 13
  • The controlled experiments use one Flask API task family; open-source validation covers four repositories in three languages, two implementation choices, and explicit guidance for one.Source: §5 · p. 13

6 Related Work

Prior work evaluates coding-agent correctness, repository context, and conflicting guidance; this study instead isolates which functionally acceptable implementation agents select and what informs that choice.

  • Earlier benchmarks progress from standalone program correctness [5] [3] to repository issue resolution [13] and long-horizon evolution [6] [15], while other work studies maintainability and pull-request acceptance [22].Source: §6 · p. 13
  • Unlike studies asking whether changes succeed or remain maintainable, this paper asks which functionally acceptable implementation agents select and what determines that choice.Source: §6 · p. 13
  • Repository-context techniques retrieve or plan over cross-file context [25] [36] [7] [31] [4]; this study holds correctness fixed while varying context.Source: §6 · p. 13
  • Prior work examines coding preferences and demonstrations [27] [39] [19] [8] [18], conflicting context [33] [34], and instruction hierarchies [28]; this study measures conflicts among defaults, code, and task intent amid lost design rationale [17] [24].Source: §6 · pp. 13–14

7 Conclusion

The study identifies implementation choice as a distinct dimension of agentic software engineering. Its findings support evaluating how agents reconcile past decisions with evolving intent and preserving the rationale and priorities that should guide future changes.

  • Repository precedent overrides shared defaults; where precedent appears can matter more than its frequency, and written priority rules redirect agents more reliably than examples.Source: §7 · p. 14
  • Evaluation should assess how agents reconcile past decisions with evolving intent, while design-decision traceability preserves the rationale and priorities guiding future evolution.Source: §7 · p. 14

Related papers

Loading 2610.38244v1…