Methodological Harness in Agentic Software Engineering: An Empirical Study on Mining Software Repositories

1 Introduction

• The proposed harness addresses the team-level mechanisms surrounding agents, but whether those mechanisms exist in practice remained unanswered by peer-reviewed research.The framework was presented as a first step toward academic-industrial consensus, not as a validated theory.Source: §1 · p. 2
• The study asks whether and how harness mechanisms are observable in repositories, rather than whether harness adoption improves development performance.Its stated scope is adoption and materialization, not effects on integration, review speed, or approvals without human oversight.Source: §1 · pp. 3–4
• RQ1 characterizes artifact prevalence, breadth, co-occurrence, visibility gradients, and timing, while RQ2 examines rule-file content against the framework’s mechanisms.The dimensions describe one adoption question rather than separate research questions.Source: §1 · p. 3
• The study analyzes 5,435 AIDev repositories for seven observable mechanisms and qualitatively codes 285 rule files from 150 repositories.H4, N-version mindset and parallel agents, was excluded because it leaves no persistent Git trace.Source: §1 · pp. 3–4

2.1 Agentic Software Engineering

• Prior literature reports a gap in evidence about whether harness mechanisms for absorbing agent capability into team practice exist and how they are adopted.The study therefore measures presence and content before studies of impact that take those conditions for granted.Source: §2.1 · p. 4

2.2 Harnessing Engineering

• The methodological harness extends the technical harness from single-agent execution to team practices and artifacts supporting reproducibility, knowledge transfer, coordination, and review.Its eight mechanisms span context engineering, shared knowledge, executable specifications, parallel agents, norms, consultation, evidence-based acceptance, and graduated autonomy; the taxonomy is conceptual, not a validated performance theory.Source: §2.2 · pp. 5–6, Figure 1 · p. 5
• The harness mechanisms are intended to work cumulatively, linking specifications, evaluation, consultation, governance, and knowledge across agent sessions.The framework is tested against repository evidence rather than assumed effective or commonly adopted.Source: §2.2 · p. 6

2.3 Repository mining: population, bias and protocol

• Visibility stratification makes popularity bias measurable, while the study uses safeguards such as a frozen codebook and a prespecified reliability threshold.The authors depart from formal preregistration because the study has no outcome variables or association hypotheses; they state it will be mandatory when outcome variables are introduced.Source: §2.3 · pp. 6–7

3 Research Methodology

• The study uses research questions and descriptive estimates rather than null-hypothesis significance testing because it has no outcome variables or association hypotheses.Every published figure is traceable to a named script and fixed dataset version.Source: §3 · p. 7
• The study measures harness adoption and implementation before asking whether adoption produces effects.The authors identify effects as outside this study’s scope.Source: §3 · p. 7

3.1 Study design

• The study combines a pilot instrument check with repository mining and qualitative content analysis using the repository as the shared unit of analysis.RQ1 uses automatic detectors for presence; RQ2 uses human coding to examine artifact content.Source: §3.1 · pp. 7–8, Figure 2 · p. 8

3.1.1 Data source

• AIDev’s PR coverage ended July 30, 2025, but artifact histories were observed through cloning in August 2026.Because inclusion depends on agent activity before the cutoff, the sample comprises early adopters; their harness prevalence may overestimate that of the general population with agentic activity.Source: §3.1.1 · pp. 8–9

3.1.2 Population and sample design

• The sample combines a census of 2,807 popular repositories with 3,000 repositories sampled across three visibility bands from a population of 116,211.Stratification supports comparisons across visibility bands; the study reports population figures both with and without repositories having only one agentic PR.Source: §3.1.2 · pp. 9–10, Table 1 · p. 10

3.1.3 Attrition

• Attrition rose from 2.7% in P to 14.4% in G3, so G3 prevalence is estimated from the repositories that remained accessible.If disappeared repositories were disproportionately short-lived and less harnessed, the observed G3 estimate would be biased upward; the direction cannot be verified.Source: §3.1.3 · pp. 10–11

3.2.1 Artifact extraction

• The analysis clones repositories and extracts artifact presence and commit dates in one pass, using GitHub’s API only when cloning fails.Clones are discarded after extraction, while rule-file text is retained for qualitative analysis.Source: §3.2.1 · p. 11

3.2.2 Deduplication (data cleansing)

• Deduplication excludes symbolic links, artifacts without resolvable first-appearance dates, and byte-identical duplicates; mechanism counts are per mechanism, not per path.Stub files are flagged and handled separately for H1 prevalence because their implementation status is debatable.Source: §3.2.2 · p. 12

3.2.3 Operationalization of harness mechanisms

• Detectors use canonical paths or textual patterns, so prevalence is a lower bound on harness-like mechanisms under other names.H4 is excluded because its internal agent-tool implementation leaves no retained repository artifact; H6 is observable only in popular stratum P, where pull-request comments are available.Source: §3.2.3 · pp. 12–14

3.2.5 Temporal estimands

• E1 measures artifacts present at cloning, whereas E2 counts agentic pull requests only when an artifact appeared by the pull request’s creation date.The two estimands are reported separately because their values can differ substantially.Source: §3.2.5 · pp. 14–15

3.2.6 Descriptive measures

• The study reports prevalence, visibility gradient, breadth, co-occurrence, and temporal evolution to characterize observable adoption.Cross-stratum figures are reweighted by population size, while stratum-level comparisons remain the key comparison.Source: §3.2.6 · p. 15

3.2.7 Validation of the instrument

• H5 is counted as modal clauses within H1 rule files, so it is reported conditional on H1 rather than as an independent mechanism.This avoids counting the same artifact twice in prevalence and breadth.Source: §3.2.7 · p. 16
• A detector’s zero is treated as measured only after a synthetic-repository test confirms that the detector triggers when its artifact is present.The H8 workflow pattern was removed after it flagged 88% of pull requests and behaved like a saturated constant.Source: §3.2.7 · p. 16

3.2.8 Corpus language, and why it is measured

• Because the H5 modal-clause pattern is English-only, non-English rule files can be recorded as lacking normative content despite containing it.Language detection estimates the scope of this measurement error; the sensitivity analysis focuses on H1 repositories with zero detected clauses.Source: §3.2.8 · p. 17

3.3 Qualitative study: RQ2 — alignment with the framework

• The qualitative method analyzes complete rule-file content from repositories, not isolated files, preserving the repository as the sampling unit.Repositories can contribute multiple rule files to the coded corpus.Source: §3.3 · p. 17, §3.3.3 · p. 18

3.3.1 Protocol; 3.3.2 Iterative procedure

• Two independent coders work from a frozen deductive codebook, with at least 20% overlap, a κ ≥0.60 threshold, and third-author adjudication of disagreements.If agreement misses the threshold, the codebook is revised and another block is coded.Source: §3.3.1 · p. 18, §3.3.2 · p. 18

3.3.3 Qualitative data

• The sample contains 150 repositories stratified across visibility bands, corpus-size terciles, and adoption eras; its size is intended to assess saturation, not statistical inference.The complete rule-file corpus from each selected repository is coded, yielding 285 files.Source: §3.3.3 · p. 18

3.3.4 Codebook

• The codebook maps rule-file content to harness mechanisms and permits inductive coding when content does not fit its categories.It also explicitly codes confusion between procedural instructions and normative specifications.Source: §3.3.4 · p. 19

3.3.5 Categories (M1–M8)

• The categories distinguish agent guidance, persistent knowledge, executable and normative specifications, prescribed consultation, evidence-based acceptance, and graduated autonomy.M6 captures written policy to pause and ask a human, unlike the separate PR-comment classifier for observed consultation.Source: §3.3.5 · pp. 19–20

3.3.6 Descriptive measure

• Category prevalence is measured in the qualitative sample conditional on H1, using human-coded content rather than automatic artifact detection.The study does not calculate breadth, co-occurrence, or temporal evolution for these categories.Source: §3.3.6 · p. 20

3.4 Reproducibility and Content Handling

• The replication package includes code, the codebook, coding labels and adjudications, the consultation-detector validation sample, and content-addressed pointers to coded rule files.The sampling frame is regenerated deterministically from the pinned dataset and recorded seed.Source: §3.4 · p. 21
• The authors do not republish third-party repository content beyond brief citations, given heterogeneous licenses and many repositories without declared licenses.Rule-file text is analyzed locally, and the qualitative sample’s license composition is reported.Source: §3.4 · p. 21
• Results are reported at repository level, without publishing identifiable-individual results or constructing developer profiles.Source: §3.4 · p. 21

4.1 Sampling design

• 5,435 of 5,807 repositories yielded usable clones; 39 others remained accessible but uncloned, and 333 were deleted or private.The remaining 372 repositories comprise the 39 API-fallback and 333 unavailable cases.Source: §4.1 · p. 21
• Attrition rose from 2.7 % in P to 14.4 % in G3, and mining identified 35,749 deduplicated artifacts.A total of 1,067 items, including symlinks, artifacts without resolvable dates, and byte-for-byte duplicates, were discarded.Source: §4.1 · p. 21
• H3 dominates artifact-file totals, while rule files (H1) are a minority; a single Gherkin suite can contribute hundreds of files.Artifact totals count files, whereas prevalence counts repositories.Source: Table 4 · p. 22

4.2.1 Prevalence

• Among repositories with rule files, normative clauses range from 90.9 % in P to 74.1 % in G3, but English-pattern detection misses non-English clauses.The reported H5 indicator is conditioned on H1; excluding unreadable files reduces the apparent decline, though a monotonic decrease remains.Source: §4.2.1 · pp. 23–24, Table 5 · p. 23, Table 6 · p. 24

4.2.2 Visibility gradient

• All five mechanisms become more prevalent with visibility, but H2 has no discernible G1–G3 gradient because its confidence intervals overlap.Persistent knowledge remains rare even among the most visible repositories; the proposed team-size explanation for differing gradients is an interpretation, not a measured result.Source: §4.2.2 · pp. 24–25

4.2.3 Breadth / 4.2.4 Co-occurrence

• Four- or five-mechanism adoption is uncommon, and the most frequent pairs combine rule files with governance artifacts rather than methodological mechanisms.H1 & H7 appears in 708 repositories, compared with 182 for H1 & H3 and 66 for H1 & H2; Figure 3 shows raw pair counts and individual prevalence on its diagonal.Source: §4.2.3 · pp. 25–26, §4.2.4 · p. 26, Figure 3 · p. 26

4.2.5 Temporal adoption

• For 33,596 P-stratum PRs, measuring artifacts at cloning rather than PR creation would count 17.7 % (5,954) as harnessed when they were not at opening.E2 is PR-weighted and unavailable for the G strata; E1 is repository-weighted, so the two measures are not directly interchangeable.Source: §4.2.5 · p. 27, Table 9 · p. 27
• Of dated H1 artifacts, 58.3 % appeared after the PR window closed; the monthly first-appearance histogram peaks in June 2025, but saturation and selection distort its shape.Figure 4 marks the window closure; cumulative incidence among the at-risk population with right-censoring is identified as the appropriate future analysis.Source: §4.2.5 · p. 28, Figure 4 · p. 28

4.2.6 Validation of H6 (structured consultation)

• The H6 detector recovered 26 of 53 coder-confirmed consultations, below the preset 0.70 recall threshold, so H6 was excluded from quantitative analysis.The sample nevertheless confirmed 53 structured consultations.Source: §4.2.6 · p. 29, Table 10 · p. 29

4.2.7 Sensitivity Analysis

• Retaining stub-files changes the any-mechanism estimate by less than half a percentage point, while restricting to repositories with at least two agentic PRs raises levels without changing the gradient.The restricted estimates are 73.4 % in P and 27.9 % in the reweighted population.Source: §4.2.7 · p. 29, Table 11 · p. 30, Table 12 · p. 30

4.2.8 Answer to RQ1

• The harness is observable, but adoption is narrow, uneven, and visibility-dependent; isolated mechanisms are more common than coherent configurations.Mechanisms specific to the agentic era remain less prevalent than rule files and pre-existing governance artifacts.Source: §4.2.8 · p. 30

4.3 RQ2: Correspondence with the Framework

• RQ2 manually coded rule-file contents from 150 repositories with at least one rule file, using eight framework categories and an additional category for substantial procedural content.Unlike artifact-presence measures, this analysis examined what the files contain.Source: §4.3 · p. 30, §4.3.2 · p. 31

4.3.1 Reliability

• After a second coding iteration, all eight fields exceeded κ ≥ 0.60 or the applicable raw-agreement threshold on the 30-item overlap.M1 and M5 were validated through raw agreement (93% and 97%) because of extreme prevalence; Table 14 describes the amended-codebook iteration.Source: §4.3.1 · pp. 30–31, Table 14 · p. 31

4.3.2 Prevalence of Categories

• 10.0–26.7% of rule files include persistent knowledge, 10.0–20.0% executable specifications, 5.0–20.0% structured consultation, and 13.3–26.7% graduated autonomy across strata.The category prevalence is conditioned on repositories having at least one rule file; Table 15 reports the breakdown by stratum.Source: §4.3.2 · p. 32, Table 15 · p. 32
• 56.7% of G2 and 43.3% of G1 rule files have substantial procedural content, compared with 25.0% of P and 16.7% of G3.This category is not monotonic with visibility and is distinct from the framework’s normative-specification category.Source: §4.3.2 · p. 32

4.3.3 Answer to RQ2

• Agent guidance (M1) and normative specifications (M5) are near-universal, while shared knowledge, executable specifications, structured consultation, and graduated autonomy appear only in a minority.Evidence-based guidance (M7) is comparatively common; overall, rule files materialize selected components rather than the integrated harness.Source: §4.3.3 · p. 33

• G3 estimates use surviving repositories despite one in seven being inaccessible, versus one in thirty-seven in P; the direction of resulting bias cannot be verified.If disappeared G3 repositories were less harnessed, the observed gradient would be conservative.Source: §5 · p. 33
• H6’s lexical classifier retrieved only 49.1% of 53 human-confirmed consultations, so the authors excluded H6 from quantitative analysis rather than treating the detector’s misses as mechanism absence.The authors attribute the limitation to the instrument, not to the mechanism.Source: §5 · p. 33
• Filename-based detectors miss mechanisms implemented under other conventions, making prevalence estimates lower bounds; H2 may also be invisible when stored outside the public versioned tree.The data cannot distinguish absence of persistent knowledge from practices that leave no public repository trace.Source: §5 · pp. 33–34
• Because the sample requires agentic activity before August 2025, early harness adopters may be overrepresented; reweighting corrects popularity bias, not early-adoption bias.The authors say this may make estimated prevalence higher than among the general population of repositories with agentic activity.Source: §5 · p. 34
• The study measures presence, not whether the harness improves development performance, so these figures support no efficacy claims.The authors leave effects for subsequent work.Source: §5 · p. 34

6.1 RQ1: The harness is observable, but widespread adoption is uncommon

• 21.7% of repositories with agentic activity have any harness mechanism after reweighting, while only 2.6% of the most visible stratum combine four or five mechanisms.Adopters typically materialize one or two mechanisms; common combinations pair rule files with governance artifacts, while rule files alongside executable specifications occur in 182 of 5,435 repositories and alongside persistent knowledge in 66.Source: §6.1 · pp. 34–35

6.2 RQ1: Adoption varies heavily with repository visibility

• All five common mechanisms decrease monotonically with visibility; CODEOWNERS drops ×8.7 and PR templates ×6.2, compared with ×4.1 for rule files.The authors interpret the gradient as possibly reflecting team presence, but note that team size was not directly measured.Source: §6.2 · p. 35

6.3 RQ1: Mechanisms specific to the agentic era show substantial recent adoption

• Agent-specific mechanisms roughly doubled between observed work and data collection (×2.00 to ×2.12), while pre-existing governance artifacts changed little (×1.18).58.3% of rule files appeared after the agentic work-window closure, but the monthly peak cannot establish acceleration because of saturation and selection effects.Source: §6.3 · p. 36

6.4 RQ2: The content of the rule files corresponds only partially to the framework

• Rule files almost always guide agents and state norms, whereas persistent knowledge, executable specifications, structured consultation, and graduated autonomy appear only in a minority.Substantial procedural content does not follow the other categories’ visibility pattern: G2 is 56.7%, G1 43.3%, P 25.0%, and G3 16.7%.Source: §6.4 · pp. 36–37

6.5 Implications of the Two Research Questions for the Framework

• The authors conclude that the harness is better understood as a goal than as widespread practice, and suggest separating agent-governance mechanisms from repurposed coordination artifacts.They also recommend distinguishing whether a mechanism is prescribed in a rule file from whether it is enacted in observable behavior.Source: §6.5 · p. 37

• 21.7% of repositories with agentic activity have at least one harness mechanism, and combining four or five mechanisms is exceptional even among the most visible repositories.Rule files and pre-existing governance artifacts predominate, while several harness mechanisms appear only in a minority of rule files.Source: §7 · p. 38
• Adoption varies unevenly: visibility gradients and first-appearance timing differ between agent-specific mechanisms and pre-existing governance artifacts.The majority of observed rule files appeared after the agentic work-window closure, while growth was concentrated in agent-specific components.Source: §7 · p. 38
• The study establishes mechanism presence, not effects on development performance; whether the harness works remains a question for future research.For structured consultation, the lexical detector recovered only about half of human-confirmed cases, so the authors excluded it from quantitative analysis.Source: §7 · p. 38

Appendix A AIDev Scale by Stratum

• Table A.1 summarizes selected and successfully cloned repositories, alongside agentic pull requests, by stratum.Agentic PR counts include selected repositories regardless of whether cloning succeeded.Source: Table A.1 · p. 41, Appendix A · p. 41
• H6 is observable only in the most visible stratum, where AIDev provides PR comments, reviews, and review comments.Its detector combines PR bodies, comments, and review comments into 99,225 text units from 33,596 agentic PRs.Source: Appendix A · p. 41

Appendix B Unit of Analysis in Prevalence, E1 and E2

• The estimates vary across two independent choices: whether adoption is measured at cloning or at PR creation, and whether repositories or PRs are weighted.A repository with 500 PRs counts 500 times as much as one with a single PR under PR-based weighting.Source: Appendix B · p. 42
• E2 is meaningful only with PR-based weighting, so the comparison recalculates E1 with PR weighting as well.Only three of the four possible estimand-weighting combinations appear in the article.Source: Appendix B · p. 42
• 26.4% for E2 is lower than the E1 estimates because much recorded adoption occurred after the counted agentic work.E2 asks whether H1 existed when a specific PR was created; E1 asks whether it existed at cloning.Source: Table B.1 · p. 42, Appendix B · pp. 42–43
• 51.2% versus 55.7% reflects the same E1 definition under repository-based and PR-based weighting, respectively.The difference indicates that repositories with more agentic activity adopt H1 somewhat more frequently than average.Source: Appendix B · p. 42