Source-linked AI summary
From Traceability to Justifiability: Accountability Structures in Agentic Software Engineering
Rashid Azarang
TL;DR
The paper asks whether delivery records can express the identity and assurance claims behind AI-system promotions, and whether declared assurance is realized in public evidence. It surveys platform defaults and measures declared-versus-realized assurance depth, finding missing behavioral identity and a verifiability hole among attestation adopters. These results place the records structurally short of justifiability, while defining scope and falsification boundaries.
Problem
The paper examines what delivery platforms’ default records can express about promotions and how much declared assurance their published evidence realizes.
Method
The paper defines promotion-level assurance conditions, surveys 47 platforms under a fixed double-graded protocol, and measures realized depth from published pipeline exhaust across 30 repositories.
Results
No platform’s default record emits a content-addressed behavioral-tuple identity; seven of fifteen attestation adopters publish source-only releases, while five of seven measurable adopters realize the binding end to end.
Takeaways & Limitations
The findings locate the field’s records structurally short of justifiability because declared assurance cannot consistently be checked against an identity-bearing release surface.
Takeaways & Limitations
The study measures platform defaults and public release-pipeline evidence rather than pipelines where the behavioral tuple is itself deployable and an agent operates the loop.
Abstract
from arXiv · showhide
A pipeline promoting an AI system publishes records claiming the thing evaluated is the thing deployed and that the evidence licensed the transition. We measure, from public material only, whether those records can express that claim and whether it holds where declared. First, a two-class documentation survey of 47 delivery platforms (20 CI/CD, 27 model-serving/agent) under one fixed three-label protocol, graded twice (second pass blind), every consulted page pinned by content hash and date. Across 188 double-graded cells we found no platform whose default record emits a content-addressed identity of the behavioral tuple (model version, instructions, tool definitions, retrieval and runtime configuration); the blind pass grades that column default on zero of 47. Immutable nominal versioning is meanwhile arriving as the agent platforms' default answer (16 of 27): version integers behind mutable pointers, a layer the artifact supply chain already found insufficient. Second, an instrument computes realized assurance depth from a pipeline's published exhaust alone and compares it with the declared depth. Applied to a frozen two-stratum frame of 30 public repositories graded twice from a hashed archive (second pass blind; cell-level agreement 23 and 19 of 30, both passes independently finding the same five full realizations), the sharpest result is a verifiability hole: seven of the 15 repositories chosen for adopting attestation tooling publish source-only releases, so the binding their workflows declare cannot be checked where declared. Where checkable it mostly checks out: five of seven measurable adopters realize the binding end to end; both shortfalls fall at identity binding. Together the results locate the field's records structurally short of justifiability, the one rung that can refuse a transition. The survey carries an expiry clock; we state what would falsify each finding.
1. Introduction
The paper asks whether delivery records can express what was evaluated and whether published evidence realizes the assurance a pipeline declares. It addresses these questions through a specification, a public depth instrument, and a double-graded platform survey.
- Four continuity conditions define justifiability: artifact, behavioral, evidence, and authority continuity.The specification distinguishes these conditions across the path from evaluation to deployment.
- On a frozen 30-repository frame, the depth study found a verifiability hole: seven of 15 attestation adopters published source-only releases.Five of seven measurable adopters realized the binding end to end, and both measured shortfalls occurred at identity binding.
- The paper states falsifiers for its survey headline, depth rubric, and prevalence conjecture, while reserving deferred behavioral and temporal-order questions for registered future work.
2. What existing frameworks cannot refuse, and why
Existing frameworks verify provenance, custody, or producer-emitted evidence, but do not measure whether default records express required identity or whether published exhaust realizes declared assurance.
- Existing frameworks occupy the mechanical side of assurance, including cryptographic custody, machine-verifiable build evidence, and admissibility decisions.
- They assume cooperating producers acting under framework obligations, whereas this paper measures platforms and pipelines without producer cooperation.
- The paper measures two questions absent from those frameworks: whether default records express needed identity and whether published exhaust realizes configured depth.
- Its method follows audit discipline by grading documentation and archived exhaust only, counting unsupported evidence as undetermined.
- Adjacent accountability and AI supply-chain studies examine responsibility mappings, research-literature tracing, or registry practice rather than platform record classes and realized pipeline assurance.
3. The specification
The specification defines promotion-time justifiability through four continuity conditions and distinguishes content-addressed identity from nominal labels. It operationalizes assurance depth from intended and realized evidence.
- A promotion is justifiable only when artifact, behavioral, evidence, and authority continuity hold across evaluation and deployment.
- Artifact continuity requires content-addressed comparison showing that the evaluated artifact is the deployed artifact.
- Behavioral continuity requires the evaluated behavioral tuple to match the running tuple, beyond identical code.The tuple includes source, model version, instructions, retrieval configuration, tool definitions, runtime configuration, and environment.
- Evidence continuity links cited evidence to the behavioral tuple and requires that it was generated from that tuple rather than a predecessor.
- Authority continuity requires an entitled actor's authorization to cover the deployed identity, scope, and temporal order relative to the evidence.
- The behavioral check partitions invariant-by-contract components from environment variants, blocking mismatches only in the invariant set.The partition prevents false breaks from legitimate environment changes and false passes from comparing nothing.
- Content-addressed references identify state itself, whereas nominal labels depend on convention and can drift independently of what they name.
- The ordinal depth scale runs from no gate at 0 to bound evidence plus authority at 4, with realized depth computed from published evidence.A binding counts only when populated with a content-addressed value; rung differences carry no arithmetic magnitude.
4. The instrument, in its public form
The public instrument uses read-only checkers over platform-published artifacts to compare declared and realized assurance without producer cooperation. Its frozen, archived, blind-graded study preserves query bounds and unresolved evidence as explicit results.
- The instrument, in its public form: The instrument consists of small read-only checkers over already-published artifacts, adding no system of record, instrumentation, or producer signatures.
- The instrument, in its public form: The classifier is blind to outcomes, states declared query bounds, and counts unsupported residuals as undetermined rather than discarding them.
- 4.1 The public depth study: The frozen frame contains 30 public GitHub repositories in two declared strata of 15, including attestation adopters and high-activity release repositories.
- 4.1 The public depth study: The collector archived 1,116 raw API responses covering workflows, policies, environments, releases, attestations, and deployments.Intended depth was graded from rulesets, environments, and workflow definitions because those were available across all repositories.
- 4.1 The public depth study: The attestation surface split into seven repositories with found attestations, six with returned none, and 17 where lookup was not attempted because release assets exposed no digests.The protocol treats returned none and not attempted as distinct outcomes.
- 4.1 The public depth study: Dry-run findings made empty ruleset listings ambiguous and required attestation status to be assessed per asset rather than per repository.
- 4.1 The public depth study: Two independent blind grading passes agreed on 23 of 30 intended-depth cells and 19 of 30 realized-depth cells.The study reports raw agreement figures to bound reconciliation claims.
- 4.1 The public depth study: Archive re-inspection established that all 17 repositories without digested release assets publish source-only releases, leaving registry-side artifacts outside the study surface.Their declared bindings therefore cannot be checked on the release surface, and the correction was dated and archived.
5. The surveys
The study uses one fixed three-label protocol to assess four record dimensions across delivery platforms, with pinned public evidence and independent blind regrading. Divergences were reconciled from better-evidenced sources and preserved as method data.
- Survey protocol: Four dimensions—artifact digest, approval record, provenance emission, and behavioral-tuple identity—were graded using one fixed three-label protocol.The labels distinguish default, opt-in, and absent recording mechanisms.
- Evidence pinning: Every consulted page was content-hashed, date-pinned, and archived where possible, making page changes distinguishable from disagreements about interpretation.The pin log contained 779 records at sealing.
- Double grading: Every survey cell was graded twice, with the second pass blind to the first grades, survey record, and pin log.Agreement varied across platform classes and dimensions, with the agent tuple column showing 23 of 27 raw agreement.
- Reconciliation: Divergences were resolved by re-fetching conflicting live pages and applying recorded criteria to the better-evidenced source.Examples include vocabulary misses, criterion application differences, and stricter resolution of agent-class tuple grades.
6. Results I: what the default record can express
Default records express approval information more readily than provenance or behavioral identity, and neither platform class provides default content-addressed identity for the behavioral tuple. Agent platforms increasingly offer nominal versioning, while approval records are often absent or administratively gated.
- CI/CD class: 20 bounded absences show that no CI/CD platform’s default record vocabulary contains behavioral-tuple identity, while provenance emission is default nowhere in the class.Approval records are the most uniformly default rung, with GCP Cloud Build providing approver, time, decision, and comment on approval-gated builds.
- Cross-class finding: Zero of 47 platforms emit default content-addressed identity over the behavioral tuple, including all 27 agent platforms.The blind pass independently graded this column default on zero of 47 platforms, and CI/CD tuple-column agreement was 20 of 20.
- Agent and model-serving class: 16 of 27 agent platforms provide immutable nominal tuple versioning as a default answer, using version integers or named revisions behind mutable pointers.The versions can pin references rather than content-addressed bytes, so nominal immutability does not establish behavioral identity.
- Authority records: Agent approval records are absent on 14 of 27 platforms or gated behind administration features, unlike CI/CD systems’ more common first-class approval records.A complete deployment-event record can be unrecoverable for periods when the feature is disabled, making it opt-in under the protocol.
- Assurance conditions: Artifact continuity is determinable unevenly, with digests default on a quarter of each class, while no platform computes the temporal order of approval and evidence records.Temporal-order checking is possible where timestamps exist, but the platforms do not perform that computation themselves.
7. Results II: the measured gap, and why it is structural
The depth study finds a verifiability hole: many attestation adopters declare bindings on release surfaces that publish no artifacts, while measurable shortfalls occur at identity binding. The documentary survey explains the structural ceiling: default records do not emit content-addressed behavioral identities.
- Verifiability hole: 7 of 15 attestation adopters publish source-only releases, making their declared binding uncheckable on the release surface.Whether the binding holds on package registries is unknown because those surfaces were not examined.
- Identity binding: Both measured shortfalls fall at the identity-binding rung, while source-only releases are counted as undetermined rather than treated as failures.The corrected rows include eight undetermined cells in stratum A and two in stratum B under the uniform source-only rule.
- Measurement boundary: All realized grades are lower bounds because admin-only branch-protection records and ambiguous empty ruleset listings were excluded across all 30 repositories.These exclusions can only depress declared and realized depth as graded.
- Structural interpretation: Behavioral-depth assurance is therefore structurally out of reach in this population, while artifact-layer partial realization remains attributable to practice and surface fragmentation.The depth-study population has the needed artifact-layer records by default, so its measured floor is informative rather than record-limited.
- Structural ceiling: No platform among 47 consulted platforms documents default emission of a content-addressed identity for the behavioral tuple.Immutable nominal versions are the default on 16 of 27 agent platforms, but they do not provide content-layer identity.
8. Threats to validity
The study bounds its claims to pinned, public records and explicitly accounts for judgment, visibility, platform change, and population scope. These constraints make realized assurance a conservative lower bound rather than a complete view of repository practice.
- Grading and reproducibility: 17 of 20, 55 of 60, and 78 of 108 were the reported raw agreement figures across repeated documentation grades.The protocol required verbatim quotations for affirmative grades, named pages for absences, blind repetition, and recorded divergence resolution; no chance-corrected statistic was claimed.
- Documentation versus behavior: Documentation grades what vendors state, not behavior, so affirmative emission findings may be inflated while negative cells are strengthened.The scoped claim concerns what an operator can rely on the platform to have recorded.
- Population scope: The measured population covers default record classes across 27 agent platforms and realized release-pipeline assurance on the CI/CD substrate.The title’s broader agentic-software-engineering scope is operationalized through these two accountability surfaces.
- Temporal validity: Platform judgments are dated and pinned because one vendor rebranded during the survey and another retirement was scheduled inside the expiry window.The findings are framed to expire rather than silently becoming stale.
- Public visibility: Realized depth is a per-repository lower bound because unauthenticated observers could not read admin-only surfaces on all 30 repositories.Empty ruleset listings were graded conservatively, and a collector defect involving unencoded environment names was recorded rather than patched mid-study.
- Generalizability: The depth study uses one forge, one CI system, and 30 repositories in two frozen strata, with no generalization beyond that declared evidence class.The documentation claims likewise remain bounded to named platform populations.
- Authorship: One author adjudicated the protocol and reconciliations, while raw pass files and pinned pages preserve re-derivability without trusting the author.The AI-assisted grading process is disclosed and archived unmodified.
9. The registered test, deliberately not reported here
A prospective registered test was frozen and deposited before any in-window promotion, but it is deliberately excluded from this paper. The paper instead identifies separate instruments for testing its claims.
- Registration: The confirmatory arm freezes predictions, analyzer, and definitions before observing an in-window promotion.They were frozen on 2026-08-18 and deposited on 2026-08-19 as an immutable, timestamped OSF registration.
- Separation: The registered test is deliberately absent so a demonstration is not combined with its prospective test.This separation is presented as the failure mode registration is intended to prevent.
- Claim testing: This paper’s claims are tested through survey reruns, replication on other archived frames, and exhibition against the bounded claims.These instruments are distinct from the registered second-site intervention.
10. Conclusion
The paper measures whether public delivery records can express promotion claims and whether published evidence realizes declared assurance. It finds structural shortfalls in the records, while keeping the findings dated, pinned, and falsifiable.
- Conclusion: No content-addressed behavioral-tuple identity was found as a default record across 47 surveyed platforms.The survey covered 188 double-graded cells; nominal versioning was arriving in its place.
- Conclusion: Most attestation adopters declare binding on release surfaces carrying no artifacts to check, while most checkable adopters realize their declaration.Both measured adopter shortfalls occur at the binding rung.
- Conclusion: The records are structurally short of the level that can refuse a transition, and the shortfall is measurable from public material alone.The study dates and pins its claims and sets them to expire.
Data and artifact availability
The research repository preserves the study’s protocols, raw grading materials, reconciliation records, archives, definitions, and regeneration scripts, with public release required for publication.
- Artifacts: The repository carries the survey protocol, raw passes, reconciliations, append-only pin log, depth-study archive, analyzer, claim sets, definitions, and regeneration scripts.These materials support re-deriving the survey, depth study, figure, and bibliography.
- Release condition: The repository is private at the time of writing, and public release is a condition of publication.
AI usage disclosure
Anthropic Claude, including Claude Code, was used throughout the program for engineering, analysis, auditing, survey grading, review, and manuscript drafting under written protocols.
- Anthropic Claude, including Claude Code, supported instrument code, analysis scripts, adversarial-audit tooling, survey gradings, and blind repetitions.
- The models also supported successive draft reviews against the record and manuscript prose.
- The program applied written protocols, verified checkable claims before adoption, and archived dispositions in its repository.