PaperlayerView exact revision on arXiv

Source-linked AI summary

AgentStripes: An Evidence-Graded Maturity Model for AI Agents

Nutan Kumar Naik, Vijay Prasad Poudel, Aditya Kumar Saroj, Saurav Samantray, Abhishek Patel

arXiv:2610.18603v1cs.AIcs.CYcs.SYeess.SY

TL;DR

Task benchmarks measure what agents do, but rarely whether a released version has engineered controls and how strongly those controls are evidenced. AgentStripes specifies an evidence-graded maturity model and demonstrates it on public repositories. The demonstration does not validate the model, and re-judging changed some final stripe assignments.

  • Problem

    Task-suite evaluations rarely assess whether a released agent version has risk-reducing engineering mechanisms or how strongly those mechanisms are evidenced.

    Source: §1 · p. 1

  • Method

    AgentStripes rates 40 capabilities across 12 areas using separate capability ratings and evidence grades, then applies severity-weighted scoring and gates to assess one released agent version.

    Source: §3.1 · p. 8, §12 · p. 27

  • Results

    26 of 30 repositories ended at one stripe and 4 at two; re-judging changed final stripes for 4 of 22 repositories.

    Source: §9.1 · p. 19, §9.3 · p. 21

  • Takeaways & Limitations

    The evaluation demonstrates the model end to end, but the stripes remain a proposal to be tested rather than a validated or calibrated instrument.

    Source: §12 · p. 27

    The evaluation uses a purposive sample with static evidence and one judge family; stripes have not been shown to predict operational outcomes, and parameters remain uncalibrated.

    Source: §11 · p. 26

Abstract

from arXiv · show

Benchmarks report what an Artificial Intelligence (AI) agent can do on a task suite, but not whether one released version has been engineered with the controls that make it reliable, safe and observable, or how strongly that is shown. This paper proposes AgentStripes, an evidence-graded maturity model for AI agents. It defines levels (0 to 7 stripes), process areas (12 areas holding 40 capabilities), three gates (basic, safety, proof) that hold an agent at a level however high its score, and evidence grades E0 to E5. Each applicable capability receives a rating from 0 to 3 and a separate evidence level; these are combined into a credit, aggregated into a severity-weighted score and converted into stripes that the gates limit. Instruments gather evidence, language-model judges interpret it under explicit criteria, and the arithmetic is deterministic. The model is compared with standards at topic level without a claim of conformance, and an unexecuted validation protocol is specified. An empirical evaluation (exploratory, non-confirmatory) of 30 public Python agent repositories, with static evidence only, is a demonstration and not a validation: static evidence caps every final stripe at 2, 26 repositories end at one stripe and 4 at two, and this distinction is unreliable, since re-judging changed the final stripe of 4 of 22 repositories (engine, dossier and panel changed), whereas none of 8 panel-only repeats, all at stripe 1, changed. The parameters are expert conventions without outcome based calibration, and the stripes have not been shown to predict reliability or safety.

1 Introduction

AgentStripes addresses the gap between task-suite performance and evidence that a released agent version has been engineered with risk-reducing controls. It specifies an evidence-graded maturity model and reports an exploratory repository evaluation, not a validation.

1 Introduction

  • Task-suite evaluations rarely assess whether a released agent version has risk-reducing mechanisms or how strongly those mechanisms are evidenced.The paper frames this as a gap between measuring what agents do and assessing how they are engineered.Source: §1 · p. 1
  • AgentStripes defines 0–7 stripes across 12 process areas and 40 capabilities, with separate evidence grades and basic, safety, and proof gates.The gates can hold an agent at a level regardless of its score.Source: §1 · p. 1

1.1 Maturity Model Overview and Terminology

  • The paper specifies a capability inventory, rating and evidence separation, gated aggregation, assessment reporting, a judge threat model, and a topic-level standards crosswalk.It also specifies a validation protocol that has not yet been executed.Source: §1.1 · p. 2
  • The evaluation of 30 public repositories is a demonstration, not validation, and the paper makes no claim that stripes predict reliability or safety.AgentStripes is not claimed to be equivalent to CMMI or to establish compliance or certification.Source: §1.1 · p. 2

2 Background and Related Work

Prior agent research measures task performance, behavior under test, and system properties across specialized frameworks with differing units and evidence. AgentStripes positions itself as a complementary, release-specific engineering-maturity lens, while identifying gaps in the reviewed corpus.

2.1 From LLM Evaluation to Agent-System Evaluation

  • Agent evaluation has expanded from answer scoring to trajectories, environments, attacks, and system reliability, but benchmarks generally do not assess release engineering controls.The reviewed work distinguishes performance reliability under test from engineering mechanisms that control runaway execution, interruption, and recovery.Source: §2.1 · pp. 2–4

2.2 Holistic Evaluation, Maturity, Readiness and Governance

  • Holistic frameworks mostly assess agent systems through behavioral or runtime measures, while maturity, readiness, and governance artefacts differ in their units and functions.The reviewed corpus found no verified agent-level, evidence-graded scoring procedure; some standards were known only through catalogue records.Source: §2.2 · p. 5, Table 1 · p. 6

2.3 AgentStripes in Relation to Existing Approaches and the Methodological Gap

  • The model combines practices found in prior work rather than claiming uniqueness, and whether its combination predicts runtime behavior remains an open empirical question.The reviewed corpus also shows gaps in joining source-level controls with runtime performance and in grading repositories against maturity criteria.Source: §2.3 · p. 7
  • AgentStripes assesses one released agent version on a common capability and evidence scale, complementing rather than replacing outcome benchmarks.Its model separates capability rating from evidence strength and combines instrument and judge evidence with mechanically verified quotes, severity-weighted scoring, and gates.Source: §2.3 · pp. 6, 8

3 Specification of the AgentStripes Model

AgentStripes assesses the engineering maturity of a pinned AI-agent release by separating how well capabilities are implemented from how strongly they are evidenced. It combines instrument findings and language-model judgments into evidence-adjusted scores, then applies gates that can limit the resulting stripes.

3 Specification of the AgentStripes Model

  • AgentStripes assigns a release 0–7 stripes to express both how well it is built and how strongly that is shown.The assessment concerns engineering maturity, not underlying-model capability, usefulness for a purpose, legal compliance, certification, or security auditing.Source: §3 · p. 8, §3.1 · p. 8

3.1 Scope, Unit of Assessment and Capability Inventory

  • The inventory contains 40 capabilities across 12 areas, with fixed severity and applicability; nine emerging capabilities count only when they raise the score.Ratings range from 0 (absent) to 3 (strong), while nine capabilities can act as blockers.Source: §3.1 · p. 8

3.2 Rating and Evidence

  • Rating measures achievement from 0 to 3, while evidence level E0–E5 separately describes how the rating is known.Evidence credit is a fixed convention; code-reading ratings receive 0.90 in the described static setting, above the 0.85 credit for E3 run evidence.Source: §3.2 · pp. 8–9, Table 3 · p. 9

3.3 Assessment Procedure

  • Instruments extract findings and set rating bounds; three language-model judges rate within them, and accepted ratings are checked against quotes and instrument results.The median accepted rating is used, at least two accepted ratings are required, and disagreement is reported as settled or contested.Source: §3.3 · pp. 9–10

3.4 From Capability Assessment to Final Stripes

  • Capability credits combine rating and evidence strength into a severity-weighted score, converted to one score stripe per 12.5 points and then limited by applicable gates.Basic, safety, and proof blockers can cap stripes; proof above two requires safety behaviours to be shown at specified evidence levels.Source: §3.4 · pp. 10–11
  • Unassessed capabilities are excluded from the counted set and reported separately, with coverage and a conservative score treating them as zero also reported.Unassessed safety behaviours also limit the proof blocker.Source: §3.4 · pp. 10–11

3.5 Sandbox Testing, Reliability of the Assessment and Planned Study

  • The planned confirmatory study fixes design rules before assessment and specifies a repository sampling frame, pinned commits, repeated ratings, and reliability analyses.Static assessments cover all included agents; only a runnable subset is used for analyses requiring runs, with both populations reported.Source: §3.5 · pp. 12–13
  • The model’s credits and severity weights lack calibration, and higher stripes have not been shown to predict fewer production failures.The evaluation described elsewhere departs from the planned protocol; repositories are not deployed agents, and runnable-subset filtering selects for well-engineered agents.Source: §3.5 · p. 13

4 Methodological Analysis and Validity

AgentStripes separates capability maturity and evidence strength from operational performance, governance context, and assessment confidence. Its scoring parameters and harm-based blockers are provisional conventions, and the proposed validity hypotheses remain untested.

4.1 Reported Elements and Status of the Parameters

  • The model reports capability maturity and evidence strength separately from operational performance, governance context, and assessment confidence.Only capability maturity and evidence strength enter the score; the other elements are reported alongside it.Source: §4.1 · pp. 13–14
  • All listed scoring parameters are expert conventions without outcome-based calibration, so the values remain provisional.The conventions include severity weights, evidence credits, stripe width, blocker thresholds, proof requirements, and the counted capability set.Source: §4.1 · p. 14

4.2 Kinds of Validity, Hypotheses and Blocker Rules

  • Four validity types frame assessment of whether stripe counts reflect an agent release's engineering maturity, but none of the specified hypotheses is claimed to hold.The planned checks cover construct predictions, assessor agreement, ranking stability, blocker behavior, agreement with independent references, and prediction of operational outcomes.Source: §4.2 · p. 14
  • Blockers are expert-specified on a harm-based rationale and have not been validated; predicting harm requires independent reference judgements.The paper distinguishes checking whether blocker rules behave as intended from showing that they predict harm.Source: §4.2 · pp. 14–15

5 Reproducibility, Report Schema and Judge Robustness

Assessment reports preserve the artefact, method, judges, run, evidence dossier, results, confidence information, and overrides needed to interpret a result. The judge threat model has mitigations and fixture-based test protocols, but adversarial tests have not been run.

5 Reproducibility, Report Schema and Judge Robustness

  • Each report records the artefact, assessment method, judges, run conditions, dossier, results, confidence information, and any manual override.The dossier includes files read, accepted quotes with line numbers, instrument findings, and rejected judgements.Source: §5 · p. 15
  • Reproducing a stored result is distinct from reproducing model behavior, which may be impossible if a provider changes or retires a model behind a stable identifier.The report therefore stores the dossier and accepted quotes, not only scores.Source: §5 · p. 15
  • Adversarial tests have not been run, so the paper claims no robustness against threats such as repository prompt injection or forged quotes.The repositories are untrusted code, so tests must run only in an isolated environment.Source: §5 · p. 15

6 Standards and Framework Alignment

The paper presents a topic-level crosswalk to relevant standards and frameworks, not a claim of conformance. It finds overlap in security and runtime-governance topics while identifying capabilities and outcomes AgentStripes does not cover.

6 Standards and Framework Alignment

  • The crosswalk is a topic-level comparison and makes no claim that AgentStripes conforms to, implements, or is endorsed by any cited document.The sources examined include standards, voluntary frameworks, guidance, prototypes, specifications, and tools; some standards projects have no published requirements.Source: §6 · p. 15, §6.1 · p. 15

6.1 Sources and Crosswalk

  • No crosswalk row is directly mapped with supporting evidence because no conformity evidence exists.The table uses five mapping classes, and contains only the four classes other than directly mapped with supporting evidence.Source: §6.1 · p. 16

6.2 Implications of the Crosswalk

  • Overlap is greatest for security and runtime governance, but AgentStripes grades mechanisms and evidence while the compared sources describe risks or task-level outcomes.The paper also identifies gaps in identity lifecycle, inter-agent communication security, human factors, task performance and consistency, and context of use.Source: §6.2 · p. 16

7 Prospective Validation Protocol

The proposed confirmatory study separates assessment reliability from whether stripes predict operational outcomes. It specifies independent assessment, multiple outcome measures, and preregistered analyses, but has not been conducted.

7 Prospective Validation Protocol

  • The proposed study is not yet conducted and is designed to separate assessment reliability from whether stripes predict operational outcomes.It would sample pinned agent releases across frameworks, domains, autonomy profiles, and capacity for consequential actions.Source: §7 · p. 16
  • Independent assessors, including at least two human experts per agent, would compare ratings with a reference panel, deterministic instruments, and language-model judges from at least two providers.Human raters would not see judge ratings before rating; public repositories and consented private deployments would be analyzed separately.Source: §7 · p. 16
  • The protocol defines five outcome groups: assessor agreement, repeat-assessment stability, sensitivity to scoring choices, controlled fault and attack outcomes, and operational failure or incident rates where governed data exist.Outcomes involving controlled tests or operational data would be measured by parties that do not see the stripes.Source: §7 · p. 16
  • The analysis would preregister hypotheses and thresholds, treat missing evidence as missing, and evaluate calibrated credits and weights on a held-out sample.Reliability analyses can proceed without outcome data, whereas usefulness analyses require outcomes; results from the first group do not establish the second.Source: §7 · pp. 16–17

8 Empirical Evaluation of the Model: Design and Provenance

The exploratory evaluation applies AgentStripes to 30 purposively selected public Python agent repositories using static evidence. It demonstrates the method’s outputs, not a confirmatory validation or a test of whether stripes predict reliability or safety.

8 Empirical Evaluation of the Model: Design and Provenance

  • The evaluation demonstrates what the method produces on real repositories; it tests none of the confirmatory hypotheses and makes no prediction claim about reliability or safety.The sample was purposive, with no human raters or outcome data.Source: §8 · p. 17
  • The engine was revised three times after results from the same sample had been seen, and the reported sensitivities were also specified after results were known.Only final-engine results are reported; the authors characterize the evaluation as development on a test sample, not a preregistered study.Source: §8 · p. 17
  • The 30-repository sample required public Python projects with at least 500 stars, a recent push, an eligible licence, and a verified agent that uses tools in a loop.The authors describe it as a popularity-filtered convenience sample that supports no inference about agents in general.Source: §8 · p. 17
  • Three judges rated applicable capabilities from repository dossiers, while unassessed capabilities were excluded from the score and no runtime evidence or independent review entered the scores.The engine parsed pinned commits and produced ranked dossiers; because agents were not run, the highest reachable evidence level was E2.Source: §8 · pp. 17–18
  • The engine credited all accepted code-reading ratings at 0.90 rather than the model table’s 0.70, making absolute scores conditional on that broader static-assessment rule.The rule gave static code evidence more credit than developer-run evidence at E3.Source: §8 · p. 18

9 Empirical Evaluation: Findings

In this exploratory evaluation, gates compressed pre-gate scores into one or two final stripes, and re-judging showed that the one-versus-two distinction was unreliable. Runtime probes demonstrated an evidence path, but did not validate the model or establish repository-level runtime capabilities.

9.1 Sample, Coverage and Distribution of Stripes

  • 26 repositories ended at one stripe and 4 at two, although their pre-gate scores mapped to three, four, or five score stripes.The paper states that the one-versus-two distinction is unreliable and does not rank repositories.Source: §9 · p. 18, Table 4 · p. 19, §9.1 · p. 19
  • For repositories with an accepted rating, mean coverage was 81.1% (median 85.0%, range 35.0%–97.5%); scores counted 12–36 of 40 capabilities per repository.Source: §9.1 · p. 19

9.2 The Two-Stripe Ceiling and Sensitivity Analyses

  • Static evidence capped every repository at two final stripes because repository-only evidence reached E2, below the proof gate’s E3 threshold.The proof gate held all 30 repositories at two or fewer stripes; the safety gate also held 23 at one.Source: §9.2 · pp. 19–20, Figure 5 · p. 21, Figure 6 · p. 22
  • If unassessed safety capabilities were rated zero, 3 of the 4 two-stripe results would fall to one; changing E2 credit to 0.70 changed no final stripe.The latter sensitivity changed 28 score stripes, despite leaving final stripes unchanged.Source: §9.2 · pp. 20–21

9.3 Stability on Re-judging, Judge Agreement and Quote Verification

  • Across 22 engine-and-panel comparisons, 4 final stripes changed; in 8 panel-only repeats, none changed, but all eight repositories were already at one stripe.In the 22 comparisons, engine, dossier, and panel could not be separated; quote checks verified text presence, not whether it supported ratings.Source: §9.3 · pp. 21–22, Figure 7 · p. 23

9.4 Runtime-Evidence Probe Runs and Cost

  • The scripted runtime variant produced no final-stripe or proof-limit changes; its E3 evidence came from study-authored probes, not developers’ runs.The probe evidence could not reach E4 or lift the proof limit.Source: §9.4 · pp. 22–23
  • The shim variant attempted all 30 repositories in 75 runs; 20 yielded spans, no final stripe changed, and one proof limit changed through study-authored scaffolding.Span coverage depended on tracing-library instrumentation, not repository properties.Source: §9.4 · p. 23
  • Shim runs used a non-deterministic responder and study stubs, so real-tool behavior was not observed and guardrail-dependent observations could describe the probe scaffolding.No repository was assigned an E3 capability claim, and no E4 evidence was obtained.Source: §9.4 · pp. 23–24

10 Discussion of the Empirical Evaluation

The discussion treats the static-evidence evaluation as a method demonstration, not a measure of repository maturity: proof requirements cap the results, and the remaining one-versus-two-stripe distinction is unreliable. It identifies effects from evidence credits, dossiers, instruments, judges, and sampling, then specifies changes needed for a stronger study.

10.1 What the Scores Do and Do Not Say, and the 1-versus-2 Distinction

  • 26 of 30 repositories ended at one stripe and 4 at two, but static evidence caps final stripes at 2, so this distribution does not establish agent maturity.Score stripes were three to five, while stripes surviving proof requirements were one or two; interpreting the one-stripe majority as immaturity reverses the intended reading.Source: §10.1 · p. 24
  • 4 of 22 re-judged final stripes changed, while none of 8 panel-only repeats changed; all eight repeats were at one stripe and could not test the one-versus-two distinction.The remaining distinction depends on a safety flag formed by an OR-rule over five capabilities, with single ratings and unassessed capabilities contributing to its instability.Source: §10.1 · p. 24

10.2 The Evidence-Credit Ladder, Instrument and Dossier Effects

  • Static code evidence earns 0.90 credit versus 0.85 for developer-run evidence, so moving a rating from E2 to E3 can lower credit and score.In the shim variant, the median score change was −0.173 points (range −1.01 to +0.30); the authors flag the broader code-reading rule and its incentive as issues for calibration.Source: §10.2 · p. 25
  • Judges assess ranked excerpts rather than repositories, and 18.8% of capability cells were unassessed; omitted guards and uneven coverage can therefore affect ratings and scores.Quotes are checked for existence, not relevance, so the discussion recommends relevance checks before using ratings to decide a blocker.Source: §10.2 · p. 25

10.3 Runtime Evidence and Threats to Validity of the Empirical Evaluation

  • Probe runs showed that the trace-to-evidence path works, but instrumentation missed events in 10 of 30 repositories and no ratings or stripes changed.The probes used assessor-written scenarios and tools, so testing whether runtime evidence changes ratings requires richer instrumentation or developers’ own run traces.Source: §10.3 · p. 25

10.4 Implications and Next Steps

  • The evaluation’s purposive Python-repository sample, single judge family, tuned engine, and lack of outcome measurements support no inference to agents generally or to reliability and safety.The authors call for independent human raters and multiple judge providers, runtime evidence, treatment of unassessed capabilities, and a defined sampling frame.Source: §10.3 · pp. 25–26, §10.4 · p. 26

11 Limitations and Roadmap

The paper presents AgentStripes as a benchmark complement, while emphasizing that its empirical demonstration does not establish validity. It identifies specification and validation gaps and lays out a roadmap for addressing them.

Limitations and Roadmap

  • AgentStripes complements task benchmarks and maps topic overlap with standards without establishing conformance.The paper proposes a prospective validation protocol but has not shown that stripes predict operational outcomes.Source: §11 · p. 26
  • The 30-repository evaluation used purposive sampling, static evidence, and one judge family; parameters remain uncalibrated, blockers unvalidated against harm, and independent-assessor agreement unmeasured.A study to fill these gaps is specified, not conducted.Source: §11 · p. 26
  • Before a large-scale study, the paper calls for closing specification gaps, publishing capability-specific anchors, independent sampling and human reference ratings, adversarial tests, and E3–E5 evidence.The roadmap also calls for the prospective study and standards-crosswalk updates; until that study is complete, claims that more stripes mean fewer failures are unsupported.Source: §11 · pp. 26–27

12 Conclusion

AgentStripes is an evidence-graded maturity model for one released AI-agent version, with explicit aggregation and gates. Its evaluation demonstrates the specification but does not validate the stripes; the paper calls for a confirmatory study with outcome measurements.

12 Conclusion

  • AgentStripes rates one released agent version across 40 capabilities in 12 process areas, converting ratings and evidence through severity-weighted scoring and gates into 0 to 7 stripes.The five-stage calculation is deterministic after assessment inputs are accepted.Source: §12 · p. 27
  • The model separates capability maturity and evidence strength from operational performance, governance context, and assessment confidence.Its standards comparison identifies topic overlap and gaps without claiming conformance.Source: §12 · p. 27
  • The 30-repository static-evidence evaluation ran the specification end to end, but static evidence capped final stripes at 2 and re-judging changed 4 of 22 final stripes.None of the 8 panel-only repeats changed, but all were at stripe 1 and could not show changes between one and two stripes.Source: §12 · p. 27
  • The parameters are conventions, independent-assessor agreement is unmeasured, and the stripes have not been validated against operational outcomes.The paper proposes a confirmatory study with independent human raters and outcome measurements.Source: §12 · p. 27

S1 Model, Related-Work, Design and Protocol Material

The supplement gathers material supporting the paper’s Sections 2 to 7, including model design, related work, standards comparison, validation design, and reporting materials.

S1 Model, Related-Work, Design and Protocol Material

  • Supplementary materials cover the evaluation corpus and prior-work comparisons, model elements and parameters, rating and evidence axes, blockers, standards crosswalk, planned validation, report schema, judge threat model, and limitations.Source: S1 Model, Related-Work, Design and… · p. 33

S2 Additional Results of the Empirical Evaluation

The supplement collects aggregate empirical-evaluation figures, none of which ranks repositories.

S2 Additional Results of the Empirical Evaluation

S3 Empirical Evaluation in Detail

The supplement documents the empirical evaluation’s artefacts, aggregate results, shim-study limits, costs, and a worked scoring example. Its named repository scores are panel-specific, and the worked example is synthetic rather than a repository finding.

S3 Empirical Evaluation in Detail

  • The supplement provides pinned-commit repository results, engine hashes, capability counts, and a worked calculation; artefact paths are relative to the study package.Source: S3 Empirical Evaluation in Detail · p. 33

S3.1 Per-Repository Results and Reproduction Table

  • Table S18 lists 30 repository scores alphabetically, not as a ranking; scores are specific to this panel, dossier, and engine, and single-repository stripes are unstable.A second panel changed 4 of 22 final stripes, while a panel-only repeat changed none of the other 8.Source: S3.1 Per-Repository Results and… · p. 33
  • The assessment used static evidence and three Claude-subagent judges per repository, but unpinned models and seeds prevent exact reproduction of judge answers.Source: S3.1 Per-Repository Results and… · p. 33

S3.2 Aggregate Counts per Capability and Additional Figures

S3.3 Workflow 2 Shim Study: Artefacts and Limits

S3.4 Cost and Token Usage

  • Estimated pricing for the 30 current results ranged from $1.08 to $13.27 under alternative OpenAI model prices, while shim token totals are a lower bound.The paper says these are estimates under assumptions, not measured assessment-service costs.Source: S3.4 Cost and Token Usage · pp. 41, 43

S3.5 Worked Calculation

  • The synthetic example scores 63.333, or five score stripes, but its E2 safety evidence binds the proof cap at two final stripes.The invented ratings are not a finding about any repository.Source: S3.5 Worked Calculation · pp. 43, 45

Related papers

Loading 2610.18603v1…