PaperlayerView exact revision on arXiv

Source-linked AI summary

Executable Code Knowledge: Code as a Native, Validation-Carrying Knowledge Representation for AI Coding Agents

Xueping Gao

arXiv:2608.16295v1cs.CL

TL;DR

AI coding agents need repository knowledge that includes semantics, evidence, relations, and freshness, not only relevant snippets. This paper introduces source-bound, executable knowledge units and finds that explicit evidence improves precise test-selector recovery, while ECK can project knowledge into rules without losing provenance.

  • Problem

    AI coding agents need software knowledge about business rules, authoritative tests, affected relations, and context freshness beyond similar-looking code snippets.

  • Method

    The paper introduces ECKUs, source-bound code units that carry semantics, executable behavior, contracts, evidence, relations, provenance, validation state, and query operations.

  • Results

    Explicit evidence raises exact task success from 1/11 to 9/11, while AST-bounded freshness classification achieves sensitivity and specificity of 1.000 across controlled cases.

  • Takeaways & Limitations

    ECK can generate rules for agent consumption while preserving executable provenance, source binding, validation state, impact, and freshness for selected units.

  • Takeaways & Limitations

    Freshness perturbations are synthetic, and the evaluation covers agent preparation and patch review rather than full patch-generation success.

Abstract

from arXiv · show

AI coding agents need more than relevant snippets: they need business semantics, validation evidence, relations, and assurance that their context is current. Existing systems usually infer or externalize this knowledge through retrieval, summaries, graphs, rules, or reverse specifications. We investigate a complementary representation in which selected code units directly carry agent-usable knowledge. We introduce Executable Code Knowledge (ECK) and define an Executable Code Knowledge Unit (ECKU) as a source-bound object combining stable identity, semantics, executable behavior, contracts, evidence, relations, provenance, validation state, and a query interface. Our Python prototype supports code-local authoring, manifest export, evidence execution, exact changed-line impact, freshness checking, and agent-facing projections. Across three real Python repositories and 26 controlled patch tasks, direct ECK provides executable test coverage for 11/11 evidence-bearing tasks and exact selectors for 9/11; hiding declared evidence reduces exact recovery to 1/11 (paired exact McNemar p=0.0078). ECK-derived rules recover 11/11 exact selectors, showing that rules are effective delivery artifacts while ECK supplies source binding, validation state, impact, and freshness. Exact changed-line impact matches independently authored labels on all 26 patches (12 unit links; precision, recall, and F1 all 1.000). AST-bounded fingerprints classify 50 positive changes and 17 unrelated same-file controls correctly, whereas static rules snapshots detect none of the 50 stale cases. Model-backed patch-review and cross-layer studies measure projection fidelity rather than independent impact discovery. These results support a hybrid architecture: retrieval for coverage, ECK for source and evidence governance, and projections for delivery.

1 Introduction

AI coding agents need repository knowledge that includes semantics, authoritative evidence, relations, and freshness—not merely relevant snippets. ECK addresses this gap by making selected high-value code units source-bound, executable knowledge objects with validation and queryable context.

  • Motivation: Agents need business semantics, authoritative tests, affected relations, and fresh context in addition to snippets that resemble a request.Repository-level evaluation has exposed this representation gap.
  • Motivation: Retrieved chunks suggest locations, while static graphs, summaries, and reverse specifications generally lack declared authority, validation, traceability, or freshness.The extractive pipeline remains useful for coverage but has an authority problem.
  • Representation: ECK makes selected implementation units directly carry domain semantics, executable behavior, contracts, evidence, relations, provenance, validation state, and query operations.The representation targets high-value units such as business rules, security checks, transformations, API behavior, and workflow transitions.
  • Representation: Broad extractive methods provide coverage, whereas direct executable code knowledge provides source-bound identity, declared executable support, and freshness for selected high-value units.ECK is complementary to retrieval rather than a replacement for whole-repository retrieval.
  • Contributions: The paper contributes ECKU framing, a Python prototype, and mechanism-level evaluation covering evidence recovery, changed-line impact, report consumption, projection fidelity, and freshness sensitivity and specificity.The prototype includes decorators, manifest generation, evidence validation, AST-bounded freshness, candidate migration, context generation, and patch-evidence reports.

2 Motivation

The motivation for ECK is that code alone leaves agents guessing about business meaning, validity conditions, relationships, tests, and freshness. ECKUs address this through source-bound, code-authored assertions with executable evidence and checkable freshness rather than unverified summaries.

  • Motivation: Code inspection leaves agents guessing whether a function is a business rule, what preconditions validate it, which systems it relates to, which tests to run, and whether documentation is fresh.These uncertainties include endpoints, data fields, and the validity of previously generated documentation or summaries.
  • ECKU structure: An ECKU directly records stable business semantics, relationships to APIs, data, and policies, executable evidence, and formal preconditions and postconditions.The pricing example specifies VIP large-order discount semantics, related endpoint and fields, a pytest evidence command, and conditions requiring VIP status and orders above 100.
  • Source binding and validation: ECK is a source-bound, code-authored assertion whose declared evidence can be executed and whose freshness can be checked.It is not a summary generated from code, and annotation alone does not establish semantic truth.

3 Related Work

Related work represents coding knowledge through repository retrieval, external graphs, persistent rules, reverse specifications, and executable protocols. ECK is complementary: selected implementation units authoritatively carry source-bound semantics, evidence, provenance, validation state, freshness, and impact, then project that knowledge for agents.

  • Repository retrieval: Repository-level benchmarks frame agentic coding as requirement-driven search over repositories rather than isolated snippet retrieval.ContextBench evaluates retrieval across 1,136 issue-resolution tasks from 66 repositories using recall, precision, and efficiency; CORE-Bench similarly emphasizes repository search.
  • External representations: ECK differs from code graphs and search indexes by authoring source-bound knowledge with code, from which manifests, relations, rules, and context packs are projected.External systems extract representations from ordinary code, whereas ECK places knowledge identity and validation provenance in the code-authored unit.
  • Rules and memories: Persistent rules and memories are effective delivery artifacts, but ECK adds source spans, contracts, evidence fingerprints, validation state, freshness, and patch-overlap queries.Fresh rules can match ECK on validation-command recovery, while ECK treats them as projections whose freshness and declared-unit overlap remain queryable.
  • Reverse specifications: ECK contrasts with reverse engineering, which reconstructs specifications from existing systems, by having selected implementation units directly carry operational knowledge and freshness state.The approaches can combine: reverse engineering may propose candidate ECKUs, while human-reviewed ECKUs become authoritative code-native knowledge.
  • Scope and authority: ECK is narrower than general agent harnesses, protocol-sovereign development, and broad structural knowledge graphs, focusing on selected Python units, declared semantics, executable evidence, and freshness.Unlike PDD, ECK does not define the full admissible implementation space or treat passing evidence as proof of semantic truth.
  • Foundations: ECK extends contracts, documentation-as-code, and change-impact traditions into structured, queryable, freshness-aware agent knowledge with direct mapping from changed lines to declared identities.Contracts are one ECKU field, while prose documentation typically lacks a typed agent interface, patch-impact mapping, and persisted validation fingerprints.

4 Executable Code Knowledge

Executable Code Knowledge Units bind source code to semantics, executable behavior, contracts, evidence, relations, provenance, validation state, and query interfaces. Their conservative validity model requires fresh state and passing declared executable evidence, while the prototype exposes construction, querying, validation, freshness, impact, and context operations.

  • ECKU definition: An ECKU combines stable identity, domain semantics, executable behavior, contracts, evidence, relations, provenance, validation state, and a query/composition interface.Relations may connect APIs, data entities, tests, documents, or other units.
  • Validation model: Valid(u) requires Fresh(u) and Supported(u), with Supported(u) requiring non-empty executable evidence whose every declared command passes.This all-evidence rule records passing declared support without claiming semantic truth.
  • Validation model: ECK separates relevance from validity: retrieved code may be relevant, while an ECKU additionally carries declared executable support and freshness state.The prototype uses conservative all-evidence aggregation, with claim-level required or supporting evidence identified as a natural extension.
  • Runtime interface: The prototype implements build, query, validation, freshness checking, direct patch impact, context generation, and patch evidence reporting; relation-based composition remains future work.The runtime interface also includes show(unit_id), freshness(unit_id), impact(patch), and context(task).

5 System Design and Implementation

The Python prototype implements ECK through five layers, with decorator-based authoring, repository-wide discovery and export, executable evidence validation, freshness checking, and typed context projection. Its design preserves source provenance and records field-specific validation state so agents receive directly actionable knowledge.

  • Authoring: Developers author ECKUs with @knowledge_unit(...), @contract(...), and @evidence(...) decorators.The prototype preserves the original executable function as the source span, even when contract wrappers are applied.
  • Build and export: The builder discovers registered ECKUs, records source spans and hashes, and exports knowledge, evidence, and validation manifests.The exported files are .eck/knowledge_units.jsonl, .eck/evidence.jsonl, and .eck/validation_state.json.
  • Validation: Validation executes evidence commands and records status, source, contract, and evidence hashes, stale reasons, and a validation timestamp.These records provide persisted evidence and validation state for each knowledge unit.
  • Freshness: Freshness checking compares executable-body, agent-facing knowledge, contract, and evidence fingerprints with persisted validation state.AST-derived function boundaries and decorator-excluding source fingerprints preserve field-specific stale reasons, while KnowledgeHash covers identity, semantics, relations, and provenance.
  • Agent-facing context: The context command emits compact typed packs containing intent, symbol, source span, contracts, executable evidence commands, and relations.Unlike generated summaries, the pack directly identifies which command validates a knowledge claim.

6 Evaluation

The evaluation investigates four questions covering agent validation planning, patch impact and validation coverage, cross-layer projection, and freshness. It uses controlled change requests across real Python repositories and compares multiple context representations with explicit safeguards around metric interpretation.

  • Evaluation questions: Four research questions assess validation planning, patch impact and coverage, cross-layer projection fidelity, and freshness detection.The questions examine whether ECK supports executable validation planning, identifies impacted knowledge units, transports fields faithfully, and detects stale knowledge.
  • Evaluation setup: 3 real Python repositories provide local tests and manually authored ECKUs covering high-value parsing, normalization, CLI, and safety functions.Examples include dotenv resolution, slug normalization and truncation, CLI behavior, TOML parsing, and parse-float safety.
  • Evaluation setup: 26 issue-style change requests compare repository-only, BM25, EKP Retrieval, Rules Context, Direct ECK without Evidence, and Direct ECK conditions.Models produce target files, symbols, validation commands, and rationale, with gold labels derived from held-out patch effects and direct ECK evidence commands.
  • Interpretation safeguards: Patch-review and cross-layer ECK-ID or field-recall results measure projection fidelity rather than independent impact discovery.The patch-review workflow shares the deterministic impact generator with its ECK targets and report, so it evaluates report consumption and projection fidelity.
  • Metrics: The evaluation reports file recall, exact-selector recall, executable-coverage recall, impacted-ECK-ID recall, cross-layer recall, and strict or lenient output rates.Exact selectors require normalized command and test-selector equality, while executable coverage uses test-file and class/test containment rather than arbitrary substring matching.

7 Results

Results show that executable evidence is the main advantage of Direct ECK for precise validation planning, while rules can deliver equivalent hints without ECK’s operational guarantees. ECK also provides exact impact and freshness behavior in controlled studies, though projection results measure fidelity rather than independent discovery.

  • Validation planning: Rules Context exactly recovers 11/11 selector sets, Direct ECK recovers 9/11, BM25 5/11, and Direct ECK without evidence 1/11.Direct ECK supplies commands covering gold tests for 11/11 tasks.
  • Validation planning: Exact task success falls from 9/11 to 1/11 without evidence, with p=0.0078125 and coverage recall dropping from 1.000 to 0.636.The ablation has identical file recall, isolating explicit evidence as the primary validation-precision mechanism.
  • Validation planning: Test-file BM25 reaches executable coverage 0.636 but exact selector recall 0.000, while test-case BM25 reaches 0.091 on both metrics.Lexical retrieval can identify broad test files but rarely recovers precise executable selectors.
  • Freshness and delivery: Across 50 positive perturbations, ECK detects 50/50 stale cases while an unchanged rules snapshot detects 0/50; 17 unrelated same-file controls remain fresh.Rules are effective agent-delivery artifacts, but lack ECK’s source binding and freshness mechanism.
  • Freshness and delivery: Across 50 positive and 17 negative cases, sensitivity is 1.000, specificity is 1.000, and false-positive rate is 0.000 for AST-bounded freshness classification.The result is a controlled mechanism test; AST-bounded spans and KnowledgeHash address complementary source and knowledge blind spots.

8 Discussion

The discussion positions ECK as a selective, source-bound evidence layer within a hybrid architecture: extraction supports broad discovery, while direct ECK provides executable validation, freshness, and patch-review provenance. Rules remain useful delivery projections, but authoritative knowledge requires reviewed source binding and current executable support.

  • Hybrid architecture: A hybrid architecture combines extractive methods for broad discovery with direct ECK for high-value units requiring contracts, validation commands, and freshness checks.Extraction may propose candidate ECKUs, but human review and executable evidence are required before they become authoritative.
  • Auditability: ECK makes patch review object-centric by linking a directly overlapping unit to its declared evidence and freshness state, rather than relying only on text.This source binding is the technical basis for representing selected knowledge in ECK instead of only externalizing it into rules, memories, summaries, or graphs.
  • Rules and projections: Fresh rules can deliver validation commands effectively, while direct ECK makes those commands source-bound, patch-impactable, and freshness-checkable.ECK is therefore a source layer that projects into agent-facing rules rather than replacing every rule or memory system.
  • Source binding: Generated ECK-like identifiers recover identity as text, but do not provide the source span, evidence fingerprint, validation state, or stale-context detection of a source-bound object.The distinction is whether the identifier is backed by current executable support, not merely whether an identifier string is shown to the agent.
  • Selective adoption: ECK should annotate selective high-value units where agents repeatedly need business semantics, exact validation evidence, or patch-review provenance, rather than every function.The prototype combines extraction and lexical search for candidate functions, LLM-drafted metadata, developer approval or editing, and CI validation.

9 Threats to Validity

The evaluation demonstrates feasibility but has limited generality because tasks, repositories, patches, labels, models, and freshness cases are constrained or synthetic. Perfect projection results measure fidelity rather than independent impact discovery, while end-to-end repair and responsible deployment remain open concerns.

  • Task construction: Issue-style requests were written independently from patch filenames but still constructed from known patches, limiting task realism.Stronger evaluation would use naturally occurring issue reports or human-written tasks created without inspecting the patch.
  • Scale and coverage: Three small Python libraries and 26 controlled patches demonstrate feasibility and validation-planning signal, but not broad generality.Direct ECKUs cover selected high-value functions rather than entire repositories, so lower patch coverage than extractive EKP-RAG is expected.
  • Manual impact labels: One annotator independently authored direct-impact labels on controlled patches, leaving annotator agreement, natural commits, and direct-versus-expanded labels for future work.The gold was derived from patch hunks and ECKU declarations rather than generated by the impact implementation.
  • Model dependence and freshness: Two Qwen-family models support the qualitative validation-planning result, while broader model coverage and natural ECKU maintenance would strengthen the model and freshness claims.The freshness study uses synthetic perturbations across 50 positive and 17 same-file negative cases, testing fingerprint sensitivity and specificity rather than developer maintenance behavior.
  • Patch-review and projection construction: Perfect patch-review and cross-layer values measure projection fidelity because targets and prompts share deterministic impact or ECKU fields, not independent impact discovery or end-to-end audit improvement.The patch-review experiment evaluates report consumption from a supplied diff rather than patch generation.
  • End-to-end repair and responsible use: The study evaluates agent preparation and patch-review quality rather than full patch-generation success, and deployments should expose freshness, fail closed for high-risk units, and separate validated evidence from candidate metadata.Incorrectly authored or unvalidated ECKUs can mislead agents; improved SWE-bench-style resolution requires additional experiments.

10 Conclusion

The paper argues that selected code units can directly carry agent-usable knowledge through source-bound, evidence-carrying, freshness-checkable ECKUs. A Python prototype supports authoring, querying, validation, direct patch-impact checking, and projection into agent context.

  • Core contribution: ECKUs let selected code units directly express agent-usable knowledge instead of relying only on external representations.The paper presents this as the central argument for Executable Code Knowledge.
  • Core contribution: ECKUs combine source binding, evidence carrying, and freshness checking in code knowledge objects.These properties define the introduced ECKU representation.
  • Prototype: The Python prototype supports authoring, querying, validating, direct patch-impact checking, and projecting ECKUs into agent context.The prototype operationalizes the representation across these capabilities.
  • Evidence ablation: The paired evidence ablation indicates that explicitly declared commands primarily improve precise test-selector recovery.The supplied passage states the ablation’s direction of effect but does not provide its quantitative result.

11 Data Availability Statement

A commit-pinned anonymous repository provides the ECK implementation and reproducibility materials, while minimal reproduction is CPU-only and model-backed reruns require the reported model environment. The work uses no personal or human-subject data.

  • Artifact contents: The repository contains the ECK implementation, examples, tests, experiment scripts, patch-report reproduction, gold labels, and saved raw model outputs.These materials correspond to the work described in Section 8.
  • Reproduction requirements: The minimal tool reproduction runs on CPU-only infrastructure.
  • Reproduction requirements: Rerunning model-backed experiments requires the model environment reported in Section 8.
  • Data scope: No personal or human-subject data are used.
Loading 2608.16295v1…