PaperlayerView exact revision on arXiv

Source-linked AI summary

Spec2Vision: Contract-Guided Delivery of AI-Generated Computer Vision Pipelines

Ghfran Jabour, Sergey Ivanov

arXiv:2608.26400v1cs.SEcs.PL

TL;DR

Generated computer-vision code can run without satisfying downstream task contracts, motivating a bounded study of delivery rather than runnability alone. Spec2Vision carries structured specifications through staged synthesis, screening, testing, and repair, and outperforms weaker variants and single-agent baselines on evaluator-facing delivery within the benchmark’s represented task-contract space.

  • Problem

    Generated CV code can be runnable while failing downstream evaluator requirements for data layout, outputs, metrics, and interfaces.

  • Method

    Spec2Vision produces specification-grounded CV bundles by carrying an explicit task contract through staged generation, screening, testing, and bounded repair.

  • Results

    Spec2Vision substantially outperforms executable single-agent baselines under the benchmark’s downstream evaluator, reaching 81/85 evaluator-test passes in the primary evaluation.

  • Takeaways & Limitations

    Contract-carrying staged generation, deterministic checks, testing, and bounded repair yield more evaluator-tested deliveries than runnable code generation alone.

  • Takeaways & Limitations

    The evidence is limited to 17 fixed CV tasks sharing one specification language and one evaluator-facing contract, representing the benchmark’s task-contract space.

Abstract

from arXiv · show

Generated computer-vision code can be runnable without satisfying the task contract enforced by a downstream evaluator. We study that gap with Spec2Vision, an experimental framework for producing and evaluating specification-grounded CV pipeline bundles through a staged runtime that keeps the task contract explicit across synthesis, screening, testing, and bounded repair. The benchmark evaluates 17 CV tasks, 10 executable conditions, and 5 repeats per task-condition cell, for 850 primary runs. In the primary 850-run evaluation, Spec2Vision reaches 81/85 evaluator-test passes; removing structural repair drops to 55/85, compatibility scaffolding to 58/85, and generator preflight to 39/85. The executable single-agent baselines expose progressively richer task specifications to the model, culminating in direct source-spec exposure, yet remain much weaker overall, from 17/85 for lightweight task grounding to 35/85 evaluator-test passes. The lightweight baseline nevertheless remains core-runnable in 85/85 runs but reaches only 17/85 evaluator-test passes and 6/85 strict-delivery successes, showing that runnability is not equivalent to delivery. Across this benchmark, the strongest evidence comes from keeping the task contract explicit across staged generation, checking, and repair. Artifacts are provided to support audit of run bundles, model-visible inputs, and derived tables.

1. Introduction

The paper studies why runnable generated computer-vision code can still fail downstream delivery, asking whether explicit task contracts carried through staged generation, testing, and repair improve outcomes. Spec2Vision addresses this gap with a contract-grounded framework and benchmark evidence that emphasizes delivery rather than runnability alone.

  • Generated CV bundles must satisfy evaluator requirements for data layout, outputs, metrics, and interfaces beyond merely surviving a dry run.
  • The study compares contract-carrying staged generation with weaker variants and executable single-agent baselines within a fixed benchmark.
  • Spec2Vision keeps a structured task specification operational across context assembly, agent handoff, preflight checks, testing, and bounded repair.
  • 81/85 evaluator-test passes were achieved by Spec2Vision, while removing compatibility scaffolding, structural repair, and generator preflight reduced results to 58/85, 55/85, and 39/85.
  • Single-agent baselines reached only 17/85 to 35/85 evaluator-test passes, despite progressively richer task-specification exposure.
  • The lightweight baseline was core-runnable in 85/85 runs but achieved only 6/85 strict-delivery successes, demonstrating that runnability is insufficient for downstream delivery.

2. Related Work

Related work commonly evaluates executable correctness, repository-level issue resolution, iterative refinement, or multi-agent organization. Spec2Vision instead focuses on delivery-state evaluation for generated CV bundles under explicit task contracts.

  • Self-contained code-generation benchmarks mainly test whether short programs satisfy hidden tests for bounded programming tasks.
  • Spec2Vision evaluates complete CV pipeline bundles against downstream requirements for files, outputs, metrics, and interfaces.
  • Repository-level coding-agent benchmarks study issue resolution, tool use, code editing, execution feedback, and sandboxed environments.
  • Iterative-refinement and multi-agent systems provide precedents for feedback loops, reasoning with interaction, role separation, and software generation.
  • ML systems, testing, reproducibility, and requirements-engineering work emphasizes failures involving interfaces, data assumptions, process gaps, traceability, and testable requirements.
  • Spec2Vision contributes a delivery-state benchmark and contract-carrying CV pipeline that separates specification exposure, role separation, screening, repair, and task-contract satisfaction.

3. Spec2Vision and the Benchmark

Spec2Vision transforms structured CV specifications into audited runnable bundles through staged planning, generation, screening, testing, and repair. Its benchmark varies contract-diverse tasks and executable conditions while judging delivered artifacts through condition-agnostic evaluator-facing checks.

  • Runtime: The staged pipeline uses an architect, generator, deterministic preflight, validator/tester, and later feedback and verification stages.
  • Runtime: Figure 1 represents the specification-to-context path, persisted contract artifacts, local preflight and repair control, and stored run-bundle evidence.
  • Runtime: The architect-to-generator handoff persists an architecture plan, module graph, interface contract, and compiled generator contract.
  • Benchmark: The benchmark covers 17 CV tasks spanning detection, segmentation, classification, tracking, pose, text, depth, flow, lane detection, stereo, LiDAR, and scene parsing.
  • Benchmark: Expected outputs, metrics, dataset layouts, and directory assumptions vary across the contract-diverse task set.
  • Benchmark: Each condition runs five repeats on every task, producing 85 runs per condition and 850 primary evaluation runs under a shared grid, artifact store, and evaluator.
  • Evaluation: The outcomes distinguish artifact completeness, core runnability, evaluator-test passes, and strict delivery, measuring contract satisfaction rather than scientific CV quality.
  • Evaluation: The evaluator checks delivered files, configuration, runtime behavior, outputs, metrics, and shallow-contract signals without inspecting Spec2Vision-specific internal artifacts.

4. Results

The evaluation compares Spec2Vision with executable single-agent baselines and mechanism-removal variants, separating evaluator-test delivery from runnability and strict-delivery failures. The strongest evidence favors contract-carrying staged generation, deterministic screening, compatibility scaffolding, and structural repair.

  • RQ1: Baseline comparison: 17/85 to 35/85 evaluator-test passes were reached by executable single-agent baselines as task-specification exposure increased.The comparison uses progressively richer specification exposure, including direct source-spec exposure.
  • RQ2: Mechanism effects: 55/85 evaluator-test passes remained after removing structural repair, compared with 58/85 without compatibility scaffolding and 39/85 without generator preflight.These larger gaps provide stronger mechanism evidence than the small differences among the strongest variants.
  • RQ3: Failure regimes: 85/85 core-runnable runs produced only 17/85 evaluator-test passes and 6/85 strict-delivery successes for the lightweight baseline.Strict delivery adds a shallow deliverable contract beyond the primary full-test outcome.
  • RQ3: Failure regimes: The failure taxonomy assigns each strict-delivery non-success to one dominant delivery-layer class using persisted run-outcome, failure-summary, and shallow-contract signals.Classes include integration/interface, evaluator-test, task-contract, metric/output-contract, placeholder/mock-substitution, data/layout, and other/unknown failures.
  • RQ3: Failure regimes: Runnable single-agent baselines typically fail later on task-contract, metric/output, placeholder/mock, data/layout, or downstream evaluator signals.Removing structural repair concentrates failures in integration/interface breakdowns, while removing generator preflight shifts many failures toward placeholder/mock and data issues.

5. Discussion

Specification exposure improves single-agent performance but does not by itself secure delivery. The discussion attributes stronger outcomes to preserving the task contract across planning, generation, checking, and bounded recovery, while limiting claims beyond this benchmark.

  • Interpretation: Specification exposure helps, but direct source-spec exposure remains far below Spec2Vision in evaluator-test performance.The discussion frames the bottleneck as both informational and structural.
  • Interpretation: The task contract must be preserved across planning, file generation, checking, and bounded recovery, rather than merely exposed to a collapsed single-agent runner.This interpretation connects the single-agent ladder with the staged pipeline’s operational design.
  • Mechanism interpretation: Structural repair provides the cleanest principal contrast, while generator preflight supports deterministic screening and compatibility scaffolding contributes more moderately.The no-history and no-architect-guidance effects are treated conservatively because they are small and may be partly redundant with other mechanisms.
  • Failure interpretation: Different mechanisms address different delivery layers: single-agent baselines show contract, metric/output, placeholder, and data-layout failures, whereas mechanism removals shift failure locations.Removing structural repair concentrates integration/interface failures; removing generator preflight shifts many failures toward later placeholder and data issues.
  • Scope: Within this benchmark, contract-carrying staged generation produces more evaluator-compliant CV bundles than runnable code generation alone.The evidence is scoped to task-contract delivery, not scientific CV quality or production readiness.

6. Threats to Validity

The study’s validity is bounded by its contract-grounded benchmark, non-external evaluator, unequal internal machinery, and residual execution and attribution uncertainty.

  • The benchmark covers 17 fixed tasks, 10 executable conditions, and 5 repeats per task-condition cell, representing a bounded task-contract space.All tasks share one specification language and one evaluator-facing contract, limiting generalization to other pipeline-generation settings.
  • The evaluator supports comparison across conditions but measures preservation of the provided task contract rather than independent computer-vision model quality.It is condition-agnostic but not an external third-party benchmark.
  • The comparison is not equal in internal machinery because Spec2Vision includes staged roles and deterministic controls absent from the single-agent baselines.The baselines test delivery under the same task grid, model path, execution envelope, and evaluator without that machinery.
  • Five repeats and two comparable reruns expose some variation but do not remove dependence on tasks, model outputs, serving configuration, or local environment.Small differences among the strongest variants are interpreted descriptively, while larger stable gaps provide stronger evidence.
  • The artifact-derived failure taxonomy assigns one dominant class per non-success run, supporting descriptive comparison rather than exhaustive causal attribution.

7. Conclusion

Within this benchmark, Spec2Vision substantially outperforms executable single-agent baselines, with strongest mechanism-level evidence for structural repair and generator preflight. The results show that preserving the task contract through staged generation, checks, testing, and bounded repair yields more evaluator-tested deliveries than runnable generation alone, while broader validation remains future work.

  • Spec2Vision substantially outperforms executable single-agent baselines under the same benchmark and downstream evaluator.The strongest mechanism-level evidence comes from structural repair and generator preflight, while compatibility scaffolding contributes less and history or architect guidance remain descriptive.
  • Many generated CV bundles are runnable but not delivered because they fail task-contract, metric/output, placeholder/mock, data-layout, or evaluator-test checks.
  • Carrying the task contract through staged generation, deterministic checks, testing, and bounded repair yields far more evaluator-tested deliveries than runnable code generation alone.
  • Future work should test generalization across more tasks, CV families, specification styles, model and serving settings, external evaluators, and richer failure taxonomies.

Artifact Availability

The anonymized repository and artifact archive provide the code, configurations, run bundles, manifests, reports, summaries, and derived tables used to support audit and reproduction.

  • The repository and GitHub Releases archive contain the task catalog, specifications, experiment configuration, runtime and harness code, persisted bundles, manifests, reports, summaries, and derived tables.Full reruns require compatible model serving and local data resources.

Declaration on Generative AI

The authors used generative AI tools for language editing and restructuring, reviewed the final manuscript, and take responsibility for its content.

  • Generative AI tools were used for language editing and restructuring, with the authors reviewing the final manuscript and taking responsibility for its content.
Loading 2608.26400v1…