PaperlayerView exact revision on arXiv

Source-linked AI summary

Instant Software Factories

Wenxin Jiang, Mikola Lysenko

arXiv:2610.36254v1cs.SE

TL;DR

Coding agents can build prototypes, but the long tail of fixes and releases raises questions about factory coordination and measurement. The paper proposes a two-input factory, reports early observations of endogenous workload and evaluation, and frames factory development as multi-objective optimization under that feedback.

  • Problem

    The paper asks how much infrastructure software factories need and how their progress should be judged when factory actions can change the work and evaluation process.

    Source: §I · p. 1

  • Method

    It proposes cron-job agents coordinating through a shared board, a one-session bootstrap prompt, and a multi-objective formulation of factory development under endogenous feedback.

    Source: §I · p. 1, §III · p. 2, §V · p. 4

  • Results

    In the CLI factory, agents filed 95% of issues and authored 40 of 58 CI workflow PRs, while at least 25 of 450 issues marked fixed were later found to have incorrect fixes.

    Source: §I · p. 1, §IV · p. 3

  • Takeaways & Limitations

    Factories should measure independently valued, validated, and delivered changes because throughput can overstate progress when agents change the workload and evaluation process.

    Source: §I · p. 1

    The two factories came from one organization and were observed for two to three weeks; changing CLI instructions and rollout prevent causal conclusions from before-and-after differences.

    Source: §IV · p. 3

Abstract

from arXiv · show

Coding agents can quickly build prototypes, but a long tail of fixes, optimizations, tests, and releases separates a prototype from a product. A software factory handles this ongoing work through a repeatable process. We ask whether this work needs dedicated orchestration and how such factories should be judged. Given a tested repository and a coding agent, an instant software factory uses only two coordination inputs, a swarm of cron-job agents with standing objectives and a shared communication board, such as GitHub, where agents coordinate through the traces they leave. We developed a bootstrap prompt designed to set up both inputs from one round of human input, with governance as a separable layer. We distilled it from two manually configured industrial factories, observed for 15 and 17 days, and have not yet evaluated it. In the GitHub-coordinated CLI factory, daily merges rose 15x during agent rollout, an observed association. Its agents also filed 95% of new issues and authored 40 of 58 PRs that changed CI workflows, and at least 5.6% of issues marked as fixed were later found to have incorrect fixes. Unlike an agent scored on a fixed benchmark, a factory thus changes its own workload, evaluator, and measured progress. We formulate factory development as multi-objective optimization under this endogenous feedback. Over five months, we plan to evolve both factories, test the bootstrap on new repositories, and test five falsifiable hypotheses.

I. INTRODUCTION

The paper asks how much infrastructure a software factory needs and how its progress should be judged. It proposes a two-input factory and a one-session bootstrap, while arguing that factory actions change the work and evaluation used to measure progress.

I. INTRODUCTION

  • The paper asks how much infrastructure a software factory needs and how its progress should be judged, against a background of substantial internal and research factory systems.Coding agents can build prototypes, but bugs, regressions, flaky CI, and releases leave a long tail of work.Source: §I · p. 1
  • A bootstrap prompt is designed to configure both coordination inputs in one session from a single round of human answers, but this feasibility claim is untested.One factory provides early evidence for the sufficiency hypothesis.Source: §I · p. 1
  • Factories create endogenous feedback because their actions change the product, workload, and evaluation process, so throughput can overstate progress.The CLI factory's agents filed 95% of issues and authored 40 of 58 PRs changing CI workflows; seven incorrect fixes shipped unrepaired.Source: §I · p. 1
  • A factory may need only cron-job agents with standing objectives and a shared board, while mandatory governance remains a separable layer.Agents coordinate through traces on the board rather than an orchestrator.Source: §I · p. 1
  • The paper contributes a multi-objective formulation of factory development under endogenous feedback, observations from two factories, and a released bootstrap prompt that has not yet been evaluated.The formulation includes a progress measure that agents cannot edit and five testable hypotheses.Source: §I · p. 1

II. AGENT SWARMS AND SOFTWARE FACTORIES

Prior work explores stigmergy, shared blackboards, and orchestrated agent factories, while practitioner reports also describe scheduled agents coordinated through GitHub.

II. AGENT SWARMS AND SOFTWARE FACTORIES

  • Stigmergy and blackboard systems provide precedents for specialists coordinating through traces or shared information, while persistent artifacts can support specialized LLM-agent roles.Teams without centralized verification can propagate errors, motivating a shared board and merge gate.Source: §II · p. 2
  • Research factory systems and toolkits often use message passing or dedicated orchestrators, whereas practitioners have also coordinated scheduled agents through GitHub issues and PRs.Those practitioner reports typically configure agents individually and report activity or merge rates.Source: §II · p. 2
  • This paper tests whether simple coordination primitives can initialize a whole factory through one prompt and judges progress by changes that hold after merge and reach users.Source: §II · p. 2

III. THE BOOTSTRAP

The bootstrap prompt configures a scheduled-agent factory for PR-sized work in one session, using cron-job agents and a board of task and verdict traces. Governance is mandatory but separate, and the setup targets a limited operating scope.

III. THE BOOTSTRAP

  • A 1,400-word prompt guides a coding agent in the target repository through a one-session setup, after which it opens a setup PR.Schedules are enabled only after governance controls are tested and approval is obtained.Source: §III · p. 2
  • Eight scheduled roles divide discovery, implementation, review, merging, and management, with runs launched at most hourly and retaining no memory beyond the board.The manager steers work through the board, including assigning tasks and balancing objectives within budget.Source: §III · p. 2
  • The board records tasks and evidence in issues, changes in PRs, and verdicts in checks and reviews; any board holding these trace types can suffice.The bootstrap targets GitHub and scheduled Claude Code tasks.Source: §III · p. 2
  • The bootstrap targets work that fits one PR and can wait an hour; production deployment, multi-repository work, and tightly coupled tasks may require other approaches.Tightly coupled work may need planners.Source: §III · p. 2
  • For user-installed software, merges by default require passing checks, independent review of the latest commit, and human approval; agents must check a stop switch before writing.The governance layer restricts actions without changing how agents coordinate, and controls scale to risk.Source: §III · p. 2
  • Hand-built routines informed the platform notes, but tool lists alone did not enforce GitHub permissions, and CI capacity and GitHub reliability were the main bottlenecks.The prompt tests each permission restriction, and bottlenecks motivated adaptive run rates.Source: §III · p. 2

IV. TWO WORKING FACTORIES

Two industrial factories sustained work with different architectures, while their observed operation exposed trade-offs across governance, correctness, and delivery.

IV. TWO WORKING FACTORIES

  • The CLI factory was observed for 15 days, with escapes tracked through day 17; the contrasting agentic factory was observed for 17 days.Neither factory evaluates the bootstrap’s automated setup.Source: §IV · p. 2
  • After the CLI swarm started, daily merged PRs rose from 2.2 to about 35, while the passage also reports a peak of 115 without establishing its timing.The CLI setup used about 30 scheduled agents and GitHub as a shared board, with no orchestrator; the contrasting agentic system used service loops and a shared database work list.Source: §IV · p. 2
  • 77% of CLI agent merges changed user-installed code; a second person approved 534 of 541 merged PRs, while the merge queue rejected 29 failing PRs.The queue delayed other PRs by a median of 26 minutes as daily merges doubled.Source: §IV · p. 3
  • Agents authored 40 of 58 CLI PRs changing CI workflows, including 7 that loosened checks; humans reviewed these changes.Two LLM raters classified CI effects with κ = 0.91.Source: §IV · p. 3
  • At least 25 of 450 fixed CLI issues (5.6%) were escapes; none reopened, while 3 of 14 replayed agentic fixes (21%) recurred.Escapes were incorrect fixes that either broke another case of the issue or left part of it unfixed.Source: §IV · p. 3
  • Two CLI releases shipped 94% of fixed issues after a median 3.9-day integration delay; reports identified 22 of 25 escapes within a day of merge.Release delays allowed 14 escapes to be corrected before shipment, but 7 shipped unfixed, including 4 before being reported.Source: §IV · p. 3

V. THE SOFTWARE FACTORY AS A MULTI-OBJECTIVE OPTIMIZATION PROBLEM

The paper frames factory configuration as multi-objective optimization under feedback that can distort measured progress. It proposes an independent evaluator and safeguards, then defines measures and hypotheses for testing factory settings.

V. THE SOFTWARE FACTORY AS A MULTI-OBJECTIVE OPTIMIZATION PROBLEM

  • Factory state comprises product, backlog, and validation process; agents can change workload, the evaluator, and when independent evaluation occurs.Because agents can grow the backlog, loosen checks, or merge early, the paper proposes measuring progress with an evaluator agents cannot edit and safeguards for each channel.Source: §V · p. 3
  • Velocity counts unprompted changes per week that an independent evaluator judges correct, reach users, and weights by severity; merge rates measure throughput only.The manager seeks Pareto-optimal settings under cost, human-time, escape-rate, and governance constraints; hidden escapes make measured progress an upper bound after a fixed audit window.Source: §V · p. 4
  • For the pooled CLI study, throughput counts 450 fixed issues, delivery counts 425 shipped, and correctness leaves at most 418 after excluding 7 shipped with detected escapes.The maintainer would assign each issue an objective and severity from a preset rubric and divide objective counts by preset targets; agent CI PRs count only toward the CI objective.Source: §V · p. 4
  • The five hypotheses test specialist versus generalist objectives, schedule adaptation, step size, memory notes, and held-out tests against agent-modified CI checks.H5 tests whether hidden held-out tests outside the agent-modifiable evaluator reduce escapes more than agent-modified CI checks.Source: §V · p. 4
  • The planned experiments use equal-cost arms; H1–H4 test both factories, while H5 tests only the CLI factory.Issues per PR approximate step size; 40% of fixed issues were in multi-issue PRs, whose 3.9% escape rate versus 6.6% for single-issue PRs was inconclusive (Fisher’s p = 0.29, ignoring PR clustering).Source: Table II · p. 4, §V · p. 4

VI. FUTURE PLANS

Over five months, the authors plan to evolve both factories, evaluate bootstrap setup on new repositories, and test five hypotheses. The program also examines cross-factory transfer and manager self-tuning.

VI. FUTURE PLANS

  • Over five months, the authors will evolve both factories, restart the agentic swarm, and test the bootstrap on new repositories against manual setup.Bootstrap feasibility measures setup effort, control-test failures, and time to the first independently validated merge; sufficiency tracks whether added coordination is needed to sustain progress for a month.Source: §VI · p. 4
  • H1–H4 test settings within each factory, while H5 randomizes CLI PRs and uses arm-blind scouts to judge escapes for 14 days after merge.Detecting a halved 5.6% escape rate at 80% power and α = 0.05 requires 800+ fixes per arm, about three months at late-study rates, and more under clustering.Source: §VI · p. 4
  • The authors will test whether security-factory lessons transfer to other domains; settings supported across both factories and similar public-board systems may become candidate best practices.Generalizable settings may become bootstrap defaults, while domain-specific settings may remain setup questions.Source: §VI · p. 4
  • An adaptation skill is planned to let the manager tune factory settings from cost, correctness, and delivery within governance.The paper describes this as self-tuning and evolution of the factories.Source: §VI · p. 4

DATA AVAILABILITY

The artifact contains the bootstrap and will include its revisions and experimental scripts; case records remain private for confidentiality.

DATA AVAILABILITY

  • The artifact contains the bootstrap and will include its revisions and experimental scripts; case records remain private for confidentiality.Source: DATA AVAILABILITY · p. 5

Related papers

Loading 2610.36254v1…