PaperlayerView exact revision on arXiv

Source-linked AI summary

RepoMAS: Solving Progressively Specified Tasks with Issue-Driven Multi-Agent Systems

Yuchen Song, Andong Chen, Wenxin Zhu, Muyun Yang, Tiejun Zhao

arXiv:2609.32490v2cs.AI

TL;DR

Many multi-agent systems assume task requirements are specified before execution, although additional requirements may emerge during problem solving. The paper introduces ProgSpec and RepoMAS, an issue-driven framework that revises task specifications from execution feedback; RepoMAS reports the highest scores across ProgSpec and five conventional benchmarks.

  • Problem

    Most multi-agent systems assume requirements are sufficiently specified before execution, despite requirements sometimes emerging during reasoning, tool use, or execution.

    Source: §1 · p. 1

  • Method

    RepoMAS records execution feedback as structured Issues that revise the task specification and execution structure, while affected tasks are selectively re-executed.

    Source: §1 · p. 2

  • Results

    RepoMAS reports the highest scores on ProgSpec and five conventional benchmarks; on ProgSpec it achieves 44.7% weighted requirement coverage, 1.8 percentage points above the strongest evaluated baseline.

    Source: §7 · p. 10

  • Takeaways & Limitations

    The results motivate multi-agent systems that revise their explicit representation of task requirements as new evidence becomes available.

    Source: §7 · p. 10

    The formulation is scoped to requirements that become identifiable through reasoning, retrieved evidence, tool use, or execution feedback.

    Source: §3.1 · p. 4

Abstract

from arXiv · show

LLM-based multi-agent systems (MASs) have shown strong potential for solving complex tasks, but most assume that task requirements are sufficiently specified before execution. In practice, user requests are often incomplete, and additional requirements may only become clear during reasoning, tool use, or execution. We refer to such problems as progressively specified tasks. To systematically study this setting, we introduce ProgSpec, a benchmark that evaluates final outputs against requirements explicitly stated in the initial request and additional requirements supported by the available task evidence. We further propose RepoMAS, an issue-driven multi-agent framework inspired by open-source project management. RepoMAS records newly discovered requirements, conflicts, and failures as structured Issues and uses them to revise the task specification and execution structure during problem solving. Across ProgSpec and five existing benchmarks, RepoMAS achieves the best performance. Further analyses show that its issue-driven revision and repository maintenance mechanisms consistently contribute to performance. These results highlight the importance of allowing MASs to revise not only how a task is solved, but also revise their explicit representation of task requirements during execution.

1 INTRODUCTION

The paper addresses tasks whose requirements emerge during problem solving, not just those fully specified upfront. It introduces ProgSpec and RepoMAS, which uses structured Issues to revise task specifications and execution, and reports strong results across six benchmarks.

1 INTRODUCTION

  • Progressively specified tasks reveal requirements during reasoning, tool use, evidence retrieval, or execution, requiring revisions to both the solution and task understanding.Existing MASs mostly assume task requirements are sufficiently specified before execution begins.Source: §1 · p. 1
  • ProgSpec evaluates whether systems satisfy requirements missing from the initial request and discovered during problem solving.The benchmark is introduced to systematically study progressively specified tasks.Source: §1 · pp. 1–2
  • RepoMAS turns execution feedback into structured Issues that guide revisions to the current task specification.Issues update the task structure and agent organization; RepoMAS then locally re-executes affected tasks.Source: §1 · p. 2
  • With GPT-4o-mini, RepoMAS outperforms evaluated baselines on ProgSpec and five established benchmarks.Experiments with additional backbones also demonstrate applicability across different model families and tasks.Source: §1 · p. 2

2 RELATED WORK

Prior work studies multi-agent collaboration, adaptive planning and organization, and benchmarks for agent capabilities or incomplete specifications. RepoMAS explicitly links requirements to tasks and manages revisions through Issues and validated patches; ProgSpec evaluates autonomous requirement recovery across multiple domains.

2.1 LLM-BASED MULTI-AGENT SYSTEMS

  • Multi-agent research studies specialized roles, collaboration, communication structures, and the computational cost of coordination.Examples include CAMEL, DyLAN, AgentVerse, MacNet, and AgentPrune.Source: §2.1 · pp. 2–3

2.2 DYNAMIC TASK PLANNING AND AGENT COLLABORATION

  • Prior work adapts task plans and agent organization during execution, while RepoMAS links newly identified constraints to coordinated specification and execution-structure revisions.RepoMAS manages these revisions through structured Issues and validated patches.Source: §2.2 · p. 3

2.3 BENCHMARKS FOR AGENTS AND INCOMPLETE SPECIFICATIONS

  • Existing benchmarks assess agent capabilities and incomplete specifications, but ProgSpec evaluates requirement satisfaction across Coding, Mathematics, and document-grounded QA using initial-context and problem-solving evidence.Its focus complements benchmarks centered on omissions in software proposals or requirements added through user interaction.Source: §2.3 · p. 3, Table 1 · p. 3

3 PROGSPEC: PROGRESSIVELY SPECIFIED TASKS BENCHMARK

ProgSpec formalizes tasks in which the initial request omits requirements needed for success and evaluates systems against an annotated, evidence-grounded reference. It covers three domains, assigns requirements by how directly they follow from the input, and weights coverage to emphasize implicit and latent requirements.

3.1 PROGRESSIVELY SPECIFIED TASKS

  • Progressively specified tasks begin with requests that omit some requirements needed for success, while systems refine their identified requirements as evidence emerges.The benchmark treats its idealized requirement set as fixed and evidence-grounded, approximated by human annotations.Source: §3.1 · pp. 3–4

3.2 BENCHMARK CONSTRUCTION

  • ProgSpec instances pair an initial request with an annotated reference requirement set and annotations of how directly each requirement follows from the input.It covers Coding, Mathematics, and document-grounded QA, using repository evidence for coding tasks and documents for QA tasks.Source: §3.2 · p. 4

3.3 DATA STATISTICS

  • ProgSpec contains 270 tasks and 1,551 atomic requirements, averaging 5.74 per task, with 46.2% classified as L3.The benchmark has 90 tasks in each of its three domains.Source: §3.3 · p. 4, Table 2 · p. 4
  • Weighted requirement coverage is the primary metric, assigning L1–L3 requirements weights of 1, 2, and 3 to emphasize implicit and latent requirements.The score aggregates satisfied requirements using their level-specific weights.Source: §3.3 · p. 4

4 REPOMAS: AN ISSUE-DRIVEN MULTI-AGENT SYSTEM FOR PROGRESSIVELY SPECIFIED TASKS

RepoMAS treats the task specification and execution structures as revisable repository state, using evidence discovered during execution to guide validated, targeted repairs.

4.1 REPOSITORY STATE INITIALIZATION

  • The shared repository tracks a revisable Task Draft alongside task and agent structures, discovered Issues, and generated artifacts.Task nodes are linked to requirements so affected subtasks can be located when the specification changes.Source: §4 · p. 5, §4.1 · p. 5

4.2 ISSUE-DRIVEN REPOSITORY MAINTENANCE

  • Execution feedback becomes structured Issues that drive repository revisions, while candidate patches remain isolated until validation and accepted changes trigger local re-execution.Rejected candidates may be revised; the process continues until no further revision is needed or the execution budget is reached.Source: §4.2 · p. 5, §4.3 · p. 6
  • Issues about missing or incorrect requirements revise the Task Draft and affected execution structures, while execution failures may instead change tasks or agent assignments.Each Issue records its type, affected repository component, problem description, and supporting evidence.Source: §4.2 · p. 6

4.3 PATCH VALIDATION, MERGE, AND RE-EXECUTION

  • After a patch is accepted, RepoMAS reruns directly affected tasks and their downstream dependents, reusing unaffected outputs only when their inputs, requirements, and agents remain unchanged.Artifacts from removed tasks are invalidated, and re-execution may reveal further Issues before the process stops or reaches its execution budget.Source: §4.3 · p. 6

5 EXPERIMENTS

RepoMAS is evaluated against a single-agent baseline and seven multi-agent systems on ProgSpec and five conventional benchmarks. With GPT-4o-mini, it achieves the highest score on all six.

5.1 EXPERIMENTAL SETUP

  • The evaluation uses ProgSpec and five conventional coding, reasoning, and QA benchmarks, comparing GPT-4o-mini with seven multi-agent systems under shared inputs, tools, and evaluation protocol.Reported metrics are weighted requirement coverage on ProgSpec, pass@1 on HumanEval and MBPP, solve rate on GSM8K, and F1 on HotpotQA and DROP.Source: §5.1 · p. 6

5.2 MAIN RESULTS

  • 44.7 weighted requirement coverage on ProgSpec gave RepoMAS the highest score across all six benchmarks, 1.8 points above Meta-Agent and 6.4 above GPT-4o-mini.The overall performance trend is statistically significant (p = 0.024).Source: Table 3 · p. 7, §5.2 · p. 7

6 ANALYSIS

The analyses find benefits from RepoMAS’s components and show that it captures more latent requirements, with discovery timing varying by domain. Those gains come with token overhead that grows with repository maintenance needs.

6.1 ABLATION STUDY

  • Removing any component reduces performance across all three benchmarks; structural revision has the largest effect on ProgSpec and HotpotQA, while patch validation has the largest effect on HumanEval.Removing structured Issues also substantially degrades performance; local re-execution helps, though its effect is smaller on HotpotQA.Source: §6.1 · p. 7, Table 4 · p. 7

6.2 EFFECT OF STRUCTURED ISSUE REPRESENTATION

  • The full structured Issue representation outperforms direct revision, free-form Issues, and structured Issues without evidence across all three benchmarks.Neither an unstructured intermediate Issue nor structure alone provides the same improvement.Source: §6.2 · pp. 7–8, Table 5 · p. 7

6.3 EFFECTIVENESS OF PATCH VALIDATION

  • 86.2% recall and 62.5% precision; good patches rise from 13.8% initially to 69.0% after revision.The authors suggest validation both filters unreliable patches and helps improve candidates that initially fail review.Source: §6.3 · p. 8, Table 6

6.4 EFFECTS OF STRUCTURAL REVISION

  • 44.7 to 25.8 on ProgSpec when either Task Graph or Agent Graph revision is disabled; disabling both yields 30.3, while full revision performs best across benchmarks.The authors attribute the result to keeping task structure and agent organization synchronized with specification changes.Source: §6.4 · p. 8, Table 7 · p. 8

6.5 REQUIREMENT-LEVEL COMPOSITION

  • 24.4% of RepoMAS’s satisfied requirements are L3, versus 19.6% for the single-agent baseline; L1 shares are 42.8% and 46.5%, respectively.The within-method distributions suggest RepoMAS satisfies a larger proportion of latent requirements; L2 remains relatively stable.Source: Figure 3 · p. 8, §6.5 · p. 9

6.6 REQUIREMENT DISCOVERY CONTINUES THROUGHOUT EXECUTION

  • Requirement discovery varies by domain: Coding is led by Tool/Verifier Feedback (62.0%), Mathematics splits between Execution (46.2%) and verifier feedback (53.8%), and QA is led by Execution (82.9%).No Mathematics requirements were first identified during Final Verification; Coding also has 24.1% first identified then.Source: §6.6 · p. 9, Figure 4 · p. 9

6.7 TOKEN COST ANALYSIS

  • RepoMAS consumes more tokens than baselines on ProgSpec, HumanEval, and HotpotQA, with the largest overhead on ProgSpec and HumanEval and a substantially smaller gap on HotpotQA.The reported token cost is tied to repository maintenance; benchmarks requiring more revisions and validation use more tokens.Source: Figure 5 · p. 9

7 CONCLUSION

The paper studies tasks whose requirements emerge during reasoning, tool use, or execution, and introduces ProgSpec and RepoMAS to address them. RepoMAS achieves 44.7% weighted requirement coverage on ProgSpec, 1.8 percentage points above the strongest evaluated baseline, and the highest reported scores on five conventional benchmarks.

7 CONCLUSION

  • RepoMAS achieves 44.7% weighted requirement coverage on ProgSpec, 1.8 percentage points above the strongest evaluated baseline, and the highest reported scores on five conventional benchmarks.The paper concludes that systems should revise their explicit representation of task requirements as new evidence becomes available.Source: §7 · p. 10

AI USE STATEMENT

The authors report limited AI assistance for editing and routine programming, while the supplied material details ProgSpec’s construction, requirement annotation, and evaluation. The benchmark uses evidence-grounded atomic requirements and weights coverage by requirement level.

AI USE STATEMENT

  • AI tools supported language polishing, code assistance, formatting, limited technical-description editing, and routine programming; authors made and verified the research and implementation decisions and reviewed outputs.The statement attributes dataset construction, analysis, and interpretation to the authors.Source: AI USE STATEMENT · p. 10

A CASE STUDY

  • Three ProgSpec-Coding examples show execution evidence revealing unstated requirements, which RepoMAS records as Issues before validated revisions and local re-execution.A third case identifies a requirement from repository structure, followed by a separate documentation and testing repair.Source: A CASE STUDY · pp. 13–14

B PROGSPEC CONSTRUCTION DETAILS

  • ProgSpec contains 270 tasks across Coding, Mathematics, and document-grounded QA; three graduate students independently annotated all tasks and resolved disagreements against task descriptions and evidence.The final labels form a consolidated requirement set.Source: B PROGSPEC CONSTRUCTION DETAILS · p. 14

B.1 CODING TASK CONSTRUCTION

  • Coding tasks use change requests and fixed public-repository snapshots, evaluating implementation plans against atomic requirements rather than executing code patches.Repository selection requires cross-file reasoning and development records; tasks exclude unavailable services or insufficient evidence and restrict scope to relevant files and dependencies.Source: §B.1 · p. 14

B.2 MATHEMATICS TASK CONSTRUCTION

  • Mathematics tasks are selected for well-defined answers and independently verifiable requirements; annotations accept alternative correct derivations and exclude requirements tied only to one reference path.Candidate requirements are drawn from the problem and reference solution, including targets, assumptions, conditions, boundary cases, and validity checks.Source: §B.2 · pp. 14–15

B.3 QA TASK CONSTRUCTION

  • QA tasks require combining information across documents, and gold requirements must be recoverable from the supplied documents rather than later dialogue.Unsupported requirements are excluded and each retained requirement has a supporting document passage.Source: §B.3 · p. 15

C.1–C.5 REQUIREMENT ANNOTATION AND QUALITY CONTROL

  • Gold requirements are necessary, atomic, independently verifiable, and evidence-grounded; L1 marks explicit requirements, L2 direct implications or standard conditions, and L3 requirements established through task-specific evidence or analysis.Annotation records evidence and access actions, then reviews requirements for omissions, duplication, validity, and level before acceptance.Source: §C.1 · p. 15, §C.2 · p. 15

D.1–D.2 EVALUATION

  • Weighted requirement coverage gives greater weight to less-explicit requirements and pools requirement weights across tasks; satisfaction is judged against each final artifact and available evidence.Coding is evaluated on implementation plans, Mathematics accepts equivalent correct derivations and answers, and QA requires document-supported information without contradiction.Source: §D.1 · pp. 16–17, §D.2 · p. 17

E PATCH VALIDATION EVALUATION

The evaluation compares the online validator’s accept-or-revise decisions with offline labels for candidate patches, using patch quality and regression-related measures.

E PATCH VALIDATION EVALUATION

  • A patch is good only if it resolves its target Issue and keeps the revision consistent with task requirements and evidence; existing requirements stand unless new evidence supports revision.The online validator’s decisions are compared with offline quality labels.Source: E PATCH VALIDATION EVALUATION · p. 17
  • Accepted-patch error rate is the fraction of accepted patches labeled bad, equal to 1 − Precision.Source: E PATCH VALIDATION EVALUATION · p. 17
  • Regression rate measures accepted patches that break previously valid requirements as a fraction of all accepted patches.Source: E PATCH VALIDATION EVALUATION · p. 17

F BACKBONE ROBUSTNESS

Additional-backbone experiments test RepoMAS with the same benchmark splits, metrics, prompts, and settings, varying the backbone model. Improvements appear across most model families and tasks.

F BACKBONE ROBUSTNESS

G ANNOTATION QUALITY AND HUMAN VALIDATION

Independent review assesses ProgSpec annotation quality, while the accompanying configuration describes the execution, prompting, evaluation, and statistical-testing protocols used in experiments.

G ANNOTATION QUALITY AND HUMAN VALIDATION

  • Among 500 reviewed requirements, 95.4% were valid; task-level annotation agreement was 91.4% (Gwet’s AC1 0.91).Five graduate students independently reviewed a fixed snapshot covering 90 randomly sampled tasks and 500 requirements, judging requirement quality and task-level support, completeness, atomicity, and level assignment. Agreement was highest for contextual support (98.9%), followed by completeness and atomicity (90.0%) and level assignment (86.7%).Source: G ANNOTATION QUALITY AND HUMAN… · pp. 18–19, Table 10 · p. 19

H EXPERIMENT CONFIGURATION

  • Unless otherwise specified, experiments use the full RepoMAS framework, including structured Issues, patch validation, structural revision, and local re-execution.Source: H EXPERIMENT CONFIGURATION · p. 19

H.1 EXECUTION ENVIRONMENT

  • Experiments run on Linux with Python 3.11; the RepoMAS runtime does not require GPU computation, and structured outputs use JSON schemas.ProgSpec-Coding provides a repository snapshot for each task.Source: §H.1 · p. 19

H.2 RUNTIME SETTINGS

  • Unless otherwise specified, RepoMAS runs use at most three execution–revision iterations per task and one task attempt, with calls limited to 4,096 tokens.ProgSpec uses temperature 0, while other benchmarks use temperature 0.2; additional-backbone runs retain the RepoMAS configuration where possible and change only the backbone.Source: §H.2 · p. 19, Table 11 · p. 19

H.3 PROMPT PROTOCOL

  • Prompts provide the original task and stage-relevant Task Graph information, while excluding gold requirements, hidden tests, and reference answers; an independent validator reviews proposed revisions.Accepted patches update repository state, and only affected workflow parts are re-executed.Source: §H.3 · pp. 19–20

H.4 EVALUATOR PROTOCOL

  • A separate GPT-5.5 post-hoc judge evaluates ProgSpec responses against evaluator-only atomic requirements and available references, returning requirement-level coverage decisions with supporting evidence.Source: §H.4 · p. 20

H.5 STATISTICAL SIGNIFICANCE

  • Statistical significance is tested with two-sided paired t-tests on 10 paired runs per dataset–baseline comparison, using α = 0.05.Source: §H.5 · p. 20

Related papers

Loading 2609.32490v2…