OpenDraco: A Platform for Engineering and Experimenting with Configurable Multi-Agent Architectures for Software Engineering

I. INTRODUCTION

• Without an integrated workflow, comparing MAS configurations across benchmarks and evaluation results is harder to organize and reproduce.The experimentation cycle includes adjusting configurations, running comparable tasks, evaluating outputs, and tracking results.Source: §I · p. 1
• OpenDraco stores agents, communication topology, prompts, tools, models, and parameters in versioned, exportable JSON specifications translated into executable LangGraph workflows.Researchers can reuse agent definitions and run alternative configurations with local or cloud-based agents.Source: §I · p. 1
• On the 77-instance SWE-bench Lite test subset, the sequential baseline resolves 12 issues; on the 23-instance development subset, it matches hyperagent_star’s two repairs with 39.78% less wall-clock time and 47.32% fewer tokens.The evaluation also covers synthetic and real-world bugs using the same locally hosted model.Source: §I · p. 1
• The paper also reports proof-of-concept workflows beyond APR, illustrating the platform’s applicability to additional tasks.A demonstration video and source code, documentation, and replication artifacts are reported as available.Source: §I · p. 1

II. RELATED WORK

• MAS research has explored predefined interaction patterns, topology-focused studies and optimization, and automatic workflow generation, including work by Li et al. [3], Yang et al. [11], GPTSwarm, G-Designer [6], and AFlow.Li et al. [3] investigate whether aggregating multiple agents can improve task performance.Source: §II · p. 2
• Existing platforms provide visual workflow composition, reusable components, declarative specifications, graph visualization, and execution debugging.The passage names Flowise, Langflow, Crew Studio, AutoGen Studio, and LangGraph Studio.Source: §II · p. 2
• OpenDraco combines configurable agents and communication structures with versioned specifications, execution, task-specific evaluation, and result comparison for SE experimentation.Its APR instantiation connects reusable agents and topology templates to SWE-bench evaluation.Source: §II · p. 2

III. OPENDRACO ARCHITECTURE

• The architecture figure presents OpenDraco’s components, and the framework follows an n-tier design.The supplied description does not specify the individual tiers.Source: §III · p. 2, Figure 1 · p. 2

A. User Workflow

• Users select or define a topology, configure agent roles and settings, save configurations for reuse and versioning, then execute and evaluate them on selected instances.They can modify or replace a configuration and rerun it on the same instances to compare designs under the same setting.Source: A. User Workflow · p. 2, Figure 2 · p. 2

B. Frontend Layer

• The Angular-based frontend supports topology and agent configuration, JSON import/export and Git versioning, inference, task-specific evaluation, and result inspection.For APR, evaluation uses the SWE-bench harness to apply patches and execute tests.Source: B. Frontend Layer · p. 2

C. Backend Layer

• The Python backend exposes services through a FastAPI server used by the frontend and accessible to third-party components.Source: C. Backend Layer · p. 2

1) Agentic System Modeler:

• The modeler records each agent’s model, prompts, parameters, and tools, plus the topology connecting agents, in a structured specification such as JSON.It validates specifications against LangGraph constraints and checks for structural anomalies, including unreachable agents or no path from START to END.Source: 1) Agentic System Modeler · p. 2
• The modeler supports reuse and change tracking through a role-organized agent catalog, Git history, and file-system persistence of specifications.Configuration history allows changes to be tracked and previous versions restored.Source: 1) Agentic System Modeler · p. 3

2) Agentic System Executor:

• The executor converts a JSON MAS specification into a LangGraph StateGraph and runs it on a problem instance to produce a prediction.Its graph builder, shared state, agent runtime, and workflow runner support fan-out and conditional edges; agents alternate LLM calls and tool invocations until a final response.Source: 2) Agentic System Executor · p. 3

3) Evaluation Runner:

• For APR, the Evaluation Runner offers the official SWE-bench Docker harness, remote execution via sb-cli, and a local evaluator that applies generated patches and runs repository tests.The local evaluator also supports repositories outside SWE-bench, including those containing synthetic instances.Source: 3) Evaluation Runner · p. 3

4) Additional Layers:

• The Storage Layer saves execution registries, predictions, reports, metrics, configurations, logs, and results as inspectable, editable local files.Inference supports localhost Ollama backends or cloud services through OpenAI-compatible APIs.Source: 4) Additional Layers · p. 3

D. Customization of OpenDraco for APR

• The APR instantiation adds a catalog of role-classified agents from 16 public repositories, the SWE-bench harness, and an APR-specific UI with topology templates and results.The catalog includes only artifacts with permissive MIT, BSD, or Apache-2.0 licenses and records their provenance.Source: D. Customization of OpenDraco for APR · p. 3

A. Evaluation Goals

• The evaluation asks whether OpenDraco can execute existing APR topologies end-to-end, repair SWE-bench Lite bugs, scale to more real-world instances, and assess temperature-related run variability.Synthetic bugs are used for the initial end-to-end assessment.Source: A. Evaluation Goals · p. 3

B. Evaluation Metrics and Setup

• The study uses five synthetic instances across five difficulty tiers, a 23-instance SWE-bench Lite dev subset, and 77 test instances whose gold patches are shorter than 600 characters.The synthetic set is used to evaluate all eight configurations before the more expensive real-world experiments.Source: B. Evaluation Metrics and Setup · p. 3
• Eight configurations vary in topology and/or agent definitions, covering chain, star, tree, and hybrid patterns; the chain baseline links localization, patch generation, and validation agents.The seven alternatives draw on leaderboard systems, including two topology-inspired configurations and five reconstructions; the table reports results with qwen3.5:9b.Source: B. Evaluation Metrics and Setup · p. 3, Table I · p. 3

C. Preliminary Results

• 12 of 77 SWE-bench Lite test instances (15.58%) were resolved by chain; on development, chain and hyperagent_star each resolved two, with chain using 39.78% less time and 47.32% fewer tokens.The results use the same qwen3.5:9b model across configurations.Source: C. Preliminary Results · p. 4
• On the synthetic set, openhands_star resolved all five instances and had the shortest wall-clock time and lowest token usage among the eight configurations.On the 23-instance development subset, chain and hyperagent_star each resolved two issues.Source: C. Preliminary Results · p. 4
• At temperature 0, chain resolved the same three synthetic instances in all five runs; at temperature 1, resolved counts ranged from two to four and runtime and token use varied more.The authors recommend repeated evaluation for configurations using nonzero temperature.Source: C. Preliminary Results · p. 4

1) Framework Extensibility:

• A translation workflow was modeled and evaluated through the UI using task-specific components, without modifying the framework core.It reads a text file, translates it, and evaluates the output with BLEU against a reference.Source: 1) Framework Extensibility · p. 4

2) Limitations of the Experiments:

• Reconstructed APR configurations approximate the original agents, prompts, tools, and communication structures, but not their exact runtime behavior.Results therefore characterize OpenDraco instantiations, and configuration differences cannot be attributed to topology alone.Source: 2) Limitations of the Experiments · p. 4
• All agents ran sequentially on a 6 GB VRAM GPU, so reported wall-clock times omit potential parallelization gains.Experiments used only the locally hosted qwen3.5:9b, limiting generalizability to other models.Source: 2) Limitations of the Experiments · p. 4
• Most configurations ran once, and only three were evaluated on real-world instances, with chain alone tested on the larger subset.These choices limit conclusions about run-to-run variability and configuration differences.Source: 2) Limitations of the Experiments · p. 4

• OpenDraco captures agents, prompts, tools, models, parameters, and communication topology in versioned, shareable configurations that can be modified and reproduced without framework changes.Source: §V · p. 4
• OpenDraco supports end-to-end comparison of configurations by effectiveness, time, and token usage.Source: §V · p. 4
• Planned work includes runtime topology evolution, more models and repeated runs, expanded catalogs and evaluators, a user study, and sustainability metrics.Source: §V · p. 4