Source-linked AI summary
Tokenomics: Quantifying Where Tokens Are Used in Agentic Software Engineering
Mohamad Salim, Jasmine Latendresse, SayedHassan Khatoonabadi, Emad Shihab
TL;DR
LLM-MA systems can automate software engineering, but their operational efficiency and resource consumption remain poorly understood. This paper analyzes 30 ChatDev execution traces with GPT-5, maps internal phases to SDLC stages, and quantifies token distributions. Code Review is the dominant consumer, while input tokens form the largest share, motivating more efficient collaboration protocols.
Problem
Resource-efficiency patterns in LLM-MA systems remain insufficiently understood specifically for software-development tasks, despite their growing use across the SDLC.
Method
The study analyzes 30 ChatDev execution traces, maps framework-specific phases to universal SDLC stages, and aggregates input, output, and reasoning tokens.
Results
Code Review accounts for an average of 59.4% of tokens across the 30 tasks, making it the largest-consuming development phase.
Takeaways & Limitations
The findings direct future work toward token-efficient collaboration protocols for verification and refinement beyond naive full-context passing.
Takeaways & Limitations
The study uses one LLM-MA system, ChatDev, one GPT-5 reasoning model, and 30 tasks that may not represent all software-development scenarios.
Abstract
from arXiv · showhide
LLM-based Multi-Agent (LLM-MA) systems are increasingly applied to automate complex software engineering tasks such as requirements engineering, code generation, and testing. However, their operational efficiency and resource consumption remain poorly understood, hindering practical adoption due to unpredictable costs and environmental impact. To address this, we conduct an analysis of token consumption patterns in an LLM-MA system within the Software Development Life Cycle (SDLC), aiming to understand where tokens are consumed across distinct software engineering activities. We analyze execution traces from 30 software development tasks performed by the ChatDev framework using a GPT-5 reasoning model, mapping its internal phases to distinct development stages (Design, Coding, Code Completion, Code Review, Testing, and Documentation) to create a standardized evaluation framework. We then quantify and compare token distribution (input, output, reasoning) across these stages. Our preliminary findings show that the iterative Code Review stage accounts for the majority of token consumption for an average of 59.4% of tokens. Furthermore, we observe that input tokens consistently constitute the largest share of consumption for an average of 53.9%, providing empirical evidence for potentially significant inefficiencies in agentic collaboration. Our results suggest that the primary cost of agentic software engineering lies not in initial code generation but in automated refinement and verification. Our novel methodology can help practitioners predict expenses and optimize workflows, and it directs future research toward developing more token-efficient agent collaboration protocols.
1 Introduction
The paper addresses limited knowledge about resource efficiency in software-engineering LLM-MA systems by introducing tokenomics and analyzing token consumption across SDLC stages. It contributes an empirical analysis, a 30-trace dataset, and a replication package.
- Research gap: Prior taxonomies describe token distribution and systemic coordination failures, but do not specifically characterize resource efficiency in software-development contexts.AGENTTAXO introduces communication tax, while MAST identifies issues such as step repetition and incomplete verification.
- Concept and question: Tokenomics denotes the study of operational efficiency and resource consumption in LLM-MA systems.
- Concept and question: The study asks what token consumption patterns LLM-MA systems exhibit for software-development tasks.
- Approach: The analysis maps ChatDev’s internal phases to distinct development stages and examines token distributions across them.ChatDev models a virtual software company in which agent roles collaborate through multi-turn dialogues across the SDLC.
- Contributions: The paper contributes an empirical analysis, a curated dataset of 30 execution traces, and a complete replication package.
2 Study Design
The study analyzes ChatDev as an initial, sequential multi-agent system using diverse software tasks, GPT-5 reasoning agents, and a trace-processing pipeline. The pipeline maps framework-specific phases to universal development stages and aggregates input, output, and reasoning tokens across 30 runs.
- System and tasks: ChatDev was selected because its sequential waterfall chat-chain architecture has distinct phases suitable for mapping to software-development stages.The framework was also chosen for its popularity and open-source status.
- System and tasks: The study executed ChatDev on 30 tasks ranging from simple algorithms to complex applications, using prompts from the ProgramDev Dataset.Reasoning-token consumption ranged from 17,280 to 40,000 across the tasks.
- Model selection: GPT-5 reasoning was used as the backbone model for all agents, with version gpt-5-2025-08-07 and default temperature 1.0.
- Analysis pipeline: The analysis pipeline logs each task’s LLM calls, including prompts, responses, and input, output, and reasoning token counts.
- Analysis pipeline: Phase mapping converts ChatDev-specific phases into universal development stages, after which scripts aggregate token totals across all 30 runs.The mapping is intended to support generalizable analysis across software-engineering LLM-MA frameworks.
3 Study Results
The study examines token consumption across software development stages and token types in ChatDev’s multi-agent workflow. Results show highly uneven usage, with Code Review dominant and distinct tokenomic profiles across activities.
- Analysis dimensions: The analysis focuses on total-token distribution across development stages and the input, output, and reasoning-token ratios within each stage.The results are based on aggregated token data from the study pipeline.
- Token distribution by stage: 59.4% of tokens are consumed by Code Review on average across all 30 tasks.Code Completion averages 26.8% in the 6 tasks where it occurred, while Testing occurred in 12 tasks.
- Token distribution by stage: 20.1% of tokens go to Documentation and 10.3% to Testing, compared with 8.6% for Coding and 2.4% for Design.These averages reflect the reported phase-level distribution across the task set, with phases not necessarily executed in every task.
- Interpretation: The findings indicate that refinement and verification, rather than initial code generation, account for the primary cost of agentic software engineering.The conclusion follows the concentration of tokens in Code Review and other refinement-focused stages.
- Stage-specific token profiles: Coding is output-heavy at 58% output versus 6.9% input, whereas Code Review and Documentation are input-heavy at 51.4% and 80.2% input.The profiles reflect the differing demands of source-code generation versus verification and documentation using existing code as context.
4 Discussion
The preliminary cost map shows token use concentrated in iterative Code Review, where full-context conversational refinement creates substantial resource demands. These findings motivate cost-aware process decisions and more efficient verification protocols.
- Code Review carries an immense token cost because agents repeatedly pass full code context back and forth during refinement.The paper interprets this pattern as the “Cost of Conversation.”
- Verification protocols may be highly inefficient when iterative dialogue consumes vast resources for minor corrections.The paper connects this concern to verification and step-repetition failures identified by MAST.
- Projects focused on refactoring and debugging existing code are expected to have a different cost structure from greenfield projects with heavy initial coding.Refactoring and debugging are described as dominated by the input-heavy Code Review cycle.
- A human-in-the-loop checkpoint before Code Review could prevent costly iterative loops and improve economic and computational efficiency.
- More token-efficient verification protocols and standardized evaluation frameworks are needed to compare the efficiency of different LLM-MA architectures.The proposed framework would translate framework-specific operations into universal software engineering activities.
5 Threats to Validity
The findings are constrained by the systems, models, tasks, and development stages represented in the study. Infrequently triggered stages provide especially limited evidence for generalizing their tokenomic profiles.
- The analysis covers one LLM-MA system, ChatDev, and one GPT-5 Reasoning Model, so patterns may differ with other architectures or models.
- The 30 diverse tasks may not represent all software development scenarios and complexities.The dataset size reflects the lack of public, large-scale benchmarks for SE-specific agent traces.
- Code Completion (n=6) and Testing (n=12) were triggered infrequently, limiting the representativeness of conclusions about those stages.
- The proposed mapping of ChatDev’s internal phases to software development stages is also identified as a validity consideration.
6 Conclusion and Future Work
The study finds that token use is concentrated in iterative code review and that input tokens form the bulk of usage. It establishes an initial basis for cost prediction while outlining broader validation and benchmarking needs.
- Token costs are overwhelmingly concentrated in the iterative, conversational phase of Code Review.
- Input tokens form the bulk of token usage and represent a key area for future optimization.The paper characterizes these input tokens as the “communication tax.”
- Future work: Expanding the dataset with more tasks is proposed to improve generalizability.
- Future work: Extending analysis to other LLMs is proposed to assess model-dependent tokenomics.
- Future work: Extending analysis to other LLM-MA systems would enable comparison of how architectural differences affect tokenomics.
- Future work: The development-stage mapping should be further developed and validated as a robust, universal framework for benchmarking SE agent efficiency.