Source-linked AI summary
Software Project Management with LLM-Based Automation: Coordination, Validation, and Governance in Practice
Ronnie de Souza Santos, Cleyton Magalhaes, Italo Santos
TL;DR
Prior software-engineering research has focused mainly on task-level and developer-centered LLM use, leaving limited empirical evidence about how project managers govern and integrate automation. This exploratory case study examines managers in a large software organization and finds that automation reshapes established management work, increasing the need for judgment, validation, and coordination rather than introducing new formal practices.
Problem
Research has paid comparatively limited attention to how project managers govern, validate, and integrate LLM automation in real software projects.
Source: §1 · p. 1Method
An exploratory case study examined managers’ perceptions of LLM automation in existing workflows, without evaluating a specific tool or establishing causal relationships.
Source: §3 · p. 3Results
LLM automation intensifies established management demands, with variability, uneven productivity effects, and reduced visibility increasing reliance on managerial judgment and contextual interpretation.
Source: §5 · p. 7, §6 · pp. 7–8Takeaways & Limitations
Managers’ work remains embedded in established practices but requires greater attention to uncertainty, coordination, learning, and accountability.
Source: §6 · p. 8The study relies on self-reported accounts from ten managers in one bounded organization and does not claim statistical generalization or causal inference.
Source: §3.7 · p. 5
Abstract
from arXiv · showhide
As software engineering has evolved, development environments have increasingly integrated automated support for tasks such as testing, analysis, and code generation, requiring project management to coordinate both human work and automated workflows. In this paper, we investigate how software project managers perceive changes in their work and learning demands associated with LLM-based automation. Motivated by a gap in software engineering research, which has largely focused on task-level and developer-centered uses of LLMs, we adopted an exploratory case study approach to capture managerial perspectives on automation in practice. Based on the experience of software project managers working in a large, multi-project software organization, our analysis indicates that LLM-based automation influences planning, estimation, coordination, monitoring, and governance activities rather than introducing new formal management practices. Participants described LLMs as becoming embedded in everyday project work, producing uneven effects on productivity, increasing the need for review and validation, and reducing visibility into task execution. These effects contribute to greater reliance on managerial judgment and coordination. Learning demands were perceived as experiential and incremental, centered on understanding LLM capabilities and limitations, critically assessing generated artifacts, and guiding responsible use within teams. The findings provide empirical evidence on how software project management work is adapted in contexts where LLM-based automation is integrated into ongoing software development practice.
1 Introduction
The introduction frames LLM-based automation as a project-management concern because managers must coordinate human work alongside automated workflows. It identifies limited empirical evidence on how managers govern and integrate automation, then states the study’s question and contribution.
1 Introduction
- Existing research gives limited attention to how managers govern, validate, and integrate LLM automation in real projects.The study asks how project managers perceive changes to their work and learning demands.Source: §1 · pp. 1–2
- An exploratory case study finds that LLMs affect planning, estimation, coordination, monitoring, and governance rather than creating entirely new management practices.Participants reported uneven productivity effects and greater attention to review, validation, accountability, and managerial judgment.Source: §1 · p. 2
- The paper characterizes trustworthy adoption as relying on managerial judgment and everyday practices for validation, responsible use, accountability, and governance.These practices operate through planning, coordination, validation, and risk management, rather than new formal project-management practices.Source: §1 · p. 2
2 Background
The background presents software management as adaptive, socio-technical work and reviews LLM use across software tasks and coordinated workflows. It identifies limited empirical understanding of how managers operationalize and govern those workflows.
The Role of Software Managers
- Software managers handle planning, estimation, tracking, quality control, and coordination, relying on adaptive judgment across technical and organizational boundaries.Agile and hybrid settings also require prioritization and alignment across teams and stakeholders.Source: §2 · p. 2
LLMs in Software Engineering Practice
- LLMs support activities from requirements and coding to testing, estimation, reporting, and coordination, while humans retain interpretation, integration, and validation responsibilities.Prior work describes LLMs mainly as assistive resources, with comparatively less attention to software engineering automation.Source: §2 · p. 2
LLMs for Software Engineering Automation
- LLM automation spans isolated tasks and coordinated workflows, but generated artifacts raise reliability, evaluation, and control challenges before project integration.The literature offers comparatively little empirical evidence on how managers coordinate, govern, and operationalize these workflows in ongoing projects.Source: §2 · p. 2
3 Methodology
The study uses an exploratory case study to examine managers’ perceptions of LLM automation in a large organization where the tools are already embedded in development work. It combines a qualitative questionnaire and thematic analysis with descriptive statistics, while acknowledging limits on generalization and direct observation.
3 Methodology
- The study examines managerial adaptation and perceived challenges in context, without evaluating a specific tool or establishing causal relationships.The exploratory case study focuses on how LLM automation is incorporated, coordinated, and governed within existing workflows.Source: §3 · p. 3
3.1 Case Study Context
- The case is a large, multi-region organization with about 1,200 employees, more than 70 teams, and at least 70 concurrent projects.LLM tools were already used in code generation, documentation, testing support, and internal knowledge access.Source: §3.1 · p. 3
3.2 Participant Selection
- Ten of 25 invited managers participated, collectively overseeing at least 30 active projects within the organization.The self-selected qualitative sample prioritized managers directly involved in projects influenced by LLM adoption, not statistical representativeness.Source: §3.2 · p. 3
3.3 Pilot
- Two experienced researchers piloted the questionnaire, prompting clearer wording and refinement of concepts and overlapping questions.Data collection took place in January 2026.Source: §3.3 · p. 3
3.4 Data Collection
- Managers’ time constraints led the researchers to adapt the planned interview guide into an asynchronous online questionnaire with open-ended questions.The format reduced opportunities for real-time probing but enabled access to otherwise difficult-to-reach participants.Source: §3.4 · p. 4
3.5 Data Analysis
- Open-ended responses were analyzed thematically, with descriptive statistics used to summarize participant characteristics and selected response frequencies.Statistics supported contextual interpretation rather than inferential claims or generalization beyond the case.Source: §3.5 · p. 4
3.6 Ethics
- The researchers obtained electronic informed consent, collected no personally identifiable information, and analyzed data in aggregate.Participants were told about voluntary participation, anonymity, and their right to withdraw.Source: §3.6 · p. 4
3.7 Threats to Validity
- The findings reflect ten managers’ self-reports in one organizational case and are not statistically generalizable or based on direct observation.The study does not include quantitative measurement or causal inference; its transfer-oriented analysis provides context for assessing applicability elsewhere.Source: §3.7 · p. 5
4 Results
Managers described LLM automation as changing the conditions of project management rather than creating new formal practices. Its effects include uneven productivity, added validation and coordination demands, incremental learning, and case-by-case governance.
4 Results
- Five participants reported always using LLMs, four frequently, and one occasionally; reported uses centered on test generation, code writing or review, and technical problem resolution.The activity counts were 8 mentions for test generation, 7 for code writing/review, and 6 for technical problem resolution; design exploration and requirements clarification had 3 mentions each, and experimental code generation had 1.Source: §4 · p. 5
4.1 Perceived Changes in Software Project Management Practices with LLM Usage
- Planning and estimation became less predictable as productivity varied by task, while managers accounted for review, validation, and integration effort.Participants also described greater coordination needs, reduced visibility into LLM-mediated work, and uncertainty about deadlines, quality, and rework; established management practices remained in use.Source: §4.1 · pp. 5–6
4.2 Learning Demands and Skill Adaptation in LLM-Mediated Software Project Management
- Managers learned mainly through direct tool use, building knowledge of capabilities and limitations and critically evaluating generated outputs.They also adapted planning to include validation and integration effort.Source: §4.2 · p. 6
- Learning also involved guiding responsible use, addressing legal and confidentiality constraints, and extending rather than replacing established managerial skills.Participants mentioned prompt writing, specification files, and configuring automated review tools.Source: §4.2 · p. 6
4.3 Governance, Responsibility, and Managerial Decision Making in LLM-Mediated Projects
- Managers addressed artifact quality, acceptable use, data boundaries, and team responsibility through contextual judgment, generally on a case-by-case basis.Some participants reported not yet needing specific LLM decisions in their projects.Source: §4.3 · p. 6
4.4 Answering RQ
- Managers reported selective changes rather than new formal practices, with greater reliance on judgment, coordination, and contextual interpretation.Learning was incremental and experience-based, while governance was generally handled case by case.Source: §4.4 · pp. 6–7
5 Discussion
The discussion situates LLM use beyond individual tasks, showing how validation and interpretation become project-level coordination and governance concerns. It recommends managing variability, review, and project constraints rather than treating automation solely as a productivity tool.
Comparison with the Literature
- Task-level LLM capabilities translate into project-level effects on planning, estimation, monitoring, and governance as validation and interpretation become coordination challenges.The discussion connects this extension to prior work on task-level and collaborative uses of LLMs [14] [18] [29]; [7] [24] [28].Source: §5 · p. 7
Discussion
- Participants associated LLM use with variable task execution, uneven productivity, and reduced work visibility, intensifying existing managerial judgment rather than creating fundamentally new practices.These patterns are consistent with socio-technical perspectives emphasizing judgment and decisions under partial information.Source: §5 · p. 7
- Learning and governance involve evaluating artifacts, aligning expectations, setting acceptable-use boundaries, and exercising judgment about quality, responsibility, and compliance.These concerns extend beyond individual prompt-writing and tool-proficiency skills.Source: §5 · p. 7
Implications
- The authors recommend adjusting estimates for variability, assigning review responsibilities, accounting for experience differences, and aligning tool use with confidentiality and compliance constraints.They also recommend monitoring both delivery speed and quality because productivity gains may require additional validation effort, and integrating automation through coordination, validation, and governance mechanisms.Source: §5 · p. 7
6 Conclusions and Future Work
The study concludes that LLMs are embedded in existing project management work and change its operating conditions without automating managerial work or introducing new roles. The authors plan broader survey research and training materials for software managers.
Conclusions
- Managers perceived LLMs as embedded in existing project management practice rather than introducing new roles or formal management approaches.Management remains important and is not automated; its conditions increasingly involve uncertainty, coordination, learning, and accountability.Source: §6 · pp. 7–8
Future Work
- The authors plan a survey to assess how identified patterns vary across a broader range of organizational contexts.They also plan training materials based on the study's identified practices, learning needs, and governance challenges.Source: §6 · p. 8