PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Human-Bot Collaborative Software Architecting with ChatGPT

Aakash Ahmad, Muhammad Waseem, Peng Liang, Mahdi Fehmideh, Mst Shamima Aktar, Tommi Mikkonen

arXiv:2302.14600v1cs.SEcs.AI

TL;DR

Research had not yet investigated ChatGPT’s role as a DevBot in software architecting. The paper examines it in a scenario-based case study and reports that ChatGPT performed several architecting activities, while noting that response variation can affect consistency.

  • Problem

    Although intelligent architecting could benefit ACSE, research had not investigated ChatGPT’s role as a DevBot in the architecting process.

    Source: §I · p. 2

  • Method

    The authors used a process-centric, scenario-based case study in which a novice architect supplied ChatGPT with a natural-language architecture story for CampusBike analysis, synthesis, and evaluation.

    Source: §I · p. 2, B. Research Method · p. 3, §III · p. 3

  • Results

    Preliminary results show ChatGPT processing an architecture story to articulate requirements, specify models, recommend and apply tactics and patterns, and develop architecture-evaluation scenarios.

    Source: §I · p. 2

  • Takeaways & Limitations

    The study presents ChatGPT’s DevBot role as a potential way to assist and empower an architect-led process through collaboration with a human.

    Source: §VI · p. 6

    ChatGPT may give varied responses to identical queries, potentially reducing the consistency of architectural analysis, synthesis, and evaluation.

    Source: A. Socio-Technical Issues of ChatGPT… · p. 5

Abstract

from arXiv · show

Architecting software-intensive systems can be a complex process. It deals with the daunting tasks of unifying stakeholders' perspectives, designers' intellect, tool-based automation, pattern-driven reuse, and so on, to sketch a blueprint that guides software implementation and evaluation. Despite its benefits, architecture-centric software engineering (ACSE) inherits a multitude of challenges. ACSE challenges could stem from a lack of standardized processes, socio-technical limitations, and scarcity of human expertise etc. that can impede the development of existing and emergent classes of software (e.g., IoTs, blockchain, quantum systems). Software Development Bots (DevBots) trained on large language models can help synergise architects' knowledge with artificially intelligent decision support to enable rapid architecting in a human-bot collaborative ACSE. An emerging solution to enable this collaboration is ChatGPT, a disruptive technology not primarily introduced for software engineering, but is capable of articulating and refining architectural artifacts based on natural language processing. We detail a case study that involves collaboration between a novice software architect and ChatGPT for architectural analysis, synthesis, and evaluation of a services-driven software application. Preliminary results indicate that ChatGPT can mimic an architect's role to support and often lead ACSE, however; it requires human oversight and decision support for collaborative architecting. Future research focuses on harnessing empirical evidence about architects' productivity and exploring socio-technical aspects of architecting with ChatGPT to tackle emerging and futuristic challenges of ACSE.

I. INTRODUCTION

The introduction frames ACSE as valuable but challenging, and proposes investigating whether ChatGPT can support collaborative architecting through conversation between an architect and a bot. A preliminary case study examines ChatGPT’s architectural analysis, synthesis, and evaluation capabilities while identifying human oversight and socio-technical concerns as important considerations.

Introduction

  • ACSE combines architectural knowledge, languages, tools, and human decisions to guide software development, but faces challenges including stakeholder alignment, architectural drift, technical debt, and limited expertise,,.The introduction describes the unsupported engineer as a ‘lonesome architect’ who needs non-intrusive process and tool support.Source: §I · p. 1
  • Although AI-based ACSE research targets decision support, the authors identify no prior research on collaborative architecting in which a bot leads tasks under an architect’s conversational supervision,,,.Such collaboration could let architects delegate tasks to the bot and supervise it through natural-language dialogue.Source: §I · p. 1
  • The study preliminarily investigates whether ChatGPT can use an architect-conversed architecture story to analyse, synthesise, and evaluate software architecture,.Although not developed specifically for software engineering, ChatGPT can generate architectural requirements, UML scripts, source-code libraries, and test cases [11], [12].Source: §I · pp. 1–2
  • Using a process-centric approach and scenario-based method for microservices, the study reports that ChatGPT can articulate requirements and models, recommend or apply tactics and patterns, and develop evaluation scenarios,.These capabilities are presented as preliminary results of architectural analysis, synthesis, and evaluation.Source: §I · p. 2
  • The case study investigates human-bot architecting that combines ChatGPT outputs with architects’ decisions to automate ACSE,.The authors present it as a preliminary investigation.Source: §I · p. 2
  • The study aims to identify ChatGPT-assisted ACSE’s potential and perils, including ethical, governance, and socio-technical constraints, and to ground future evidence on capabilities and architect productivity.The authors also propose guidelines for practitioners to experiment with delegating tedious ACSE tasks and call for research on emergent and futuristic software.Source: §I · p. 2

II. RESEARCH CONTEXT AND METHOD · A. Human-Bot Collaborative Architecting

The section frames software architecting as a systematic process for turning requirements into architecture models and evaluating them, then motivates collaboration between architects and DevBots. It presents ChatGPT as a conversational bot that can help create architectural artifacts under human supervision.

A. Human-Bot Collaborative Architecting

  • Software architecture abstracts implementation complexity into component-and-connector blueprints, and architecture-centric approaches reuse patterns, styles, languages, and frameworks,.The section introduces these concepts as context for the paper’s research method.Source: A. Human-Bot Collaborative Architecting · p. 2, §II · p. 2
  • The architecting process separates concerns through analysis, synthesis, and evaluation, with evaluation focusing on scenarios,.The process described in is derived from five industrial projects.Source: A. Human-Bot Collaborative Architecting · p. 2
  • Architects identify Architecturally Significant Requirements (ASRs), map them to implementations through models in UML or ADLs, and evaluate those models with methods such as SAAM or ATAM,.The requirements express required functionality and desired software quality.Source: A. Human-Bot Collaborative Architecting · p. 2
  • DevBots are AI-driven conversational agents or recommender systems that assist software engineers through automation or intelligence, but architecting use has been limited to answering questions or recommending responses to erosion and maintenance issues,.The section identifies no prior research or solution demonstrating a DevBot-integrated architecting process.Source: A. Human-Bot Collaborative Architecting · pp. 2–3
  • Human-bot architecting is proposed to combine architects’ rationale with bots’ automated architecting, extending beyond question-answering and recommendations.The collaboration could help novice architects specify requirements in natural language for translation into ASRs, architecture models, and evaluation scenarios.Source: A. Human-Bot Collaborative Architecting · p. 3
  • ChatGPT, an LLM-based conversational bot, can dialogue with an architect to lead the creation of architectural artifacts under human supervision.The paper positions this collaboration as a way to enrich the architecting process.Source: A. Human-Bot Collaborative Architecting · p. 3

B. Research Method

The method has three phases: develop a natural-language architecture story, collaborate with ChatGPT on architectural analysis, synthesis, and evaluation, then extend the work with empirical validation. The first two phases structure the study’s exploration of ChatGPT’s role in collaborative software architecting.

Phase 1 - Developing the Architecture Story

  • Phase 1 creates a natural-language architecture story from domain scenarios, describing core functionality, desired quality attributes, and constraints, then feeds it to ChatGPT as pre-processing.The story provides the foundation for the architecting process.Source: B. Research Method · p. 3

Architectural analysis

  • Architectural analysis articulates ASRs through ChatGPT recommendations, architect-specified requirements, or ongoing dialogue to refine requirements, following activities adopted from.Source: B. Research Method · p. 3

Architectural synthesis

  • Architectural synthesis consolidates ASRs into a model showing structural decomposition and runtime scenarios; the study uses UML and refines the architecture with reuse tactics and patterns.UML was selected for its documentation, ease of use, diagram diversity, tool support, and adoption.Source: B. Research Method · p. 3

Architectural evaluation

  • Architectural evaluation incrementally validates all or part of the synthesized architecture against ASR scenarios, with SAAM used to supervise ChatGPT.Source: B. Research Method · p. 3

Phase 3 - Conducting the Empirical Validations

  • Phase 3 extends the initial phases with empirical validation; future work will develop empirically grounded guidelines on socio-technical issues in ChatGPT-driven architecting.The study’s current scope explores ChatGPT’s role in collaborative architecting, with empirical guidelines discussed as future work.Source: B. Research Method · p. 3

III. CASE STUDY ON COLLABORATIVE ARCHITECTING

The section illustrates collaborative architecting through a CampusBike case study led by a motivated novice engineer, with a scenario focused on eco-friendly campus bicycle rentals.

Case study setup

  • The case study develops CampusBike, a browser- and app-accessible service for visitors to register, find bikes, reserve them, pay, and view usage reports.The novice architect has working knowledge of software design and implementation.Source: §III · p. 3

Scenario example

  • The scenario lets potential bikers find nearby bikes within 500 meters and reserve one for a specified time after payment.The campus administration introduces the pay-per-use bicycle service to support a Green Campus and enhance mobility.Source: §III · p. 3

A. Formulating the Architecture Story

The architecture story is a natural-language narration of the intended software’s core functionality and constraints, developed from analysis of its domain and scenarios. The architect feeds this story to ChatGPT as the foundation for architectural analysis.

A. Formulating the Architecture Story

B. Architectural Analysis

Architectural analysis used ChatGPT to propose ASRs for CampusBike, which the architect refined through dialogue into requirements covering functionality, quality, and constraints.

B. Architectural Analysis

  • ChatGPT outlined ASRs and constraints when queried, while the architect refined its proposals by adding, removing, or modifying requirements.ASRs specified required functionality, desired quality, and constraints for CampusBike.Source: B. Architectural Analysis · p. 4
  • The architect refined “Reserve Bike” to specify nearby-bike viewing, reservation within 90 seconds, reservation-token encryption, and data minimization on registration.The constraint was identified as a GDPR requirement.Source: B. Architectural Analysis · p. 4
  • ASRs were iteratively refined through architect–ChatGPT dialogue into a final list [15], as illustrated in Figure 4.Source: B. Architectural Analysis · p. 4, Figure 4 · p. 4

C. Architectural Synthesis · D. Architectural Evaluation

The sections describe UML-based synthesis of CampusBike’s architecture and its evaluation with SAAM. ChatGPT supports refinement and scenario-based assessment, while the architect provides instructions and oversight.

C. Architectural Synthesis

  • The architecture is modeled in UML, using component diagrams for the overall architecture and class diagrams for fine-grained design.Source: C. Architectural Synthesis · p. 4
  • The class diagram applies the singleton pattern to ‘UserLogin’ to restrict users to a single login session across devices.Source: C. Architectural Synthesis · p. 4
  • The design applies caching to ‘ViewBikes’ and a data-minimization constraint to ‘User Location’.Source: C. Architectural Synthesis · p. 4
  • The architect instructs ChatGPT to create the UML class-diagram script, and their dialogue enables application of the pattern, tactic, and constraint.Source: C. Architectural Synthesis · p. 4

D. Architectural Evaluation

  • SAAM evaluates whether the synthesized architecture satisfies ASRs and constraints, producing a report on functionality, quality, and constraints.The evaluation is illustrated using the ‘View Bike’ component.Source: D. Architectural Evaluation · p. 4
  • For ‘View Bike’, ChatGPT presents SAAM scenarios for the component individually and in interactions with other components.Source: D. Architectural Evaluation · p. 4

IV. RELATED WORK · A. AI in Software Engineering and Architecting · B. ChatGPT Assisted Software Engineering

Prior work examines AI’s role in software engineering and architecting, while emerging ChatGPT studies address education, programming, testing, and debugging. The paper identifies conversational, human-bot collaboration for architecture-centric software engineering as an unexplored focus.

IV. RELATED WORK

  • Related work reviews AI applications in software engineering and architecting alongside ChatGPT’s role in software development.Source: §IV · p. 5

A. AI in Software Engineering and Architecting

  • AI for software engineering and software engineering for AI are distinct dimensions, with AI for SE urged to balance machine intelligence and human intellect.This intelligence equilibrium is proposed for processes, patterns, and tools, including in emergent software such as blockchain and quantum applications.Source: A. AI in Software Engineering and… · p. 5
  • Prior work identifies needs for AI in automating complex SE activities and analyzing data, while machine learning has been proposed to mitigate architecture degradation.Barenkamp et al. identify debugging, pattern discovery from big data, and evaluation of data in neural and software-defined networks; Herold et al. propose an architecture-focused conceptual framework.Source: A. AI in Software Engineering and… · p. 5

B. ChatGPT Assisted Software Engineering

  • ChatGPT is described as a capable chatbot but an unexplored prospect for software development, with studies focused on education, programming, and testing [17],, [11],,,, [12].Source: B. ChatGPT Assisted Software Engineering · p. 5
  • Conversational use of ChatGPT has supported programming tasks, with reported human oversight and intervention alongside well-scripted solutions and reduced developer time and effort [9],.Examples include an online behavioral task implemented with HTML, CSS, and JavaScript [9] and incremental dialog for genetic programming to solve the traveling salesman problem.Source: B. ChatGPT Assisted Software Engineering · p. 5
  • For automated bug fixing, ChatGPT offers software testers dialogue for incrementally identifying and fixing bugs, unlike the status quo of automated techniques [12],.Source: B. ChatGPT Assisted Software Engineering · p. 5
  • The reviewed literature contains no research or development on ChatGPT-led or supported conversational ACSE; this study instead focuses on architecture-centric development and proposes human-bot collaborative architecting [12],.The proposed collaboration combines architects’ knowledge and supervision with bot capabilities to architect software-intensive systems and services.Source: B. ChatGPT Assisted Software Engineering · p. 5

V. DISCUSSION AND VALIDITY THREATS · A. Socio-Technical Issues of ChatGPT in ACSE

The discussion identifies socio-technical concerns in collaborative architecting with ChatGPT, including inconsistent responses, ethical and intellectual-property risks, and biased architectural recommendations. The authors call for dedicated research and emphasize architect oversight.

V. DISCUSSION AND VALIDITY THREATS

  • The discussion covers socio-technical aspects of collaborative architecting and potential validity threats.Source: §V · p. 5

A. Socio-Technical Issues of ChatGPT in ACSE

  • The authors frame these concerns as social issues and technical limitations, and say systematic investigation requires dedicated research.Source: A. Socio-Technical Issues of ChatGPT… · p. 5
  • ChatGPT can vary its recommendations or architectural artifacts for the same query, potentially undermining consistency across architecture analysis, synthesis, and evaluation.The authors suggest iterative dialogue and architect oversight to refine outputs and keep artifacts consistent and coherent.Source: A. Socio-Technical Issues of ChatGPT… · p. 5
  • ChatGPT-generated specifications, scripts, or code can raise ethical, copyright, and intellectual-property concerns, while location-sensing components may risk privacy and regulatory noncompliance.The authors say architects must vigilantly ensure generated architectures respect ethics, intellectual-property rights, and guidelines such as GDPR and CCPA.Source: A. Socio-Technical Issues of ChatGPT… · pp. 5–6
  • Output bias may arise from user input, training data, or algorithms and can favor widely adopted architectural approaches over contextually optimal choices.Source: A. Socio-Technical Issues of ChatGPT… · p. 6
  • Bias can affect recommendations about notations, tactics, patterns, or styles, as well as design decisions and validation scenarios, producing sub-optimal ACSE artifacts.Source: A. Socio-Technical Issues of ChatGPT… · p. 6

B. Threats to the Validity

The study identifies internal, external, and conclusion-validity limitations and notes that further work is needed to strengthen methodological rigor and generalizability.

Threats to validity

  • Future work can minimize validity threats to improve methodological rigor and generalization.Validity threats are limitations, constraints, or potential flaws that may affect study generalization, replicability, and validity.Source: B. Threats to the Validity · p. 6

Internal validity

  • The study used a systematic approach, a well-known architecting process, an architecture evaluation method, and incremental case-study architecting, but other processes and evaluation methods remain untested.The authors say more work is needed to determine whether the study can be validated using different architecting processes or evaluation methods.Source: B. Threats to the Validity · p. 6

External validity

  • A single moderately complex case study limits generalizability across cross-organizational development, mission-critical software, and novice or experienced engineers.Future work will engage architecting teams and analyze their feedback to assess external validity.Source: B. Threats to the Validity · p. 6

Conclusion validity

  • A three-step process and scenario-based case study supported fine-grained architecting and validation, but productivity and ChatGPT efficacy need testing across multiple case studies, diverse teams, and real contexts.The authors identify these as conclusion-validity concerns requiring further experimentation.Source: B. Threats to the Validity · p. 6

VI. CONCLUSIONS AND FUTURE RESEARCH

The paper examines ChatGPT’s potential and perils as a DevBot in human-bot collaborative architecting, advocating AI that enriches software-engineering processes. It calls for empirical validation and further study of architects’ experiences, including in quantum software engineering.

VI. CONCLUSIONS AND FUTURE RESEARCH

  • The study examines ChatGPT’s underexplored role as a DevBot that can generate textual artifacts and assist an architect leading collaborative ACSE.The paper characterizes ChatGPT as a disruptive conversational bot that produces recommendations, scripts, and source code.Source: §VI · p. 6
  • Human-bot collaboration should enrich AI-supported software-engineering processes, while accounting for response variability, ethics, human oversight, and legal and socio-technical issues.The paper calls for evidence-based validation of effects on engineer productivity, process optimization, and support for novice developers tackling complex and emergent software.Source: §VI · p. 6
  • Future research will explore architects’ feedback and ChatGPT-assisted development of software services for quantum computing systems, where expertise in software engineering and quantum physics is scarce.The planned inquiry addresses a field facing a lack of human expertise to combine software-engineering skills with quantum-physics knowledge.Source: §VI · p. 6
  • Planned controlled experiments with diverse software teams will document architects’ responses and use interviews or documents to assess usefulness, rigor, acceptance, productivity impact, and potential perils.The teams will vary in geography, expertise, experience, and software-system class.Source: §VI · p. 6

Related papers

Loading 2302.14600v1…