PaperlayerView exact revision on arXiv

Source-linked AI summary

LLMs for Generation of Architectural Components: An Exploratory Empirical Study in the Serverless World

Shrikara Arun, Meghana Tedla, Karthik Vaidhyanathan

arXiv:2502.02539v1cs.SE

TL;DR

LLM code generation has focused on snippets, leaving architectural-component generation unexplored. This study masks serverless functions in open-source repositories and regenerates them with LLMs using varying context. LLMs often fail to produce fully functional functions autonomously, while minimal human intervention significantly improves functionality.

  • Problem

    LLM code generation has focused on low-level snippets, leaving generation of software architecture components unexplored.

    Source: §I · p. 1

  • Method

    The study masks serverless functions in selected open-source repositories and uses LLMs with varying prompt context to regenerate them, evaluating functionality and code quality.

    Source: §I · p. 1

  • Results

    LLMs often fail to generate fully functional serverless functions autonomously, but minimal human intervention significantly improves test passing; Type 3 prompts achieved 73% of tests passed for the entire codebase.

    Source: §V · pp. 6–7, §VII · p. 11

  • Takeaways & Limitations

    In this study, generated serverless functions can serve as a starting point because minimal human intervention improves their functionality.

    Source: §VII · p. 11

    Model choices and generation parameters for summarization and function generation threaten external validity, despite systematic leaderboard-based selection.

    Source: E. Threats to Validity · p. 10

Abstract

from arXiv · show

Recently, the exponential growth in capability and pervasiveness of Large Language Models (LLMs) has led to significant work done in the field of code generation. However, this generation has been limited to code snippets. Going one step further, our desideratum is to automatically generate architectural components. This would not only speed up development time, but would also enable us to eventually completely skip the development phase, moving directly from design decisions to deployment. To this end, we conduct an exploratory study on the capability of LLMs to generate architectural components for Functions as a Service (FaaS), commonly known as serverless functions. The small size of their architectural components make this architectural style amenable for generation using current LLMs compared to other styles like monoliths and microservices. We perform the study by systematically selecting open source serverless repositories, masking a serverless function and utilizing state of the art LLMs provided with varying levels of context information about the overall system to generate the masked function. We evaluate correctness through existing tests present in the repositories and use metrics from the Software Engineering (SE) and Natural Language Processing (NLP) domains to evaluate code quality and the degree of similarity between human and LLM generated code respectively. Along with our findings, we also present a discussion on the path forward for using GenAI in architectural component generation.

I. INTRODUCTION

Architectural-component generation with LLMs remains unexplored beyond low-level code snippets. This study explores generating serverless functions, a small architectural unit, and evaluates them for functionality and code quality.

I. INTRODUCTION

  • LLM code generation has focused on low-level snippets, leaving generation of software architecture components unexplored.Prior approaches using ADLs and DSLs have had limited practical application due to steep learning curves and other barriers.Source: §I · p. 1
  • The study masks and regenerates serverless functions using three prompt types, four repositories, and five LLMs, evaluating 145 generated functions with and without human intervention.Functionality and code quality are the evaluation targets.Source: §I · p. 1

A. Function-as-a-Service (FaaS) (Serverless Functions)

  • FaaS functions are fine-grained, event-driven units whose infrastructure and execution environment are managed by the cloud provider.They are ephemeral and elastic, running when triggered and scaling automatically with load.Source: A. Function-as-a-Service (FaaS)… · p. 2

B. Large Language Models and Code Generation

LLMs generate text token by token, and prompting can supply task context or examples without updating model parameters. Prior architecture-generation work largely uses ADLs and model transformations, while LLM code generation has focused on snippets rather than architectural components.

B. Large Language Models and Code Generation

C. Zero-Shot Prompting and Few-Shot Prompting

  • Zero-shot prompts request a function from supplied context, while few-shot prompts also provide repository examples and descriptions.Few-shot prompting is also called in-context learning.Source: C. Zero-Shot Prompting and Few-Shot… · p. 2

III. RELATED WORK

  • Prior ADL and model-transformation approaches include serverless deployment-code generation, but still require developers to write application logic.Other architecture-generation work covers Go, cyber-physical systems, robotics, and microservices.Source: §III · p. 2
  • LLM code-generation tools target snippets such as classes or methods, whereas this study evaluates LLM generation of serverless architectural components for functionality, code quality, and similarity to human-written code.Prior serverless GenAI work considered selecting an architectural pattern rather than generating its components.Source: §III · pp. 2–3

IV. STUDY DESIGN

The study evaluates LLM generation of FaaS architectural components for functional correctness and code quality. It masks an existing function in a serverless repository and asks an LLM to recreate it using different amounts and types of context.

A. Goal

  • The study evaluates LLM effectiveness for generating FaaS architectural components in terms of functional correctness and code quality.The stated perspective is that of software architects and developers.Source: A. Goal · p. 3

B. Research Questions

  • RQ1 asks whether LLMs can generate functional serverless functions with and without minimal human intervention; RQ2 compares generated functions’ code quality and similarity to human-written code.For RQ1.2, the study measures tests both before and after fixing minor errors.Source: B. Research Questions · p. 3

C. Experiment Workflow

  • The experiment workflow includes function masking as a stage for generating architectural components.The subsection introduces the stages of the workflow shown in Figure 1.Source: C. Experiment Workflow · p. 3

1) Function Masking:

  • Function masking adapts masked-language modeling by replacing one repository function and asking an LLM to recreate it using context from the repository.The study varies the types and amounts of context provided.Source: 1) Function Masking · p. 3

2) LLM Selection:

The study uses a separate model for codebase summarization and function generation, and selects repositories with tests suitable for evaluation. Four repositories were ultimately selected.

2) LLM Selection:

  • Gemini-1.5-Pro was selected for codebase summarization, while code-generation models were chosen from the top ten of EvalPlus at study time.The authors selected distinct models for summarization and generation because the tasks differ and to avoid propagating a single model’s biases.Source: 2) LLM Selection · p. 3, Table I · p. 3

3) Repository Selection:

  • Repository selection prioritized popular, non-archived projects with substantial tests and coverage, leading to four repositories.Repositories were screened for meaningful tests and serverless functions with test coverage; selection also sought JavaScript, TypeScript, and Python representation.Source: 3) Repository Selection · p. 4, Table II · p. 4

4) Function Selection:

The study selects ten test-covered functions across four repositories, then supplies either README-level or target-excluding codebase context for generation.

4) Function Selection:

  • Up to three functions per repository were considered, prioritizing the highest statement coverage and using source lines of code as a tiebreaker; ten functions were selected across four repositories.The selected functions had associated tests to support evaluation.Source: 4) Function Selection · p. 4

5) Context Selection:

  • README context provides a concise overview but often lacks implementation details, testing generation with minimal context.It includes repository descriptions, functionality, and usage instructions.Source: 5) Context Selection · p. 4
  • Codebase-summary context excludes the masked function and uses Gemini-1.5-Pro to summarize architecture, functionality, other functions, paths, and code.The summary is structured with terms such as “components,” “connectors,” and “relationships.”Source: 5) Context Selection · p. 4
  • The masked function’s description is generated from its code and later included in the generation prompt.Gemini-1.5-Pro produces the description.Source: 5) Context Selection · pp. 4–5

6) Generation Methods:

Three generation prompts vary the repository context and whether examples accompany it. The study also checks repeat-generation similarity before evaluating 145 generated functions.

6) Generation Methods:

  • Zero-shot README prompting supplies the repository README and masked-function description, without examples of other serverless functions.The model is asked to generate the function code from that context.Source: 6) Generation Methods · p. 5
  • Zero-shot codebase-summary prompting supplies architectural information and the masked-function description, but no examples of other functions.The prompt uses the codebase summary as context.Source: 6) Generation Methods · p. 5
  • Few-shot codebase-summary prompting combines architectural context with descriptions and code from other serverless functions as in-context examples.The examples guide generation of the masked function from its description without modifying model parameters.Source: 6) Generation Methods · p. 5
  • 145 serverless functions were generated for evaluation after repeated outputs for each model were found to be quite similar by CodeBLEU.The consistency check compared multiple generations using the same context because LLM outputs can vary between runs.Source: 6) Generation Methods · p. 5

7) Evaluation Metrics:

The evaluation combines repository tests, code-quality metrics, and CodeBLEU, distinguishing functional correctness from code similarity and quality.

7) Evaluation Metrics:

  • Generated functions were evaluated with existing repository tests, recording passing and failing cases for the whole codebase and selected functions after regeneration.The evaluation also compared the original and generated code.Source: 7) Evaluation Metrics · pp. 5–6
  • Unmodified generations were tested after manual code-only cleaning, while a separate evaluation fixed frequently observed errors considered solvable with minimal intervention.Researchers spent an average of 15 minutes fixing errors per function before recording updated test counts; Table III identifies the errors addressed.Source: 7) Evaluation Metrics · p. 6, Table III · p. 6
  • Code quality was measured with SLOC, Halstead Volume, cyclomatic complexity, and cognitive complexity to assess size, control-flow complexity, testability, and understandability.The chosen metrics also reflect reading difficulty and testing difficulty.Source: 7) Evaluation Metrics · p. 6
  • CodeBLEU measures similarity between generated and human-written functions, while repository tests assess semantic correctness because high similarity can coexist with functional errors.Examples include incorrect imports or exchanged variables that may not substantially lower CodeBLEU.Source: 7) Evaluation Metrics · p. 6

V. RESULTS

Type 3 prompts had the highest average codebase test pass rate, and larger models outperformed smaller ones without human intervention. Generated functions differed from human-written code in code-quality metrics and CodeBLEU similarity.

V. RESULTS

  • 73% of codebase tests passed on average with Type 3 prompts, which had the highest average pass rate; Type 2 prompts also outperformed Type 1 for function tests.Table IV compares codebase and individual-function test pass rates across models and prompt types.Source: §V · p. 6, Table IV · p. 7
  • Type 3 examples more closely matched the original function than Type 1 examples, while the Type 2 version replaced hard-coded strings with a variable but still lacked complete logic.Figure 3(d) shows findTag() returning the tag object, whereas the Artigenz-Coder-DS-6.7B Type 3 output is a stub.Source: §V · pp. 6–7, Figure 3 · p. 8
  • Over 81% of codebase tests and 13% of individual-function tests passed for DeepSeek-Coder-V2 with Type 3 prompts without human intervention, outperforming smaller models across prompt types.GPT-4 passed 79% of codebase tests and 10% of function tests with Type 3 prompts.Source: §V · p. 7
  • JavaScript-based repositories achieved higher pass rates than repositories in the other studied languages.Source: §V · p. 7
  • After minimal human intervention, DeepSeek-Coder-V2 with Type 3 prompts achieved over 90% success on codebase tests and 71% on individual-function tests.Smaller models' codebase test pass rates rose from 67% to 74% for Artigenz-Coder-DS-6.7B and from 69% to 80% for CodeQwen1.5-7B-Chat.Source: §V · p. 7
  • Generated functions generally had lower SLOC, cyclomatic complexity, and Halstead volume than originals, while cognitive complexity varied by model.CodeQwen1.5-7B-Chat and GPT-4 had higher cognitive complexity than originals; GPT-3.5-Turbo generated the most understandable code.Source: §V · p. 8
  • More prompt context produced no appreciable change in average SLOC or Halstead volume, but reduced average cyclomatic and cognitive complexity.The lowest values for both complexity measures came from Few Shot Prompt with Codebase Summarization.Source: §V · p. 8
  • Type 3 prompts made every model's functions more CodeBLEU-similar to human-written functions, and DeepSeek-Coder-V2 had the highest similarity score.In many cases, Type 2 codebase summaries reduced CodeBLEU scores versus Type 1 even though Type 2 produced functions that passed more tests.Source: §V · p. 8, Figure 4 · p. 8

VI. DISCUSSION

The discussion covers lessons learned from the study and threats to validity.

VI. DISCUSSION

  • The discussion addresses lessons learned and threats to validity.Source: §VI · p. 8

A. Can LLMs generate functional serverless functions? (RQ1)

LLMs can generate functional serverless functions, but performance varies by model and prompt and remains below human capability. Comprehensive repository context can improve generation, while human refinement remains important.

A. Can LLMs generate functional serverless functions? (RQ1)

  • Type 3 prompts produce the most functional functions; their comprehensive repository context particularly benefits larger models.Smaller models tend to generalize more, reducing the precision needed for function generation.Source: A. Can LLMs generate functional… · p. 8
  • DeepSeek-Coder-V2 performs best overall, while GPT-4 adheres to function requirements better than GPT-3.5-Turbo.The overall comparison spans multiple functions and prompt types without human intervention.Source: A. Can LLMs generate functional… · p. 9
  • Smaller models require substantial corrections, whereas larger models need relatively minor adjustments; GPT-3.5-Turbo can be easier to refine than GPT-4.GPT-3.5-Turbo’s simpler initial outputs can be easier to refine despite often lacking full functionality; GPT-4 can be harder to adjust with minor changes.Source: A. Can LLMs generate functional… · p. 9
  • JavaScript-based repositories achieve higher pass rates than other repositories in the study.The authors suggest this may relate to JavaScript’s prevalence in serverless functions.Source: A. Can LLMs generate functional… · p. 9
  • Generated functions still fall short of fully functional human-written components, so the authors emphasize keeping a human in the loop.Human refinement is needed to address functionality, complex logic, and practical applicability.Source: A. Can LLMs generate functional… · p. 9
  • LLMs generate functional serverless functions with varying performance, but not at human capability.The study concludes that they can aid architectural component generation.Source: A. Can LLMs generate functional… · pp. 8–9

B. How does the code quality of LLM-written serverless functions compare to human-written code? (RQ2)

LLMs can produce simpler, more understandable serverless code, especially with larger models and architectural and functional context, but code quality does not guarantee functionality. The paper also argues that software architecture must be considered at both design-decision and component-constraint levels.

B. How does the code quality of LLM-written serverless functions compare to human-written code? (RQ2)

  • LLMs produce simpler code than human-written code, but omitted essential logic can compromise functionality.Generated code shows reductions in Source Lines of Code, Cyclomatic Complexity, and Halstead Volume; the authors also report that larger models with context can produce functionally accurate code, with tradeoffs.Source: B. How does the code quality of… · pp. 9–10
  • Type 3 prompts reduce Cyclomatic Complexity and Cognitive Complexity and increase similarity, with minimal impact on Source Lines of Code and Halstead Volume.The authors associate reduced complexity with simpler, more understandable code.Source: B. How does the code quality of… · p. 9
  • Type 2 prompts sometimes lower CodeBLEU scores even when more tests pass than with Type 1 prompts.The authors note that functional accuracy does not always align with similarity to human-written code.Source: B. How does the code quality of… · p. 10
  • The authors identify a trade-off between simpler, more understandable code and autonomous handling of complex logic, requiring further refinement.The paper says LLMs’ limited ability to handle complex logic autonomously constrains functionality.Source: B. How does the code quality of… · p. 10

C. Defining Software Architecture in the Age of GenAI

  • For component generation, Jansen and Bosch’s design-decision view applies at the black-box level, while Garlan and Shaw’s view addresses existing components, connections, and constraints.The authors argue that architectural descriptions should capture design decisions and constraints for component generation.Source: C. Defining Software Architecture in… · p. 10

D. The Path Forward for GenAI in SA

The authors identify scarce, imperfect data and a lack of architecture-specific benchmarks as challenges for GenAI in software architecture. Their serverless study finds that generated functions can improve with human intervention and argues for a human-centered approach, while noting validity threats and future research directions.

D. The Path Forward for GenAI in SA

  • High-quality architectural-component data is scarce, repository tests and components were often unsatisfactory, and architecture-specific benchmarks do not yet exist.The authors also note that methods such as retrieval-augmented generation, knowledge graphs, and agentic frameworks have not been explored for software-architecture-specific tasks.Source: D. The Path Forward for GenAI in SA · p. 10

E. Threats to Validity

  • The authors identify external-validity threats from model and generation-parameter selection and possible training-data contamination, which they cannot mitigate.They also cite test effectiveness, metric selection, and repository and function selection as internal- and construct-validity threats, alongside steps taken to address them.Source: E. Threats to Validity · p. 10

VII. CONCLUSION AND FUTURE WORK

  • In the serverless study, LLMs often fail to generate fully functional functions autonomously, but minimal human intervention improves functionality and test passing.Generated functions also show better cyclomatic and cognitive complexity metrics; the authors say humans should not be completely removed from component generation.Source: §VII · pp. 10–11
  • Future work includes retrieval-augmented generation, multiagent frameworks, and architectural styles such as microservices and monoliths with larger components.The study evaluates serverless functions, selected because their basic architectural units are small.Source: §VII · pp. 10–11

Related papers

Loading 2502.02539v1…