PaperlayerView exact revision on arXiv

Source-linked AI summary

Investigating Software Aging in LLM-Generated Software Systems across Generation-and-Execution Environments

Cesar Santos, Michele Vitagliano, Roberto Natella, Ermeson Andrade

arXiv:2608.26391v1cs.SE

TL;DR

Long-term reliability of LLM-generated software under sustained execution remains less studied than correctness and other short-term quality attributes. This paper generates and validates BaxBench-derived backend services across JavaScript, Python, and Rust, runs them for 48 hours, and combines runtime, static, and exploratory comparison analyses. Memory usage shows the most consistent potential aging signal, while response time and throughput vary more across configurations.

  • Problem

    Evidence about the long-term reliability of LLM-generated software during sustained execution remains limited despite extensive evaluation of correctness, security, maintainability, and robustness.

  • Method

    The study generates BaxBench-derived backend applications across JavaScript, Python, and Rust, validates them functionally, executes 48-hour workloads, and applies runtime, static-analysis, and exploratory human-written comparisons.

  • Results

    Memory consumption is the most consistent potential software-aging indicator, with statistically significant upward trends in most application-language combinations, while response time and throughput behave more heterogeneously.

  • Takeaways & Limitations

    Functional correctness alone is insufficient for assessing the operational reliability of LLM-generated software intended for continuous execution.

  • Takeaways & Limitations

    Because complete environments combine generated code with frameworks, runtimes, libraries, operating-system behavior, and generation platforms, degradation cannot be attributed to a single factor.

Abstract

from arXiv · show

Large Language Models (LLMs) are increasingly used to generate executable software systems from natural language specifications, accelerating development and reducing manual implementation effort. Although recent studies have investigated the functional correctness, security, maintainability, and robustness of LLM-generated code, little is known about the long-term reliability of such systems under sustained execution. In this paper, we experimentally investigate software aging symptoms in LLM-generated service-based applications across different programming languages. Using backend scenarios derived from BaxBench, we generated applications targeting JavaScript, Python, and Rust through LLM-based generation platforms, validated them with BaxBench-derived tests, and subjected them to 48-hour workload executions. We monitored memory usage, response time, and throughput and analyzed them using the Mann--Kendall test and Sen's slope estimator. We further complemented the runtime evaluation with static analysis of the generated source code and an exploratory comparison with human-written implementations of related backend scenarios. The results show that memory usage is the most consistent indicator of potential software aging, with statistically significant upward trends in most application-language combinations, while response time and throughput exhibit more heterogeneous behavior. Static analysis identified plausible code-level aging mechanisms, and the comparison with human-written systems showed that aging trends can also emerge in manually developed implementations. These findings indicate that functional correctness alone is insufficient to assess the operational reliability of LLM-generated software before deployment in continuously running environments.

I. INTRODUCTION

Existing evaluations of LLM-generated software emphasize correctness and other short-term quality attributes, leaving long-term reliability under sustained execution less examined. This paper addresses that gap by evaluating aging symptoms across environments and complementing runtime analysis with static analysis and human-written comparisons.

  • Motivation: Existing studies mainly evaluate functional correctness, security, maintainability, robustness, and performance under bounded conditions.
  • Motivation: Software aging describes progressive degradation during execution that may arise from leaks, unreleased memory, fragmentation, accumulated errors, or other runtime effects.
  • Approach: The study generates equivalent BaxBench-derived backend services in JavaScript, Python, and Rust, then validates them functionally before long-duration execution.
  • Approach: The runtime methodology monitors memory usage, response time, and throughput across diverse generation-and-execution environments.
  • Complementary analyses: Static analysis examines plausible mechanisms including unbounded persistent state, missing cleanup paths, and conditional resource retention.
  • Findings and implications: Human-written related implementations can exhibit aging trends comparable to or stronger than those in generated applications, while functional validation alone does not establish continuous operational reliability.

III. RELATED WORK

Prior work evaluates generated software primarily through correctness, security, and related quality attributes, while software-aging research has focused largely on traditional long-running systems. This study extends the intersection by examining sustained execution, cross-environment behavior, code-level mechanisms, and human-written comparisons.

  • Automatically generated software: Automatic software-generation research spans symbolic tools, model transformations, code generators, and modern LLM-based generation from natural-language specifications.
  • LLM-generated software: Studies of LLM-generated software commonly assess correctness, maintainability, robustness, security, and code quality rather than long-term operational reliability.
  • Benchmarks and open questions: BaxBench provides realistic backend scenarios and functional validation tasks but focuses on correctness and security instead of sustained execution or degradation over time.
  • Software aging: Traditional software-aging studies document degradation in web servers and machine-learning systems, establishing sustained execution as a relevant reliability concern.
  • Prior evidence: Earlier studies of LLM-generated applications reported memory growth as the main symptom and variable response-time degradation, but used specific generation settings.
  • This study: This paper broadens the investigation across JavaScript, Python, and Rust environments and combines runtime, static-analysis, and comparison-based research questions.

B. Experiment Overview

The experiment generates and functionally validates equivalent backend applications from BaxBench-derived prompts, then subjects them to controlled 48-hour workloads while monitoring runtime trends. Statistical analysis and complementary source-code and comparison procedures support assessment across complete generation-and-execution environments.

  • Runtime execution: The client uses Apache JMeter to issue sustained HTTP workloads while server-side monitoring collects resource usage and client-side recording captures response time and throughput.
  • Generation environments: Applications are generated in JavaScript with Node.js and Express, Python with FastAPI, and Rust with Actix Web, preserving equivalent APIs across environments.
  • Inputs and subjects: Standardized BaxBench-derived prompts specify application behavior, APIs, formats, and outputs for four backend scenarios with distinct operational characteristics.
  • Validation: BaxBench-derived functional tests cover endpoints, input conditions, response statuses, and expected outputs before applications enter long-duration experiments.
  • Runtime execution: Each experiment runs for 48 hours with 10 concurrent threads and a 0.1-second request interval after workload calibration.
  • Analysis: Mann–Kendall tests detect monotonic trends at p < 0.05, and Sen’s slope estimates their direction and magnitude.

V. RESULTS

The experiments evaluate memory usage, response time, and throughput during long-duration execution. Memory usage shows the clearest and most widespread evidence of aging across applications and languages.

  • 1) Memory usage:: All three Credit Card implementations showed statistically significant upward memory trends, with Python having the highest Sen’s slope.JavaScript and Rust also exhibited increasing memory usage, though at lower slopes.
  • 1) Memory usage:: JavaScript and Python showed significant Image Converter memory growth, whereas Rust’s trend was non-significant with p = 0.205.JavaScript had the highest slope, followed by Python.
  • 1) Memory usage:: All three Monitor implementations exhibited statistically significant upward memory trends, with Python having the highest slope.The observed growth patterns differed across languages, including step-like and later-phase increases.
  • 1) Memory usage:: 11 of 12 application-language combinations presented statistically significant upward memory trends.Rust for Image Converter was the only non-significant case.
  • 1) Memory usage:: Python had the highest memory-growth slope in Credit Card, Monitor, and Uptime, while JavaScript led only in Image Converter.Rust generally had lower slopes but still showed significant growth in Credit Card, Monitor, and Uptime.

2) Response time:

Response time provides complementary evidence of aging but is less uniform than memory usage. Significant trends vary by application and language, and isolated spikes can strongly influence the results.

  • 2) Response time:: Python and Rust showed significant positive Credit Card response-time trends, while JavaScript did not with p = 0.772.Python had the highest slope; Rust’s significant slope was very small.
  • 2) Response time:: Rust’s Image Converter response-time trend was significant with p = 1.74 × 10^-14, but an extreme latency spike complicates interpretation.Python’s trend was non-significant with p = 0.343, while JavaScript’s positive trend had p = 0.032.
  • 2) Response time:: JavaScript and Rust showed significant positive Monitor response-time trends, while Python showed no significant trend.JavaScript had the highest slope, whereas Rust’s significant slope was much smaller.
  • 2) Response time:: Response time showed a less uniform degradation pattern than memory usage across applications and languages.Latency trends were sometimes associated with transient spikes or workload-specific instability.
  • 2) Response time:: Response time should be interpreted with visual time-series behavior rather than used alone as evidence of long-term degradation.Transient contention, I/O delays, and isolated request failures can affect this metric.

3) Throughput:

Throughput degradation appears in selected application-language combinations rather than uniformly across services. Its interpretation complements response-time analysis and should be combined with other aging indicators.

  • 3) Throughput:: Rust Credit Card throughput had a significant negative slope despite achieving the highest throughput level among the three languages.This indicates that high initial processing capacity did not prevent reduction during prolonged execution.
  • 3) Throughput:: Image Converter throughput remained statistically stable for Python and Rust, while JavaScript showed a significant negative trend with a very small slope.The workload operated at lower throughput levels than the other applications.
  • 3) Throughput:: Monitor throughput showed significant negative trends for JavaScript and Rust, but not for Python.Python exhibited only small oscillations without a clear degradation pattern.
  • 3) Throughput:: For Uptime, JavaScript and Rust showed significant throughput degradation, whereas Python remained relatively stable with p = 0.383.The results indicate reduced processing capacity in the significant cases.
  • 3) Throughput:: Significant negative throughput trends occurred in Python and Rust for Credit Card, JavaScript and Rust for Monitor, and JavaScript and Rust for Uptime.The strongest reductions were Rust Credit Card at -2.754, JavaScript Uptime at -2.186, and Rust Uptime at -1.340.

B. Comparison (RQ2)

The comparison across environments is organized by memory usage, response time, and throughput. It examines mean values, Sen’s slopes, and confidence intervals across four application scenarios.

  • B. Comparison (RQ2): The comparison covers programming languages, runtime environments, frameworks, and LLM-based generation platforms.
  • B. Comparison (RQ2): The analysis is organized by memory usage, response time, and throughput.
  • B. Comparison (RQ2): Each metric is examined using mean values, Sen’s slope estimates, and confidence intervals across four application scenarios.

1) Memory usage:

Memory usage provided the most consistent evidence of potential software aging, with significant upward trends across nearly all application-language combinations. Growth rates varied by workload and execution environment.

  • Almost all application-language combinations showed statistically significant upward memory trends, except Rust Image Converter (p = 0.205).The Rust Image Converter trend was not statistically significant.
  • 4.410 × 10−3 GB/hour was JavaScript Image Converter’s highest memory-growth slope, while Python Image Converter reached 3.744×10−3 GB/hour.Image Converter showed the highest growth rate in both languages.
  • Rust Image Converter averaged 2.476 GB but had a low, non-significant slope, separating high average consumption from progressive growth.Rust nevertheless showed significant increasing trends for Credit Card, Monitor, and Uptime.
  • Python had the highest memory-growth rate in Credit Card, Monitor, and Uptime, whereas JavaScript led only for Image Converter.Rust generally showed lower slopes than Python and JavaScript.
  • Confidence intervals separated language-specific growth rates most clearly for Credit Card, Monitor, and Uptime, with Python steepest and Rust lowest.Intervals overlapped more for some Image Converter comparisons.

2) Response time:

Response time and throughput showed more heterogeneous aging patterns than memory usage. Their trends varied with application scenario and language/runtime environment, and isolated spikes complicated interpretation.

  • Response time showed significant positive trends in some configurations but non-significant or negative slopes in others.This pattern made latency degradation less consistent than memory growth.
  • 6660.227 ms and 4.136 × 10−1 were JavaScript Image Converter’s mean response time and slope, respectively, the strongest JavaScript latency case.JavaScript Monitor also showed a positive slope of 1.550 × 10−1.
  • Python showed a significant Credit Card response-time slope of 9.832×10−3, while Image Converter, Monitor, and Uptime trends were non-significant.Python Image Converter nevertheless had the highest average response time at 13469.534 ms.
  • Rust Credit Card and Uptime had mean response times of 0.519 ms and 0.570 ms, with statistically significant but very small positive slopes.Rust Image Converter’s slope of 71.027 had a wide interval spanning negative and positive values, indicating high variability.
  • Response time is a weaker standalone aging indicator than memory usage and should be interpreted alongside throughput and memory.Significant slopes may reflect isolated spikes rather than smooth degradation.
  • Throughput degradation depended strongly on application and language/runtime, rather than following a uniform pattern.Significant reductions included Rust Credit Card at -2.754, JavaScript Uptime at -2.186, and Rust Uptime at -1.340.

4) Overall interpretation:

Software aging symptoms differed across generation-and-execution environments, with memory trends most consistent and response time and throughput more heterogeneous. Static analysis supplied plausible mechanisms but not definitive causal proof.

  • Memory usage was the most consistent aging symptom, with significant upward trends in almost all configurations.Rust Image Converter was the main exception.
  • Response time and throughput were less uniform than memory usage and depended more strongly on application scenario and runtime behavior.The metrics captured different aspects of performance degradation.
  • Six of 12 application-language combinations exhibited significant degradation across all three metrics.The combinations included JavaScript Image Converter, Monitor, and Uptime; Python Credit Card; and Rust Monitor and Uptime.
  • Python showed the most persistent memory growth, JavaScript was particularly affected in Image Converter, Monitor, and Uptime, and Rust generally had lower growth rates.Rust still showed degradation in several configurations, especially throughput.
  • Static analysis identified conditional or per-request mechanisms in Monitor, but none were classified as supported static aging mechanisms.The JavaScript Monitor analysis included a possible child-process accumulation mechanism if external commands hang.
  • Static findings provided useful explanatory evidence, but correspondence with runtime symptoms was not uniform and did not establish causality.Some mechanisms depended on runtime conditions absent from the workload, while some trends lacked a visible application-level mechanism.

2) Credit Card:

Credit Card implementations exposed persistent-state and database/concurrency patterns that could relate source-level behavior to runtime degradation. The clearest mechanism was potentially unbounded state growth in shared data structures.

  • JavaScript Credit Card repeatedly created database connections and performed request-level database operations, which may introduce overhead and contention.These patterns were not characterized as persistent aging mechanisms.
  • Python Credit Card used an in-memory phone-number-to-card-number structure with no deletion, eviction, size cap, or time-to-live mechanism.New unique pairs can increase heap-resident memory, while synchronization may introduce contention.
  • Rust Credit Card exhibited significant degradation across memory usage, response time, and throughput.The unbounded state provides a plausible explanation for progressive memory consumption, while state size and synchronization may contribute to overhead.
  • Credit Card was among the scenarios where static findings most strongly corresponded with runtime degradation across multiple indicators.The strongest correspondence involved Python/Flask and Rust Credit Card implementations.

D. Human-written implementations (RQ4)

The Python Open applications show software-aging signals across memory usage, response time, and throughput, with the strongest trends varying by metric and application.

  • Memory usage: All three Python Open applications show increasing memory consumption over time, with Image Converter exhibiting the highest growth slope.Monitor and Uptime also show statistically significant upward trends.
  • Memory usage: Monitor and Uptime present statistically significant increasing memory trends, indicating progressive resource accumulation.The supplied evidence reports this conclusion for the Python Open applications.
  • Memory usage: Image Converter has the highest Sen’s slope and average memory usage, while the applications exhibit distinct memory growth rates.The confidence intervals of the slopes do not overlap.
  • Response time: Uptime shows the clearest response-time degradation, with a Sen’s slope of 4.112 and a confidence interval entirely above zero.Image Converter’s confidence interval crosses zero, so its response-time trend requires cautionary interpretation.

3) Throughput:

Throughput degradation is clearest for the Python Open Monitor and Uptime applications, while comparisons with LLM-generated systems indicate stronger aging evidence in selected human-written implementations.

  • Throughput: Image Converter has the lowest average throughput, but its negative slope is very small and its confidence interval crosses zero.Its throughput degradation pattern is therefore weak and should be interpreted with caution.
  • Throughput: Uptime exhibits the strongest throughput degradation, with a Sen’s slope of −2.860 × 10−1 and a confidence interval entirely below zero.Monitor also shows a statistically significant negative throughput trend, but with a smaller degradation rate.
  • Comparison with LLM-generated systems: Python Open implementations show stronger memory growth than corresponding LLM-generated implementations in all three scenarios.The memory growth slope rises from 3.744 × 10−3 to 8.239 × 10−3 for Image Converter, from 2.324 × 10−3 to 3.255 × 10−3 for Monitor, and from 2.810 × 10−3 to 5.043 × 10−3 for Uptime.
  • Comparison with LLM-generated systems: Python Open Monitor and Uptime show statistically significant negative throughput trends, whereas their LLM-generated counterparts remain statistically stable.For Uptime, the strongest throughput degradation has a Sen’s slope of −2.860 × 10−1.
  • Comparison with LLM-generated systems: Human-written implementations can exhibit aging trends comparable to or stronger than those observed in LLM-generated implementations.The comparison is exploratory because the systems are functionally related but not exact replicas.

E. Threats to validity

The study’s validity is constrained by fixed workloads, complete-environment attribution, limited monitoring and application diversity, and non-controlled human-written comparisons. Static findings and runtime trends provide complementary but not definitive causal evidence.

  • Attribution of degradation causes: Because applications were evaluated as complete environments, degradation cannot be attributed solely to generated code or programming language.Frameworks, runtimes, libraries, operating-system behavior, external tools, and generation platforms may all contribute or interact.
  • Static-analysis interpretation: Static analysis identifies plausible aging mechanisms but cannot establish whether they were activated during experiments or prove causality.Runtime conditions may not trigger every mechanism, and LLM-assisted review can produce false positives or miss implementation details.
  • Human-written comparison: The exploratory human-written comparison is not a controlled human-versus-LLM experiment because the systems differ in frameworks, libraries, architecture, and implementation choices.Its purpose is to contextualize whether similar symptoms appear in manually developed systems under comparable procedures.
  • Experiment scope and future validation: The study covers four BaxBench-derived service applications, so its findings may not generalize to database-intensive, batch-processing, stateful-microservice, or externally dependent systems.Longer executions, repeated runs, broader application classes, and finer-grained monitoring are identified as directions for strengthening the evidence.
  • Monitoring granularity: Operating-system-level memory monitoring may include effects from runtimes, background activity, auxiliary processes, and other server-environment components.Accordingly, memory trends represent server-environment-level evidence rather than isolated application-process behavior.
  • Workload configuration: Fixed 48-hour workloads with 10 concurrent threads and 0.1-second dispatch intervals may not represent production profiles or activate all aging mechanisms.Different request rates, concurrency levels, input sizes, failure conditions, and workloads with more unique inputs could reveal different patterns.
Loading 2608.26391v1…