Beyond Functional Correctness: Static Code Quality Profiling of LLM-Generated Code

• Passing benchmark tests does not establish that generated code is readable, maintainable, or structurally sound.A program may pass tests while remaining unnecessarily complex or difficult to maintain.Source: §1 · p. 1
• Static code-quality profiles remain underexplored at scale, including which dimensions distinguish models and how profiles vary with task difficulty.The study combines Pylint, Radon, and SonarQube to examine style, refactoring signals, structural complexity, maintainability, reliability, and security-related warnings across 65,663 programs from 73 LLMs.Source: §1 · p. 2
• The study analyzes correct programs separately and reports length-normalized metrics to assess whether patterns reflect incomplete implementations or code volume.These are supporting analyses rather than separate research objectives.Source: §1 · p. 2
• Maintainability is a key dimension distinguishing model behavior, while harder tasks generally produce longer, more structurally involved programs.Static-analysis changes with task difficulty depend on whether raw counts or length-normalized measures are considered.Source: §1 · p. 2
• The contributions include characterizing cross-model and cross-difficulty profiles and releasing code, analysis scripts, and aggregated results for reproducibility.Source: §1 · p. 3

2.1 LLMs for Software Engineering and Code Generation

• Short or structurally simple generated code may reflect omitted logic or incomplete implementation rather than higher quality.Source: §2.1 · p. 3

2.2 Limits of Correctness-Based Evaluation

• Passing a limited test suite does not establish maintainability or robustness, while comprehensive tests are costly and dynamic methods can be difficult to scale.Static analysis offers a complementary way to examine code characteristics beyond benchmark pass rates.Source: §2.2 · p. 3

2.3 Static Analysis and Code Quality Metrics

• Static analysis scales across large corpora, but raw issue counts can reflect code volume and low complexity can reflect incomplete implementations.Length-normalized measures help distinguish issue burden from concentration; these metrics describe structural characteristics, not inherently positive or negative quality.Source: §2.3 · p. 4

3.1 Research Questions

• The study asks how static code-quality metrics vary across LLMs and what trade-offs emerge between correctness and quality metrics.Source: §3.1 · p. 5
• To answer these questions, the authors build a Python-program dataset, apply established static-analysis tools, and analyze the resulting metrics.Source: §3.1 · p. 5

Dataset

• LiveCodeBench submissions provide the study’s source of generated code.Source: Dataset · p. 5
• The final dataset contains 65,663 code samples, selected as the first generated sample for each prompt and model from LiveCodeBench submissions.The benchmark includes submissions from up to 73 LLMs across up to 1,055 prompts, with 10 generated samples per prompt.Source: Dataset · p. 5
• Programs are analyzed separately for LiveCodeBench’s Easy, Medium, and Hard difficulty levels.Source: Dataset · p. 5
• The analysis compares all 65,663 programs with a correct-only subset that passes the corresponding LiveCodeBench tests.The correct-only subset reduces the influence of incomplete incorrect programs that may appear artificially short or simple.Source: Dataset · p. 5

Non-functional Qualities

• Maintainability is assessed through indicators of complexity, coding conventions, refactoring, comment usage, and maintainability-related static-analysis issues.It concerns how easily code can be understood, modified, and extended.Source: Non-functional Qualities · p. 5
• Reliability concerns execution without failure under expected conditions and is assessed using warnings, potential errors, fatal errors, and reliability-related issues.Source: Non-functional Qualities · pp. 5–6
• Security assessment focuses on static-analysis findings for potentially unsafe coding patterns that could affect confidentiality, integrity, or availability.Source: Non-functional Qualities · p. 6

Quality Construct Operationalization

• Quality constructs are represented by individual metrics, while code size and organization provide structural context rather than direct quality indicators.Separate measures avoid subjective weighting across metrics with different meanings and scales.Source: Quality Construct Operationalization · p. 6

Tools Selection

• Pylint findings cover convention, refactoring, warning, and error categories, capturing maintainability- and reliability-related signals.The study counts findings in each category and computes the average per code sample.Source: Tools Selection · p. 6
• Radon provides cyclomatic complexity, maintainability index, function and line counts, and comment and blank-line counts.The maintainability index is score-based; the other metrics describe code structure.Source: Tools Selection · p. 6
• SonarQube findings are grouped into maintainability, reliability, and security issues, with average issue counts computed per code sample.Source: Tools Selection · p. 6

Length-Normalized Metrics

• Count-based metrics are normalized by lines of code to account for their increase with program length.Source: Length-Normalized Metrics · p. 7
• The resulting density expresses findings per 100 SLOC and is applied to count-based Pylint and SonarQube metrics.Where appropriate, normalization also applies to complexity, comments, blank lines, and function counts.Source: Length-Normalized Metrics · p. 7
• Code-size metrics and the Maintainability Index remain in their original forms rather than being normalized.Source: Length-Normalized Metrics · p. 7

Experiments

• The study analyzes 65,663 code samples with three tools and records each sample’s generating model, task difficulty, correctness label, and static-analysis results.Source: Experiments · p. 7
• RQ1 compares raw metric distributions across models and Easy, Medium, and Hard tasks, then checks correct programs and length-normalized measures.These analyses assess whether patterns persist after excluding potentially incomplete implementations and accounting for code volume.Source: Experiments · p. 7
• RQ2 compares static quality metrics for correct and incorrect programs across difficulty levels and examines raw and length-normalized measures for correct programs.The analyses distinguish effects associated with implementation completeness from those related to code length.Source: Experiments · p. 7

4.1 RQ1: How do static code quality metrics vary across different LLMs?

• Maintainability-related and structural metrics differentiate models most clearly; reliability-related findings vary more moderately, while security findings remain sparse.The raw profiles summarize representative models across Easy, Medium, and Hard tasks.Source: §4.1 · p. 7, Table 1 · p. 8
• Raw findings reflect both structure and code volume, but normalized differences in convention, warning, and maintainability findings remain visible across models.OpenReasoning-Nemotron-32B retains high warning density across difficulty levels.Source: §4.1 · pp. 8, 10
• Harder tasks generally produce longer, more complex programs and more raw findings, but several issue densities remain stable or decrease after length normalization.The normalized analysis is reported for functionally correct programs, with Table 3 values per 100 SLOC.Source: §4.1 · pp. 8–9, Table 3 · p. 11
• Reliability findings vary less than maintainability findings: warnings contribute more to model variation than errors, and reliability counts rise less sharply with difficulty.Security findings are rare, likely because function-level competitive-programming tasks offer limited security-relevant functionality and attack surface.Source: §4.1 · p. 9
• Correct-only programs retain major cross-model differences in size, complexity, coding conventions, Maintainability Index, and maintainability findings.Correct solutions to harder tasks also remain longer and more structurally involved, though excluding incorrect programs changes some difference magnitudes.Source: §4.1 · p. 9, Table 2 · p. 10

4.2 RQ2: What trade-offs emerge between functional correctness and static code quality metrics in LLM-generated code?

• Pylint Convention most clearly separates passed from failed samples; Error generally decreases with correctness, while Warning is less stable across models.Convention and Refactor findings tend to be higher for stronger models, consistent with longer, more complex implementations creating more opportunities for such findings.Source: §4.2 · p. 13
• Correct solutions are often longer and more complex yet score better on maintainability, so correctness and static code quality are related but not interchangeable.The figure compares raw metric distributions for passing and failing samples; the prose reports higher Maintainability Index and fewer SonarQube Maintainability Issues for passed code.Source: Figure 1 · p. 12, §4.2 · p. 13

5 Discussion

• Similar functional performance does not imply similar static quality profiles; maintainability distinguishes models, and correct code may be more developed yet more complex.The discussion treats correctness and static quality as complementary evaluation dimensions.Source: §5 · pp. 13–14

5.1 Generalization beyond LiveCodeBench

• HumanEval more often links higher correctness with fewer Pylint findings and better maintainability, while LiveCodeBench shows weaker or less consistent alignment.The relationship between correctness and structural properties is comparatively weaker in HumanEval; the profiles depend on benchmark characteristics.Source: §5.1 · p. 14

5.2 Implications

• Evaluations should complement pass rates with multidimensional static profiles, using correctness-conditioned and length-normalized metrics to help separate code volume and completeness effects.Model developers can use these profiles as secondary signals when models have similar functional performance; generated code that passes tests should still receive static-analysis checks before integration.Source: §5.2 · pp. 14–15
• Correctness, complexity, maintainability, and issue counts can vary in different directions, so associations with larger or more complex code should not be read as evidence that such code is inherently better.Correct solutions may be more complete and structurally developed, creating more opportunities for static analyzers to report findings.Source: §5.2 · p. 15

5.3 Limitations and Future Directions

• LiveCodeBench mainly covers short, function-level competitive-programming tasks that rarely exercise security-sensitive operations, limiting security evaluation and the study’s reach beyond that setting.HumanEval also shows zero SonarQube security issues; future work proposes larger, multi-file benchmarks with greater security-relevant coverage.Source: §5.3 · pp. 15–16, §5.2 · p. 15
• Correct-only groups may have fewer, less stable samples, especially for lower-correctness models on harder tasks, so those results complement rather than replace all-code results.Source: §5.3 · p. 15
• Per-100-SLOC normalization controls for code volume but not strategy, style, or task demands; group-level ratios also give longer programs greater weight and other normalization may summarize differently.Source: §5.3 · pp. 15–16

5.4 Threats to Validity

• Findings depend on tool configurations, sample selection, benchmark labels, and proxy metrics; conclusions are limited to Python function-level settings and may not transfer to project-level or other-language code.Static analyzers can produce false positives or miss issues, and observed relationships are descriptive rather than causal.Source: §5.4 · pp. 16–17

6 Conclusion

• Models with similar functional performance can exhibit distinct static quality profiles, with maintainability and structural metrics providing the clearest differentiation.Reliability varies more moderately, security findings remain sparse, and harder tasks generally produce longer, more complex code.Source: §6 · p. 17
• The study profiles 65,663 Python programs from 73 LLMs using Pylint, Radon, and SonarQube across coding conventions, complexity, size, maintainability, reliability, and security-related findings.Correct-only and length-normalized analyses show that several patterns persist after accounting for incomplete implementations and code volume.Source: §6 · p. 17
• The findings support complementing correctness evaluation with multidimensional static quality profiles and extending analysis to project-level, multi-file, and multi-language benchmarks.Source: §6 · p. 17