Source-linked AI summary
Benchmarking the Titans: A Multi-Dimensional Empirical Evaluation of LLM Code Generation Quality in the .NET Ecosystem
Seyed Mohammad Mahdi Ghalandarian, Majid Bazargani, Masoumeh Taromirad
TL;DR
Existing benchmarks emphasize Pass@k and largely focus on Python, leaving multidimensional C#/.NET evaluation underdeveloped. The paper evaluates GPT, Gemini, Claude, and Grok on 85 HumanEval-derived tasks using automated correctness testing, Roslyn analysis, and adversarial BenchmarkDotNet profiling. It finds that correctness and composite quality are nearly orthogonal, with Pearson r = 0.075 within this benchmark.
Problem
Existing benchmarks emphasize Pass@k and Python, while C#/.NET lacks unified evidence covering correctness, structural quality, and runtime efficiency.
Method
The study evaluates 340 generated solutions from four LLMs across 85 tasks using unit testing, Roslyn AST analysis, and adversarial BenchmarkDotNet profiling.
Results
Correctness and composite quality are empirically orthogonal within the benchmark, with Pearson r = 0.075 across 340 records.
Takeaways & Limitations
Pass@k rankings are insufficient as a sole criterion, so correctness and structural quality should be reported as independent dimensions.
Takeaways & Limitations
The 85-task HumanEval subset does not represent complex, multi-file enterprise .NET architectures, and the orthogonality finding may not generalize unconditionally.
Abstract
from arXiv · showhide
Evaluating Large Language Model (LLM) code generation quality requires examining not just whether the generated code is correct, but whether it is maintainable, efficient, and stylistically sound, all of which are qualities of direct importance to software engineering practitioners. Existing benchmarks reduce evaluation to a single Pass@k metric, which obscures critical trade-offs between functional correctness and structural quality. A further limitation is the near-exclusive focus on Python, leaving enterprise-relevant ecosystems such as C# and .NET without dedicated evaluation. This paper presents an automated, multi-dimensional evaluation framework for C# code generation, applying it to four state-of-the-art LLMs: GPT, Gemini, Claude, and Grok. We conduct a controlled experiment across 85 algorithmic tasks derived from HumanEval, generating and evaluating 340 solutions in total, in which each solution is assessed across three independent dimensions: functional correctness via automated unit testing, static code quality via Roslyn AST analysis, and runtime efficiency via adversarial BenchmarkDotNet profiling. Our central finding reveals a substantial gap between correctness and quality attributes (Pearson r = 0.075), demonstrating that Pass@k rankings systematically misrepresent the full LLM performance profile in software engineering contexts. We further characterize GPT's bimodal failure behavior.
1. Introduction
Existing LLM code benchmarks emphasize test-passing correctness, while overlooking maintainability and the under-evaluated C#/.NET ecosystem. This paper introduces an automated framework that evaluates correctness, structural quality, and runtime efficiency together.
- Motivation: Pass@k cannot distinguish equally correct solutions with materially different complexity and maintainability.A function with cyclomatic complexity 19 and one with complexity 2 can receive the same Pass@k result.
- Research gap: Python dominates existing benchmarks, while C#/.NET lacks dedicated infrastructure combining correctness, static quality, and runtime evaluation.Existing C# tools provide static analysis but do not unify all three dimensions for LLM outputs.
- Contribution: The framework evaluates GPT, Gemini, Claude, and Grok through dataset preparation, code generation, dynamic evaluation, and result collection.It integrates .NET Reflection testing, Roslyn AST analysis, and adversarial BenchmarkDotNet profiling.
- Contribution: Correctness and structural code quality are empirically orthogonal, challenging Pass@k as the sole evaluation criterion.The paper frames this result as its central contribution for software-engineering evaluation.
2. Related Work
Prior work established Pass@k and expanded evaluation toward repository complexity and integrity, but important quality dimensions remain insufficiently integrated. This paper targets that gap in the .NET ecosystem.
- Benchmark scope: HumanEval and MBPP established Pass@k as a reproducible correctness signal, while repository-level studies expose limitations of isolated function generation.DevEval uses samples from 119 real-world projects, and HumanEvo reports 10% to 61% performance overestimation under evolution-aware evaluation.
- Evaluation integrity: Studies of evaluation integrity report structural and logical deficiencies that correctness metrics alone do not capture.Related work notes that generated code can be shorter yet more complex than canonical solutions, alongside hallucination and contextual-dependency failures.
- Quality measurement: TASTY predicts complexity classes for C++ and Python but does not empirically measure runtime efficiency or naming quality.Its scope differs from a multidimensional measurement framework for generated C# code.
- Research gap: No prior work identified here integrates runtime benchmarking, static analysis, and functional correctness testing in one automated .NET pipeline.This integration defines the paper’s closest literature gap.
3. Methodology and Framework Architecture
The study implements a fully automated four-phase pipeline for evaluating generated C# solutions across correctness, static quality, and runtime efficiency. It applies task-controlled generation and multidimensional scoring to 340 solution–model pairs.
- Evaluation dimensions: The framework measures unit-test correctness, AST-based static quality, and adversarial runtime efficiency as separate evaluation dimensions.Static analysis uses Roslyn; runtime profiling uses BenchmarkDotNet with execution time and memory allocation metrics.
- Research questions: The study investigates correctness, task-driven structural complexity, correctness–quality orthogonality, and runtime performance across GPT, Gemini, Claude, and Grok.These correspond to RQ1–RQ4.
- Evaluation metrics: The metrics include Success Rate, pairwise Pearson correlations for complexity and nesting, BenchmarkDotNet time and memory, and a 0–100 Composite Quality Score.The composite score aggregates normalized correctness, complexity, nesting, style, and performance.
- Pipeline architecture: The four pipeline phases are dataset preparation, code generation, dynamic evaluation, and result collection.The architecture maps these phases to the study’s research questions.
- Phase 1: Data Preparation: HumanEval records are transformed into an Excel database with task fields, prompts, and tests for each problem.The prepared-record example is task csharp_1.
- Phase 1: Data Preparation: Each task receives engineered solution and benchmark prompts, with independent worst-case harness generation intended to prevent input bias and early exits.DeepSeek generated the benchmark harnesses used for identical profiling conditions.
- Phase 2: Code Generation: All four models are queried in parallel at temperature 0, and outputs are compiled into shared .NET 8.0 projects containing three files per task.The projects include generated solutions, a benchmark harness, and a test runner with a 5,000 ms timeout.
- Phase 3: Evaluation: Dynamic evaluation runs correctness testing through Reflection, Roslyn static analysis, and BenchmarkDotNet profiling.Correctness compares full assertion results, while static analysis extracts complexity, nesting, naming violations, and professionalism.
4. Experimental Results
The experiment evaluates GPT-4, Gemini 1.5 Pro, Claude 3.5 Sonnet, and Grok 3 on C# generation using correctness, structural complexity, and runtime-performance measures. Results show differences in correctness, code complexity, and resource use, including strong runtime outliers for Claude.
- Functional Correctness (RQ1): 98.53% mean correctness led Gemini, followed by Claude at 98.32%, Grok at 97.98%, and GPT at 96.37%.Gemini had 2 failures, Claude and Grok 4 each, and GPT 6; GPT also had the highest variance, with SD = 15.37.
- Task Complexity and Model Behavior (RQ2): Claude generated the simplest code, with cyclomatic complexity 2.47 and nesting depth 1.13.Its cyclomatic complexity was approximately half GPT’s 5.16, while nesting depth was lower than GPT’s 1.68.
- Task Complexity and Model Behavior (RQ2): Pearson correlations of r = 0.63 to 0.69 across model pairs indicate that task difficulty substantially contributes to observed complexity.Recursive logic and multi-branch control elevated complexity across all four models.
- Runtime Performance (RQ4): 4,823 ns was GPT’s fastest median execution time, while Grok used the least memory at 6.0 MB.Claude had the highest median execution time at 12,859 ns and memory allocation at 156.6 MB.
- Runtime Performance (RQ4): Claude’s median memory allocation was roughly 26× Grok’s and 13× GPT’s, driven primarily by two anomalous profiling tasks.On csharp_12, Claude reached approximately 2.56×10^11 ns, roughly 2,500× slower than the next worst model; on csharp_16, it allocated approximately 3.94 GB.
5. Discussion
Across 340 records, correctness is nearly independent of structural quality, while model profiles reveal distinct trade-offs in reliability, failure behavior, complexity, and runtime efficiency.
- Inter-Metric Correlation Analysis: 0.075: Correctness–Quality correlation is near zero, so functional correctness and structural quality must be evaluated independently.The benchmark reports |r| < 0.18 for correctness against all other metrics.
- GPT’s Bimodal Failure Behavior: GPT’s failed-task Quality Score is 22.44 versus 39.51 on successful tasks, indicating a pronounced quality collapse.GPT’s failed-task quality is less than 57% of its successful-task quality and reflects broad structural deficiencies.
- Model Correctness and Reliability: 98.53%: Gemini has the highest mean correctness with 2 failures, while GPT has the lowest mean at 96.37% and highest variance.GPT’s SD is 15.37, indicating a bimodal distribution between fully correct and substantially failed solutions.
- Task Difficulty and Complexity: 0.63–0.69: Cross-model complexity correlations show that task difficulty primarily drives cyclomatic complexity on identical tasks.Comparisons that omit task-level difficulty can produce biased conclusions about model complexity.
- Multi-Dimensional Model Profiles: Grok has the most balanced profile, but no model dominates across all six evaluation dimensions.The radar comparison supports selecting models according to the quality dimension prioritized by the deployment context.
- Runtime and Structural Trade-offs: Claude combines the simplest structure with the highest memory footprint, showing that low complexity does not imply runtime efficiency.Claude’s complexity is 2.47 and nesting is 1.13, while its median memory footprint is 156.6 MB; two anomalous tasks primarily drive these outliers.
6. Threats to Validity
The paper identifies internal, construct, external, and conclusion-validity threats arising from deterministic decoding, performance outliers, metric design, limited task scope, and dataset size.
- Internal validity: Temperature = 0 mitigated API non-determinism, while multi-run variance analysis under non-zero temperature remains future work.
- Internal validity: Claude’s BenchmarkDotNet outliers were addressed with median performance metrics and explicit identification of anomalous tasks csharp_12 and csharp_16.
- Construct validity: Equal weighting of the five Composite Quality Score components may not reflect real-world priorities, although decomposed metrics are reported separately.
- External validity: The 85-task HumanEval subset excludes complex multi-file enterprise .NET architectures, and the orthogonality finding may not generalize across languages or task types.
- Conclusion validity: The 85-task dataset may limit statistical power, mitigated by correlating all 340 solution records and reporting standard deviations.
7. Conclusion and Future Work
The paper concludes that correctness and code quality are nearly independent in its C# benchmark, while the four models exhibit distinct trade-offs. Future work extends evaluation toward enterprise patterns, calibrated quality weights, prompt-injected style constraints, variance analysis, and robustness checks.
- Conclusion: Correctness and quality are virtually orthogonal (r = 0.075) within this C# HumanEval benchmark, challenging Pass@k as a sole production model-selection metric.
- Conclusion: The four models show distinct profiles: Gemini leads correctness, Grok composite quality, Claude structural simplicity, and GPT execution speed.
- Future Work: Future work will evaluate enterprise-representative C# patterns, calibrate Quality Score weights through expert studies, and test explicit style and naming constraints.
- Future Work: Future work also includes multi-run variance analysis and validating orthogonality with a four-component quality score that excludes correctness.
- Replication: The evaluation harness, adapted task specifications, and raw results dataset are publicly available on GitHub for reproducible experimentation.
Declaration on Generative AI
The authors used Claude for writing assistance, grammar and spelling correction, and LaTeX formatting, then reviewed and edited the content.
- Claude was used for writing assistance, grammar and spelling correction, and LaTeX formatting; the authors reviewed and edited the content.