PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-LCB: Extending LiveCodeBench to Multiple Programming Languages

Maria Ivanova, Pavel Zadorozhny, Rodion Levichev, Ivan Petrov, Adamenko Pavel, Ivan Lopatin, Alexey Kutalev, Dmitrii Babaev

arXiv:2606.20517v1cs.AIcs.PL

TL;DR

LiveCodeBench evaluates code generation only in Python, leaving multilingual generalization uncertain. Multi-LCB extends it to twelve languages and evaluates 24 LLMs, revealing Python overfitting, language-specific contamination, and substantial cross-language performance disparities.

  • Problem

    LiveCodeBench evaluates only Python, leaving unresolved whether LLMs generalize coding competence across multiple programming languages.

  • Method

    Multi-LCB extends LiveCodeBench to twelve languages by replicating identical tasks, preserving contamination controls and evaluation protocols, and evaluates 24 instruction- and reasoning-oriented LLMs.

  • Results

    Models show substantial cross-language performance gaps, Python overfitting, and language-specific contamination, with weaker results in statically typed or less prevalent languages.

  • Takeaways & Limitations

    Multi-LCB establishes a contamination-aware benchmark for rigorous cross-programming-language code-generation evaluation and exposes multilingual capability gaps.

  • Takeaways & Limitations

    The evaluation excludes proprietary models, so its results may not represent the real-world multilingual code-generation leaderboard.

Abstract

from arXiv · show

LiveCodeBench (LCB) has recently become a widely adopted benchmark for evaluating large language models (LLMs) on code-generation tasks. By curating competitive programming problems, constantly adding fresh problems to the set, and filtering them by release dates, LCB provides contamination-aware evaluation and offers a holistic view of coding capability. However, LCB remains restricted to Python, leaving open the question of whether LLMs can generalize across the diverse programming languages required in real-world software engineering. We introduce Multi-LCB, a benchmark for evaluating LLMs across twelve programming languages, including Python. Multi-LCB transforms Python tasks from the LCB dataset into equivalent tasks in other languages while preserving LCB's contamination controls and evaluation protocol. Because it is fully compatible with the original LCB format, Multi-LCB will automatically track future LCB updates, enabling systematic assessment of cross-language code generation competence and requiring models to sustain performance well beyond Python. We evaluated 24 LLMs for instruction and reasoning on Multi-LCB, uncovering evidence of Python overfitting, language-specific contamination, and substantial disparities in multilingual performance. Our results establish Multi-LCB as a rigorous new benchmark for multi-programming-language code evaluation, directly addressing LCB's primary limitation and exposing critical gaps in current LLM capabilities.

1 INTRODUCTION

Multi-LCB extends LiveCodeBench from Python to twelve programming languages while preserving its contamination controls, evaluation protocol, and automatic updates. Evaluating 24 instruction- and reasoning-oriented LLMs reveals cross-language performance gaps, Python overfitting, and language-specific contamination.

  • Motivation: LCB evaluates only Python, leaving unresolved whether models generalize coding competence across languages or overfit to Python.This limitation overlooks software engineering’s use of languages with distinct syntax, semantics, and idiomatic practices.
  • Benchmark: Multi-LCB extends LCB to 12 programming languages without task loss, preserving contamination controls and evaluation protocol for identical cross-language problems.The benchmark replicates every LCB task across supported languages and updates automatically as LCB evolves.
  • Evaluation: 24 instruction- and reasoning-oriented LLMs are evaluated across Multi-LCB, exposing systematic multilingual performance gaps and language-specific contamination.The evaluation covers both reasoning- and instruction-oriented models across all supported languages.
  • Findings: Models that perform strongly in Python often degrade sharply in other languages, and Python is not always a reliable proxy for individual non-Python languages.In several cases, models stronger on Python do not retain their advantage elsewhere.
  • Significance: Multi-LCB establishes a rigorous benchmark for multi-programming-language code evaluation and addresses LCB’s Python-only limitation.The benchmark provides a foundation for developing more robust, programming-language-agnostic coding models.

2 RELATED WORK

Prior benchmarks largely evaluate Python code generation, while newer multilingual datasets translate functional-format tasks or provide broader execution frameworks. Multi-LCB extends contamination-aware evaluation by retaining natural-language descriptions and converting hidden tests into language-agnostic STDIN/STDOUT inputs.

  • Single-language code benchmarks: HumanEval and MBPP established Python-focused benchmarks using short natural-language exercises, fixed function formats, and hidden unit tests.HumanEval contains 164 hand-written problems, while MBPP provides small exercises aimed at introductory programming.
  • Multi-programming-language benchmarks: MBXP and MultiPL-E broaden language coverage by translating HumanEval and MBPP tasks, including language-specific signatures, tests, or execution formats.Such translation must be repeated for each language and is sensitive to syntax and runtime differences.
  • Multi-programming-language benchmarks: HumanEval-XL expands HumanEval into additional languages with a standardized execution harness while preserving its functional, unit-test format.Multi-LCB instead keeps only natural-language descriptions and converts hidden tests into language-agnostic STDIN/STDOUT inputs.
  • Multi-programming-language benchmarks: Ag-LiveCodeBench-X, xCodeEval, McEval, and BigCodeBench broaden multilingual evaluation through expanded language coverage or unified execution frameworks, but differ in task sources and update practices.Ag-LiveCodeBench-X adds rarer targets such as Lua, R, Julia, OCaml, and Fortran; xCodeEval draws exclusively from Codeforces and is not continuously updated.
  • Contamination-aware evaluation: LiveCodeBench introduced release-date filtering and continuous collection from LeetCode, AtCoder, and Codeforces for contamination-aware Python evaluation.By harvesting new tasks and filtering post-training releases, LCB supports live evaluation and has become a de-facto standard for robust single-language assessment.

3 BENCHMARK DESIGN

Multi-LCB converts every LCB code-generation release into a multilingual benchmark while preserving its tasks, metadata, contamination-aware filtering, and evaluation protocol. It unifies native and adapted task formats through standardized prompts, automated hidden-test evaluation, and language-specific execution across twelve programming languages.

  • Data Source: Multi-LCB converts every LCB code-generation release without modification, retaining Python tasks from LeetCode, AtCoder, and Codeforces with metadata for contamination-aware filtering.Each task includes a natural-language description, input/output examples, and contest release date.
  • Task Conversion: Functional LeetCode tasks are reformatted into STDIN/STDOUT problems, with public and hidden tests converted for evaluation by one unified harness.This addresses the challenge of extending Python-specific functional starter code and testing harnesses across languages.
  • Code Generation: Prompts use the original LiveCodeBench zero-shot protocol: a target-language system instruction, complete problem specification with examples, and a code-block placeholder.Models must output only a complete program that reads standard input and writes standard output.
  • Automatic Testing and Evaluation: Correctness is measured with Pass@1, the fraction of tasks whose first generated solution passes every public and hidden test without runtime errors or timeouts.The pipeline securely compiles and executes each candidate before grading its output against official hidden tests.
  • Language Selection: The benchmark covers C++, C#, Python, Java, Rust, Go, TypeScript, JavaScript, Ruby, PHP, Kotlin, and Scala, balancing popularity, reproducible infrastructure, and paradigmatic diversity.The selection spans differences in compilation strategies, type systems, and memory-management models.

4 EXPERIMENT SETUP

The experiments evaluate 24 publicly available LLMs across diverse model families and sizes using standardized multilingual execution environments, zero-shot inference, and Pass@1 averaged over 10 runs. Correctness requires successful compilation or interpretation and passing all hidden official tests without runtime errors or timeouts.

  • Models: 24 publicly available LLMs spanning 7B to 685B parameters are evaluated, including instruction-tuned and reasoning-augmented models from several general-purpose and code-specialized families.Representative models include GPT-OSS-120B∗(Medium), Qwen3-235B-A22B-Thinking-2507∗, DeepSeek-R1-0528∗, and OpenReasoning-Nemotron-32B∗.
  • Hardware and Environment: Experiments run on 16 NVIDIA H100 80 GB GPUs with CUDA 12.3 and Python 3.11, using isolated language sandboxes with bundled compilers or interpreters.Each test case is limited to 6 s wall time and 4 GB memory, with external network access disabled.
  • Inference Protocol: Following LiveCodeBench, models use zero-shot prompts, nucleus sampling with temperature = 0.2 and top-p = 0.95, and triple-backtick stopping.Inference uses vLLM or SGLang for efficient batched decoding, with model-specific token budgets.
  • Evaluation Metric: Pass@1 (%) averaged over 10 runs is the primary metric, measuring the fraction of problems solved correctly by the first generated solution.A solution is correct only when it compiles or interprets successfully and passes all hidden official tests without runtime errors or timeouts.

5 EXPERIMENTS AND RESULTS

Experiments evaluate 24 recent models on post-cutoff Multi-LCB tasks across 12 languages using 10-run Pass@1, revealing substantial cross-language disparities and persistent Python bias. Reproduction results preserve LCB fidelity, while time-wise trends still show residual contamination on older problems.

  • 5.1 Dataset v6 Results: 24 recent models are evaluated on tasks released after 2025-02-01 across 12 programming languages using Pass@1 averaged over 10 runs.Dataset v6 covers February 2025 through May 2025, with temperature t = 0.2 reported in Table 1.
  • 5.1 Dataset v6 Results: GPT-OSS-120B∗(Medium) and DeepSeek-R1-0528∗ outperform Qwen3-235B-A22B-Thk-2507∗ on several non-Python languages despite Qwen3-235B-A22B-Thk-2507∗ being consistently stronger on Python.The language-specific reversals occur on Go, Javascript, Typescript, Rust, Ruby, Kotlin, and Scala.
  • 5.1 Dataset v6 Results: Almost every model favors Python; OpenRsn-Nmt-32B∗ and OpenCodeRsn-Nmt-1.1-32B∗ exceed 60% on Python while remaining below 30% across other languages.Qwen3-235B-Thk and DeepSeek-R1 also retain a measurable, though less pronounced, Python bias.
  • 5.1 Dataset v6 Results: Python has the highest mean Pass@1 at 0.482, followed by Java and C++ at about 0.44, while most other languages form lower-performing tiers.C#, Ruby, PHP, Go, Rust, Kotlin, and JavaScript/TypeScript have means near 0.33-0.39; only a handful of reasoning-augmented variants exceed 50%, and most models remain below 40%.
  • 5.2 Comparison with LCB: Multi-LCB reproduces original Python LCB results closely, with a mean absolute deviation of about 3% and consistent model rank ordering.Qwen3-235B-A22B-Thinking-2507 has 74.0% versus 74.1% originally (∆= −0.1), while DeepSeek-R1-0528 has 66.3% versus 68.7% (∆= −2.4).
  • 5.3 Contamination Analysis: Monthly trends show residual contamination on older problems, with higher pre-cutoff scores and step-like drops after evaluation windows cross model cutoffs.Lower performance is sustained on post-cutoff problems, despite Multi-LCB’s release-date filtering design.

6 LIMITATIONS AND THREATS TO VALIDITY

Multi-LCB’s validity is limited by incomplete language coverage, competitive-programming-centered tasks, and protocol and conversion effects that can confound cross-language comparisons. Results are further constrained by public-model selection and possible residual contamination or training-period bias.

  • Language Coverage and Selection: Multi-LCB covers 12 languages but excludes important languages, bases selection on 2025 popularity rankings, and does not account for dialects or versions.The selection may not reflect specialized domains or emerging languages.
  • Task Complexity and Domain: Because tasks remain rooted in competitive programming, Multi-LCB does not directly assess API integration, legacy-code debugging, or collaborative development workflows.The benchmark spans systems programming, web development, and data science domains, but algorithmic problem-solving only indirectly relates to industrial coding.
  • Evaluation Protocol Constraints: Strict STDIN/STDOUT evaluation can confound multilingual coding assessment when failures arise from syntax unfamiliarity, input parsing, or output-format noncompliance rather than problem-solving.These constraints may introduce performance degradation beyond algorithmic reasoning limitations.
  • Model Selection Bias: Evaluating only publicly available models excludes proprietary systems and may not represent the real-world leaderboard of multilingual code-generation capabilities.The findings therefore reflect only a subset of available models.
  • Construct Validity: Automatic conversion from functional format to STDIN/STDOUT may alter task complexity unevenly across languages, creating unequal conditions for cross-language comparisons.Some languages may be more naturally suited to particular problem types.
  • Internal Validity: Despite date-based filtering, hidden contamination may persist through shared problem patterns or solution templates, while training exposure can create temporal bias across languages.These factors may affect internal validity even when explicit release-date controls are applied.

7 FUTURE WORK

Future work will expand Multi-LCB to additional languages, evaluate proprietary models, and extend its STDIN/STDOUT framework to LCB-Pro and other format-conversion benchmarks.

  • Language expansion: Multi-LCB plans to add Swift, Haskell, R, and Julia by defining their compilation commands and runtime environments.Its modular design is intended to make language expansion straightforward.
  • Proprietary-model evaluation: The authors will evaluate GPT-4, Claude, and Gemini to establish comprehensive multilingual leaderboards reflecting current state-of-the-art performance.These evaluations target proprietary models.
  • Benchmark compatibility: The STDIN/STDOUT framework supports LCB-Pro and other benchmarks requiring format conversion, enabling broader contamination-aware evaluation.The passage identifies direct compatibility with LCB-Pro and potential application to other benchmarks.

8 CONCLUSIONS … D TASKS DISTRIBUTION

Multi-LCB is introduced as an extensible, contamination-aware benchmark for multilingual code generation, with live updates and standardized task conversion. The appendices document its public-data basis, interactive interface, and prompt adaptations across programming languages.

  • 8 CONCLUSIONS: Multi-LCB evaluates multilingual code generation across twelve programming languages and continuously incorporates newly released problems.Its conversion methodology can also extend to other Python benchmarks, such as LCB Pro, while inheriting LiveCodeBench’s live evaluation protocol.
  • A LEGAL COMPLIANCE AND LICENSE: The benchmark contains no personally identifiable information, offensive content, or proprietary code and derives entirely from publicly released LCB materials.The source materials include contest problems, reference solutions, and test cases from LeetCode, AtCoder, and Codeforces.
  • B UI OF MULTI-LCB: The Multi-LCB web interface filters tasks by interactively selected time windows and visualizes performance on newly released problems.Figure 6 shows tasks released between January 2024 and December 2024, emphasizing the benchmark’s live and continuously updated design.
  • C PROMPT EXAMPLES: Multi-LCB prompts distinguish original platform problem text from added instructions that standardize the STDIN/STDOUT format.Original content appears in blue boxes, while added instructions appear in red boxes.
  • C.1 ATCODER/CODEFORCES EXAMPLE (NATIVE STDIN/STDOUT): AtCoder and Codeforces examples require reading inputs from STDIN, solving the problem, writing output to STDOUT, and enclosing code within specified delimiters.The example asks for the number of positive integers not greater than N having exactly 9 positive divisors.
  • C.2 LEETCODE EXAMPLE (ADAPTED INTO STDIN/STDOUT): LeetCode examples adapt array-based problem statements into the same STDIN/STDOUT prompting format.The examples specify inputs and outputs for enemyEnergies and currentEnergy, while 2D arrays are encoded with a row count followed by newline-separated rows.
  • C.2 LEETCODE EXAMPLE (ADAPTED INTO STDIN/STDOUT): For non-Python settings, only the code-block header changes, while the rest of the prompt structure remains identical.The examples give C++ and Java headers as language-specific replacements.

D.1 TASK DISTRIBUTION BY DIFFICULTY AND PLATFORM … F.3 PERFORMANCE ON THE MULTI-LCB (JUL 2024-MAY 2025 SUBSET)

The paper extends LCB’s live, contamination-aware competitive-programming benchmark across diverse task sources, input/output structures, languages, models, temperatures, and evaluation subsets. Experiments report Pass@1, Pass@5, and Pass@10 results, including a July 2024–May 2025 Multi-LCB subset.

  • D.1 TASK DISTRIBUTION BY DIFFICULTY AND PLATFORM: LCB continuously aggregates Python problems from LeetCode, AtCoder, and Codeforces, with monthly distributions showing a steady inflow of tasks by difficulty and platform.The source tasks include natural-language descriptions, examples, hidden tests, and editorial review, supporting correctness and clarity.
  • D.2 TASK DISTRIBUTION BY I/O DATA DIMENSIONALITY (LEETCODE FUNCTIONAL FORMAT): LeetCode tasks span functional input/output structures including scalar values, one-dimensional arrays, and two-dimensional arrays over time.This distribution captures the variety of functional tasks inherited from LeetCode.
  • E PROGRAMMING LANGUAGE RANKINGS AND RUNTIME CHARACTERISTICS: The evaluated languages were selected for 2025 popularity and industrial relevance, spanning compiled/JIT, interpreted, and transpiled paradigms.They also cover static and dynamic typing, multiple memory-management models, and native, JVM, .NET, interpreter, and JavaScript runtimes.
  • F.1 MODELS OVERVIEW: The study evaluates models in instruction and reasoning settings, with the included-model details provided in Table 4.Reasoning-mode models are marked with an asterisk in the model overview and performance tables.
  • F.2.1 PASS@1 AVERAGED OVER 10 RUNS PERFORMANCE AT VARIOUS SAMPLING TEMPERATURES: Pass@5 results are reported at sampling temperatures t = 0.2, t = 0.6, and t = 1.0, with higher scores indicating better performance.The tables identify the best and second-best results and mark reasoning-mode models with an asterisk.
  • F.2.3 PASS@10 PERFORMANCE AT DIFFERENT SAMPLING TEMPERATURES: Pass@10 results are reported at sampling temperatures t = 0.2, t = 0.6, and t = 1.0, measuring the percentage of problems solved correctly on the 10th attempt.Higher Pass@10 values reflect better performance, and the tables mark best, second-best, and reasoning-mode results.
  • F.3 PERFORMANCE ON THE MULTI-LCB (JUL 2024-MAY 2025 SUBSET): On the July 2024–May 2025 Multi-LCB subset, Table 13 reports Pass@1 at sampling temperature t = 0.2 for all evaluated models.Scores represent the percentage of problems solved correctly on the first attempt, with higher values indicating better performance.

F.4 PERFORMANCE ON THE COMPLETE MULTI-LCB BENCHMARK … J DIFFICULTY ANALYSIS

Multi-LCB evaluates models across twelve languages using first-attempt Pass@1 on 1,055 tasks per language, while evaluation cost, platform, and difficulty substantially shape observed performance. The benchmark’s controlled runtimes support reproducibility, and performance declines most sharply on Hard problems.

  • G COMPUTATION TIME: 106 hours of total wall-clock time are required to evaluate all twelve languages, averaging about 8 min 50 s per language across 90 parallel CPUs.The average covers a full run of 1,050 tasks per language and includes compilation and execution.
  • G COMPUTATION TIME: 17 min 37 s is Ruby’s highest mean evaluation time, while Kotlin, PHP, and JavaScript each finish in under 4 minutes.Go and Python each exceed 11 minutes on average; differences primarily reflect compilation overheads and runtime performance.
  • H LANGUAGES AND COMPILER VERSIONS: Controlled language runtimes and compiler versions were used consistently for compilation, execution, and evaluation to ensure reproducibility across Multi-LCB.Specified versions include JavaScript’s Node.js 20.19.4, TypeScript’s Deno 2.3.4, and general compilers 1.11.0.
  • I PLATFORM ANALYSIS: Models vary by platform and may excel on LeetCode’s interview-style problems or AtCoder’s competitive programming tasks, depending on the programming language.Figures 10–12 compare overall and platform-specific Pass@1 performance across the twelve languages.
  • J DIFFICULTY ANALYSIS: Hard problems show the largest performance gaps between models, with performance degrading significantly as problem complexity increases.Figures 13–15 break down Easy, Medium, and Hard results across all programming languages using Pass@1 scores.

K TEMPORAL ANALYSIS

Monthly Pass@1 performance trends from 2023 to 2025 show a consistent decline across all evaluated programming languages and models. Top-performing models fell from approximately 80% to 60% Pass@1 scores, indicating a universal degradation pattern rather than a language-specific one.

  • Cross-language trend: Approximately 80% to 60% Pass@1 scores: top-performing models declined over time across all programming languages.The trends span monthly evaluations from 2023 to 2025.
  • Cross-language trend: Python, C++, C#, Java, and Go all exhibit monthly Pass@1 performance trends in the evaluated period.These language-specific trends are shown in Figures 16, 17, 18, 19, and 20.
  • Cross-language trend: Rust and TypeScript are likewise included in the monthly Pass@1 trend analysis.Their trends are shown in Figures 25 and 27, respectively.

L LANGUAGES ERRORS TYPE

Wrong-answer failures dominate across languages and models, indicating algorithmic correctness is the main bottleneck. Compiled languages and explicit input-parsing requirements introduce additional compiler, type, and runtime-error burdens, explaining multilingual performance gaps.

  • Error types: Wrong-answer errors are the largest failure source for every model in both Python and non-Python languages.This indicates that algorithmic correctness, rather than compilation or parsing, remains the primary bottleneck.
  • Error types: Compiled languages such as C++, Java, Rust, and Go produce substantially more compilation and type-related errors than Python.Common failures include missing imports, type mismatches, and incorrect signatures under strict compilation pipelines.
  • Error types: Runtime exceptions are far more frequent in Java, C#, and Go than in Python when explicit input parsing is required.Uniform STDIN/STDOUT formats expose weaknesses in input handling and data conversion.
  • Error types: TimeoutExpired failures occur more often in Java, Rust, and Go, particularly for reasoning-heavy models generating inefficient multi-step solutions.R1-0528 and Nemotron-32B are cited as examples of reasoning-tuned models prone to long-running implementations.
  • Error types: Error profiles remain remarkably consistent across models: Python is least error-prone, compiled languages add predictable errors, and verbose I/O handling amplifies runtime failures.These distributions mechanistically explain Pass@1 gaps, especially underperformance in Rust and C++ due to syntactic and type-level correctness challenges.
Loading 2606.20517v1…