PaperlayerView exact revision on arXiv

Source-linked AI summary

Dynamic Benchmarking of Reasoning Capabilities in Code Large Language Models Under Data Contamination

Simin Chen, Pranav Pusarla, Baishakhi Ray

arXiv:2503.04149v2cs.SEcs.AIcs.CL

TL;DR

Static, human-created benchmarks can be contaminated, and existing approaches may require manual effort or fail to preserve problem complexity. DyCodeEval uses agents to vary seed-problem context while preserving its core logic. The authors report that it reflects reasoning under contamination and that some models’ performance degrades on dynamic benchmarks, suggesting potential contamination.

  • Problem

    Static benchmarking can be contaminated, while existing approaches to contamination-resistant evaluation require manual effort and may not preserve semantic complexity.

    Source: §1 · p. 2

  • Method

    DyCodeEval uses LLM-based agents to generate diverse contexts for seed problems while preserving inherent complexity, with a validation agent checking correctness and consistency.

    Source: §1 · p. 2

  • Results

    DyCodeEval reflects reasoning capabilities in a manually crafted contamination environment, while some Code LLMs’ performance degraded significantly on its dynamic benchmarks, suggesting potential contamination.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors conclude that DyCodeEval-generated questions with DivPass provide a reliable evaluation tool, whereas Pass@k on current benchmarks has caused inflated model scores.

    Source: §6 · p. 9

    Generating problems has significant computational cost, and a very large LLM is required to achieve a high consistency rate.

    Source: §6 · p. 9

Abstract

from arXiv · show

The rapid evolution of code largelanguage models underscores the need for effective and transparent benchmarking of their reasoning capabilities. However, the current benchmarking approach heavily depends on publicly available, human-created datasets. The widespread use of these fixed benchmark datasets makes the benchmarking process to be static and thus particularly susceptible to data contamination, an unavoidable consequence of the extensive data collection processes used to train Code LLMs. Existing approaches that address data contamination often suffer from human effort limitations and imbalanced problem complexity. To tackle these challenges, we propose \tool, a novel benchmarking suite for evaluating Code LLMs under potential data contamination. Given a seed programming problem, \tool employs multiple agents to extract and modify the context without altering the core logic, generating semantically equivalent variations. We introduce a dynamic data generation methods and conduct empirical studies on two seed datasets across 21 Code LLMs. Results show that \tool effectively benchmarks reasoning capabilities under contamination risks while generating diverse problem sets to ensure consistent and reliable evaluations.

1. Introduction

Static benchmarks can be contaminated, while existing contamination-resistant approaches still require manual effort and may change problem complexity. DyCodeEval uses agents to vary problem context while preserving complexity, and its evaluations reflect reasoning under contamination, though some models’ scores degrade significantly.

1. Introduction

  • Code LLMs’ varied architectures and training algorithms make effective reasoning-capability benchmarking a key research focus.Without a standardized, transparent suite, assessing performance and driving improvements becomes challenging.Source: §1 · p. 1
  • Static benchmarking suites can be contaminated through unintended data crawling, while existing alternatives still need substantial manual input and may produce problems with different semantic complexity.Those differences make it hard to tell whether lower scores reflect reduced capability or increased benchmark complexity.Source: §1 · pp. 1–2
  • DyCodeEval uses LLM-based agents to generate diverse contexts for seed problems and a validation agent to check the resulting variants’ correctness and consistency.It changes complexity-unrelated context while preserving inherent complexity.Source: §1 · p. 2
  • Some Code LLMs’ performance degraded significantly on dynamic benchmarks, suggesting potential data contamination.The method also effectively reflects reasoning capabilities in a manually crafted contamination environment.Source: §1 · p. 2
  • DyCodeEval’s inherent randomness generates diverse problems and makes identical generations extremely unlikely, reducing contamination risk.The passage points to §4.4 for this result.Source: §1 · p. 2

4. Despite its randomness, DyCodeEval consistently

The paper characterizes static benchmarks as insufficient when contamination occurs and training processes are opaque. DyCodeEval instead generates context-varied problems without added dataset complexity, with the authors reporting reliable results under contamination scenarios.

4. Despite its randomness, DyCodeEval consistently

  • Static benchmarking schemas are insufficient for evaluating modern Code LLMs when contamination occurs and training processes lack transparency.Source: §4 · p. 2
  • DyCodeEval separates context from algorithm and dynamically generates benchmark problems without adding dataset complexity, aiming to mitigate data contamination.The authors describe the approach as supporting transparent and reliable benchmarking.Source: §4 · p. 2
  • DyCodeEval provides consistently reliable results under data-contamination scenarios and generates semantically diverse problems while maintaining stable benchmarking results.The authors contrast this with traditional static benchmarks, which they say can create a false sense of accuracy.Source: §4 · p. 2

2. Background & Related Work

Prior work covers human-created code-generation benchmarks, strategies for reducing contamination, and LLMs used as evaluators. The approaches differ in how they construct or protect evaluation data and assess model outputs.

2.1. Benchmarking Code LLMs

  • HumanEval introduced a human-crafted code-generation benchmark; EvalPlus addressed its and MBPP’s limited test coverage, and HumanEval-XL extended HumanEval to multilingual settings.A programming problem typically includes a prompt, canonical solution, and test cases; generated code is evaluated against hidden tests.Source: §2.1 · pp. 2–3

2.2. Data Contamination Free Benchmarking

  • Contamination-mitigation work includes protecting benchmark data, regularly updating datasets with recent questions, and dynamically generating or rewriting evaluation sets.Examples include encryption and privatization, LiveBench and LiveCodeBench, and DyVal, TreeEval, and ITD.Source: §2.2 · p. 3

2.3. LLM as Judgment Agent

  • LLMs have been used as examiners for training and testing data generation and for evaluating and comparing model outputs.The cited examples include PandaLM and DyVal, alongside a claim that LLMs can offer a faster, equally accurate alternative to human evaluation.Source: §2.3 · p. 3

3. Methods: DyCodeEval

DyCodeEval starts from a seed problem and varies its context while preserving algorithmic complexity. Its agent workflow proposes scenarios, assigns input contexts, rewrites prompts, and validates variants; the paper also analyzes generation collisions under stated randomness assumptions.

3.1. Design Overview

  • DyCodeEval’s metamorphic relationship treats algorithmic abstraction as complexity-related and context description as complexity-unrelated, so changing context preserves canonical solutions and complexity while varying semantics.The design overview uses four agents to propose scenarios, assign contexts, rewrite the prompt, and validate the variant against the original.Source: §3.1 · pp. 3–4

3.2. Detailed Design

  • The Scenario Proposer expands a pool of practical scenarios, and the Context Generation Agent selects one and assigns context to each input variable.For languages such as Python, the type-inference procedure gathers concrete inputs from ASSERT statements, recursively abstracts composite values, and produces types guaranteed to appear in the canonical solution, though it may miss return-value types.Source: §3.2 · p. 4
  • Rather than generating a prompt from scratch, the rewriting agent tailors the seed prompt to the chosen scenario and input-variable contexts.This uses a simpler rewriting task to create semantically diverse prompts.Source: §3.2 · p. 5
  • The Validator compares seed and rewritten prompts for preserved concept and factual accuracy, then checks their alignment with the seed’s canonical solutions; both responses must be “YES.”The paper also includes a human verification step.Source: §3.2 · p. 5
  • A generated recommendation-system problem using a user’s blog illustrates semantic diversity with equivalent complexity, which the authors attribute to the combination space of scenarios and contexts.Source: §3.2 · p. 5

3.3. Theoretical Collision Analysis

  • The collision analysis assumes uniform sampling and ignores randomness in rewriting, while its theorem statements bound repeated-generation and collision probabilities when M is much smaller than the scenario-context combination space.The stated analysis considers repeated runs on one seed problem and on a seed dataset.Source: §3.3 · p. 5

4. Evaluation

The evaluation tests DyCodeEval on standard seed datasets, simulated contamination, and in-the-wild models, then assesses generated-problem diversity, result stability, and foundation-model choice. The results indicate that dynamic benchmarking distinguishes contaminated from in-the-wild performance, produces diverse problems with stable scores, and depends on a capable foundation model for consistency.

4.1. Experimental Setup

  • The evaluation uses HumanEval and MBPP-Sanitized seeds and generates datasets with CLAUDE-3.5-SONNET using 50 scenarios and 50 randomly generated contexts per scenario.The study also evaluates three publicly available models in the contamination experiment.Source: §4.1 · p. 5, §4.2 · p. 6

4.2. Benchmarking Contaminated Model

  • Dynamic benchmarking prevents the artificially high Pass@1 scores seen on static benchmarks after fine-tuning on leaked data, while producing results comparable to manually curated datasets for uncontaminated models.A DEEPSEEK-CODER anomaly at 25% leaked data is hypothesized to reflect destabilized overfitting.Source: Figure 4 · p. 6, §4.2 · p. 6

4.3. Benchmarking In-the-Wild Model

  • For both seed datasets, in-the-wild models’ Pass@1 scores maintain a linear relationship, while overfitted models appear as outliers; QWEN2.5-CODER-7B falls outside the 95% confidence interval.The authors suggest it may be contaminated on both datasets.Source: Figure 5 · p. 7, §4.3 · p. 7

4.4. Problem Diversity

  • DyCodeEval generates greater syntactic and semantic diversity than rule-based baselines, which show high BLEU-4 and semantic similarity scores.The authors attribute the increased diversity to an LLM agent suggesting different scenarios and contexts.Source: §4.4 · p. 7, Table 1 · p. 8

4.5. Benchmarking Stability

  • Across 10 random runs, benchmarking-score variance is minimal compared with the mean, indicating stable results across generated datasets.The mean and standard deviation of Pass@1 scores are presented in Figure 6.Source: Figure 6 · p. 7, §4.5 · p. 8

4.6. Impact of Foundation LLM

  • Replacing CLAUDE-3.5-SONNET with CLAUDE-3.5-HAIKU lowers the generated-problem consistency rate from 95% to 83%.The authors conclude that robust and capable LLMs are needed as foundation models.Source: §4.6 · p. 8

5. Dynamic Evaluation Metrics

Pass@K evaluates solutions in a fixed problem context, which can make it difficult to distinguish reasoning from memorization under contamination. DyPass evaluates semantic prompt variants of seed problems; on contaminated models it is more sensitive to leakage, while aligning with static metrics when models are not trained on the leaked dataset.

  • DyPass creates semantic prompt variants that preserve the seed problem’s complexity by changing its description while retaining the underlying algorithmic abstraction.Unlike Pass@K’s fixed prompt context, these variants expand the input space and make full coverage more challenging.Source: §5 · p. 8
  • DyPass is designed to better distinguish whether a model memorizes problem context or genuinely reasons to solve the problem.Source: §5 · p. 8
  • When trained on leaked data, Pass@K scores rise from 0.82 to 0.89, whereas DyPass@K shows a slight decrease; without leakage, the metrics are consistent.The comparison covers contaminated and in-the-wild models.Source: Table 2 · p. 8, §5 · pp. 8–9

6. Conclusion

DyCodeEval dynamically generates semantically equivalent, diverse problems to address data contamination, and the experiments report reliable evaluation with DivPass. The authors identify generation cost and potentially confusing excess information as limitations.

6. Conclusion

  • DyCodeEval dynamically generates semantically equivalent, diverse problems to combat data contamination, and DivPass provides a reliable evaluation tool compared with inflated Pass@k scores on current benchmarks.The authors describe this as a promising path forward.Source: §6 · p. 9
  • A very large LLM is required for a high consistency rate, making generation cost a significant concern; generated questions may also contain excessive, potentially confusing information.The authors suggest improving generation efficiency and experimenting further with prompt generation.Source: §6 · p. 9

Impact Statement

The Impact Statement links contamination with inflated evaluation accuracy and proposes DyCodeEval for measuring capabilities. The accompanying proofs analyze collision probabilities in a finite sampling space.

Impact Statement

  • Data contamination can inflate evaluation accuracy and obscure LLMs’ true performance; DyCodeEval is proposed to measure capabilities more accurately and provide deeper behavioral insights.Source: Impact Statement · p. 9

A.1. Proof of Theorem 3.1

  • Theorem 3.1’s proof models a space of N = ||S|| × ||C|| outcomes and bounds the probability of distinct samples using a logarithm inequality.The proof assumes independent draws and develops a bound through the supplied inequality steps.Source: §A.1 · p. 12

A.2. Proof of Theorem 3.2

  • Theorem 3.2 considers independent uniform samples from a space of size N, derives the probability that all M samples are distinct, and treats collision probability as its complement.Its bound assumes M << ||S|| × ||C||.Source: §A.2 · pp. 12–13

A.3. Proof of Theorem 3.3

Theorem 3.3’s proof represents each sample as a tuple of draws and considers the chance that a later sample exactly matches the initial one. It then relies on Theorem 3.1.

A.3. Proof of Theorem 3.3

  • Each sample is a D-tuple drawn uniformly from the possible sets, and the proof evaluates the probability that a later draw exactly matches the initial sample before invoking Theorem 3.1.Source: §A.3 · p. 13

B. Dataset Description.

The appendix describes HumanEval and MBPP-Sanitized, then presents prompts and examples for proposing scenarios, contextualizing variables, rewriting problems, and validating generated instructions and code.

B. Dataset Description.

  • HumanEval has 164 Python problems with prompts, canonical solutions, and tests; MBPP-Sanitized has 427 crowdsourced problems and is zero-shot, with prompts reformatted for the experiments.Source: B. Dataset Description · p. 13

C.1. Template for Scenario Proposer Agent

  • The Scenario Proposer Agent is prompted to suggest practical, general real-world scenarios across specified areas and return each on a separate line.Source: §C.1 · p. 14

C.3. Prompt for Context Generator Agent

  • The Context Generator Agent receives a problem, input types, and a scenario, then supplies one scenario-tailored context for each variable without extra reasoning.The example contextualizes temperatures as daily city readings and a threshold as a critical heat level.Source: §C.3 · p. 15

C.4. Example for Context Generator Agent

  • The example shows variable contexts attached to a problem and scenario, illustrating how input meanings are made relevant to the selected context.Source: §C.4 · p. 16

Prompt for Prompt Rewriter Agent

  • The rewrite should preserve the seed problem’s complexity and constraints, remain solvable with the same approach, and be clear, concise, and logically consistent in the selected scenario.The description is limited to 1–3 sentences; the example provides a bank-account operations problem, a social-networking recommendation scenario, and contextualized user-interaction variables.Source: Prompt for Prompt Rewriter Agent · p. 17, Example for Prompt Rewriter Agent · p. 18

Prompt for Validation Agent 1

Prompt for Validation Agent 2

D. Human Verification

Manual verification sampled 30 original/generated problem pairs from each dataset, with two graduate-level reviewers assessing preservation of the core algorithm and complexity. They resolved all initial disagreements, yielding 95% overall agreement.

D. Human Verification

  • 95% overall agreement followed two reviewers' assessment of 60 original/generated problem pairs, with all three initial disagreements resolved through discussion.Each reviewer independently assessed whether the core algorithm and complexity were preserved.Source: D. Human Verification · p. 20
  • Reviewers assessed 30 randomly sampled problem pairs from each of the two datasets, for 60 pairs total.Each pair matched a benchmark problem with its generated variant.Source: D. Human Verification · p. 20
  • Two graduate-level students independently checked whether each pair preserved the core algorithm and complexity, discussing disagreements until consensus.Source: D. Human Verification · p. 20
Loading 2503.04149v2…