PaperlayerView exact revision on arXiv

Source-linked AI summary

CodeHacker: Automated Test Case Generation for Detecting Vulnerabilities in Competitive Programming Solutions

Jingwei Shi, Xinxiang Yin, Jing Huang, Jinman Zhao, Shengyu Tao

arXiv:2602.20213v2cs.SEcs.AIcs.CR

TL;DR

Existing code-generation evaluations can miss subtle implementation vulnerabilities because their test cases lack targeted coverage. CodeHacker addresses this gap with an automated, program-specific adversarial test-generation framework and reports stronger evaluation robustness, while its generated cases also support training and adversarial-reasoning assessment.

  • Problem

    Existing benchmarks may lack coverage for subtle corner cases, allowing incorrect program solutions to pass evaluation.

  • Method

    CodeHacker searches the valid input space for submission-specific counterexamples using stress testing, targeted semantic generation, and other adversarial strategies, with checker refinement before evaluation.

  • Results

    CodeHacker improves true negative rate (TNR) and true positive rate (TPR) on existing code datasets, strengthening evaluation robustness.

  • Takeaways & Limitations

    Generated hacks provide a basis for CodeHackerBench and distinguish advanced reasoning models through their ability to produce valid adversarial inputs.

  • Takeaways & Limitations

    The agent may miss subtle algorithmic or performance-related failures, and the current benchmark covers only a subset of languages and problem domains.

Abstract

from arXiv · show

The evaluation of Large Language Models (LLMs) for code generation relies heavily on the quality and robustness of test cases. However, existing benchmarks often lack coverage for subtle corner cases, allowing incorrect solutions to pass. To bridge this gap, we propose CodeHacker, an automated agent framework dedicated to generating targeted adversarial test cases that expose latent vulnerabilities in program submissions. Mimicking the hack mechanism in competitive programming, CodeHacker employs a multi-strategy approach, including stress testing, anti-hash attacks, and logic-specific targeting to break specific code submissions. To ensure the validity and reliability of these attacks, we introduce a Calibration Phase, where the agent iteratively refines its own Validator and Checker via self-generated adversarial probes before evaluating contestant code.Experiments demonstrate that CodeHacker significantly improves the True Negative Rate (TNR) of existing datasets, effectively filtering out incorrect solutions that were previously accepted. Furthermore, generated adversarial cases prove to be superior training data, boosting the performance of RL-trained models on benchmarks like LiveCodeBench.

1 Introduction

CodeHacker addresses the limited coverage and weak targeting of existing code-evaluation tests by automating program-specific adversarial test generation. The framework formalizes hacking as counterexample search and improves evaluation robustness while supporting a new benchmark for adversarial reasoning.

  • Motivation: Evaluation outcomes depend sensitively on test-case quality and coverage, motivating better construction and validation procedures.The introduction frames test design as a central challenge for assessing whether LLM-generated programs satisfy constraints and execute correctly.
  • Motivation: Increasing test counts or input-space coverage coarsely is insufficient because effective cases must target vulnerabilities in concrete implementations.Successful hacks require reasoning about algorithmic assumptions, boundary conditions, and complexity constraints.
  • Approach: CodeHacker treats individual programs as first-class objects and searches for failure-inducing counterexamples within a competitive-programming setting.Its code-aware adversarial perspective couples test construction with the evaluated program’s execution behavior rather than relying only on static or heuristic tests.
  • Approach: The framework formalizes hacking as adversarial test generation and uses an LLM-driven agent to search for high-value corner cases and logical counterexamples.The stated target is to find cases that traditional mutation-based or prompt-based generation methods may miss.
  • Results: CodeHacker improves both true negative rate (TNR) and true positive rate (TPR), strengthening evaluation robustness on existing code datasets.The introduction reports substantial improvements in both metrics without providing their numerical values in the supplied passage.
  • Results: CodeHackerBench provides a new evaluation setting for characterizing models’ ability to reason about incorrect code and extreme failure scenarios.The benchmark is built around the adversarial cases generated by the framework.

2 Related Work

Prior work spans handcrafted benchmarks, mutation-based input generation, mined competitive-programming hacks, and reinforcement-learning methods for code generation. These approaches differ in scalability, adversarial intent, and training objectives.

  • Code Benchmarks: Manual benchmarks such as MBPP, HumanEval, and LiveCodeBench use handcrafted tests that target problem-specific corner cases but are expensive and difficult to scale.The passage positions manual construction as more suitable for small evaluation sets than large training corpora.
  • Code Benchmarks: Mutation-based approaches automatically recombine or mutate existing inputs to improve coverage, but the supplied passage does not detail their full limitations.The passage introduces mutation-based testing as one of three major paradigms in existing code datasets.
  • Adversarial Data: Codehacks mines historical Codeforces hacks, but inaccessible victim submissions require post-hoc matching based on observed execution failures.Mutation- or randomly generated tests expose false negatives but do not model adversarial intent.
  • RLHF and RLVR: PPO, GRPO, and related methods provide reinforcement-learning foundations increasingly used with verifiable rewards for code generation.PPO uses on-policy rollouts and value estimation, whereas GRPO removes the explicit critic through group-based baselines.

3 Method

CodeHacker models hacking as submission-specific counterexample search: an LLM agent generates valid inputs that expose target-program failures under a calibrated judging process. It combines explicit success conditions, iterative Validator and Checker refinement, and complementary stress, LLM, and anti-hash generators.

  • Problem formulation: CodeHacker searches the valid input space for submission-specific counterexamples that trigger non-AC verdicts, unlike classical perturbation-based adversarial attacks.The agent generates inputs while the environment supplies ground-truth outputs and judge verdicts.
  • Problem formulation: A successful hack requires a valid input, acceptance by the standard solution, and a non-AC verdict for the target submission.These conditions separate valid counterexamples from malformed tests or failures of the reference solution.
  • Agent formulation: The agent interacts with problem–submission history, proposes test inputs, receives judge feedback, and stops after T trials or the first non-AC verdict.Hack success is the indicator that at least one proposed input succeeds; overall HSR averages this indicator over evaluated pairs.
  • Evaluation tool calibration: Before generation, CodeHacker iteratively attacks and patches the Validator and Checker to eliminate invalid-input acceptance and false verdicts.Validator attacks test bypass and rejection cases, while Checker attacks deceptive wrong outputs and rejection of valid outputs.
  • Evaluation tool calibration: Checker updates use small boundary inputs, explicit reasoning, and independent cross-verification to reduce contamination from incorrect generated valid outputs.The supplied passages state that the checker is updated only after passing cross-verification.
  • Scope boundary: Human experts patch difficult Validators or Checkers in fewer than 5% of cases, while Phase II hack-case generation remains fully autonomous.The intervention is concentrated on high-difficulty problems and is described as a one-time per-problem setup cost.
  • Hack case generation: Stress, LLM-based, and anti-hash generators target complementary vulnerabilities, including extreme-size failures, semantic bugs, resource limits, and hash collisions.Stress testing explores boundary structure and size; LLM generation follows the analyst’s vulnerability plan; anti-hash generation targets fixed-parameter polynomial rolling hashes.

4 Experiments

Experiments evaluate CodeHacker across benchmark construction, adversarial hacking, metric correction, reinforcement learning, and component ablations. Results show stronger discrimination, improved robustness, and substantial dependence on iterative refinement and targeted adversarial cases.

  • Experimental Setup: The study samples 2,000 CodeContest+ problems, split evenly between traditional-judge and special-judge problems, with refined validation and checking used for fairness.The experiments focus on C++ submissions and use LiveCodeBench’s 287 AtCoder problems for reinforcement-learning evaluations.
  • Main Results: 64.83% HSR is achieved by DeepSeek V3.2, compared with 51.40% for GPT-5-Mini and 35.65% for Gemini-3.0.DeepSeek V3.2 falls to 23.76% without explicit reasoning, a 2.7× decline.
  • Main Results: CodeHackerBench reveals metric inflation by filtering incorrect solutions that pass weak tests, including a rank reversal where GPT-5-Mini is more robust than Gemini-3.0-Flash.The authors interpret lower Pass@1 scores under adversarial evaluation as correction rather than capability loss.
  • Main Results: The augmented training data consistently improves reinforcement-learning performance, including on out-of-distribution LiveCodeBench problems.The authors characterize adversarial inputs as dense reward signals that promote boundary-condition handling and logical robustness.
  • Ablation Study: The refinement loop is the most critical ablated component, with HSR dropping from 51.40% to 46.60% when removed.The Code Analyst raises HSR from 49.86% to 51.40%, while Stress Test raises it from 50.12% to 51.40%.
  • Ablation Study: The Anti-Hash Generator breaks 100% of the rolling-hash submissions on which it is triggered, despite applying to approximately 0.4% of submissions.This module targets a narrow algorithmic shortcut rather than broad vulnerability patterns.
  • Ablation Study: Hack-case augmentation raises TNR to 96.05% on Special Judge problems, while the refined checker raises TPR to 96.62%.The refined validator slightly lowers TNR from 82.18% to 82.08% by removing invalid inputs that masked weaker discrimination.

5 Conclusion

CodeHacker strengthens competitive-programming evaluation by refining judging components and adding targeted adversarial tests. Its results also position valid hack generation as a discriminator of advanced reasoning and a possible component of future evaluation infrastructure.

  • CodeHacker iteratively refines competitive-programming evaluations through Self-Hacking and targeted adversarial inputs.
  • The augmented subset achieves the highest True Negative Rate, providing a more trustworthy standard than existing benchmarks.
  • CodeHackerBench evaluates adversarial reasoning, with valid hack generation differentiating advanced reasoning models.
  • The framework is envisioned for quality assurance, personalized training, and transfer to broader issue-solving benchmarks.

Ethic Statement

The paper emphasizes ethical constraints for automated hacking research, especially respecting platform data policies and protecting public judging infrastructure. It requires adversarial evaluation to run locally and offline rather than through public online judges.

  • The research prohibits unauthorized bulk scraping of hack data from platforms such as Codeforces and QOJ.The paper directs researchers to use officially released datasets or compliant access methods.
  • Automated evaluation must not submit generated test cases to public online judges because this creates denial-of-service risk.
  • CodeHacker is designed to conduct adversarial evaluation entirely offline in a local, sandboxed environment.

Limitations

The paper acknowledges that CodeHacker is not infallible and may miss some difficult failures. Future work therefore must expand its detection capabilities and keep pace with evolving algorithmic challenges.

  • CodeHacker can generate valuable test cases but does not identify every vulnerability.
  • The agent may miss edge cases and subtle algorithmic flaws.
  • Future work will expand failure detection and adapt the framework to changing competitive-programming challenges.

D RL Training Hyperparameters

The reinforcement-learning setup trains Qwen3-4B with DAPO using long-context generation and specified exploration and optimization settings. Evaluation separates Codeforces-derived training data from AtCoder-derived LiveCodeBench problems to reduce contamination risk.

  • Qwen3-4B is trained with DAPO using a 24,000-token maximum generation length and global batch size 32.The actor learning rate is 5 × 10^-7, with N = 8 sampled solutions per prompt.
  • The training uses dual clipping with ϵ_low = 0.2 and ϵ_high = 0.
  • Experiments were conducted on NVIDIA H100 GPUs, with detailed DAPO settings reported in Table 8.
  • Training uses Codeforces problems, while RL evaluation uses LiveCodeBench problems from AtCoder.
  • The distinct source platforms are intended to prevent overlap between training and evaluation data.The paper attributes observed improvements to generalization rather than memorization of specific problem patterns.

G Detailed Reinforcement Learning Results

The section combines anti-hash collision generation, checker and validator analysis, and stress-oriented test construction to expose weaknesses in competitive-programming evaluations.

  • Anti-hash attacks: The lattice method weights nonzero hash remainders heavily, reduces the constructed basis, and converts a short vector into colliding strings.The resulting vector is intended to have remainder R = 0 and a bounded, nonzero difference array d.
  • Anti-hash attacks: CodeHacker targets hash vulnerabilities using lattice reduction for polynomial rolling hashes and birthday-paradox sampling for generic or smaller-modulus hashes.Lattice reduction seeks a small nonzero difference array satisfying modular constraints, while probabilistic sampling finds collisions when approximately 75,000 inputs suffice for a typical 32-bit hash.
  • Checker analysis: Checker analysis exposes weak parsing that mishandles hyphens, illegal characters, and multiple valid output formats.The refined checker performs character-level validation before verifying the grouping structure and content.
  • Validator analysis: Validator analysis identifies mismatches between stated input constraints and implemented ranges, including rejection of valid zero-valued memory exponents.The described memory-block constraint permits bj = 0, but the original validator excludes it.

I.2.2 Vulnerability Analysis

The vulnerability analysis presents validator and checker failures caused by incorrect constraint handling, overly permissive ranges, and weak output validation.

  • Validator flaws: The original validator rejects valid Codeforces 309_C cases by enforcing bj ∈ [1, 60] instead of allowing bj = 0.This excludes valid memory blocks of size 2^0 = 1 and can mask solution bugs involving the smallest block size.
  • Validator flaws: The revised validator changes the accepted range to [0, 29], allowing bj = 0 and restoring the stated input domain.The adjustment is intended to process all valid edge cases correctly.
  • Validator flaws: A separate validator is too permissive for Codeforces 177_C2, accepting up to roughly 2 million friendship edges although the problem limits k to 100,000.This can expose solutions optimized for the stated bound to unexpectedly large inputs and cause TLE.
  • Checker flaws: The Codeforces 25_B checker is vulnerable to malformed hyphen placement and incomplete character validation, motivating character-level structural checks.The examples distinguish weak and corrected checker behavior for the same problem.
  • Solution flaws: The analysis also examines a submission using modulo heuristics and floating-point square checks instead of tracking factorial prime-factor parity.The described vulnerability is a false generalization in the solution logic.

J.3 Vulnerability Analysis

The analysis shows CodeHacker exposing false generalizations and numerical heuristics that fail on cases requiring exact structural or number-theoretic reasoning.

  • Heuristic number theory: The factorial-product submission contains a logic error because its arithmetic square checks have no guarantee of covering the required prime-factor parity conditions.Floating-point operations add precision risk, but the primary failure is logical.
  • Heuristic number theory: N = 998787 is identified as a failure case where the heuristic does not find the optimal removal set, unlike a correct prime-factor-parity hashing approach.The case lies in the branch where N mod 4 = 3.

K.2 Vulnerability Analysis

The analysis illustrates how CodeHacker generates adversarial inputs for checker, validator, arithmetic, and solution-logic vulnerabilities, while enforcing valid test-case construction and execution-based diagnosis.

  • Validator and checker prompts: Checker prompts test missing checks and overly rigid comparisons, including invalid outputs accepted and valid alternative outputs rejected.The prompt framework explicitly distinguishes permissive and strict checker failures.
  • Validator and checker prompts: Validator prompts extract stated constraints, audit enforcement, and generate either illegal accepted inputs or legal rejected inputs.These correspond to false positives and false negatives in validator behavior.
  • Execution-based diagnosis: The execution workflow supports arbitrary Python probing, compilation and execution of target C++ code, and a final vulnerability report.The example identifies integer overflow by comparing a large computed result with 32-bit limits after a small-case check succeeds.
  • Stress testing: Stress-test generators must produce valid inputs, bias toward large instances for TLE detection, and occasionally include small edge cases.The generator requirements also prohibit undefined behavior in the generator itself.
  • Agent role: CodeHacker targets wrong answers, runtime errors, time-limit exceedances, and memory-limit exceedances by generating inputs designed to expose specific submission weaknesses.The agent operates against a problem statement and contestant submission.
Loading 2602.20213v2…