Noisy Test-Time Reinforcement Learning for Code LLMs

1 Introduction

• Code generation quality can drop significantly after even minor prompt changes, and vague or erroneous instructions from less-experienced users pose further challenges.The cited discussion notes changes such as rephrasing with equivalent descriptions.Source: §1 · p. 1
• Paired clean-noisy training data is costly to prepare and can encourage overfitting to specific noise patterns.Iterative self-denoising may improve noisy-prompt generation, but the cited discussion says it does not fundamentally improve robustness and increases inference time.Source: §1 · p. 2
• Entropy-based and self-rewarding test-time methods lack prompt-perturbation-specific designs, and noisy inputs can impair their signals.For code, raw majority voting can also split logically equivalent programs that differ in variable names or formatting.Source: §1 · p. 2
• NTRL-Code adapts from unlabeled noisy test samples, using denoised rollouts to estimate structure-aware proxy targets and a hybrid reward to update the policy.The reward combines format validity, code similarity, and no-repetition signals.Source: §1 · p. 2

2.1 LLM Robustness

• Robustness research evaluates prompt perturbations, including adversarial textual attacks, and has introduced tools such as a Prompt Sensitivity Index.The cited discussion also reports that LLMs are not robust to adversarial prompts.Source: §2.1 · p. 2
• Proposed ways to mitigate prompt perturbations include iterative self-denoising, hidden-state alignment, adversarial training, and prompt optimization.Agrawal et al. (2025) found iterative self-denoising effective for improving generation quality with noisy prompts.Source: §2.1 · p. 3

2.2 AI coding Agent

• Few studies have focused on improving coding-agent robustness; cited approaches use adversarial prompts for instruction-tuning data or edit unstable model layers.Hossen and Hei (2024) used GPT models to generate adversarial prompts, while Liu et al. (2025b) updated unstable layers using paired original and perturbed prompts.Source: §2.2 · p. 3

3.1 Overview of NTRL-Code Framework

• NTRL-Code uses denoised-prompt rollouts to select a proxy target, then updates the policy with GRPO on candidates generated from the original noisy prompt.The reward compares noisy-branch candidates against the proxy target through a hybrid reward function.Source: §3.1 · p. 3

3.2 Prompt Correction via Self-denoising

• A length-change threshold rejects unreliable denoising and reverts to the original prompt when the rewrite is too aggressive.The failure threshold is τlen = 0.5; rejected rewrites include cases of over-deletion, hallucinated requirements, or semantic drift.Source: §3.2 · p. 3

3.3 AST-based Structural Aggregation

• AST aggregation canonicalizes parsed programs to cluster structural matches and selects a representative from the largest cluster as the proxy target.For unparsable programs, lexical normalization and token-level Jaccard similarity provide a fallback; candidates cluster when similarity is at least ϵ = 0.92.Source: §3.3 · p. 4

3.4 Test-time RL with Hybrid Rewards

• A hybrid reward combines code similarity, format validity, and no-repetition signals to compare noisy-branch candidates with the proxy target.Similarity combines normalized Levenshtein similarity and Jaccard similarity; the no-repetition reward measures repeated code using sliding windows.Source: §3.4 · pp. 4–5

3.5 Why Decoupled Test-Time RL Improves Robustness

• Decoupling target estimation from policy updates anchors noisy-prompt outputs to denoised rollouts, encouraging noise-invariant mappings rather than increased confidence on noisy inputs.The authors state that the denoised anchor, structure-aware aggregation, and small number of update steps make overfitting to a specific perturbation less likely.Source: §3.5 · pp. 5–6

4.1 Experiment Setup

• Across character-, word-, and paragraph-level perturbations, NTRL-Code is evaluated on LeetCode-Bench, HumanEval, and MBPP using Pass@1.The experiments use three perturbation levels across the three datasets, producing nine noisy benchmarks.Source: §4.1 · p. 6

4.2 Main Results

• Across nine settings, gains averaged 4.0%, 3.5%, and 3.0% for DeepSeek-Coder-1.3B, DeepSeek-Coder-6.7B, and Qwen2.5-Coder-32B, respectively.NTRL-Code consistently improved over the original models across datasets and perturbation levels; iterative self-denoising remained inferior overall.Source: §4.2 · pp. 6–7, Table 1 · p. 7

4.3 Self-Denoising Ablation

• Removing self-denoising reduced performance by 2.5%, 3.1%, and 2.1% under character-, word-, and paragraph-level perturbations, respectively.The ablation evaluates DeepSeek-Coder-1.3B across all three datasets; some results fell below the original model.Source: §4.3 · p. 7, Table 2 · p. 7

4.4 Rewards Ablation

• The full reward combining format, no-repetition, and code similarity achieved the best performance among tested configurations.Under paragraph-level perturbations, omitting similarity reward yielded a 5.5% drop.Source: §4.4 · p. 8, Table 3 · p. 8

4.5 Performance Evolution on RL Steps

• Figure 3 tracks NTRL-Code's step evolution on HumanEval using DeepSeek-Coder-1.3B.Source: Figure 3 · p. 8

4.6 Results on Mixed-Type Perturbations

• On HumanEval, NTRL-Code improved performance across four mixed-perturbation settings, with its largest improvement of 6% when all three perturbation types were combined.The base model's performance deteriorated as perturbations became more complex.Source: Table 4 · p. 8, §4.6 · p. 8

4.7 Generalization Analysis

• NTRL-Code improved performance in cross-dataset tests and when evaluated on a noise type different from the one used for training.The cross-dataset test fine-tuned on HumanEval and evaluated on LeetCode Bench.Source: §4.7 · pp. 8–9, Table 5 · p. 9

• NTRL-Code is a label-free test-time reinforcement learning framework for improving code LLM robustness under noisy prompts.Source: §5 · p. 9
• It estimates structure-aware proxy targets from rechecked denoised rollouts while updating the policy on original noisy prompts with a hybrid reward.Source: §5 · p. 9
• Experiments across three benchmarks, multiple perturbation types, and several base models show consistent robustness gains, while ablations support the roles of self-denoising, AST-based aggregation, and hybrid reward.Source: §5 · p. 9

Limitations

• The evaluated noise covers character-, word-, paragraph-level, and mixed perturbations, but not all realistic user-side noise.Requirement-level issues such as missing specifications or ambiguous constraints are beyond the stated scope.Source: Limitations · p. 9
• Proxy targets may be incorrect when candidates share a structural bug, denoising changes a requirement, or correct programs split across AST clusters.The method does not use executable tests during optimization; execution-based verification could improve reliability.Source: Limitations · p. 9
• Test-time adaptation requires multiple rollouts and GRPO updates, so it costs more than standard single-pass inference.The authors describe batching unlabeled prompts to update a shadow policy in the background, which reduces user-facing latency but still uses additional compute.Source: Limitations · p. 9
• The framework requires access to model parameters for test-time updates and therefore cannot be directly applied to closed-source APIs.The authors identify black-box or training-free variants as future work.Source: Limitations · p. 9

A Recap of Test-time Reinforcement Learning

• Multiple policy samples produce candidate outputs, whose majority vote estimates a proxy ground truth for test-time reinforcement learning.The policy can then be updated using GRPO, with relative performance within a group represented by the advantage and the reward function.Source: A Recap of Test-time Reinforcement… · p. 13

B.1 Details of Perturbation Simulation

• The benchmarks use keyboard typos, synonym or near-synonym substitutions, and language-model rephrasing to simulate character-, word-, and paragraph-level perturbations.The evaluations draw on LeetCode-Bench, HumanEval’s Python subset, and selected MBPP tasks.Source: §B.1 · p. 13

B.2 Experiment Details for NTRL-Code

• Experiments ran on one node with 8 NVIDIA H200 GPUs, each with 140 GB of memory.The maximum response length was 3,072 tokens.Source: §B.2 · p. 13
• Training used a 5 × 10^-7 base learning rate with cosine decay and warmup, while rollouts used temperature 0.6.The denoising agent used deterministic decoding and a 512-token prompt cap; inference used greedy decoding and a 1,024-token response cap.Source: §B.2 · p. 13
• AST-parsable programs were canonicalized and clustered by exact serialized-tree hash, while unparsable programs used normalized token-level Jaccard similarity.The lexical clustering threshold was 0.92; denoised prompts were rejected when their length-change ratio exceeded 0.5.Source: §B.2 · p. 13
• Table 6 reports Pass@1 on original and character-, word-, and paragraph-perturbed HumanEval, LeetCode-Bench, and MBPP benchmarks.The caption identifies the table’s scope but gives no cell values.Source: Table 6 · p. 14
• Table 7 covers frontier closed-source models on HumanEval under clean and perturbed prompts.The caption does not state the performance values or a comparison outcome.Source: Table 7 · p. 14

C.1 Robustness Evaluation for Code LLMs

• All six evaluated open-source code models show noticeable performance drops across character-, word-, and paragraph-level perturbations.Frontier closed-source models also show nontrivial drops under most noise types, including GPT-5-Codex under character- and word-level noise.Source: §C.1 · p. 14

C.2 Analysis of the Noise Severity

• Perturbation severity increases from character to word to paragraph level on every benchmark, while semantic similarity remains high.The authors interpret the high cosine similarity as indicating that the perturbations preserve underlying meaning while varying surface form.Source: §C.2 · p. 14, Table 8 · p. 15

C.3 Effect of Number of Voting Samples

• N = 32 generally gives the worst results; gains diminish as the majority-voting sample count grows, so most experiments use 64 samples.Figure 4 presents performance for 32, 64, 96, and 128 samples on HumanEval with DeepSeek-Coder-1.3B.Source: §C.3 · p. 14, Figure 4 · p. 15

C.4 Performance on the Clean Dataset

• About 3.2% average gains appear across three clean benchmarks for DeepSeek-Coder-1.3B.The authors attribute the improvements to test-time training effects.Source: §C.4 · p. 15, Table 9 · p. 15

C.5 Hyperparameter Sensitivity Analysis

• The length-threshold sweep tracked pass@1 and revert rate, while lexical-clustering settings kept pass@1 within a 0.6-point band (54.9–55.5%).The lexical threshold values tested were 0.80, 0.85, 0.90, 0.92, and 0.95.Source: §C.5 · p. 15, Table 11 · p. 15, Table 10 · p. 15
• The default reward weights (α, β) = (0.5, 0.25) performed best; equal weighting and similarity-only weighting performed worse.The comparison supports balancing similarity guidance with the additional reward components.Source: §C.5 · p. 15, Table 12 · p. 15
• Equal weighting of normalized Levenshtein distance and Jaccard similarity performed best among the tested Levenshtein weights.The tested weights were 0.4, 0.5, and 0.8; 0.5 is the equal-weight setting.Source: §C.5 · p. 16, Table 13 · p. 16
• A window/step ratio of 0.5 achieved the best pass@1 among the tested ratios of 0.25, 0.5, and 1.0.Source: §C.5 · p. 16, Table 14 · p. 16

C.6 Additional Ablation Results

• On DeepSeek-Coder-6.7B with HumanEval word-level perturbation, NTRL-Code improves over the base model, and adding self-denoising gives a further consistent gain.This repeats the trend observed at 1.3B.Source: §C.6 · p. 16, Table 15 · p. 16

C.7 Proxy Target Analysis

• The proxy-target audit sampled 32 candidates per HumanEval problem and executed both candidates and the structurally aggregated proxy against benchmark tests.It defined proxy pass rate, random-candidate pass rate, reward–correctness correlation, and consensus–correctness correlation as audit measures.Source: §C.7 · p. 16, Table 16 · p. 16
• The audit’s candidate pass rate measures the expected pass rate of a uniformly random candidate, making its gap from proxy pass rate the aggregation selection gain.The audit also tracks reward–correctness and consensus–correctness correlations.Source: §C.7 · p. 16

C.8 NTRL-Code under Requirement-level Noise

• NTRL-Code shows positive transfer to requirement-level noise, especially in intent recovery and robustness, but contradiction and missing-information handling remain limited.The authors characterize this as a positive trend rather than conclusive evidence.Source: §C.8 · p. 17, Table 17 · p. 17

C.9 Cross-language Evaluation

• +1.2 to +5.6 points: NTRL-Code improves over the base model in all four C++ and JavaScript HumanEval settings, including clean and character-noise conditions.For these languages, aggregation falls back to normalized-Jaccard clustering because the Python AST parser fails.Source: §C.9 · p. 17, Table 18 · p. 17

D Summary of Prompts

• Figures 5–7 show prompts for paragraph-level perturbation, NTRL-Code self-denoising, and the baseline’s few-shot denoising, respectively.Source: D Summary of Prompts · p. 17, Figure 5 · p. 17, Figure 6 · p. 17

E Case Study

• Figures 8–10 compare Qwen2.5-Coder-32B outputs for original prompts, perturbed prompts, and NTRL-Code-enhanced perturbed prompts across character-, word-, and paragraph-level cases.Source: E Case Study · p. 17

F.1 Deployment and Computational Cost

• Adaptation is asynchronous: a shadow policy updates in the background from accumulated unlabeled noisy prompts, and validated updates can be deployed periodically.Rollout and update costs are amortized over future requests rather than added to each query’s response latency.Source: §F.1 · pp. 17–18
• Multiple rollouts make adaptation cost scale as O(TB(Kd + Kn)); this may not suit applications requiring immediate per-query personalization.The framework shifts this compute to background adaptation rather than adding synchronous inference-time overhead to each query.Source: §F.1 · p. 18

F.2 Reliability and Failure Modes of Proxy Targets

• The AST-based proxy target is approximate and can fail when sampled programs share a structural bug, denoising changes an important constraint, or correct implementations split across clusters.In such cases, the largest cluster may reflect model prior bias rather than the true solution.Source: §F.2 · p. 18
• NTRL-Code mitigates proxy-target risk by rechecking denoised prompts, canonicalizing ASTs, and combining similarity, format-validity, and anti-repetition reward signals.Without executable feedback, it cannot guarantee that the selected proxy target is correct.Source: §F.2 · p. 18

F.3 Discussion on the Surface-level Metrics in Reward Function

• Levenshtein and Jaccard are lightweight reward-shaping signals after target selection; they do not select the proxy target for parsable programs.Canonicalized AST clustering selects those targets, while Jaccard also serves as a fallback for unparsable outputs.Source: §F.3 · p. 18

F.4 Discussion on the AST-based Structural Aggregation

• Canonicalized ASTs are hashed only as exact-match clustering keys; the policy neither sees nor reasons about the MD5 hashes.Canonicalization replaces variable and function names with placeholders before serialization and hashing.Source: §F.4 · p. 22

F.5 Potential Risks

• Without security mechanisms, intentionally injected harmful instructions could enter NTRL-Code’s learning process and shape its updates.Source: §F.5 · p. 22