Source-linked AI summary
Confidence-Gated Transductive Test Generation for Code Reranking
Sungjae Lee, Youngsik Yoon, Seockbean Song, Siwei Wang, Wei Chen, Jungseul Ok
TL;DR
Synthesized tests can provide execution-based signals for evaluating and ranking code, but reliable expected outputs are difficult to obtain. CoTT uses inductive prediction and invokes transductive generation only for low-confidence inputs. Across two reranking benchmarks, it offers a favorable efficiency–effectiveness trade-off relative to always-transductive generation.
Problem
Synthesized tests can supply execution-based signals for code evaluation, but reliable expected outputs are difficult to obtain, especially when human-written tests are sparse.
Source: §1 · p. 1Method
Results
From $0.077 to $0.065 average inference cost, CoTT preserves most TT w/o Gating benefit on HumanEval-R+ and performs better on MBPP-R+, where it leads three reranking metrics.
Source: §4 · p. 4Takeaways & Limitations
Across both datasets, confidence-gated allocation maintains strong reranking performance at lower cost than always-transductive generation.
Source: §6 · p. 5Evaluation uses a single efficient LLM on two function-level Python reranking benchmarks, so cross-model robustness and generalization to repository- or project-level tasks are not established.
Source: Limitations · p. 5
Abstract
from arXiv · showhide
Test case synthesis is crucial for evaluating and ranking programs generated by large language models (LLMs). However, constructing high-quality test cases remains challenging because reliable expected outputs are often difficult to obtain. We propose Confidence-Gated Transductive Test Generation (CoTT), which first uses an efficient inductive procedure and invokes transductive generation only when inductive confidence is low. This adaptive design improves output reliability while allocating extra computation only when needed. On code reranking benchmarks, CoTT outperforms prior baselines across the reported metrics while reducing cost relative to applying transductive generation to every input. These results show that confidence-based allocation of test-time computation provides a favorable efficiency-effectiveness trade-off with a single efficient LLM.
1 Introduction
Synthesized tests can provide execution-based signals for identifying stronger code candidates, especially when high-quality human-written tests are sparse. CoTT combines inductive expected-output prediction with transductive refinement, invoking the latter only when inductive confidence is low.
1 Introduction
- Reliable synthesized tests are especially useful when high-quality human-written tests are sparse or unavailable.Execution-based signals from tests help identify better program candidates.Source: §1 · p. 1
- Prior approaches use agreement among generated programs or input-conditioned and verifier-guided procedures, which may still be unreliable or require extra resources.Generated programs can themselves be unreliable, while alternatives may require additional training, search, or auxiliary-model computation.Source: §1 · p. 1
- CoTT first predicts expected outputs inductively and invokes input-specific transductive generation only when inductive output agreement indicates low confidence.This confidence-gated design allocates additional computation only when needed.Source: §1 · p. 1, Figure 1 · p. 2
- CoTT improves synthesized-test expected-output reliability and code reranking across datasets using a single efficient LLM.The method allocates additional computation only when stronger expected-output estimation is needed.Source: §1 · pp. 1–2
2 Related Work
Related work uses synthetic tests and execution feedback to score or rerank generated code, with reliability of synthesized tests a key concern. Existing approaches to expected outputs rely on either execution-based consensus or stronger input-conditioned procedures.
Synthetic verification and scoring for code
- Synthetic tests, execution feedback, and learned critics support scoring and reranking code candidates, particularly when human-written tests are limited.This work treats synthesized-test reliability as a bottleneck and evaluates it directly on reranking benchmarks.Source: §2 · p. 2
Test synthesis and the oracle problem
- Consensus methods estimate expected outputs by executing general solution candidates and aggregating their outputs.These methods suit settings where execution-based consensus is sufficiently reliable.Source: §2 · p. 2
- Other approaches use stronger or input-conditioned procedures to reason about target inputs or select and verify candidate outputs.They are tailored to settings with access to stronger auxiliary models.Source: §2 · p. 2
3 Method
CoTT predicts expected outputs inductively from general solution candidates and uses their output agreement as confidence. It invokes target-specific transductive generation when that confidence is low, then uses the selected output in a test case for code reranking.
3 Method
- The inductive output distribution is estimated by executing sampled general solution candidates on each synthesized test input.Inductive confidence is the majority ratio of the most frequent output; high confidence indicates the candidates provide a reliable estimate.Source: §3 · p. 2
- When inductive confidence is low, transductive generation creates target-input-specific programs, executes them on that input, and forms an output distribution.The transductive distribution counts how many samples produce each output.Source: §3 · pp. 2–3
- CoTT uses the inductive prediction by default and selects the transductive prediction when confidence falls below threshold τ.The two distributions are not combined; the selected output is paired with the test input for candidate-program scoring by pass rate.Source: §3 · p. 3
- Main experiments use a confidence threshold τ = 0.8 across datasets.Hyperparameter settings are provided in Table A1.Source: §3 · p. 3
4 Experiments
CoTT is evaluated for code reranking on HumanEval-R+ and MBPP-R+ using Llama-3.1-8B-Instruct. It offers a favorable efficiency–effectiveness trade-off, though it does not uniformly beat always-transductive generation.
Experimental Setup
- Evaluation uses the full HumanEval-R+ and MBPP-R+ code-reranking datasets with Llama-3.1-8B-Instruct and five candidate solutions per task.Methods predict outputs for shared synthesized inputs; candidates are ranked by pass rate against the reference ranking induced by reference test suites.Source: §4 · p. 3
Evaluation Metrics
- Top-1 accuracy, Bottom-1 accuracy, and Spearman’s ρ measure endpoint matches and correlation with the full reference ordering; cost averages input and output LLM tokens per problem.Unless specified otherwise, methods use the same synthesized inputs, isolating differences in test-output generation.Source: §4 · p. 3
Baselines
- The baselines include single-pass CoT, transductive SOL-VER and SYNTRA, inductive rStar-Coder, and TT w/o Gating, which applies transduction to every input.TT w/o Gating predicts with arg max_y ptrans(y | x) and isolates the effect of confidence gating.Source: §4 · pp. 3–4
Experimental Results
- $0.077 to $0.065 average inference cost for CoTT versus TT w/o Gating, while CoTT performs best across all three MBPP-R+ metrics and retains most of its HumanEval-R+ benefit.On HumanEval-R+, TT w/o Gating leads in Top-1 accuracy and Spearman’s ρ, while CoTT leads in Bottom-1 accuracy.Source: §4 · p. 4
5 Analysis: When Does CoTT Help?
Inductive confidence distinguishes cases where the inductive prediction is reliable from those where transductive refinement is useful. The examples show how CoTT can retain a correct inductive output or recover a boundary-case output through transduction.
Effectiveness of Confidence Gating
- When only induction is correct, inductive confidence is typically high; when only transduction is correct, it is much lower, supporting confidence as a trigger for extra computation.Figure 2 compares inductive and transductive majority ratios for inputs where the predictions disagree.Source: §5 · p. 4, Figure 2 · p. 4
Case Study: When Transduction Helps
- For eulerian_num(0, 10), transduction predicts the correct output 0 by adding an n=0 boundary condition, although neither generated program is generally correct.The inductive prediction is 1; the transductive prediction provides a complementary, input-specific signal.Source: Figure 2 · p. 4
Case Study: When Induction Suffices
- For find(10, -2), induction correctly applies dividend // divisor while transduction adds an erroneous negative sign; CoTT retains induction at c(x) = 1.0.This illustrates the efficient inductive path when its prediction is reliable.Source: §5 · pp. 4–5
6 Conclusion
CoTT selectively invokes transductive generation when inductive confidence is low. Across both datasets, it maintains strong reranking performance at lower cost than always-transductive generation.
Conclusion
- Across both datasets, CoTT provides a favorable efficiency–effectiveness trade-off by maintaining strong reranking performance at lower cost than always-transductive generation.The conclusion identifies co-evolving code and test generation using CoTT’s constructive feedback as a future direction.Source: §6 · p. 5
Limitations
The evaluation and use case have boundaries that limit how broadly the results apply.
Scope
- Evaluation uses one efficient LLM and two function-level Python code-reranking benchmarks, so cross-model robustness and repository- or project-level generalization are unestablished.The controlled setting isolates test-oracle generation.Source: Limitations · p. 5
Execution Setting
- CoTT targets sandboxed code-reranking settings with execution timeouts and may be less applicable when execution is limited, unavailable, or computationally expensive.Source: Limitations · p. 5
Future Integration
- The study does not integrate CoTT with code generation to co-evolve programs and tests.The authors identify this as an unexplored direction.Source: Limitations · p. 5
External Information
- CoTT may benefit from external information sources, including relevant code documentation or retrieval-augmented context, when available.Source: Limitations · p. 5
AI Assistants
The paper limits LLM assistance to improving manuscript clarity, grammar, and style. CoTT’s algorithm forms output distributions by executing inductive and transductive samples.
AI Assistants
- LLMs were used only to improve the manuscript’s clarity, grammar, and style; the authors report the scientific work as their own.The stated assistance was within the scope of language editing.Source: AI Assistants · p. 5
A CoTT Algorithm
- CoTT’s algorithm executes inductive samples on each input and forms an output distribution from their output frequencies.The algorithm passage also identifies confidence gating and transductive generation as components.Source: A CoTT Algorithm · p. 7
- When transductive samples are used, the algorithm executes them on the input and forms a second output distribution from their output frequencies.The supplied algorithm fragments also specify sampling transductive programs.Source: A CoTT Algorithm · p. 7
B Experimental Settings
The appendix lists experimental hyperparameters and describes two code-generation prompts: one seeks a general solution, while the other specializes to a target input. Examples illustrate the different generation settings.
C Prompt
- The inductive prompt requests a general-purpose solution to the original problem, whereas the transductive prompt requests code specialized to the target input.The inductive prompting style follows Yu et al. (2024).Source: C Prompt · p. 7
B Experimental Settings
- Table A1 lists the experiment’s hyperparameters, grouped into test construction and code generation settings, and LLM settings.The supplied text does not give the individual hyperparameter values.Source: Table A1 · p. 7, B Experimental Settings · p. 7
D Generated Output
- For MBPP problems 103 and 292, paired examples illustrate general-purpose inductive outputs and input-specialized transductive outputs.The examples correspond to eulerian_num(0, 10) and find(10, -2), respectively.Source: D Generated Output · p. 7
E Threshold Sensitivity Analysis
Increasing the confidence threshold routes more inputs to transductive generation and raises inference cost, while reranking gains level off at higher thresholds. The cost analysis also reports paired savings for CoTT relative to SYNTRA.
E Threshold Sensitivity Analysis
- On HumanEval-R+, reranking improves through approximately τ = 0.8–0.9 and then largely plateaus; always-transductive generation has the highest Top-1 accuracy and Spearman’s ρ at higher cost.On MBPP-R+, τ = 0.8 provides a strong quality–cost balance, with no consistent improvement from increasing it further.Source: E Threshold Sensitivity Analysis · pp. 7–8, Table A2 · p. 8
- As τ increases, both the transductive rate and inference cost increase monotonically.Table A2 defines the transductive rate as the fraction of test inputs routed to transductive generation and identifies inductive-only and always-transductive endpoints.Source: E Threshold Sensitivity Analysis · p. 7, Table A2 · p. 8
F Cost Statistics
- $0.0039 per problem is the mean paired cost difference between SYNTRA and CoTT, with a standard error of $0.0002.Per-problem costs vary substantially; Table A3 reports their mean and standard deviation across HumanEval-R+ and MBPP-R+.Source: F Cost Statistics · p. 8, Table A3 · p. 8
G Confidence-Score Ablation
CoTT can gate transductive generation using several confidence scores computed from inductive samples. Maximum probability, top-2 margin, and negative entropy perform similarly, while subset agreement offers a lower-cost, slightly weaker operating point.
G Confidence-Score Ablation
- The compared confidence scores are maximum probability, top-2 margin, negative entropy, and subset agreement, with larger values indicating higher inductive confidence.Subset agreement measures how often two random halves of the inductive samples select the same modal output.Source: G Confidence-Score Ablation · p. 8
- Maximum probability, top-2 margin, and negative entropy achieve similar performance across both benchmarks; subset agreement lowers cost further but is slightly weaker.Thresholds are swept separately for each score, and Table A4 reports each score’s best-performing operating point.Source: G Confidence-Score Ablation · p. 8, Table A4 · p. 9
- The inductive prompt requests a general-purpose solution, whereas the transductive prompt conditions generation on a target input.The supplied examples pair inductive and transductive outputs for MBPP problems 103 and 292.Source: Figure A1 · p. 10, Figure A2 · p. 11, Figure A3 · p. 12