PaperlayerView exact revision on arXiv

Source-linked AI summary

Execution-Anchored Hallucination Calibration Reranking for Verilog Code Generation

Guang Yang, Xing Hu, Xiang Chen, Terry Yue Zhuo, Xin Xia

arXiv:2608.22938v1cs.SEcs.AR

TL;DR

Verilog generation suffers from limited data and hardware-specific complexity, and existing rerankers struggle with weak testbench coverage or inconsistent reasoning. EAHC independently acquires execution and reasoning signals and fuses them at ranking time. It improves average Pass@1 on both benchmarks and ranks first in 15 of 18 configurations, while remaining bounded by Verilog-only evaluation and testbench limitations.

  • Problem

    Verilog generation has a substantial performance gap caused by scarce training data and hardware-specific semantics, while existing reranking signals have complementary limitations.

  • Method

    EAHC independently acquires execution and reasoning signals, then fuses them only at the final ranking decision.

  • Results

    Average Pass@1 increases from 53.99% to 65.10% on VerilogEval-v2 and from 53.18% to 68.25% on ResBench, with first place in 15 of 18 configurations.

  • Takeaways & Limitations

    Independent fusion anchors reasoning judgments to execution behavior so execution-equivalent candidates receive consistent scores while reasoning covers behaviors tests may miss.

  • Takeaways & Limitations

    Evaluation covers only Verilog benchmarks, generated datasets may inherit teacher-model biases, and correctness depends on testbenches that can miss subtle bugs.

Abstract

from arXiv · show

Large Language Models (LLMs) have demonstrated remarkable capabilities in code generation, yet their performance degrades significantly on low-resource Hardware Description Languages such as Verilog. While multi-candidate sampling improves the likelihood of generating correct solutions, au-tomatically selecting the optimal candidate remains an open challenge. Through a systematic empirical study across nine models and two benchmarks, we identify two critical limitations:(1) existing execution-based reranking methods, which rely on testbench pass/fail outcomes, exhibit poor domain transferability due to low-quality generated testbenches; and (2) LLM-as-a-Judge suffers from reasoning hallucination, producing incon-sistent judgments for execution-equivalent code. These findings reveal two signal types with orthogonal errors: execution signals(deterministic but testbench coverage limited)and reasoning signals (semantically rich but hallucination-prone). Their orthog-onality suggests combining the two signals, yet in our experiments letting the reasoner directly observe execution results merely anchors its judgments on test outcomes; we therefore acquire the two signals independently and fuse them only at the decision stage. Based on these insights, we propose EAHC, an Execution-Anchored Hallucination Calibration reranking framework that anchors reasoning judgments to execution behavior so that execution-equivalent candidates receive consistent scores, which implements a dual-channel architecture: EAHC-R, a 4B reasoning discriminator; and EAHC-T, a testbench generator leveraging RAG for execution verification.

I. INTRODUCTION

Verilog generation remains difficult because low-resource data and hardware-specific semantics limit LLM performance, while selecting among sampled candidates remains underdeveloped. The paper studies these reranking failures and proposes EAHC, which independently combines execution and reasoning signals at the decision stage.

  • Motivation: Verilog Pass@1 remains 30–50%, far below the over-90% Pass@1 reported for leading LLMs on Python benchmarks.The gap reflects both scarce Verilog data and hardware semantics involving temporal logic, parallel execution, and signal propagation.
  • Motivation: Pass@10 reaches 71.30% on VerilogEval-v2 and 77.78% on ResBench, while Pass@1 reaches only 53.99% and 53.18%, respectively.The 17–25 percentage-point gap indicates substantial candidate-generation capability that reranking could help realize.
  • Empirical Findings: Execution rerankers transfer poorly to Verilog because self-generated testbenches often lack sufficient hardware-domain quality and coverage.Probability and semantic-matching methods also struggle with distribution shift and hardware-specific constructs.
  • Empirical Findings: LLM-as-a-Judge can produce inconsistent correctness judgments for candidates with identical execution behavior, revealing reasoning hallucination.The inconsistency arises despite execution equivalence on the evaluated test suite.
  • Proposed Framework: EAHC acquires execution and reasoning signals independently and fuses them only at the decision stage.EAHC-R provides reasoning judgments, while EAHC-T generates testbenches for execution verification.
  • Results: EAHC raises average Pass@1 from 53.99% to 65.10% on VerilogEval-v2 and from 53.18% to 68.25% on ResBench.It outperforms the strongest baseline by +5.91% and +4.96%, respectively, and ranks first in 15 of 18 configurations.

B. Code Reranking

Code reranking selects the best solution from multiple LLM-generated candidates using probability, semantic, execution, or reasoning signals. EAHC differs from coupled approaches by keeping execution and reasoning acquisition separate until ranking, targeting Pass@1 improvement.

  • Problem Formulation: Code reranking aims to select the best candidate from multiple LLM-generated solutions.The formal objective is to maximize correctness probability for the top-ranked candidate and thereby improve Pass@1.
  • Reranking Paradigms: Existing reranking paradigms use generation probability, semantic matching, execution verification, or LLM-based correctness judgments.These paradigms provide distinct signals for ranking sampled implementations.
  • Dual-Channel Design: EAHC keeps execution and reasoning acquisitions separate, allowing them to meet only when the final ranking is decided.Prior coupled methods let verifiers consume execution results or rewrite code after test failures, making judgment depend on execution outcomes.
  • Generation Probability: CodeReviewer combines forward generation probability with backward reconstruction likelihood through Rrev(x, ˆy) = log p(ˆy | x) + log p(x | ˆy).The backward term measures how well code reconstructs the original requirement.
  • Semantic Matching: CodeRank uses cosine similarity between requirement and code embeddings, evaluated with Qwen3-Embedding and Jina-Code-v2.The embedding function is a pretrained code representation model.
  • Execution Verification: CodeT generates testbenches alongside candidates, clusters candidates by execution outcomes, and scores clusters using code and test counts.The evaluated variants differ in whether the same model or GPT-5 generates the testbenches.
  • LLM-as-a-Judge: LLM-as-a-Judge aggregates multiple correctness judgments through majority voting.The comparison evaluates Code-DiTing-1.5B and Code-DiTing-7B judges.

B. Results and Analysis

The analysis finds poor transferability in existing rerankers and reasoning hallucination in LLM-as-a-Judge, then introduces EAHC to combine execution anchoring with reasoning signals.

  • Method-wise Analysis: 35.47% and 35.36% average Pass@1: Prob and CodeReviewer underperform the original 53.99% on VerilogEval-v2.The passage attributes this degradation to distributional shift between Verilog syntax and high-resource programming-language training data.
  • Method-wise Analysis: 59.19% average Pass@1: CodeT-GPT is the strongest baseline on VerilogEval-v2 but remains 12.11 percentage points below the 71.30% oracle.On ResBench, CodeT-GPT reaches 63.29% against a 77.78% oracle.
  • Method-wise Analysis: Self-generated Verilog testbenches exhibit low coverage and frequent compilation failures, limiting execution-based reranking.The limitation reflects a mismatch between reranking assumptions and HDL characteristics.
  • Method-wise Analysis: LLM-as-a-Judge produces inconsistent judgments for execution-equivalent candidates, including opposite verdicts for counter implementations with identical waveforms.The passage attributes this to token-level reasoning that is sensitive to superficial code variations rather than functional behavior.
  • Framework: EAHC selects candidates through execution anchoring, equivalence clustering, hybrid fusion scoring, and hierarchical selection.Execution vectors cluster candidates before the hybrid score combines execution and reasoning signals.

A. Framework Overview

EAHC selects among k candidate implementations with a two-stage framework that groups execution-equivalent candidates and then combines execution and reasoning evidence for hierarchical selection.

  • Framework Overview: EAHC receives a requirement x and candidate set Y_k, then selects an optimal candidate through a two-stage process.The framework first forms execution-equivalence clusters and then evaluates them for selection.
  • Framework Overview: Execution-equivalence clusters C_e are defined by shared execution vectors e, while S_hybrid fuses execution and reasoning signals.F_ϕ denotes the reasoning discriminator.
  • Framework Overview: The workflow generates testbenches, obtains candidate execution vectors, clusters candidates, computes fusion scores, and selects the best cluster before the best individual.This hierarchical procedure separates cluster-level execution behavior from individual candidate ranking.

1) Channel Independence:

EAHC acquires execution and reasoning signals independently because their errors differ systematically: execution misses uncovered behaviors, while reasoning hallucinations vary across equivalent implementations.

  • Channel Independence: The dual-channel model treats correctness judgment as acquiring execution and reasoning signals without either channel observing the other.This separation preserves the distinct information sources before fusion.
  • Channel Independence: The execution channel generates testbenches from requirement x alone, using pass/fail outcomes as execution signal E.Candidate code is excluded when acquiring this signal.
  • Channel Independence: The reasoning channel judges (x, ŷ) pairs through semantic analysis without observing execution results, producing signal R.Its output includes a Yes/No judgment and reasoning chain.
  • Channel Independence: Execution errors are systematic consequences of limited coverage, whereas reasoning errors are stochastic consequences of hallucination.Their differing error patterns motivate combining the signals while preserving separate acquisition.
  • Channel Independence: The approximation P(E, R | Y) ≈ P(E | Y) · P(R | Y) licenses additive fusion but is a modeling choice rather than a guaranteed property.The paper explicitly distinguishes this approximation from the unconditional mutual-information identity.
  • Channel Independence: Interactive post-hoc fusion cannot exceed the information in independently acquired signals, and an execution-echoing reasoner contributes no conditional information.The theorem bounds information content, not the accuracy of every interactive design.

C. EAHC-R: Reasoning Discriminator

EAHC-R supplies execution-independent semantic judgments, while EAHC-T supplies execution signals from RAG-generated testbenches; the framework keeps these channels separate during acquisition.

  • EAHC-R: Reasoning Discriminator: EAHC-R estimates each candidate’s functional correctness probability F_ϕ(x, ŷ) from the requirement and candidate code.It excludes testbenches, execution results, and pass rates to preserve channel independence.
  • EAHC-R: Reasoning Discriminator: VeriJudge-47K is curated through multi-teacher distillation, domain prompting, and compiler-in-the-loop verification.DeepSeek-R1 and GLM-4 provide reasoning chains and verdicts, while Icarus Verilog filters samples against ground-truth testbenches.
  • EAHC-R: Reasoning Discriminator: Qwen3-4B is fine-tuned on VeriJudge-47K, and n=3 reasoning chains per candidate are aggregated by majority voting at inference.The lightweight model supports single-GPU deployment and lower inference latency than commercial API calls.
  • EAHC-R: Reasoning Discriminator: The resulting F_ϕ score is an endorsement frequency rather than a correctness probability, and execution anchoring later enforces cross-candidate consistency.Voting reduces sampling variance for an individual candidate.
  • EAHC-T: Testbench Generator: EAHC-T generates high-quality testbenches for execution anchoring and receives only the natural-language requirement, excluding candidate implementations.The same testbench therefore evaluates all candidates without access to reasoning signal R.
  • EAHC-T: Testbench Generator: EAHC-T uses two-stage RAG: BM25 retrieves top-K examples, then retrieved demonstrations are injected into generation with K=5.The generator reuses the EAHC-R-tuned 4B model without additional fine-tuning.

4) Execution Anchoring:

EAHC anchors reasoning judgments to execution behavior by grouping execution-equivalent candidates and combining cluster-level execution and reasoning signals. This preserves execution consistency while using reasoning to distinguish candidates where testbenches are silent.

  • Execution Equivalence: Each candidate is executed against a generated testbench, producing a vector that records passes, failures, and compilation errors.The vector contains one outcome per test case, with values 1, 0, or −1.
  • Execution Equivalence: Candidates with identical execution vectors form execution-equivalence clusters, although generated tests may leave different implementations indistinguishable.Reasoning remains necessary for differences that fall outside testbench coverage.
  • Signal Fusion: EAHC computes execution scores from cluster pass rates and fuses them with reasoning scores at the cluster level.The final stage combines the two signals through hierarchical selection.
  • Signal Fusion: Within each cluster, the reasoning score uses the maximum individual judgment, representing the most optimistic reasoning estimate.This choice allows one confident judgment to carry the cluster but can tension with the consistency goal.
  • Signal Fusion: The default fusion weight is α = 0.6, giving slightly greater weight to the execution signal.The setting reflects the stated higher reliability of execution relative to reasoning.

3) Two-Level Selection:

EAHC selects candidates in two levels: it first chooses the best execution-equivalence cluster using a hybrid score, then selects the highest-scoring candidate within that cluster. Across two benchmarks and nine models, this design achieves the strongest aggregate reranking results while retaining some configuration-specific limitations.

  • Two-Level Selection: EAHC first selects the cluster with the highest hybrid score, breaking ties by hybrid, execution, reasoning, then cluster-size priority.This ordering makes execution-aware cluster selection precede individual candidate selection.
  • Two-Level Selection: Within the selected cluster, EAHC chooses the candidate with the highest reasoning score, balancing execution reliability with reasoning-based quality assessment.The output therefore combines cluster-level execution behavior with candidate-level reasoning.
  • Overall Effectiveness: EAHC attains the highest average Pass@1 on both benchmarks and the best reranking accuracy in 15 of 18 configurations.The evaluation covers VerilogEval-v2 and ResBench across nine code generation models.
  • Overall Effectiveness: 65.10% average Pass@1 on VerilogEval-v2 exceeds CodeT-GPT’s 59.19% by +5.91% absolute improvement.EAHC also recovers 64.2% of the gap from original Pass@1 to the Pass@10 oracle on this benchmark.
  • Exceptions and Significance: CodeT-GPT remains ahead in three strong-generator configurations, where permissive testbenches and residual reasoning errors limit EAHC’s advantage.The reported false-accept and reasoning-error rates identify execution and reasoning failure modes behind these exceptions.
  • Exceptions and Significance: EAHC’s aggregate improvements over CodeT-GPT are statistically significant, with p-values of 1.1 × 10−10 on VerilogEval-v2 and 1.8 × 10−8 on ResBench.The authors note that CodeT-GPT still wins three individual configurations.

2) Ablation Study:

Ablations show that both execution and reasoning channels contribute to EAHC, while hierarchical selection and intermediate fusion weights improve performance over single-channel or extreme configurations. The study also tests the execution-equivalence premise and examines complementarity with training-based optimization.

  • Channel Ablations: EAHC-T alone achieves 57.98% on VerilogEval-v2 and 62.50% on ResBench, trailing full EAHC by 7.12% and 5.75%.The gap indicates complementary value from reasoning signals beyond execution feedback.
  • Channel Ablations: EAHC-R alone achieves 62.77% on VerilogEval-v2 and 63.10% on ResBench, trailing full EAHC by 2.33% and 5.15%.The degradation supports execution anchoring as additional grounding for reasoning-based selection.
  • Hierarchical Selection: Hierarchical selection raises performance from 61.82% to 62.77% on VerilogEval-v2 and from 62.70% to 63.10% on ResBench.The comparison is between EAHC-R with one and three reasoning samples.
  • Fusion Weight: Fusion performance peaks for α ∈[0.1, 0.6], while reasoning-only and execution-only settings are suboptimal and performance degrades sharply when α > 0.8.The result supports using complementary information from both channels.
  • Execution-Equivalent Inconsistency: Reasoning scores disagree for 15.3% and 27.4% of execution-equivalent pairs, and byte-identical code receives different scores for 5.2% and 16.4% of pairs.These measurements motivate clustering candidates by execution behavior before reasoning-based selection.
  • Execution-Equivalent Inconsistency: Majority voting lowers verdict flips from 7.5% to 4.8% and from 12.8% to 7.1%, but cross-candidate disagreement persists.Execution anchoring absorbs this residual drift and yields +2.33% and +5.15% over EAHC-R.
  • Training-Based Optimization: EAHC is evaluated for complementarity with SFT and RL across Base, SFT, and RL stages of the CodeV training pipeline.The experiment compares reranked Pass@1 at each stage and includes Pass@10 as an oracle upper bound.

3) Results Analysis:

Across training stages and nine-model fusion comparisons, EAHC’s independent acquisition consistently complements generation improvements and outperforms interactive execution-informed judging.

  • Training-stage orthogonality: EAHC improves Pass@1 at every training stage, adding +12.82%, +10.26%, and +8.98% on VerilogEval-v2 for Base, SFT, and RL models.The corresponding ResBench gains are +19.64%, +10.72%, and +8.93%.
  • Training-stage orthogonality: EAHC recovers 64.5% of the Pass@1-to-Pass@10 gap for Base, 80.0% for SFT, and 56.0% for RL models.Even the RL-optimized model benefits from inference-time selection.
  • Fusion strategy: Independent fusion outperforms interactive fusion on every model and benchmark, with average gains of +4.49% on VerilogEval-v2 and +4.56% on ResBench.Both strategies use the same EAHC-R checkpoint and differ only in whether execution results are supplied before judgment.
  • Fusion strategy: The fusion gap is smallest for stronger models, such as GPT-5 at +1.28 and GLM-4 at +1.92, and reaches +7.70 for VeriPrefer on VerilogEval-v2.Interactive fusion appears most harmful when reasoning must distinguish lower-quality candidates.
  • Fusion strategy: Interactive fusion exhibits pass anchoring and fail anchoring, causing judgments to echo execution outcomes despite uncovered bugs or testbench-induced failures.The reasoning channel therefore contributes less independently and approaches execution-only behavior.

D. RQ4: Validity of the Execution Anchor

The generated execution anchor is informative but conservative: it rarely accepts incorrect candidates, often rejects correct ones, and varies with testbench construction. EAHC therefore fuses rather than filters it, while acknowledging correlated channel errors and bounded theoretical scope.

  • Anchor quality: At k=5, false acceptance is 12.8% and 6.6%, whereas false rejection is 47.3% and 49.6%, with AUC values of 0.743 and 0.727.The anchor is conservative because it often fails to certify correct candidates.
  • Anchor quality: Because filtering would discard about half of correct candidates, EAHC fuses the execution score instead of treating it as an oracle.Roughly a quarter of candidate clusters still mix correct and incorrect solutions, leaving reasoning to separate them.
  • Anchor construction: At k=1, only 38.4% of VerilogEval-v2 testbenches compile and AUC falls to 0.618, motivating the k=5 setting.Retrieval improves every ResBench measure against GPT-5 testbenches but not VerilogEval-v2, where GPT-5 testbenches compile more often.
  • Anchor informativeness: On 1,117 instances where the generated testbench gives every candidate the same verdict, EAHC reaches 79.43% versus 78.51% for EAHC-R and 74.92% for EAHC-T.On the remaining 755 instances, EAHC reaches 43.74% versus 40.31% and 33.79%, showing gains when the anchor separates candidates.
  • Theoretical boundaries: The theoretical analysis supports independent acquisition as a default but does not establish that independent fusion beats every interactive design.The preference ultimately rests on the empirical RQ3 comparison rather than a general optimality claim.
  • Channel dependence: Both channels select an incorrect candidate on 16.1% of mixed instances, versus 9.4% predicted under independence, with the ratio rising to 1.73 on ResBench.Sequential logic and state machines show especially strong dependence.
  • Theoretical boundaries: Mutual information measures verdict-level utility rather than whether the reasoning chain itself is sound.Compiler filtering and majority voting reduce verdict noise but do not inspect reasoning chains.

C. Broader Implications and HDL Specificity

EAHC’s practical scope is Verilog, where benchmark overlap with its retrieval corpora appears domain-related rather than problem-level leakage. The paper reports benefits alongside dataset, validity, cost, and transfer boundaries.

  • Contamination and diversity: Benchmark problems have no retrieval neighbour above 0.9 on either overlap measure; the closest reaches 0.667 ROUGE-L and 0.862 cosine similarity.The authors interpret this as domain relatedness rather than problem-level leakage.
  • Contamination and diversity: Recurring port signatures reflect standard HDL skeletons rather than shared problems, because an 8-bit input-to-output interface does not determine the implemented function.Such matches occur in 52/156 VerilogEval-v2 problems but only 3/56 ResBench problems.
  • Contamination and diversity: The retrieval corpora contain 47,375 records over 12,195 problem statements and 53,015 distinct testbench questions, with exact repetitions collapsed.Benchmark problems lie further from the corpora than corpus items lie from one another.
  • Validity boundaries: The training records pair requirements with sampled candidates and Yes/No verdicts rather than reference answers, while pre-training exposure remains uncontrolled.The rerankers nevertheless score the same candidates and therefore share any such exposure.
  • Scope and transfer: The evaluation is limited to Verilog, and transferring to another HDL requires rebuilding both corpora and retraining.The authors make no claim beyond Verilog.
  • Practicality: The method incurs approximately 15–20 seconds per problem versus approximately 2 seconds for greedy decoding, a roughly 10× latency increase.The cost is dominated by testbench generation and reasoning calls.
  • Broader implications: EAHC raises average Pass@1 by over 11% on VerilogEval-v2 and over 15% on ResBench, ranking first in 15 of 18 configurations.The conclusion identifies extension to other low-resource HDLs and industrial-scale benchmarks as future work.
Loading 2608.22938v1…