CodeTD: Topology of Attention Detects Hallucinations in Code LLMs

1 Introduction

• Functional tests can be unavailable, incomplete, expensive, or unsafe, motivating pre-execution verification of generated code.The paper defines hallucination as code that does not solve the problem in the prompt.Source: §1 · p. 1
• CodeTD analyzes attention-map topology to detect hallucinations in LLM-generated code.The paper names the approach Code Topology Divergence.Source: §1 · p. 1
• Experiments cover 10 Code LLMs up to 34B parameters, four benchmarks, and five programming languages, with results reported as outperforming baselines.The paper also reports transfer between code benchmarks and Code LLMs, and attention-head indicators across several languages.Source: §1 · pp. 1–2

2 Related Work

• Code LLMs vary in training data, tokenizers, training and fine-tuning protocols, and attention mechanisms.The passage lists CodeLlama, StarCoder2, DeepSeek-Coder, and Qwen2.5-Coder as examples.Source: §2 · p. 2
• Prior transformer-attention studies include analyses of attention maps and the use of merged human and machine attention for neural code summarization (Zhang et al., 2024c).The passage also cites Clark et al. (2019), Htut et al. (2019), and Michel et al. (2019) on attention maps.Source: §2 · p. 2
• Code-hallucination research includes taxonomies and benchmarks, while Jiang et al. (2024) report lower token probabilities and higher generation-step entropy for hallucinations.The passage also describes CodeHalu and related categorization and benchmark work by Tian et al. (2025), Liu et al. (2024), and Jiang et al. (2024).Source: §2 · p. 2
• Broader hallucination approaches include grounding, self-evaluation, model collaboration, representation editing, and query reconstruction; TOHA applies attention-graph topology to RAG.The passage identifies these approaches with Peng et al. (2023), Zhang et al. (2024b), Feng et al. (2024), Zhang et al. (2024a), Yehuda et al. (2024), and Bazarova et al. (2026), respectively.Source: §2 · p. 2

3 Code Hallucination as Functional Incorrectness

• A code hallucination is code that is not functionally correct and does not solve the problem provided in the prompt.Functional tests can check correctness, but may be unavailable, incomplete, expensive, or unsafe to execute.Source: §3 · p. 2
• The paper groups hallucinations into syntactic, runtime, and logical errors, with logical errors corresponding to approximately 78% of incorrect code cases for modern Code LLMs.The 78% figure is attributed to Appendix O.Source: §3 · p. 2
• The verifier targets incorrect outcomes regardless of root cause, and the paper says static code analysis tools are of little help for this focus.The paper notes that its definition almost coincides with Tian et al. (2025), while other definitions also exist.Source: §3 · p. 2

4 Background. Transformer-based LLMs

• A transformer has L layers of multi-head self-attention blocks, each with H heads, and each head takes an input matrix X ∈Rn×d.The head output is given as Xout = A(XW V ), with projection matrices and an attention map A ∈[0, 1]n×n.Source: §4 · p. 2
• Figure 1 depicts prompt and generation tokens as red and blue vertices in weighted graphs, with correct code shown as having higher prompt-generation connectivity than hallucinated code.The caption identifies MTD0 as an integral characteristic of prompt-generation connectivity.Source: Figure 1 · p. 3
• An attention map represents how each token interacts with every other token in the input sequence, and each element ai,j is interpreted as an interaction force.Tokens may attend more to other contextually related tokens.Source: §4 · p. 3

• CodeTD models each token as a vertex in a fully connected weighted graph, with edge weights representing token interaction strength.Source: §5 · p. 3
• Hallucinations may arise when attention drifts from the prompt, which the authors hypothesize corresponds to low connectivity between prompt and generation tokens.Source: §5 · p. 3
• Thresholding attention scores is a baseline approach, but it requires an unknown optimal threshold and produces discontinuous topological changes as thresholds or weights change.Source: §5 · p. 3

• MTD evaluates connectivity between disjoint prompt and generation token sets by adding edges in ascending weight order and tracking topological patterns as they appear and disappear.The resulting Cross-Barcode is a multiset of birth-death moments.Source: §6 · p. 3
• MTD_k sums birth-death interval lengths; k indexes pattern dimension, and higher values indicate greater dissimilarity between token sets.The persistence-barcode feature is stable to perturbations of weights.Source: §6 · p. 3
• Setting edge weights to 1−a_i,j makes high-attention token pairs connect early in the filtration, prioritizing strong semantic interactions between prompt and generation tokens.Source: §6 · p. 3
• Low MTD indicates that a generation is structurally detached from the prompt and hallucinates.Source: §6 · p. 4
• Features are calculated per layer and head using normalized directional MTD_0 and MTD_1 values, with additional normalization by prompt and generation set sizes and diagonal-attention averages.Source: §6 · p. 4
• A supervised classifier is trained on topological features, requires functional-test labels for training, but needs no test execution at inference and transfers across benchmarks.Figure 2 depicts the pipeline from concatenated prompt and generated code through attention maps and graph features to a binary classifier.Source: §6 · p. 4, Figure 2 · p. 4

7.1 Dataset Construction

• Experiments used five-fold stratified group cross-validation, keeping solutions to each coding problem within one fold, across four benchmarks and models up to 33B parameters.The evaluation included five programming languages and multiple generation and prompting settings.Source: §7.1 · pp. 4–5, §7.2 · p. 5

7.3 Ranking Ability

• CodeTD ranking produced significantly higher pass@1 than random generation selection for all evaluated Code LLMs.For each problem, the generation with the highest predicted probability of correctness was selected.Source: §7.3 · p. 6, Table 4 · p. 8

7.4 Transferability

• CodeTD’s attention features transferred across HumanEval and MBPP, outperforming alternatives in 80% of transfer cases; Magicoder-S-DS-6.7B was the strongest feature extractor for most generators.Transfer performance was lower than when training and testing on the same benchmark; Magicoder-S-DS-6.7B also outperformed DeepSeek-Coder-33B as an extractor.Source: §7.4 · pp. 6–7, Table 3 · p. 7

7.2 Code Hallucination Detection

• CodeTD significantly outperformed baselines in most ROC-AUC comparisons and significantly beat LapEigvals on BigCodeBench (Fisher’s combined test, p = 0.017).BigCodeBench is described as a stress test, with correct generations as low as 5.9%.Source: §7.2 · p. 5, Table 1 · p. 6

7.5 Feature Importance

• On MBPP, retaining 5% of attention features caused no significant loss in classification quality, while the classifier’s feature importance indicated that about 25% were meaningful.The feature-selection procedure ranked features with a classifier trained on all attention features, then retrained using the selected subset.Source: §7.5 · p. 7, Figure 4 · p. 8

7.6 A Fine-Grained Error Type Classification

• Across models, CodeTD identified the majority of fine-grained error types; CodeLlama-7B accuracy ranged from 75.7% for NameError to 100% for RecursionError.Some low-frequency error types were grouped.Source: §7.6 · pp. 7–8, Table 5 · p. 8

7.7 Ablation Study

• Adding 0-dim and 1-dim topological features to diagonal attention improved detection in 92% of cases and yielded more robust, transferable classifiers.Features were added progressively in order of computational complexity.Source: §7.7 · p. 8

• CodeTD transforms attention maps into weighted graphs and assesses prompt–generation connectivity as a pre-execution check when functional tests are unavailable, incomplete, expensive, or unsafe.Source: §8 · p. 8
• The authors report that CodeTD outperformed recent baselines and transferred across coding benchmarks and Code LLMs.Source: §8 · p. 8
• Proposed future mitigation paths include best-of-N selection, regeneration from the first wrong token, and abstention or fallback in human-in-the-loop workflows.The authors frame the verification layer as facilitating safer deployment in security-critical applications.Source: §8 · p. 8

Limitations

• CodeTD targets hallucinations reflected in attention geometry, does not identify all root causes, and predicts whole-code correctness rather than locating bugs.The authors leave analysis of other causes, including spurious pre-training correlations, decoding drift, and RLHF bias, for future work.Source: Limitations · pp. 8–9

A Details on Generation Procedure

• Solutions were generated at temperature 0.8, with output limits varying by benchmark; the attention submatrix covered the prompt and valid solution.HumanEval and MultiPL-E were capped at 512 total tokens; MBPP allowed 256 new tokens, while BigCodeBench followed its original repository procedure.Source: A Details on Generation Procedure · pp. 10–11

B Details on Training Procedure

• The study compares CodeTD with prompt-based self-evaluation, adapted Interrogate-LLM, and attention-based detectors using matched feature counts and XGBoost.Self-Eval was adapted by asking Code LLMs to judge functional correctness; Interrogate-LLM used CodeT5+ 110M embeddings with K = 5.Source: B Details on Training Procedure · p. 11

D Examples of Cross-Barcodes

• MTD is the average barcode length, not the number of bars; correct generations tend to show more H0 bars and more, longer H1 bars.The reported MTD distinction is not spuriously correlated with generated-code length.Source: D Examples of Cross-Barcodes · p. 12

E Ablation for the Classifier Model Selection

• XGBoost combines strong average performance across Code LLMs with about 30 seconds of training per fold and no hyperparameter tuning in this setting.The authors attribute the method’s high performance to the relevance of extracted attention features, rather than a particular classifier choice.Source: E Ablation for the Classifier Model… · pp. 12–13

F Applicability of GNNs to Attention Maps

• Storing all attention matrices requires about 578.2–996.4 GB for HumanEval and 1023.5–1718.7 GB for MBPP, whereas CodeTD computes features during generation without storing them.The authors therefore describe storing all such features as prohibitively costly even for these moderate-sized datasets.Source: F Applicability of GNNs to Attention… · p. 13

G Computational Complexity

• For CodeLlama-7B on HumanEval, generation averaged 6.2 seconds without CodeTD and 11.5 seconds with topological features computed for all layers and heads.Using only 5% of attention features can reduce inference computation without significant classification-quality loss; average memory was about 190 MB for HumanEval and 544 MB for MBPP.Source: G Computational Complexity · pp. 13–14

H On Definition of a Code Hallucination

• MTD0(P, G) integrates how long generation components remain disconnected from prompt-conditioned structure; small values indicate early integration, while large values indicate persistent fragmentation.The authors interpret the construction as a multiscale divergence under stated attention-based assumptions, and report Lipschitz stability under attention perturbations.Source: H On Definition of a Code Hallucination · p. 15

I Experiments with Larger Models

• Combining DeepSeek R1 outputs with attention features from DeepSeek-6.7B produced the best ROC-AUC, although DeepSeek R1 alone outperformed the attention-feature classifier.The authors report that the small model’s attention features improved DeepSeek R1’s performance.Source: I Experiments with Larger Models · p. 16
• CodeTD achieved better detection quality than zero-shot prompting when evaluated on Qwen2.5-Coder-32B.The same model generated code, supplied attention features, and was used for zero-shot evaluation.Source: I Experiments with Larger Models · p. 17
• In the fair comparison, CodeTD outperformed CodeJudge by a large ROC-AUC margin with CodeJudge run without reference and CodeTD using large 32–34B models; it also outperformed CodeJudge by ROC-AUC in additional setups.The additional setups included CodeJudge run with reference and CodeTD run for smaller 6.7–7B models.Source: I Experiments with Larger Models · p. 17

J Contribution of Individual Heads

• On MultiPL-E across Go, Java, Rust, and Lua, features from some individual heads had high correlation with hallucination labels and served as predictors.The paper reports ROC-AUC scores for the top-performing features.Source: J Contribution of Individual Heads · p. 17

K Evaluation with Greedy Decoding

• CodeTD achieves high performance across models and datasets with greedy decoding, consistently outperforming CodeJudge under both greedy decoding and sampling.The greedy setup uses one generation per task: 164 HumanEval and 500 MBPP samples, with five-fold cross-validation.Source: K Evaluation with Greedy Decoding · pp. 17–18

L Evaluation with Partial Generation

• Sliding-window processing reduces memory consumption for long sequences, with some loss in accuracy.Chunks of at most L tokens are processed independently, and their features are concatenated for XGBoost.Source: L Evaluation with Partial Generation · p. 18, Table 6 · p. 18

M Performance on Different Task Lengths

• CodeTD has higher mean ROC-AUC across most prompt-length categories; the StarCoder2 long-prompt estimate is fragile, based on only two positive examples.The paper reports consistent performance across prompt lengths, while noting the StarCoder2 imbalance and its 5.9% hallucination share.Source: M Performance on Different Task Lengths · p. 18, Table 7 · p. 19

N Failure Case Study

• The most common classifier errors are misclassifying test-passing code as wrong and solutions that fail tests with an AssertionError as correct.Other error categories occur at substantially lower fractions.Source: N Failure Case Study · p. 18

O Breakdown by Error Type

• Most observed execution errors are failures of functional tests recorded as AssertionError.The analysis considers exceptions after running functional tests as hallucination types.Source: O Breakdown by Error Type · p. 18

P Individual Features Detect Error Types

• Some individual features distinguish logical errors from syntax and runtime errors, including AssertionError versus SyntaxError, ZeroDivisionError, and NameError.Source: P Individual Features Detect Error Types · p. 18

Q Manifold Topology Divergence

• MTD measures dissimilarity between prompt and generation vertex sets through birth–death intervals in persistence barcodes; larger MTD indicates greater dissimilarity.The filtration adds edges by ascending weight; homology indices distinguish connected components, cycles, and higher-dimensional features, and MTD is not symmetric.Source: Q Manifold Topology Divergence · p. 19

R On Importance of Cycles

• H1 cycles represent generation-token detours between prompt nodes: correct generations collapse these loops quickly, while incorrect ones let them persist.MTD0 instead measures whether generation components attach to the prompt at all; MTD1 measures how cleanly they attach.Source: R On Importance of Cycles · pp. 19–20, Figure 14 · p. 20

S Comparison with TOHA (Bazarova et al., 2026)

• CodeTD outperforms AttnLogDet, AttnEigvals, and LapEigvals in 84% of model–language cells; under transfer, LapEigvals drops 13–20 ROC-AUC points while CodeTD is more stable.Source: S Comparison with TOHA (Bazarova et… · p. 20
• TOHA selects and aggregates individually discriminative heads, whereas CodeTD uses topological features across all layers and heads; CodeTD also outperforms both tested TOHA versions.The reported sparse optimum is a structured 5% pattern across layers, heads, and homological dimensions, found by XGBoost.Source: S Comparison with TOHA (Bazarova et… · pp. 20–21, Table 9 · p. 21

T On Statistical Significance for BCB

• On BigCodeBench, CodeTD significantly outperforms LapEigvals in 3 of 5 paired tests; Fisher’s combined p = 0.017 favors CodeTD, though LapEigvals wins the StarCoder2-7B cell.Source: T On Statistical Significance for BCB · p. 21

U On Streaming Variant of CodeTD

• A streaming variant can update MTD as tokens arrive and trigger early stopping; experiments show fewer generated tokens at the cost of a slight decrease in classification performance.The proposed update tracks about 5% of heads and uses incremental edge insertion for H0 persistence.Source: Table 10 · p. 21, U On Streaming Variant of CodeTD · p. 22

V Experiments with Prompt Masking

• Masking 50% of HumanEval prompt tokens establishes the reported causal pattern: reduced prompt attention yields more functional errors, with little change in syntactic-error rate.Source: V Experiments with Prompt Masking · p. 22

W Supervision Cost

• Reported CodeTD full-data ROC-AUC fractions are 90–93% on HumanEval, 89–91% on MBPP, and 79–91% on BCB, across the models with stated values.Source: W Supervision Cost · p. 22

X Licenses

• The pretrained models and benchmarks used in the paper permit research use.Source: X Licenses · p. 22