Do Large Language Models Pay Similar Attention Like Human Programmers When Generating Code?

• Prior work uses attention analysis to inspect model behavior, and studies associate human-aligned attention with improved performance and user trust.The introduction cites applications of attention analysis in computer vision and prior findings about alignment.Source: §1 · pp. 1–2
• Comparing human and model attention is intended to test whether LLMs grasp task semantics like programmers or rely on superficial patterns, and whether attention differences can explain errors.The authors also seek opportunities to improve code-generation models through this comparison.Source: §1 · p. 2
• The study compares attention alignment across six LLMs on HumanEval [27] and MBPP [7], asking whether attention explains errors and how calculation methods affect alignment and programmer preference.The study’s four research questions also ask how aligned model attention is with human attention.Source: §1 · p. 2
• 27% of code-generation errors were explained by five attention patterns; all six LLMs showed consistent misalignment, while perturbation-based attention was more aligned with and preferred by programmers.The analysis covered 211 incorrect snippets; the user study involved 22 participants.Source: §1 · p. 2
• The authors compare attention-calculation methods quantitatively and in a user study, and release a programmer-attention dataset covering 1,138 Python tasks.The dataset is intended for developing human-aligned models and evaluating interpretability methods.Source: §1 · p. 3

2.1 Motivation

• Prior code-model attention research covered summarization, repair, and method-name prediction, but not code generation with LLMs of billions of parameters.Earlier work in other domains motivates examining human–model attention alignment in code generation.Source: §2.1 · p. 3
• The study evaluates 12 attention-calculation methods and a user study with 22 participants amid no consensus on aggregating LLM attention scores.Prior work variously sums attention across layers and heads or uses only the first transformer layer.Source: §2.1 · p. 3

2.2 Code Generation Benchmarks and Metrics

• The task asks an LLM to complete a function from its header and a natural-language description; HumanEval has 164 tasks and MBPP has 974, both with test cases.The study focuses on these two popular benchmarks.Source: §2.2 · pp. 3–4, Figure 1 · p. 4
• Pass@k counts a task as solved when any of k generated samples passes all test cases; BLEU and CodeBLEU compare generated code with a reference.CodeBLEU adapts BLEU for code by accounting for structure and correctness.Source: §2.2 · p. 4

2.3 Model Attention

• Model attention denotes how important an LLM considers each natural-language task-description token during code generation.The concept resembles feature importance, saliency maps, and feature attribution in explainable AI.Source: §2.3 · p. 4

2.3.1 Self-attention-based Methods.

• Self-attention methods aggregate scores across heads or layers, while gradient-based methods use prediction gradients; the study tests six self-attention methods and two gradient methods.Different heads can focus differently, and prior aggregation choices include summing across layers and heads or using only the first layer.Source: §2.3.1 · p. 5, §2.3.2 · p. 5

2.3.3 Perturbation-based Methods.

• Perturbation methods mutate inputs and infer token importance from output differences, without requiring access to model internals.The study tests SHAP and BERT masking-based substitutions; LIME and SHAP may require many samples and deletion can alter input meaning or structure.Source: §2.3.3 · p. 6

3 THE CONSTRUCTION OF THE PROGRAMMER ATTENTION DATASET

• The dataset contains programmer-attention labels for 1,138 tasks: all 164 HumanEval prompts and 974 MBPP prompts.The benchmarks were selected in part because they are widely used and provide test cases for assessing code correctness.Source: §3 · p. 6
• Two Python-experienced authors independently labeled important words and phrases, beginning with 20 HumanEval prompts and four keyword categories.Their initial Cohen’s Kappa was 0.68; after discussing disagreements, they established the labeling categories.Source: §3 · p. 6
• The four inclusive keyword categories are data types, operators, conditionals, and properties.Examples include “list,” “sort,” conditions following “if,” and qualifiers such as “all” or “first.”Source: §3 · p. 6
• Labeling agreement reached Cohen’s Kappa 0.72 on the remaining HumanEval prompts, Fleiss’ Kappa 0.64 in verification, and Cohen’s Kappa 0.73 on MBPP.Disagreements were resolved, and the full labeling process took 192 person-hours.Source: §3 · p. 7
• Prompts averaged 29.6 words, with 7 considered important; properties were most frequent at 45.2%, followed by operators at 27.2%, conditionals at 25%, and data types at 2.6%.Figure 3 presents two labeled prompt examples, with the four keyword types shown in different colors.Source: §3 · p. 7, Figure 3 · p. 7

4.1 Code Generation Models

• Six models, including GPT-4, were evaluated for code generation, with Table 1 covering model size, attention layers and heads, and combined-benchmark Pass@1 performance.The study includes open and closed models with varied architectures and sizes.Source: §4.1 · pp. 7–8, Table 1 · p. 8

4.2 Model Attention Calculation

• Twelve attention-calculation methods were tested: six self-attention-based, four gradient-based, and two perturbation-based methods.Self-attention methods vary layer and generation-step aggregation; gradient methods use saliency or Input×Gradient, while perturbation methods include SHAP and BERT Masking.Source: §4.2 · p. 8, §4.2.1 · p. 8, §4.2.2 · pp. 8–9

4.2.3 Perturbation-based Methods

• SHAP masks input tokens by deletion, while BERT Masking substitutes each token with a BERT prediction and scores the regenerated code against the original using BLEU.SHAP scores were aggregated across predicted tokens; BERT Masking repeats the substitution and regeneration process for each prompt token.Source: §4.2.3 · p. 9

4.3 Attention Alignment Measurement

• Alignment was measured primarily with San Martino’s token-overlap metrics and Krippendorff’s alpha; Cohen’s kappa and keyword coverage were also tested but not reported in the paper.For token overlap, model tokens were ranked by attention and the top K (5, 10, or 20) compared with human-selected words; partial token overlap receives credit.Source: §4.3 · p. 9, §4.3.1 · p. 9

4.3.2 Krippendorff’s Alpha [48]

• Because model subword tokenization can split a human-labeled word, the study maps that word to its subtokens when forming equal-length vectors for Krippendorff’s alpha.For example, if “separate” is selected by a human labeler, both “separ” and “ate” are marked selected.Source: §4.3.2 · p. 10, Figure 4 · p. 10

4.4 User Study Design

• The user study recruited 22 students with an average of 5.62 years of programming experience and compared three attention methods on eight task descriptions using CodeGen-2.7B.GPT-4 was excluded because its closed-source API does not permit the self-attention and gradient methods used in the comparison.Source: §4.4 · p. 10
• Participants rated whether each visualization matched their reading attention, explained code-generation success or failure, and would be useful in practice.Methods were presented in randomized order without revealing their names, and participants also answered open-ended questions about explainability and trust.Source: §4.4 · pp. 11–12

5.1 RQ1: To What Extent Is Model Attention Aligned With Human Attention?

• GPT-4 was assessed only with perturbation-based methods because its internal states are unavailable; the section reports the best of these, BERT_masking.The broader comparison of attention-calculation methods is addressed separately in Section 5.3.Source: §5.1 · p. 12
• With BERT_masking, increasing K from 5 to 10 raised F1 as recall approached 90% and offset declining precision; from 10 to 20, F1 stayed stable as precision fell rapidly.Krippendorff’s alpha was stable from K=5 to K=10 except for GPT-4, then declined rapidly from K=10 to K=20.Source: §5.1 · p. 12
• Across all models and K settings, Krippendorff’s alpha remained below 0.3 and F1 below 0.6, indicating little agreement between model and programmer attention.GPT-4 had the lowest overlap on both metrics; the authors describe misalignment across all settings.Source: §5.1 · p. 12

5.2 RQ2: Can Attention Explain Errors of Code Generation Models?

• The authors sampled 211 of 920 incorrect solutions from GPT-4 and CodeGen-2.7B, then manually assessed whether model attention could explain the errors.They began with 50 errors, using BERT_masking, the most aligned method in the quantitative experiments, to identify common patterns.Source: §5.2 · pp. 12–13

5.2.3 Missing Attention to Operation Descriptions.

• When models miss operation verbs or phrases, generated code may perform the wrong action or omit the requested action.The authors identify missing attention to operation descriptions as a recurring pattern.Source: §5.2 · p. 13

5.2.4 Missing Attention to Data Types.

• Missing data-type descriptions can lead to wrong return types or methods applied to the wrong object type; one example missed “z” and failed to test for its presence.The paper reports seven errors in this example category.Source: §5.2 · p. 13, §5.2.4 · p. 14

5.2.5 Incorrect Mapping Between NL Words and Code Elements.

• A model may attend to a relevant word yet map it to the wrong code element; attending to “bell” still produced factorial logic instead of a Bell number.The authors report 25 incorrect solutions in this category.Source: §5.2 · p. 13, §5.2.4 · p. 15, §5.2.5 · p. 15

5.2 RQ2: Can Attention Explain Errors of Code Generation Models?

• Of 211 sampled errors, 57 (27%) matched five attention patterns; this covered 54/172 CodeGen-2.7B errors (31%) versus 3/39 GPT-4 errors (8%).The authors note that many errors, including syntax errors and incorrect API usage, were not readily explained by attention patterns.Source: §5.2 · p. 13

5.2.1 Missing Attention to Critical Conditions.

• Missing “followed by zero or more” led CodeGen-2.7B to mishandle the condition that zero or multiple “b” characters could follow “a”; six errors fit this pattern.The generated function instead matched one “a” followed by one or more “b”s.Source: §5.2.1 · p. 14

5.2.2 Missing Attention to Important Descriptive Words of an Operation or a Data Object.

• Missing “individual items” led to powering one list element instead of all (13 errors); missing “count” led to equality testing instead of counting (6 errors).These examples show errors from missing descriptive or operator words in the task description.Source: §5.2.2 · p. 14, §5.2.3 · p. 14

5.3 RQ3: What Is the Impact of Attention Calculation Methods on the Alignment?

• Gradient-based methods generally align better than self-attention-based methods, especially at K=5 on both reported metrics.Self-attention methods are least aligned at smaller K; the authors suggest that considering self-attention alone may miss token influences from feed-forward layers.Source: §5.3 · p. 15
• For gradient-based methods, coding-stage attention aligns better than reading-stage attention across K=5 and K=10, while Input×Gradient and Saliency differ only trivially.For self-attention-based methods, coding-stage attention also aligns better than reading-stage attention.Source: §5.3 · pp. 15–16
• BERT_masking achieves the best human-attention alignment among the methods, according to the reported averages across five models and K settings.GPT-4 is excluded from the averages; the table reports San Martino’s F-1 and Krippendorff’s alpha.Source: Table 3 · p. 15, §5.3 · pp. 15–16
• Among self-attention aggregation choices, using the last layer gives the highest alignment in both stages, except at K=5 during coding.Self-attention distributions from the coding stage are consistently more aligned than those from the reading stage.Source: §5.3 · p. 16

5.4 RQ4: Which Attention Calculation Method Is the Most Preferred?

• SHAP received higher average attention-alignment ratings than InputXGradient_coding and CODING_all: 5.13 versus 4.59 and 3.62, respectively.Participants also favored SHAP for explainability and overall preference.Source: §5.4 · pp. 16–17
• SHAP was rated more explainable than InputXGradient_coding by 0.59 points and CODING_all by 1.26 points.The reported Welch’s t-test p-values were 0.02 and 0.00001, respectively.Source: §5.4 · p. 16
• Despite seeing attention-based explanations, 14 of 22 participants lacked confidence and trust in LLMs and requested reference-code rationales or finer input-to-output attention mapping.SHAP’s mean preference rating was 5.04, compared with 4.56 for InputXGradient_coding and 3.48 for CODING_all.Source: §5.4 · pp. 16–17

6 IMPLICATIONS AND OPPORTUNITIES

• In 211 incorrect generations from GPT-4 and CodeGen-2.7B, 27% of errors could be explained by incorrect attention alignment.The authors suggest exploring an attention-alignment penalty based on KL divergence and released their human-attention dataset to support future work.Source: §6 · p. 17
• The authors propose prioritizing important prompt tokens during adversarial perturbation to help reveal code-generation robustness issues more efficiently.They contrast this with methods that perturb random prompt tokens without considering token importance.Source: §6 · p. 17
• For measuring model attention, the authors recommend BERT_masking first; among other methods, gradient-based approaches generally align more stably than self-attention-based approaches.They also recommend coding-stage attention for gradient-based and self-attention-based methods, and last-layer scores for self-attention.Source: §6 · pp. 17–18
• Most user-study participants considered SHAP the best XAI method, and the authors suggest it as a possible default for LLM code-generation explanations.Participants also requested finer-grained associations between individual input and output tokens.Source: §6 · p. 18
• The authors propose studying how LLMs interpret code so programmers might craft prompts that are more understandable to models and guide better code generation.This is presented as future work.Source: §6 · p. 18

7 THREAT TO VALIDITY

• Keyword selection was subjective and involved two programmers, although discussion established a labeling standard and a third labeler’s validation yielded Fleiss’ Kappa of 0.64.The third labeler independently assessed 164 HumanEval prompts.Source: §7 · p. 18
• Because experiments covered only Python, the authors cannot guarantee that findings generalize to other programming languages.The authors identify this as an external-validity threat.Source: §7 · p. 18
• The survey involved 22 experienced Python programmers who had used code-generation LLMs, limiting how broadly its findings may generalize.The authors note that programmers may differ in which prompt content they think a model should attend to.Source: §7 · p. 18

8.1 Code Generation From Natural Language

• Code-generation work spans natural-language-to-code models and methods that augment generation with retrieval, domain knowledge, testing, decomposition, or sketches.The section also describes models trained on code and text, including CodeBERT and more recent LLMs.Source: §8.1 · pp. 18–19

8.2 Empirical Studies on Code Generation

• Prior code-generation studies examine performance, robustness, security, usability, and explainability, while this work compares model attention with programmers’ attention to task descriptions.Earlier explainability studies probe learned code properties or analyze model self-attention and code structures.Source: §8.2 · p. 19

8.3 Model Attention Analysis in Other Domains

• Attention-analysis approaches in other domains use gradients, pooling, deconvolution, or semantic predictive control to explain model features, image regions, or action mappings.The examples span computer vision and autonomous driving.Source: §8.3 · p. 19

9 CONCLUSION

• The study finds consistent misalignment between LLMs’ and programmers’ attention in code generation.The paper presents an empirical study of attention alignment between LLM-based code-generation models and programmers.Source: §9 · pp. 19–20
• Among 12 attention calculation methods, perturbation-based methods align better with human attention and are more preferred by user-study participants.Source: §9 · p. 20
• The study discusses implications and future research opportunities for interpreting and improving the performance of LLM-based code-generation models.Source: §9 · p. 20