PaperlayerView exact revision on arXiv

Source-linked AI summary

TaskEval: Assessing Difficulty of Code Generation Tasks for Large Language Models

Florian Tambon, Amin Nikanjam, Cyrine Zid, Foutse Khomh, Giuliano Antoniol

arXiv:2407.21227v3cs.SEcs.AI

TL;DR

Benchmark-level aggregation limits fine-grained analysis of task characteristics. TaskEval evaluates code-generation tasks using different prompt information levels, and the paper reports stark differences between CodeLLM and human difficulty assessments.

  • Problem

    Benchmark-level aggregation provides an incomplete picture of individual task characteristics.

    Source: §1 · p. 2

  • Method

    TaskEval evaluates code-generation task characteristics with a framework that varies the information level in task prompts.

    Source: §1 · p. 3, §3.2 · p. 7

  • Results

    The comparison of task difficulty assessments reveals stark differences between CodeLLMs and human annotators.

    Source: §6 · p. 28, §9 · p. 32

  • Takeaways & Limitations

    Human difficulty assessments may not align with CodeLLM task difficulty in this study.

    Source: §6 · p. 28, §9 · p. 32

    The two-dataset scope limits generalisation of the study’s practical insights.

    Source: §8 · p. 31

Abstract

from arXiv · show

Large Language Models (LLMs) excel in code-related tasks like code generation, but benchmark evaluations often overlook task characteristics, such as difficulty. Moreover, benchmarks are usually built using tasks described with a single prompt, despite the formulation of prompts having a profound impact on the outcome. This paper introduces a generalist approach, TaskEval, a framework using diverse prompts and Item Response Theory (IRT) to efficiently assess LLMs' capabilities and benchmark task characteristics, improving the understanding of their performance. Using two code generation benchmarks, \textit{HumanEval}+ and \textit{ClassEval}, as well as 8 code generation LLMs, we show that \textit{TaskEval} is capable of characterising the properties of tasks. Using topic analysis, we identify and analyse the tasks of 17 and 21 topics within the benchmarks. We also cross-analyse tasks' characteristics with programming constructs (e.g., variable assignment, conditions, etc.) used by LLMs, emphasising some patterns with tasks' difficulty. Finally, we conduct a comparison between the difficulty assessment of tasks by human annotators and LLMs. Orthogonal to current benchmarking evaluation efforts, \textit{TaskEval} can assist researchers and practitioners in fostering better assessments of LLMs. The tasks' characteristics can be used to identify shortcomings within existing benchmarks or improve the evaluation of LLMs.

1 Introduction

TaskEval addresses benchmark evaluations that aggregate across tasks and typically use one prompt by introducing a multi-prompt framework for assessing task characteristics. Its analyses report differences across benchmarks, topics, programming constructs, and human versus CodeLLM difficulty assessments.

1 Introduction

  • Benchmark-level metrics aggregate across tasks, and single-prompt benchmarks do not capture task-level characteristics while accounting for prompt variability.The paper motivates more fine-grained task-level evaluation and analysis.Source: §1 · p. 2
  • TaskEval generates 18 prompts per task with varied phrasing and context, queries eight code LLMs, and uses IRT to estimate task difficulty, discrimination, and model ability.It operates on existing benchmarks, including HumanEval+ and ClassEval.Source: §1 · pp. 2–3
  • ClassEval tends to contain more extreme easy and difficult tasks than HumanEval+.The paper also compares human and CodeLLM difficulty assessments and reports a difference between them.Source: §1 · p. 3
  • Tasks involving sequences of numbers in HumanEval+ tend to have higher difficulty and discriminate between CodeLLMs.The analysis identifies 17 topics in HumanEval+ and 21 in ClassEval.Source: §1 · p. 3
  • Conditional-statement use increases with task difficulty regardless of code length.The analysis examines generated-code constructs using Abstract Syntax Trees.Source: §1 · p. 3
  • The paper finds a difference between CodeLLM and human difficulty assessments, arguing that human ratings should not be used as-is for CodeLLM task difficulty.The comparison is part of the paper’s analysis of human annotators and CodeLLMs.Source: §1 · p. 3

2 Motivating examples

The examples show that a task’s measured difficulty and model outputs can vary across prompt formulations. TaskEval uses multiple prompts to account for prompt and model variation when estimating task difficulty.

2 Motivating examples

  • The sorting task’s difficulty is 0.17, while wording the duplicate condition as “occurs more than twice” raises Pass@1 to 0.6–1.0 across models.The original prompt’s Pass@1 is 0 for all models except DeepSeek-V3 at 1.0 and QwenCoder at 0.2.Source: §2 · p. 4
  • For the evaluated task, a more thorough yet high-level description of intended functionality (the bottom prompt) improves Pass@k for all models, reaching 0.6–1.0.Source: §2 · p. 4
  • Average Pass@1 shifts only 0.01–0.03 standard deviation for HumanEval+ and 0.02–0.05 for ClassEval, while individual-task Pass@1 can double on 10%–30% of tasks for a given model.The authors argue that such prompt sensitivity prevents inferring task difficulty from a single prompt formulation.Source: §2 · p. 5
  • TaskEval’s difficulty score accounts for prompt variation and differences between models, rather than characterizing only one prompt formulation.The examples contrast the prompt-level changes in generated-code correctness with the task’s difficulty score.Source: §2 · pp. 5–6

3 TaskEval

TaskEval characterizes benchmark task characteristics by varying prompts, evaluating code generated by multiple CodeLLMs for functional correctness, and modeling test scores with β3-IRT.

3.1 General Overview of TaskEval

  • TaskEval varies prompts while preserving task semantics, evaluates code from multiple CodeLLMs on benchmark tests, and fits IRT to estimate task and model attributes.The framework targets task characteristics rather than prompt characteristics; scores are aggregated per task and CodeLLM before model fitting.Source: §3.1 · p. 6

3.2 Generating Different Prompts based on Transformations

  • The two prompt transformations are Context Information, which varies disclosed information without changing the task, and Rephrasing, which paraphrases a prompt using a Prompt LLM.Templates are tailored to each task; the workflow applies Context Information levels before applying the Rephrasing template.Source: §3.2 · pp. 7–9
  • Context Information prompts have three levels, progressing from inputs, outputs, and a high-level description to natural-language function details and direct references such as variable names and helper functions.The levels are generated incrementally; the first is intended to use consistent formatting across benchmark tasks.Source: §3.2 · pp. 7–8
  • The prompt-generation design removes examples when present to control injected information, while assessing additional techniques such as Chain-of-Thought is outside this study’s scope.The authors note that removing examples can decrease CodeLLM performance.Source: §3.2 · pp. 8–9
  • Transformed examples can range from basic cases to corner cases, and their quality may artificially bias apparent task difficulty.Source: §3.2 · p. 9
  • Authors manually checked transformed prompts, with a second-author cross-check, to preserve context levels and semantics and prevent misleading wording from making tasks seem hard.Source: §3.2 · p. 9

3.3 Collecting metrics for code snippets

  • For each prompt, evaluator CodeLLMs generate seed-sampled code, which is run against benchmark tests; functional-correctness scores are aggregated across prompts and seeds per task.Source: §3.3 · pp. 9–10

3.4 Item Response Theory modelling

  • β3-IRT models continuous task-level scores, treating CodeLLMs as respondents and tasks as items; difficulty locates expected response at 0.5, while discrimination captures curve steepness.Source: §3.4 · pp. 10–11

4 Experimental setup

The experimental setup uses two code-generation benchmarks, eight CodeLLMs, and varied prompts, fitting separate IRT models to analyze benchmark, topic, and programming-construct effects. It also analyzes programming constructs in generated code and compares IRT-based task difficulty with human effort estimates using multiple CodeBLEU thresholds and an iterative crowd-assessment process.

4.1 Hyperparameters used

  • Each benchmark task receives 18 prompts from three levels and six rephrasings, and each CodeLLM generates five samples per prompt, totaling 90n samples per model.For 200 tasks, this amounts to 18,000 generated code samples per CodeLLM.Source: §4.1 · p. 12

4.2 Benchmark used

  • HumanEval+ [52] covers 164 function-level Python problems, while ClassEval [23] provides class-context tasks drawn from 200 sampled methods across 100 classes.ClassEval’s 200 methods were stratified-sampled and allocated equally across classes; its compositional prompting strategy supplies class context while requesting one method.Source: §4.2 · p. 12

4.3 Models used

  • GPT-4 generates prompts, while eight instruction-tuned CodeLLMs provide code samples; GPT-4 is excluded as a CodeLLM to avoid bias.The CodeLLMs are YiCoder-1.5B [9], CodeLLama 7B [68], MagiCoder 6.7B [88], DeepSeekCoder 7B [32], CodeGemma 7B [76], QwenCoder-2.5 7B [91], GPT-3.5, and DeepSeek-V3 [21].Source: §4.3 · p. 13

4.4 IRT model fit

  • After fitting separate IRT models to collected HumanEval and ClassEval scores, R^2 was 0.87 for HumanEval and 0.90 for ClassEval, respectively.Source: §4.4 · p. 13

4.5 Research Questions

  • The study compares benchmarks using CodeLLMs’ expected-response distributions and Anderson-Darling tests, then profiles task difficulty and discrimination.The analyses use cumulative distributions across tasks and the IRT-estimated difficulty and discrimination parameters.Source: §4.5 · p. 13
  • BERTopic analysis of Level 1 prompts yielded 17 HumanEval+ topics and 21 ClassEval topics, excluding noise and requiring at least three tasks per topic.The researchers manually reviewed and refined topic names based on the assigned tasks and their prompts.Source: §4.5 · p. 14
  • To examine construct effects, the study analyzes ASTs of generated code and uses CodeBLEU similarity to retain fragments sufficiently like a correct solution.The analysis is intended to relate task difficulty or discrimination to constructs such as if statements, for-loops, and function calls.Source: §4.5 · p. 14
  • AST programming-construct counts were normalized by code length, averaged per task, and compared with task difficulty or discrimination using Kendall’s tau and top/bottom-group AD tests.Tasks and constructs were filtered to avoid sparse code-fragment samples and rarely represented node types.Source: §4.5 · pp. 14–15
  • Using Holm-Bonferroni to require significance across all tested similarity thresholds, the analysis retained 80%–94% of ClassEval tasks and 88%–99% of HumanEval+ tasks.Source: §4.5 · p. 15
  • The study adopted an alternative experimental setup leveraging effort-estimation techniques and implemented Alhamed et al.’s iterative method with five participants per task per round.When responses exceeded a predefined threshold, task effort was estimated by their median.Source: §4.5 · pp. 15–16
  • The first round targeted 50 developers and sampled 120 of 364 benchmark tasks into ten questionnaires, balancing task groups by oracle code length and cyclomatic complexity.Participants saw task prompts also used for the CodeLLMs, including the original prompt, then estimated completion time and subjective difficulty and explained their choices.Source: §4.5 · p. 16
  • After responses reached substantial agreement, human-rated difficulty was compared with discretized IRT scores; the final sample comprised 72 developers with at least three years’ experience.Time estimates were used in the analysis, while subjective difficulty and explanations supported response-quality checks; participants’ experience ranged from under five years to over ten years.Source: §4.5 · p. 16

5 Results

The results examine task difficulty and discriminative power across benchmarks, topics, programming constructs, and prompt formulations, alongside comparisons between human and CodeLLM assessments. They report differences in estimated difficulty and model behavior across these dimensions, as well as weak agreement between human ratings and CodeLLM-based estimates.

5.1 Validation

  • Same-level prompt rephrasings were semantically closer to one another than to rephrasings from other levels (p-value < 0.01; effect size > 0.8).Source: §5.1 · p. 17
  • Median task difficulty fell from 0.42 (Level 1) to 0.15 (Level 3) for HumanEval+ and from 0.27 (Level 1) to 0.05 (Level 3) for ClassEval.Across levels, differences were significant (p-value < 0.01), supporting the finding that more prompt information lowers estimated difficulty.Source: §5.1 · p. 17

5.2 RQ1: How do tasks’ characteristics varies across benchmark?

  • HumanEval+ shows wider performance variation across CodeLLMs; Code Gemma, Code Llama, and Yi-Coder have less than ~50% chance of success on over 40% of its tasks.Source: §5.2 · p. 17
  • ClassEval contains more very easy and very hard tasks than HumanEval+, while HumanEval+ has more medium-difficulty tasks on average.Source: Figure 3 · p. 18, §5.2 · p. 18
  • Task discriminability peaks at intermediate difficulty ranges: 0.4–0.6 for HumanEval+ and 0.2–0.4 and 0.6–0.8 for ClassEval.Tasks at extreme difficulty are less useful for distinguishing models because models tend either to fail them all or succeed at similar rates.Source: Figure 4 · p. 19, §5.2 · p. 19
  • Manual review found prompt-formulation issues in 7 of 9 anomalous ClassEval tasks and the single HumanEval+ task, including missing radians-conversion instructions that misled generated code.Source: §5.2 · p. 20
  • For integer filtering, GPT3.5 often uses isinstance while Code Llama uses type, and the latter avoids treating Python booleans as integers.The authors suggest this difference may reflect training-data specificity or, in the worst case, memorization.Source: §5.2 · pp. 20–21
  • ClassEval has more very easy or difficult tasks than HumanEval+, which has more medium-difficulty tasks; the most discriminating tasks in both benchmarks tend to separate models.Few tasks have negative discrimination, which the authors attribute to small differences caused by nondeterminism or annotation.Source: §5.2 · p. 21

5.3 RQ2: Do tasks’ topics affect their characteristics?

  • HumanEval+ topics are mostly easy-to-medium, with sequences and several string or bracket topics harder; in ClassEval, SQL requests, file reading, and data retrieval are relatively trivial.ClassEval also has harder topics, including data-structure calculations, text encryption/decryption, and string replacements.Source: §5.3 · p. 21
  • HumanEval+ sequences have high positive discrimination, while ClassEval has more low-discrimination topics, indicating less separation among models on those tasks.Topic-level accuracy varies; DeepSeek-V3 generally has the highest mean accuracy and Code Llama the lowest, though less capable models sometimes outperform more capable ones on particular topics.Source: §5.3 · pp. 21, 23
  • On some ClassEval tasks, more capable models infer unrequested assumptions and generate unnecessarily complex code, while Code Llama follows the prompt more closely.The paper illustrates this with a course-management task where GPT3.5 adds a conflict check that the prompt does not request.Source: §5.3 · pp. 23–24

5.4 RQ3: Do programming constructs affect tasks’ characteristics?

  • Across benchmarks, If and Assign constructs recur in tasks of varying difficulty, and after controlling for code-fragment size, harder tasks are associated with higher-level processing.ClassEval shows no significant correlation between task discrimination and program constructs.Source: §5.4 · p. 24
  • For HumanEval+, more discriminant tasks were on average harder and more likely to elicit variable assignments, data structures, and attribute use; the authors observed no corresponding patterns for ClassEval.They suggest that the difficulty pattern may reflect highly discriminant tasks also being more difficult on average.Source: §5.4 · p. 25
  • Conditions and variable assignment appear more often in harder tasks regardless of code length; Subscript in ClassEval and DataStruct in HumanEval+ are benchmark-specific, while discriminant patterns are absent except for HumanEval+ Assign and Attribute.The authors note that these AST findings describe how CodeLLMs attempt tasks, not necessarily what the tasks require.Source: §5.4 · p. 26

5.5 RQ4: Do human annotators and CodeLLMs rate tasks’ difficulty similarly?

  • 40% of ClassEval tasks versus 17% of HumanEval+ tasks were rated by humans as taking at least 20 minutes, with a significant distribution difference (p-value < 0.01).This contrasts with CodeLLM assessments, which tended to identify more medium-difficult tasks in HumanEval+; the sampled tasks showed no significant difficulty-distribution difference from their full benchmarks.Source: §5.5 · p. 27
  • Weighted agreement between human and CodeLLM difficulty ratings was low: κ = 0.21 for HumanEval+ and κ = 0.17 for ClassEval, indicating humans are not a reliable proxy.The authors report that shifting human ratings up or down by one category would not substantially change agreement.Source: §5.5 · p. 27
  • Prompts with convoluted wording or multiple conditions could be rated easy by humans but challenge CodeLLMs, which may lose track of conditions.In one example, CodeLLM difficulty was 0.88 while humans’ median estimated time was 5 minutes; generated code often merged two conditions incorrectly.Source: §5.5 · p. 28
  • Human annotators’ difficulty assessments differ from CodeLLMs’, suggesting human judgment may not be a valid proxy for CodeLLM task difficulty.This motivates approaches such as TaskEval that use CodeLLMs’ own assessments to establish task characteristics.Source: §5.5 · p. 28

6 Discussion

TaskEval applies IRT to multiple prompts per task to produce finer-grained benchmark analysis. The discussion argues that human judgments may not reflect CodeLLMs’ task difficulty and highlights practical uses and constraints of the approach.

Multi-prompt evaluation

  • Prompt variation can drastically alter generated outputs, so TaskEval evaluates task characteristics across formulations rather than tuning one prompt for best performance.The discussion cautions users to take care when prompting a CodeLLM, particularly when relying on a single generation.Source: §6 · p. 28

Human assessment of difficulty

  • Human-assessed difficulty may not reflect difficulty for CodeLLMs, so benchmark difficulty labels may not apply to evaluating them.The discussion suggests using IRT difficulty estimated from CodeLLM outputs as a more faithful alternative.Source: §6 · p. 28

Identifying CodeLLMs shortcomings

  • Task-level IRT analysis can reveal shortcomings, especially when difficult tasks cluster within a topic, and guide further investigation.The discussion suggests fine-tuning on similar tasks or prompt templates, or using in-context learning, as possible responses.Source: §6 · p. 29
  • A list of easy and hard tasks alone may not explain why tasks are difficult enough to support accurate generation of new tasks.The discussion proposes adding constraints to hard tasks or transferring constraints from hard to easy tasks, while noting this may reduce task diversity and increase the likelihood generated tasks are hard.Source: §6 · p. 29

Evaluating benchmark discriminability

  • For differentiating CodeLLMs within a benchmark, the most discriminant tasks are useful because models differ most on them; low-discriminant tasks reveal shortcomings when models perform similarly and all struggle.Source: §6 · p. 29

Cost and Scalability

  • Manual prompt verification is the main scalability constraint, although the authors consider the approach relatively inexpensive and scalable to more benchmarks or models.Verification is needed once for the benchmarks under study.Source: §6 · p. 29

7 Related Works

Prior CodeLLM evaluations often aggregate results at the benchmark level and may not assess prompt rephrasing or task difficulty. TaskEval instead measures task-level difficulty across diverse prompts, extending IRT analysis to code generation.

Code benchmarks

  • Many code benchmarks aggregate performance over the whole benchmark using accuracy or pass@k, despite differing in task scope and dependency.Examples mentioned include function-level HumanEval+ and class-level ClassEval, as well as APPS, CodeXGlue, and BigCodeBench.Source: §7 · p. 30

Evaluating CodeLLMs

  • Benchmark-level properties and prompt-rephrasing effects are often not evaluated, while difficulty is commonly based on human judgment.The discussion reports that human difficulty assessment might not translate to the difficulty of realizing tasks for CodeLLMs.Source: §7 · p. 30

Item Response Theory to evaluate NLP models

  • IRT has been used in NLP to compare models, select discriminative examples, assess ranking stability, and analyze dataset difficulty.The cited studies apply IRT to chatbots, leaderboards, NLP datasets, and adaptive CodeLLM testing.Source: §7 · p. 30

TaskEval’s focus

  • TaskEval focuses on code-generation task difficulty rather than prompt quality, using multiple prompts to represent possible formulations for each task.Unlike prompt engineering aimed at optimizing results on a particular task, TaskEval generates prompt variants and observes model outputs.Source: §7 · p. 30

8 Threats to Validity

The study’s validity is bounded by its modeling choices, selected benchmarks and models, and the scope of its human and topic analyses. The authors also identify prompt dependence and the limited coverage of practical tasks as constraints on interpretation and generalization.

Construct validity

  • TaskEval uses three prompt-information levels based on the authors’ observations, and multiple prompts may challenge IRT’s local-independence assumption.The authors aggregate prompts for each task into polytomous superitems or testlets to alleviate local dependence.Source: §8 · p. 31
  • Prompt generation may bias evaluation, although the authors report that modeled abilities did not appear influenced by using GPT-4 to generate prompts.Prompts were manually checked for semantic preservation; omitting examples may also decrease model performance and make tasks appear artificially hard, though the authors say this was needed for fair comparison.Source: §8 · p. 31

Internal validity

  • The study’s two benchmarks, eight models, and automatic topic labeling constrain its coverage; discarded noisy tasks may reduce the topic list.The authors report retaining over 15 topics with at least three tasks each and manually reviewing the topics.Source: §8 · p. 31
  • Human annotations capture perceived task difficulty rather than performance from annotators coding the tasks.The study used effort estimation instead of asking crowd-workers to code, and the authors report a difference between human and CodeLLM difficulty perceptions despite variation in human judgment.Source: §8 · p. 31

External validity

  • The selected tasks may not represent real-world programming, limiting practical generalization despite covering function- and class-level benchmarks.The authors limited the study to two datasets to reduce analysis complexity and prompt-generation and human-judgment costs.Source: §8 · p. 31
  • The authors identify advanced, other-language, and industry-oriented benchmarks as valuable settings for future study.They state that the approach itself should not be affected by more practical benchmarks, while practical insights from this study remain limited by the chosen datasets.Source: §8 · pp. 31–32

9 Conclusion

TaskEval uses diverse prompts and IRT to assess code-generation task characteristics, then examines topic and program-construct patterns and compares difficulty assessments by CodeLLMs and humans.

9 Conclusion

  • TaskEval assesses task characteristics with diverse prompts and IRT across HumanEval+ and ClassEval, using eight CodeLLMs.The framework estimates task difficulty and discriminant characteristics.Source: §9 · p. 32
  • Across 17 and 21 topic clusters, the analysis identifies topics with high difficulty and discrimination for CodeLLMs, including Sequences.The analysis clusters tasks from both benchmarks by thematic similarity.Source: §9 · p. 32
  • Cross-analysis relates task characteristics to program constructs and highlights similarities and discrepancies between benchmarks.Source: §9 · p. 32
  • Human annotators' difficulty assessments differ starkly from CodeLLMs'.Source: §9 · p. 32
  • The authors plan to extend TaskEval to code completion and summarisation and explore task selection and generation.Examples include selection based on discriminative features and generation using topic modelling.Source: §9 · p. 32
Loading 2407.21227v3…