InfiBench: Evaluating the Question-Answering Capabilities of Code Large Language Models

1 Introduction

• Existing code benchmarks emphasize code generation, often reuse source data, can be saturated, and may be contaminated, leaving real-world coding questions insufficiently assessed.The cited examples include HumanEval and MBPP for Python code generation and DS-1000 for Python data-science code generation.Source: §1 · p. 2
• InfiBench evaluates free-form question answering using 234 Stack Overflow questions across 15 programming languages and five major areas.Questions are selected to reflect the natural distribution of high-quality Stack Overflow questions, without restrictions on topic, language, question type, or answer form.Source: §1 · pp. 2–3
• Domain experts specify criteria for four model-free metrics, and human experiments show 85.1% agreement with InfiBench evaluation versus 77.8% with GPT-4o-based evaluation.The metrics are keyword matching, blank filling, unit testing, and dialogue similarity.Source: §1 · p. 3
• GPT-4 scores 70.64%, while the evaluation finds large differences among model families, stronger performance from coding and fine-tuned models at similar sizes, and a scaling-law turning point above 40B parameters.The scaling law is empirically verified for open-source models below 40B parameters, but not for larger models.Source: §1 · p. 3

2.1 Data Curation

• Questions with at least three positively voted answers and an accepted answer are filtered by age and views, then mapped to language and area tags to form a 13,854-question seed set.Six domain-specific or data-description languages are excluded from the seed set.Source: §2.1 · p. 4

2.2 Sampling

• Area quotas reflect a developer-demand study; language quotas follow seed-set frequencies, with a minimum of 10 questions for CSS and C/C++ to support language-level subscores.The initial quotas total 270 questions, from which a larger pool is sampled for expert selection and annotation.Source: §2.2 · p. 4

2.3 Human Annotation

• Experts select and classify questions, paraphrase prompts, and annotate correctness criteria; calibration and post-filtering yield 234 questions across 15 languages.The process includes review and consensus on controversial cases, and removes questions with incorrect or out-of-distribution criteria.Source: §2.3 · pp. 4–5

2.4 Evaluation Criteria and Evaluation Framework

• Four model-free metrics cover diverse answers: keyword matching, blank filling, unit testing, and dialogue similarity using ROUGE-based comparison to reference answers.The framework assigns each of 234 questions one point by default, with that point divisible into weighted scoring components.Source: §2.4 · pp. 5–6
• In a comparison of response-pair judgments, InfiBench agrees with human experts more often than GPT-4o evaluation, at 85.1% versus 77.8%.The authors attribute the advantage to detecting deceptive answers through required key concepts.Source: §2.4 · pp. 6–7

2.5 Mitigations on Memorization and Data Contamination

• Paraphrasing mitigates potential memorization: copying Stack Overflow answers scores 65.18%, and models using Stack Overflow data show no significant advantage over those that do not.The authors deem the effect of contamination small and release source post IDs for future deduplication and ablation studies.Source: §2.5 · p. 7

2.6 Comparison with Existing Benchmarks

• InfiBench complements existing benchmarks with diverse real-world tasks, data sources, and programming languages, but its size is limited by the cost of correctness-criterion labeling.The authors state that the benchmark is continuously expanding.Source: §2.6 · p. 7

• Over 100 proprietary and open-source code LLMs were systematically evaluated on InfiBench.Source: §3 · p. 7
• The main evaluation uses best@10, recording the best of 10 sampled responses for each question and summing those scores.Sampling uses temperature 0.2 and top p 0.9.Source: §3 · p. 7
• GPT-4 scored 76.15%±0.21% at T = 1.0 and p = 0.9, versus 70.64%±0.82% at T = 0.2 and p = 0.9.The authors retained the lower-temperature setting as more akin to real-world use where a user generates once.Source: §3 · p. 7
• Normal questions and open-ended questions use distinct system prompts, and model prompt templates are used for instruction-finetuned or chat models.Source: §3 · p. 7
• The evaluation uses API calls for proprietary models and an 8xA100 server for open-source models; some evaluations are repeated three times with standard deviations reported.The repetitions apply when budget permits for proprietary models and to open-source models within 30B parameters.Source: §3 · p. 7
• The leaderboard aggregates scores by model family, type, and size; Table 4 encodes model categories by bar color and proprietary status by icon.Figure 4 presents scores as scatters with error bars for normal models, parameter-range segments for MoE models, and horizontal lines for strong proprietary models.Source: §3 · p. 8, Table 4 · p. 8

Model performance

• InfiBench scores are 70.64% for GPT-4, 63.89% for Claude 3 Opus, 62.98% for Codestral-22b, and 62.96% for deepseek-coder-33b-instruct.The authors note that GPT-4-0613, rather than the newer GPT-4o, achieves the highest score; Codestral-22b is roughly 8 points below GPT-4.Source: §4 · p. 8
• At 7B, open-source model scores range from around 15% to 55%; deepseek-coder-1.3b-instruct scores 41.32% at 1.3B, surpassing some 70B- or 100B-scale models.Figure 4 plots model size against InfiBench score; the authors conclude that training techniques and data can matter as much as or more than scale, reducing the required scale by over 10× for a given score.Source: Figure 4 · p. 9, §4 · p. 9

Scaling projections

• Figure 4’s projected curves flatten after an initially steep segment; the authors estimate open-source models would need to exceed 300B parameters to catch GPT-4 along the flatter segment.They note that this projection differs from commonly observed scaling laws and leave investigation of the apparent barrier beyond 40B or a change in scaling behavior for future work.Source: Figure 4 · p. 9, §4 · p. 10

Problem difficulty

• InfiBench subscores decrease as problem difficulty rises, and problems difficult for GPT-4 are generally difficult for open-source models too.Hard items often involve long, domain-specific code-generation contexts or challenging blank-filling questions, a task the authors say rarely appeared in prior training data.Source: §4 · p. 9

Instruction tuning

• Instruction-finetuned models almost always perform best among similarly sized models in the same family, and instruction tuning generally improves both QA and code generation.In their preliminary Stack Overflow fine-tuning experiments, Codellama-13b-Instruct improves from 46.37% to 60.74%, while mixtral-8x7B-Instruct improves from 55.55% to 62.61%.Source: §4 · p. 9

Benchmark comparison

• For smaller models, InfiBench and HumanEval results can diverge: some models fare relatively better on QA, while more fare relatively better on code generation.The authors interpret this as evidence that some models may be optimized too heavily for code-generation benchmarks while neglecting generic code scenarios.Source: §4 · p. 10

Code Llama

• Code Llama results show a trade-off: Python fine-tuning improves HumanEval but hurts InfiBench, whereas instruction fine-tuning usually improves InfiBench but may hurt HumanEval.The authors also report that CodeLlama-70B may be overly safeguarded and decline to answer some safe InfiBench questions.Source: §4 · p. 10, Table 5 · p. 10

Scaling projections

• Code models tend to outperform general models of the same scale, although the gap shrinks at larger scales.The authors suggest code-only training could be more compute-efficient when the compute budget is heavily limited.Source: §4 · p. 10

Related benchmarks

• Code-generation benchmarks have been extended through language translation, test augmentation, task generalization, and human rewriting, but these extensions still primarily target code generation.The examples named include Multilingual HumanEval, HumanEval-X, HumanEval+, HumanEvalPack, and LBPP.Source: §5 · pp. 10–11
• Many code benchmarks focus on code generation; broader alternatives either use LLM judges without domain-specific scores or rely on existing data sources.The paper contrasts InfiBench with these approaches as a benchmark for free-form code-domain QA using automated, model-independent evaluation.Source: §5 · pp. 10–11

Conclusion

• InfiBench evaluates code-LLM question answering in real-world scenarios using 234 high-quality Stack Overflow questions and automatic model-free evaluation.The authors report evaluating more than 100 code LLMs and state that the benchmark is publicly available and continuously expanding.Source: §6 · p. 11

Reproducibility

• The authors report that code, data, and reproduction instructions are publicly available, and that the work does not involve model training.They say inference hyperparameters are listed in Section 3 and ablation studies are provided in Appendix G.Source: Checklist · p. 14

Experimental reporting

• Experiments are repeated three times when budget and computing resources permit, with error bars and standard deviations reported.Source: Checklist · p. 14

Ethics and limitations

• The checklist says the Stack Overflow assets are released under CC BY-SA 4.0, experts remove personal or offensive information by paraphrasing, and limitations and societal impacts are discussed.The authors point to Section 2 and Appendix B for limitations and societal-impact discussion.Source: Checklist · pp. 14–15

• Appendix A covers the dataset card and data accessibility details.Source: Appendices · p. 16
• Appendices B–D address limitations and societal impact, agreement statistics, and question difficulty grouping.Source: Appendices · p. 16
• Appendices E–I provide the full leaderboard, additional findings, sampling-hyperparameter analysis, prompts, and benchmark examples.Source: Appendices · p. 16

• The seed set contains Stack Overflow questions with at least three positively voted answers and an accepted answer, filtered by popularity, recency, language, and topic.Experts sampled from the seed set and each handled one area; the annotation process included question selection and type annotation, prompt paraphrasing, and correctness-criterion annotation.Source: A Dataset Card and Accessibility Details · p. 16
• Experts paraphrased prompts and cross-validated them to remove personal and sensitive information.Source: A Dataset Card and Accessibility Details · p. 17
• InfiBench’s evaluation scores depend on specified hyperparameters, and the benchmark measures response correctness rather than broader qualities such as naturalness or fairness.The authors caution that differing evaluation conditions may mislead comparisons and that scores may be overinterpreted.Source: A Dataset Card and Accessibility Details · p. 17
• The benchmark questions, evaluation tool, inference harness, raw responses, and leaderboard are publicly available under CC BY-SA 4.0.The Croissant description records the CSV of prompts and evaluation standards, while the tool and leaderboard are available separately.Source: A Dataset Card and Accessibility Details · p. 17

• InfiBench scores focus on correctness and are not a comprehensive measure of code LLM usability.The authors note that expert-annotated criteria are subjective and recommend combining InfiBench with evaluations of other aspects, including trustworthiness and system-level measures.Source: B Limitations, Societal Impacts, and… · pp. 17–18
• Because the benchmark is public, future models may deliberately use its data to raise their scores, and the current release has no contamination-detection module.The authors suggest scores around human performance without further scaling gains could indicate contamination, while noting that this is only a potential signal.Source: B Limitations, Societal Impacts, and… · p. 18
• Expert labeling limits benchmark expansion, leaving relatively few questions for less popular languages such as Rust and Ruby.Alternative metrics either risk model bias and high computing cost or diverge from domain experts’ correctness judgments.Source: B Limitations, Societal Impacts, and… · p. 18

C Agreement Statistics between InfiBench/GPT-4o Evaluation and Human

• 85.1% for InfiBench and 77.8% for GPT-4o were the respective agreement rates with humans when both evaluators had clear preferences.The authors report that InfiBench aligned with human experts better than GPT-4o with >80% confidence.Source: C Agreement Statistics between… · p. 18
• InfiBench labeled two responses as “both good” in only 29.4% of cases where humans judged both responses good.The authors attribute possible over-differentiation to pattern matching and fixed post-processing.Source: C Agreement Statistics between… · p. 19

D Difficulty Grouping

• The five difficulty groups are defined by GPT-3.5-turbo and GPT-4 score thresholds, sampling outcomes, and whether GPT-4 scores any question positively across 100 trials.Source: D Difficulty Grouping · p. 19
• Mean scores strictly decrease at higher difficulty levels, which the authors say supports the consistency and reasonableness of the grouping.Source: D Difficulty Grouping · p. 19

E Evaluation Details and Full Benchmark Results

• For open-source models over 30B parameters, evaluation used a 4,096-token context limit and one run; the limit was estimated to reduce scores by at most 0.37%.Models within 30B parameters were evaluated three times, with generation capped at min{1024, context length - prompt length}.Source: E Evaluation Details and Full… · p. 19
• 65.18% was the highest score among the three Stack Overflow answer settings, using the best-scoring of up to 10 highest-voted posts.This setting was deemed most comparable with model best@10 scores; among single posts, the highest-voted answer outscored the officially accepted answer.Source: E Evaluation Details and Full… · p. 19
• 37.82% was the mean overall score across 106 code LLMs, below human answers, which scored over 50% even with one attempt.Scores declined as difficulty increased, were relatively equivalent across problem types, and were weaker under blank-filling and dialogue-similarity metrics than under keyword-matching and unit-testing metrics.Source: E Evaluation Details and Full… · p. 20

F Additional Findings and Discussion

• The appendix introduces additional findings and discussion beyond Section 3.Source: F Additional Findings and Discussion · p. 20

F.1 Correlations between InfiBench and HumanEval Scores

• Across 66 LLMs, InfiBench and HumanEval pass@1 scores correlate positively (r = 0.8058), while most models, including all highly scored ones, lie below y = x.Mixtral-8x7B-Instruct is an outlier performing relatively better on InfiBench, while several relatively small or older models perform better on HumanEval; the authors suspect HumanEval-like optimization may explain this pattern.Source: §F.1 · p. 20

F.3 Scaling of Large Open Source LLMs

• Large open-source models over 40B parameters showed no significant advantage over smaller models and proprietary models in the comparison.The authors offer two possible explanations: unresolved barriers or changed scaling behavior beyond 40B, or the lack of released large models deliberately trained in the code domain.Source: Table 9 · p. 23, §F.3 · p. 23

F.2 Comparison of GPT-4o and GPT-4

• GPT-4, GPT-4-turbo, and GPT-4o scored 70.64%, 68.42%, and 66.19%, respectively; the authors treat the differences as small fluctuations among versions.Source: §F.2 · p. 23

F.4 Over-Safeguarding in CodeLlama-70B

• CodeLlama-70B improved on HumanEval pass@1 but deteriorated on InfiBench; CodeLlama-70B-Instruct showed refusal-pattern responses on 58 of 234 safe questions.The cited examples show the model usually refusing where GPT-4 provides answers.Source: §F.4 · p. 23, Table 10 · p. 24

G Study of Sampling Hyperparameters

• At T = 0.2, GPT-4-0613 scored 70.64% with best@10, 74.61% with best@30, and 79.75% with best@100.Three repeats were usually sufficient, top-p settings had little impact, and temperature around 1.0 produced the highest best@10 score; generalization beyond GPT-4 remains unvalidated.Source: G Study of Sampling Hyperparameters · p. 25

H.1 System Prompts

• The system prompt frames the model as a professional assistant for programmers and asks for brief Markdown answers, with a succinctness instruction for open-ended questions.Source: §H.1 · pp. 25–26

H.2 Prompt Templates by Models

• Base models use a shared system-prompt and content-prompt template, while finetuned models use family-specific templates; evaluation uses no few-shot examples.Source: §H.2 · p. 26, Table 12 · p. 26

I Examples

• The examples illustrate multiple grading criteria, including keyword matching, unit tests, and blank filling; extra text beyond a requested first paragraph reduced one example score to 0.2.The first paragraph alone scored 1.0 in that example.Source: I Examples · pp. 28–30