CRUXEval: A Benchmark for Code Reasoning, Understanding and Execution

• CRUXEval fills a benchmark gap by measuring code reasoning, understanding, and execution through input prediction and output prediction on short Python programs.Its samples are at most 13 lines, avoid complex arithmetic, and are intended to be solvable by a university-level CS graduate without extra memory.Source: §1 · p. 2
• The benchmark is generated by having Code Llama 34B produce functions and inputs, executing them, filtering for simple low-resource problems, and randomly selecting 800 samples.This generate-and-filter recipe is designed to expose failures on reasonable programs and support harder future benchmarks as models improve.Source: §1 · p. 2
• GPT-4 reaches 67% pass@1 on CRUXEval-I and 63% on CRUXEval-O, versus 47% and 44% for the best open-source models.The open-source models fail more than half the time despite training on 100G of Python code and 1T of code data.Source: §1 · p. 3
• For base models, stronger HumanEval performance correlates with stronger CRUXEval performance, but this relationship breaks for GPT-4-distilled models such as WizardCoder, Phind, and Phi.These models achieve impressively high HumanEval scores but perform no better than CodeLlama on CRUXEval.Source: §1 · p. 3
• Chain-of-thought prompting and fine-tuning on input-output assertions improve CRUXEval performance, yet remain insufficient to solve the benchmark, and GPT-4 still consistently fails on some surprisingly simple programs.The findings indicate that existing HumanEval and MBPP evaluations do not adequately measure code understanding and execution.Source: §1 · pp. 2–3

LMs for Code Generation

• Code LMs include base models such as Codex, CodeGeeX, SantaCoder, PolyCoder, InCoder, CodeGen, StarCoder, DeepSeek-Coder, and Code Llama, alongside instruction-tuned models including Phind, WizardCoder, and Phi-1/Phi-1.5.The paper evaluates a selection of these models on CRUXEval.Source: §2 · p. 3

Benchmarks for Evaluating Code LMs

• Existing code benchmarks evaluate simple Python generation, stronger programming problems, multilingual extensions, data-science applications, APIs, and broader software-engineering tasks.Examples include HumanEval, MBPP, HumanEval+, ReCode, HumanEval-X, MultiPLe, MBXP, APPS, CodeContests, LeetCode-Hard, DS-1000, ARCADE, NumpyEval, PandasEval, JuICe, APIBench, RepoBench, ODEX, and SWE-Bench.Source: §2 · p. 3
• Other benchmarks target code translation, test-case generation, code search, type prediction, summarization, security, program repair, and performance optimization.These works broaden evaluation beyond code generation and software-engineering applications to additional code-related tasks.Source: §2 · p. 4

Benchmark novelty

• CRUXEval is presented as the first publicly available benchmark measuring code LMs’ execution ability and, through CRUXEval-I, input prediction ability.CRUXEval-O enables a more thorough investigation of output prediction capabilities.Source: §2 · p. 4

Leveraging Test Cases and Code Execution

• Prior execution-focused work trains models on execution information, uses chain-of-thought or fine-tuning, and applies execution-based selection, reinforcement-learning feedback, or self-repair.Austin et al. (2021), Scratchpad (Nye et al., 2021), and CodeExecutor (Liu et al., 2023a) are especially relevant; CRUXEval is proposed as a reference point for future techniques.Source: §2 · p. 4

Failure modes of LM Reasoning

• Research on language-model reasoning documents failures involving negation, irrelevant context, counterfactual rules, identifier swaps, and theoretically constrained reasoning abilities.CRUXEval empirically examines failures in code execution and reasoning within this broader literature.Source: §2 · p. 4

3 Benchmark Construction

• 800 functions define two execution-based tasks: predicting an output for a given input or finding any input that produces a specified output.Input prediction passes when assert f(generated input) == output, while output prediction passes when assert f(input) == generated output.Source: §3 · p. 5

3.1 Generating Candidates

• 102,000 candidate functions and 489,306 input-output pairs are generated from 69 Python standard-library functions spanning str, dict, and list operations.Code Llama 34B generates candidate functions and five test inputs per function using varying few-shot examples.Source: §3.1 · p. 5

3.2 Filtering Candidates

• Filtering retains human-solvable samples by enforcing valid, compact code, bounded runtimes and arguments, no exceptions, no imports, determinism, and no side effects.Compilation filters require 75–300-character functions using all arguments and valid assertions; runtime filters exclude floating-point operations and enforce a two-second limit.Source: §3.2 · p. 5

3.3 Data size and measuring noise

• 800 examples suffice to preserve several expected model orderings, including Code Llama 34B > Code Llama 13B and Deepseek 33B > Code Llama 34B on output prediction.Bootstrap samples from approximately 1,700 examples estimate the probability that performance orderings reverse.Source: §3.3 · p. 6
• At N = 800, sampling noise is about 1.5% per model versus 0.2% from candidate sampling, while model-pair comparisons often reach statistical significance at α = 0.05.Sampling which data points enter the benchmark dominates candidate-sampling noise, and dataset size is the main factor affecting noise.Source: §3.3 · p. 6, Figure 1 · p. 6

Models and Reproducibility

• The evaluation covers StarCoder, Mistral, WizardCoder, Phi, Phind, Code Llama, DeepSeek Coder, GPT-3.5, and GPT-4 models.Non-GPT model checkpoints are provided in Appendix B, and all prompts are provided in Appendix D.2 to support reproducibility.Source: §4 · pp. 6–7

Main Results

• Code Llama 34B outperforms WizardCoder 34B on input prediction but underperforms DeepSeek Coder 33B on output prediction with over 97.5% paired-bootstrap probability.These comparisons are based on the bootstrap analysis against Code Llama 34B.Source: §4 · p. 7, Figure 2 · p. 7

Evaluation Protocol

• Results report pass@1 at T = 0.2 and pass@5 at T = 0.8, using N = 100 samples for non-GPT models and N = 10 for GPT models.Raw scores are provided in Appendix Table 2, while the main results appear in Figure 2.Source: §4 · p. 7, Figure 2 · p. 7

Statistical Analysis

• Figure 2 summarizes model performance with percentile boxes, 2.5–97.5 intervals, and a median bar approximately equal to the mean.Intervals are generated from 10,000 bootstrap samples; non-overlapping whiskers indicate significance at the 2.5% level.Source: Figure 2 · p. 7, §4 · p. 7

5 Quantitative Analysis

• HumanEval gains from distilled models do not transfer comparably to CRUXEval: WizardCoder, Phind, and Phi outperform base models on HumanEval, but no distilled model significantly beats Code Llama 34B on CRUXEval.By contrast, StarCoder, Mistral, CodeLlama, and DeepSeek-Base show a positive HumanEval–CRUXEval trend.Source: §5 · p. 8, Figure 3 · p. 8
• Input and output prediction performance is strongly correlated, suggesting that code-reasoning capabilities generalize across these distinct tasks and increase with model scale.With the potential exception of GPT models, CRUXEval-I and CRUXEval-O are especially correlated.Source: §5 · pp. 8–9, Figure 4 · p. 9
• Model prediction errors are strongly correlated by model family and size, while weaker models sometimes succeed on examples where stronger models fail.Open models cluster strongly, same-model sizes and same-size models correlate, and instruct/base pairs such as Phind or Wizard 34B with Code Llama 34B are closely associated.Source: §5 · pp. 9–10, Figure 5 · p. 10

5.1 Chain of Thought Prompting

• CoT generally produces larger output-prediction gains than input-prediction gains, but it widens the pass@5–pass@1 gap, likely because it increases generated diversity.CoT does not improve Code Llama 13B or GPT-3.5 on input prediction, and its gains vary across individual samples.Source: §5.1 · pp. 10–12, Figure 6 · p. 11
• 74.8% input and 81.9% output pass@1 are GPT-4’s highest scores with CoT, yet they remain far from acing CRUXEval.CoT helps Code Llama 34B and GPT-4 on both tasks, GPT-3.5 only on output prediction, and Code Llama 13B on neither.Source: §5.1 · p. 10

5.2 Fine-tuning Experiments

• Under 70% accuracy is the fine-tuning plateau for Code Llama 34B on both tasks, despite matching GPT-4 after training on benchmark-similar samples.The setup used nearly 140K distilled Python-function samples with weak decontamination and a 50/50 mixture of benchmark-excluded and input-output-disjoint functions.Source: §5.2 · pp. 12–13

Overview

• All models except GPT4 had failure rates above 50%, indicating that they struggled with simple executions; the analysis therefore focuses on GPT4’s remaining failures.The authors characterize GPT4’s failures as model-consistent and occurring on simple programs.Source: §6 · p. 13

Failures of GPT-4 CoT

• GPT-4 with CoT scored 0/10 on 54 output-prediction and 65 input-prediction tasks, including 22 problems where it failed on both tasks; most of these problems were manually judged simple.Two cases required counting to around 30, and two required simulating a few steps, which the authors considered within CoT’s scope.Source: §6 · p. 13
• GPT-4 with CoT consistently failed some output-prediction tasks with only two possible answers, including a mistaken comparison involving 6173; scratchpad-style prompting failed similarly.Small perturbations to the numerical condition and strings also produced failures, although other models sometimes answered correctly.Source: §6 · p. 14

Failures of GPT-4, Input Prediction

• GPT-4 with and without CoT also failed concise input-prediction programs involving character translation and case alternation.Examples include predicting “ow” instead of “lwwer case” and “LAK” instead of “lalak.”Source: §6 · p. 14

Other GPT-4 Failures

• The authors identify six additional relatively simple string-manipulation tasks that GPT-4 fails on and suspect tokenization contributes to these errors.The tasks involve string construction, rfind, slicing, joining, substring extraction, and replacement.Source: §6 · p. 15

Correlations between various code tasks

• Output prediction can use an interpreter and input prediction can benefit from one, but the authors argue that strong code models should still understand and execute code well.Base-model scores correlate reasonably across HumanEval, input prediction, and output prediction, motivating deeper study across code tasks.Source: §7 · p. 15

Distilling future execution benchmarks

• CRUXEval measures relatively simple, self-contained Python functions distilled from Code Llama 34B, motivating evaluation on longer, harder, open-domain, and multilingual code.The general distillation technique can support benchmarks from other code distributions.Source: §7 · p. 15

Variation due to prompt and temperature

• Prompt, task format, and temperature can substantially affect accuracy, so benchmark comparisons require caution; Phind v2’s 47.9% versus CodeLlama’s 46.5% is inconclusive given roughly 1.5% standard deviations.The experiments used T = 0.2 and T = 0.8 because of budget constraints, despite other temperatures potentially changing performance.Source: §7 · p. 15

Information loss due to pass@1

• Pass@k compresses substantial behavioral information into one number, so the authors recommend reporting pass@1 and pass@5 while developing complementary evaluation and analysis methods.The paper adds behavioral analysis but treats pass@k as only one perspective on execution ability.Source: §7 · p. 15

Fine-tuning

• Exact-string decontamination may leave semantically duplicated fine-tuning examples, while process supervision, correct CoT training, and program-state supervision offer more sophisticated alternatives.The authors also question whether execution-focused fine-tuning can improve code generation and whether distilled data from stronger models transfers effectively.Source: §7 · p. 16

Jointly improving code generation and code execution

• Code-generation fine-tuning in Phi, Phind, and WizardCoder does not significantly improve CRUXEval over their base models, leaving joint improvement of code generation and execution unresolved.The authors specifically propose studying whether better execution fine-tuning improves generation and how to improve both abilities simultaneously.Source: §7 · p. 16

Understanding reasoning from the lens of code

• Output prediction provides a tractable lens for pinpointing CoT failures because each reasoning step maps to a ground-truth operation, enabling systematic study of simple mistakes.Future evaluations could test recursive functions, natural-language-described execution, and compositions of two functions.Source: §7 · p. 16

Self-repair

• Because many CoT output-prediction failures are recitation errors involving information models may already understand, CRUXEval could help study self-repair capabilities.The authors expect such errors may be easier to repair than failures where the correct reasoning path is never found.Source: §7 · p. 16

8 Conclusion

• CRUXEval evaluates code LMs’ input and output prediction abilities while contributing a construction recipe, model analysis, improvement studies, and qualitative GPT-4 failure analysis.The recipe combines large-scale distillation, filtering, and statistical noise-based data-size selection; the benchmark is intended to complement HumanEval and MBPP.Source: §8 · pp. 16–17

Appendix A Benchmark Construction and Statistics

• Random-looking inputs in few-shot prompts substantially increased benchmark difficulty compared with common words and phrases.The sample prompt includes two examples: one using a string function and one using a list function, followed by generated functions and five test inputs.Source: Appendix A · p. 25

A.1 Functions used in prompt

• The benchmark’s prompt-function pool covers all non-dunder methods for Python str, list, and dict classes.The listed pools contain string, list, and dictionary methods, including operations such as center, append, and update.Source: §A.1 · p. 26
• Operator methods were excluded because most were either too simple and uninteresting or too computational for the final benchmark.This decision was motivated in part by a GPT-4 failure involving confusion between exponentiation and xor.Source: §A.1 · p. 26

A.2 Few-shot Examples

• The benchmark uses 10 handwritten few-shot examples—five string and five list examples—paired into 25 prompt combinations, with generation performed by Code Llama 34B at T = 1.Each prompt contains one string example and one list example.Source: §A.2 · p. 26
• After about 60000 generations from one fixed example pair, only about 5000 functions were unique, while all 25 prompt combinations helped overcome this duplication bottleneck.The observation indicates limited function diversity for a fixed pair of few-shot examples.Source: §A.2 · p. 26
• The full few-shot-example collection is provided in Listing 13.The examples constitute the handwritten prompt set used in the benchmark construction process.Source: §A.2 · p. 26

• The 800 benchmark samples span distributions of character counts and line counts.Source: §A.3 · p. 26, Figure 9 · p. 27
• Execution step counts are reported as approximate Python bytecode operations, excluding one 3175-step outlier.Source: Figure 10 · p. 27, §A.3 · p. 27
• Input-prediction and output-prediction pass@1 scores show little to no sample-level correlation.Source: §A.3 · p. 27, Figure 11 · p. 28
• Method frequencies are highly non-uniform despite equal str and roughly doubled list/dict distillation targets, partly because Code Llama 34B omits requested methods about 30% of the time.Other causes include method-specific filtering of removed operations and common methods appearing in functions that did not prompt them.Source: Figure 12 · p. 28, §A.3 · p. 28
• Methods such as str.rsplit, str.maketrans, and str.rfind are among the hardest, possibly because obscure functions are underrepresented in pretraining and distilled datasets.The authors hypothesize that distilling more obscure methods could improve models’ syntax and semantics learning.Source: §A.3 · pp. 28–29, Figure 13 · p. 29, Figure 14 · p. 29

Appendix B Model URLs

• The evaluation used gpt-3.5-turbo and gpt-4 on October 26, 2023, before OpenAI’s developer-day release of GPT-4-Turbo.The corresponding HuggingFace URLs for the other evaluated models are listed in Table 1.Source: Appendix B · p. 30
• Table 1 lists the evaluated models and their HuggingFace URLs.Source: Table 1 · p. 30

C.1 Main Results

• Table 2 reports pass@1 and pass@5 results for all models evaluated on CRUXEval.Source: Table 2 · p. 31, §C.1 · p. 31
• Figure 16 visualizes the main CRUXEval results in box-plot form with confidence intervals compared to Code Llama 34B.Source: Figure 16 · p. 32

• CoT improves most samples little, decreases pass@1 for many non-GPT-4 samples, and is much more effective for GPT-4 output than input prediction or other models.The limited gains are partly explained by samples that already have high pass@1 scores.Source: §C.2 · p. 32, Figure 17 · p. 33
• A granular analysis again shows that CoT often decreases pass@1 and has a starkly different impact on GPT-4 than on other models.Source: §C.2 · p. 33, Figure 18 · p. 34
• GPT-4 can succeed without CoT but fail with it: CoT misreads a substring search and derives an incorrect input list despite identifying the relevant sorting condition.For output prediction, it says ‘77777’ is absent from ‘7747777722’; for input prediction, it returns [5, 4, 3, 2, 1] after correctly stating the list must remain unchanged by sorting and reversal.Source: §C.2 · p. 34
• Although harder examples tend to challenge weaker models when stronger models fail, weaker models also solve some such examples, revealing idiosyncratic failures even for GPT-4 with CoT.Source: §C.2 · p. 34, Figure 19 · p. 35

Diversity of generations across models

• Higher temperature sharply increases generation diversity, with input prediction generally more diverse than output prediction and Code Llama models more diverse than distilled models at matched temperatures.These patterns are measured by unique answers across samples, without regard to correctness.Source: §C.3 · p. 35, Figure 20 · p. 36

CoT increase the diversity of generations

• CoT increases generation diversity at both temperatures, while input prediction remains more diverse than output prediction.Distinct chains of thought producing the same prediction are counted as identical.Source: §C.3 · p. 36, Figure 21 · p. 36

Functional diversity via distribution of pass@1 scores

• At T = 0.2, functional diversity is generally low because models tend to produce mostly correct or mostly incorrect outputs rather than intermediate pass@1 scores.Functional diversity is defined as the percentage of samples with pass@1 in (0.1, 0.9), exclusive.Source: §C.3 · p. 36, Figure 22 · p. 37
• Distilled models have lower functional diversity than base models, whereas CoT greatly increases functional diversity, especially for GPT-3.5 and GPT-4.The base-versus-distilled comparison includes Phind 34B versus CodeLlama 34B and DeepSeek Instruct 33B versus DeepSeek Base 33B.Source: §C.3 · p. 37

C.4 Difficulty of Benchmark Samples

• Fully solved and unsolved rates provide complementary views of models’ ability to solve problems and are related to diversity: higher diversity can increase the likelihood of solving a problem.The fully solved metric is analyzed at T = 0.2 for greater consistency, whereas the unsolved metric is analyzed at T = 0.8 for greater diversity.Source: §C.4 · pp. 37–38
• Output prediction has a more bimodal pass@1 distribution than input prediction, while CoT increases samples with intermediate scores between 0.25 and 0.75.These distributions are reported for selected strong models at T = 0.8.Source: §C.4 · p. 37, Figure 24 · p. 38
• Among non-CoT models, GPT-3.5 and GPT-4 have the best pass@1 but are considerably worse on the percentage of unsolved samples than Code Llama 34B and DeepSeek Base 33B.The unsolved-sample metric is measured at T = 0.8, where greater diversity is expected to improve the chance of solving a problem.Source: Figure 25 · p. 38, §C.4 · p. 38
• CoT reduces the percentage of unsolved samples for GPT models but increases it for the two Code Llama models, plausibly because CoT increases GPT diversity.This comparison uses samples with pass@1 equal to 0 at T = 0.8.Source: Figure 25 · p. 38, §C.4 · p. 38
• Instruction-tuned and distilled models have more unsolved samples than their base counterparts, suggesting instruction-tuning adherence may stifle diversity.The observation concerns the pass@1 = 0 metric at T = 0.8.Source: Figure 25 · p. 38, §C.4 · p. 38
• GPT-4 excels on fully solved samples, exceeding 60% for both input and output prediction at T = 0.2.Instruction-tuned models also outperform their base counterparts on this consistency-oriented metric, and GPT-4 + CoT preserves output-prediction consistency despite increasing diversity.Source: Figure 26 · p. 39, §C.4 · p. 39

C.5 Impact of Anonymizing Functions

• CodeLlama 7B, 13B, and 34B are evaluated on an anonymized benchmark subset, replacing variable names with x1, x2, and similar identifiers.The same non-anonymized few-shot prompt is used, with pass@1 at T = 0.2 and pass@5 at T = 0.8 reported for N = 10 samples.Source: §C.5 · p. 39
• The anonymization experiment is intended to clarify the effect of variable names on execution ability and strengthen the case against memorization effects.Results are presented in Table 4, but the supplied passage does not provide its numerical values.Source: §C.5 · p. 39, Table 4 · p. 40

C.6 Impact of Data-Generating Model

• Across datasets, the relative ordering of Code Llama 13B, Code Llama 34B, and GPT-3.5 is preserved across data-generating models, but GPT-3.5-generated data produce a significantly easier benchmark.The authors attribute the easier benchmark to more predictable, guessable inputs and find no significant advantage from evaluating models on their own generated data.Source: §C.6 · p. 40, Table 5 · p. 40

C.7 Fine-tuning

• Fine-tuning improves both prediction tasks, but test accuracy plateaus quickly despite increasing training accuracy, limiting simple fine-tuning’s ability to approach perfect CRUXEval performance.Output prediction is easier to overfit than input prediction, which the authors partly attribute to assertion format and the greater reasoning demands of input prediction.Source: §C.7 · p. 41, Figure 27 · p. 41
• Matching fine-tuning and evaluation assertion formats is crucial: misalignment prevents easy training-set overfitting and lowers testing accuracy by almost 10% for input prediction, although it still improves over the base model.The comparison uses assert output == f(input) and assert f(input) == output formats.Source: §C.7 · pp. 41–42, Figure 28 · p. 42
• Even fine-tuning on benchmark programs with different input-output pairs cannot raise test accuracy beyond 70%, suggesting that these execution tasks are difficult to learn with this scheme.The setup mixes 50% of the original fine-tuning data with 50% of 7259 benchmark-derived samples.Source: §C.7 · p. 42, Figure 29 · p. 42

D.1 Benchmark Generation Few-Shot Prompts

• Benchmark generation uses few-shot task examples spanning string and list operations, with Python functions paired with test assertions whose outputs must be completed.The examples cover operations including str.split, str.capitalize, str.rpartition, str.center, list.pop, list.insert, and list.remove.Source: §D.1 · pp. 43–45

D.2 Direct Prediction Prompts

• Evaluation prompts use few-shot examples for non-GPT models because they improve parseable formatting, whereas GPT models use zero-shot prompts after few-shot prompting reduced performance on both tasks.Phind uses a separate output-prediction prompt because the standard prompt often elicited explanatory text before the assertion.Source: §D.2 · p. 46

D.3 Chain of Thought Prompts

• Chain-of-thought prompts instruct models to reason step by step before returning a tagged answer, using one-shot prompts for Code Llama and zero-shot prompts for GPT models.The prompts separately support input prediction and output prediction while requiring constrained answer formatting.Source: §D.3 · pp. 48–49

Appendix E Qualitative Analysis

• GPT-4’s qualitative behavior is prompt-sensitive: slight prompt changes can turn correct examples into failures or incorrect examples into successes.Some examples may come from a larger generated set rather than the benchmark.Source: Appendix E · p. 50

E.1.1 GPT-4 Successes without CoT, Output Prediction

• GPT-4 achieves impressively high pass@1 on output prediction without CoT, with examples demonstrating complex reasoning and code-execution capability.Its successes include correctly handling string partitioning, character counting, frequency ordering, and sorting-based transformations.Source: §E.1.1 · pp. 50–51

E.1.2 GPT-4 Failures without CoT, Output Prediction

• Without CoT, GPT-4 still exhibits relatively simple output-prediction failures: it outputs "-2, 1, 3, 9, -1" in one case and "Not found" instead of asserted "5" in another.Source: §E.1.2 · p. 51

E.2.1 GPT-4 Successes without CoT, Input Prediction

• The cited input-prediction examples include a list transformation and deleting one character at index length // 2, with GPT-3.5 and CodeLlama 34B outputs shown for comparison.Source: §E.2 · p. 52, §E.2.1 · p. 52

E.2.2 GPT-4 Failures without CoT

• The cited examples record five GPT-4 outputs: calls involving list-like inputs, a string containing [NEGMENDS], a numeric output, and f(5, "Hello").Source: §E.2.2 · pp. 53–54

E.3.1 GPT-4 Successes with CoT, Output Prediction

• With CoT, GPT-4 successfully simulates execution across control flow, uppercase counting, string cleaning, dictionary sorting, and loop behavior.The examples include correctly computing 8 from four uppercase letters, removing selected characters from a string, sorting skills by widths, and reducing a list to [].Source: §E.3.1 · pp. 54–55

E.3.2 GPT-4 Failures with CoT, Output Prediction

• CoT can make GPT-4 contradict knowledge it demonstrates independently, such as claiming 6173 is less than 1000 while separately judging that comparison false.The passage identifies paired mistakes in which the chain of thought fails on information GPT-4 appears to know when prompted independently.Source: §E.3.2 · p. 57
• GPT-4 with CoT also misinterprets Python string operations and sorting, with reported failures involving str.islower, sorted("h e l l o"), str.strip, and list indexing.The sorting error is suggested as a possible tokenization artifact, while the str.strip example incorrectly removes characters from the wrong positions.Source: §E.3.2 · pp. 57–59
• GPT-4’s output-prediction CoT miscomputed negative indexing and string slicing, producing incorrect answers for otherwise simple Python operations.It reported -2 for index -3 in [0, 6, 2, -1, -2, 6, 6] and incorrectly concluded that text[-i:] was ’5’.Source: §E.3.2 · pp. 59–60

E.4.1 GPT-4 Successes with CoT, Output Prediction

• GPT-4 successfully reasoned through non-trivial output-prediction programs, including padding, sorting, hexadecimal conversion, list manipulation, and error-triggering inputs.Examples include recovering f(’7’, 11), an input yielding ’ DEGHIVjkptx’, the input ’nw Rome!’, and an even-length list causing an empty output; the hexadecimal example may reflect memorization or contamination.Source: §E.4.1 · pp. 60–61

E.4.2 GPT-4 Failures with CoT, Input Prediction

• GPT-4’s input-prediction failures often skipped reasoning and incorrectly validated answers against assertions, suggesting a prior that its candidate solution was correct.The examples report direct wrong answers without much step-by-step reasoning and emphasize that assertion checking itself was frequently incorrect.Source: §E.4.2 · p. 63
• Misleading variable names further challenged GPT-4: it misunderstood str.removeprefix semantics and inferred an input by treating the suffix as the relevant remainder.The passage identifies the mismatch between naming and semantics as the source of difficulty and gives ’hellohymi’ as an example input.Source: §E.4.2 · p. 64
• GPT-4’s input-prediction failures include incorrectly checking forward execution when reconstructing function inputs.One example shows GPT-4 deriving an input of ’w l ’ for the asserted output ’wlace Alc l ’.Source: §E.4.2 · pp. 64–65

E.4.3 GPT-4 Failures with CoT, Examples Requiring Insights

• Some input-prediction examples require insights beyond line-by-line execution, which GPT-4 misses across sorting, translation, string partitioning, branching, and joining operations.Observed errors include treating incompatible portions as sorted, translating only one character, assigning digits to the wrong substring, pursuing an unnecessary branch, and overlooking that a separator is absent from the output.Source: §E.4.3 · pp. 65–66

E.5 Simple and Isolated GPT-4 Failures

• GPT-4 also fails on isolated subtasks involving character counts, string search, slicing, joining, indexing, and replacement.The examples include incorrect reasoning about rfind, extended slicing, plus-separated joins, length-based indexing, and substring replacement.Source: §E.5 · pp. 66–68

E.6 Failing Gems

• The “failing gems” collection assembles examples GPT-4 often fails even with CoT to provide a lens on its input- and output-prediction behavior.Some examples overlap earlier cases, while others are new.Source: §E.6 · p. 68

E.6.1 Failing Gems, Output Prediction

• The output-prediction failing gems span loops, replacements, conditionals, date slicing, and recursive suffix handling, with asserted outputs such as ’816a’, ’3Leap and the net will appear’, and ’Not found’.The collection also includes a date transformation assertion yielding ’20090-08-1’.Source: §E.6.1 · pp. 68–69
• E.6.1 Failing Gems, Output Prediction: Failing examples include incorrect results for character counting, capitalization, filtering, and sorting strings.The assertions include outputs such as 15, “HeY !!”, [], and “ ehllo”.Source: §E.6.1 · p. 69

E.6.2 Failing Gems, Input Prediction

• E.6.2 Failing Gems, Input Prediction: Failing examples require recovering inputs for formatting, insertion, character-frequency, reordering, and list-transformation programs.The asserted cases include “*+++n+z+o+h”, a character-count dictionary, “Mmohamt”, and [2, 4, 4].Source: §E.6.2 · p. 70