Sparks of Artificial General Intelligence: Early experiments with GPT-4

1 Introduction

• GPT-4 demonstrates broad, often human-level capabilities across language, coding, mathematics, vision, medicine, law, and other domains, supporting its characterization as a significant step toward AGI.The tested model was an early, non-multimodal version queried through natural language, and its outputs were often indistinguishable from or better than human outputs; preliminary examples are shown in Figure 1.1 and contrasted with ChatGPT in Figure 1.2.Source: §1 · p. 4, Figure 1.1 · p. 5, Figure 1.2 · p. 6
• GPT-4 remains incomplete and non-human-like: it has limitations, biases, hallucinations, arithmetic errors, uncertain planning ability, and no continuous learning from experience.The authors describe its learning during development as slow and eventually halting, unlike the real-time learning expected from AGI, while noting progress on non-linguistic capabilities and common sense.Source: §1 · pp. 4, 6, §1.1 · p. 6

1.1 Our approach to studying GPT-4’s intelligence

• Because benchmark evaluation risks training-data contamination and struggles to measure open-ended, interactive abilities, the authors study GPT-4 through novel tasks designed to probe general understanding beyond memorization.Their psychology-inspired approach uses human creativity and curiosity to generate difficult questions, tests consistency, coherence, correctness, limitations, and biases, and is acknowledged as subjective and informal.Source: §1.1 · pp. 6–7
• GPT-4’s flexible understanding is illustrated by combining mathematical reasoning with poetic form and visual imagination with coding, producing outputs superior to ChatGPT and comparable to or better than human performance.Varying styles, modifying code, and asking GPT-4 to restore a removed unicorn horn further probe adaptation, code comprehension, and inference of visual features.Source: §1.1 · pp. 7–8, Figure 1.4 · p. 8

1.2 Organization of our demonstration

• The paper organizes its demonstrations around language, coding, mathematics, planning and learning, human understanding, limitations, societal impact, and future research, with preliminary scores near 80% on USMLE components and above 70% on the Multistate Bar Exam.On LeetCode’s three interview rounds, early GPT-4 scored 8.96/10, 8.69/10, and 10/10, beating 93%, 97%, and 100% of users, respectively.Source: §1.2 · pp. 8–9

2.1 Integrative ability

• GPT-4 combines art, programming, literature, mathematics, history, physics, and medicine in creative interdisciplinary outputs, often more complete and creative than ChatGPT.The authors caution that these examples do not provide a precise evaluation or rigorous model comparison.Source: §2.1 · p. 13

2.2 Vision

• GPT-4 can generate and manipulate visual content through text-produced code, including identifiable SVG objects, 2D scenes, and editable 3D landscapes, despite being non-multimodal.Its visual-task performance is presented as evidence of visual-concept understanding rather than simple copying from training data.Source: §2.2.1 · pp. 16–17, §2.2 · p. 16, §2.2.2 · p. 18

2.2.1 Image generation beyond memorization

• GPT-4 usually preserves both object and letter identities when creatively merging them, and can iteratively correct proportions, center a head, and add clothing in generated figures.These tasks require inventing how objects and alphabet letters should be combined before producing the image.Source: §2.2.1 · p. 17

2.2.2 Image generation following detailed instructions (`a la Dall-E)

• GPT-4 follows detailed 2D and 3D image-editing instructions, correctly adding, relocating, recoloring, and reordering objects in generated scenes.Examples include a frog-in-a-bank scene and a fantasy landscape with floating islands, waterfalls, bridges, dragons, and a castle.Source: §2.2.2 · p. 18

2.2.3 Possible application in sketch generation

• Combining GPT-4’s instruction-faithful sketches with Stable Diffusion produces images that better satisfy prompts while improving rendered quality over either model alone.The authors frame this combination as a promising tool-use direction.Source: §2.2.3 · p. 18

2.3 Music

• GPT-4 generates valid ABC tunes and can describe and modify melodic structure, but it fails to produce nontrivial harmony, consistent chord descriptions, or reliable recognition of well-known tunes.Across 10 generated tunes, the authors found no clear chords or arpeggios, and duet extensions lacked harmony between voices.Source: §2.3 · p. 19

3 Coding

• GPT-4 writes focused programs at roughly an average software engineer’s level, empowers skilled and non-skilled users, but still produces invalid or incorrect code and can miss instructions.It can improve code through human feedback and compiler or terminal errors, while limitations are more pronounced for longer or more complex programs; results come from an early version of GPT-4 and are illustrative [Ope23].Source: §3 · p. 21

3.1.1 Coding challenges

• GPT-4 significantly outperforms other models on fresh LeetCode problems and is comparable to human performance in pass@1 and pass@5 accuracy.The benchmark contains 100 problems posted after GPT-4’s pretraining period, reducing the likelihood that the tasks were memorized.Source: §3.1.1 · p. 21, Table 2 · p. 21

3.1.2 Real world scenarios

• Across real-world tasks, GPT-4 produces working visualizations, a zero-shot 3D game, a custom PyTorch optimizer, and compiling LaTeX from imprecise instructions, generally outperforming ChatGPT.GPT-4 correctly adapts plots, implements domain-specific operations, and follows vague requirements, although it misses some particularly unclear optimizer instructions.Source: §3.1.2 · pp. 23–26

3.2 Understanding existing code

• GPT-4 understands and reasons about existing code by reverse-engineering binaries, explaining compiler-dependent C output, simulating Python, and executing informal pseudocode.It can use debugging and reverse-engineering tools, track recursive execution states, and interpret vague instructions without translating pseudocode into a formal language.Source: §3.2 · pp. 27–29

4 Mathematical abilities

• GPT-4 successfully solves a novel multi-step population problem that ChatGPT repeatedly answers incorrectly with nonsensical reasoning.The problem required formulating an annual population-change expression, deriving a recurrence relation, and solving two equations; the question was authored to reduce memorization.Source: §4 · pp. 30–31

4.1 A mathematical conversation with GPT-4

• GPT-4 can reformulate mathematical questions and select sophisticated solution paths, including induction and polynomial-degree arguments, but its reasoning often fails under altered wording or fixed quantifiers.It produced a sound reformulation and a solid degree-based argument, yet repeatedly applied invalid induction reasoning when the modified problem prescribed specific coefficients.Source: §4.1 · p. 31, §4.1.1 · p. 32, §4.1.2 · p. 33

4.1.2 A second variant of the original question

• In the polynomial variant, GPT-4 first made calculation errors and missed that no solution exists, then correctly inferred impossibility from the composition degree.It also incorrectly claimed that exponential and logarithmic functions are closed under composition, later contradicting itself and producing incoherent arguments.Source: §4.1.2 · p. 33

4.1.3 Analysis of the limitations highlighted by conversation

• GPT-4’s mathematical knowledge is context-dependent: changes in question wording can alter the displayed knowledge, producing sophisticated answers in some settings but basic errors or incoherence in others.The section cautions that standard human-style examination performance alone cannot capture this uneven profile, which spans creative reasoning, technical proficiency, and critical reasoning.Source: §4 · p. 30, §4.1.2 · p. 34, §4.1.3 · p. 34
• GPT-4 exhibits strong creative reasoning but frequent technical mistakes and a marked deficiency in critically checking, revising, or backtracking over its own mathematical arguments.The authors attribute the latter limitation partly to linear solution-oriented training data and partly to the next-word-prediction paradigm, which lacks a mechanism for revising prior output.Source: §4.1.3 · pp. 34–35

4.2 Performance on mathematical problem datasets

• GPT-4 shows a modest accuracy improvement over other models on the evaluated math benchmarks, and manual inspection finds that it uses the correct approach on a significant majority of its MATH errors.Its remaining errors are largely arithmetic or calculation mistakes, whereas ChatGPT often produces incoherent arguments irrelevant to the solution; the reported numbers concern an early GPT-4 version.Source: §4.2 · pp. 35–36, Table 3 · p. 36, Table 4 · p. 37

4.3 Mathematical modeling in various domains

• GPT-4 builds plausible mathematical models for complex interdisciplinary systems, whereas ChatGPT fails to make meaningful progress.This demonstrates quantitative reasoning beyond strictly mathematical exercises, including profiling a professional StarCraft 2 player’s power rate over time.Source: §4.3 · p. 38, Figure 4.4 · p. 38
• GPT-4 approaches Fermi questions by layering reasonable assumptions and informed guesses, unlike ChatGPT, although its final estimates may be far off.The examples show quantitative thinking and general knowledge applied to quantities that are difficult or impossible to measure directly.Source: §4.3 · p. 38, Figure 4.5 · p. 39

4.4 Higher-level mathematics

• The cited example concludes that g(x) = x2 for all x > 0, while advanced mathematical examples showcase model capabilities but do not always succeed at this difficulty.Source: §4.4 · pp. 39–40
• GPT-4 understands abstract graph constructions and SAT reasoning with notable creativity, yet makes a human-like formula typo and a counting error that produces an incorrect probability answer.The graph-theory discussion later generalizes the formula correctly, while the number-theory and probability example contains a final counting mistake.Source: §4.4 · pp. 41–42

5.1.1 Using multiple tools to solve more complex tasks

• In a novel zoo-management scenario requiring more than 100 commands, GPT-4 solved almost all tasks, corrected command errors from system feedback, and needed only a prompt tweak to fix fabricated email content.It nevertheless repeated an established quoting error across files, suggesting that interaction enabled correction without eliminating persistent error patterns.Source: §5.1.1 · p. 45
• GPT-4 can combine search and summarization tools to answer questions accurately, including questions containing false premises, without fine-tuning or demonstrations.It identifies relevant results, examines them in greater depth, and summarizes them before answering.Source: §5.1.1 · pp. 45, 49, Figure 5.4 · p. 46, Figure 5.5 · p. 47

5.1 Tool use

• GPT-4 can identify and use external tools, parse their outputs, and respond appropriately without specialized training or fine-tuning.It can use search, calculation, and character-extraction functions with minimal instruction, and can infer which APIs a task requires.Source: §5.1 · pp. 43–44, §5.1.2 · p. 49

5.1.2 Discussion

• Tool use remains prompt-dependent: GPT-4 may use tools unnecessarily, fail to adapt to unusual APIs, or repeat errors, although environmental or user feedback can help it correct them.Without permission or expectation to use tools, its weaknesses in current knowledge and symbolic manipulation remain limiting; ChatGPT was less consistent in using tools and their responses.Source: §5.1 · p. 43, §5.1.2 · p. 49, §5.1.1 · p. 49

5.2.1 Warmup: navigating a map

• Through a restricted text interface, GPT-4 accurately described the portion of a house it explored, despite not exploring the entire house.This warmup tests whether language can support interaction with an environment without direct embodiment.Source: §5.2 · p. 49, §5.2.1 · p. 49

5.2.2 Text-based games

• In TextWorld, GPT-4 completed a navigation game in 30 actions without looping, adapted to feedback in a harder cooking game, but solved that game only after a one-shot demonstration.It learned action requirements through trial and error, generalized between actions, and inferred missing steps, while text-davinci-003 ignored environment feedback.Source: §5.2.2 · p. 52

5.2.3 Real world problems

• With a human acting as an intermediary, GPT-4 identified actions for two real-world problems, guiding a water-leak repair and proposing plausible solutions to a gas-service problem without initially finding its root cause.In the leak case, it recommended the exact actions that resolved the problem; in the gas case, it reached the cause only after the human suggested checking the stovetop.Source: §5.2.3 · pp. 52–53

5.2.4 Discussion

• Language can serve as a surrogate interface for environmental interaction, but these demonstrations cannot establish general performance because GPT-4 was tested on only a limited set of games and real-world problems.A stronger evaluation would use a larger, more diverse set of real-time problems rather than retrospective examples.Source: §5 · p. 43, §5.2.4 · p. 53

6.1.1 Testing specific aspects of theory of mind

• GPT-4 correctly reasons about false beliefs, emotions, and intentions in abstract tests, while ChatGPT generally passes basic tests and text-davinci-003 fails or gives less informative responses.The tests use unfamiliar framing and a nonsense word to reduce the possibility of memorization.Source: §6.1.1 · p. 54

6.1.2 Testing theory of mind in realistic scenarios

• In realistic social scenarios, GPT-4 identifies characters’ mental states and root causes of misunderstandings, producing more nuanced interventions than ChatGPT and text-davinci-003.The comparison is qualitative and does not imply a single correct answer for every scenario.Source: §6.1.2 · p. 54

6.1.3 Discussion

• GPT-4 outperforms ChatGPT and text-davinci-003 on basic and realistic theory-of-mind tasks, especially when reasoning about multiple actors and action effects.It handles abstract, novel scenarios such as the modernized Sally-Anne test and ZURFIN scenario, and proposes actions addressing social misunderstandings.Source: §6.1.1 · p. 54, §6.1.2 · p. 54, §6.1.3 · p. 60

6.2 Talking to Humans: Explainability

• GPT-4 generates remarkably coherent output-consistent explanations, including for wrong or nonsensical outputs, but output consistency does not guarantee process consistency.Process consistency requires explanations to predict the model’s behavior under related inputs or contexts.Source: §6.2 · pp. 60, 62, Discussion · p. 67
• Process-consistent explanations can fail when the underlying process is variable, arbitrary, insufficiently explainable, or mismatched with the explanatory process’s constraints.The chapter also notes that the theory-of-mind tests omit sarcasm, irony, humor, deception, and non-verbal social cues.Source: §6.1.3 · p. 60, §6.2 · p. 67
• 100% vs 60% faithfulness: GPT-4 was more process-consistent than GPT-3 for counterfactual rewrite explanations in a separate sentiment-analysis experiment.Process consistency is most plausible when GPT-4 simulates the task well and can faithfully approximate an explanatory process.Source: §6.2 · p. 67

7 Discriminative capabilities

• The section frames discrimination as making context-sensitive distinctions and argues that stronger generation does not automatically establish stronger discrimination.The experiments cover PII identification, misconception-prone questions, truthful-answer comparison, and judging which response better matches a reference.Source: §7 · p. 69

7.1 PII Detection

• GPT-4 matched the PII ground truth for 77.4% of sentences and missed one element about 13% of the time, outperforming the custom-built Presidio baseline.It captured context-dependent PII, such as inferring Denmark from the currency kroner, though the evaluation was not exhaustive and some errors involved equivalent strings counted separately by the ground truth.Source: §7.1 · pp. 69–70, Table 5 · p. 71

7.2.1 Why Are Current Metrics Insufficient?

• GPT-4 generated answers closer to TruthfulQA gold answers than GPT-3 by ROUGE, BLEU, and BLEURT, and was more truthful across most categories.Long, hedged answers could score poorly despite containing relevant information, exposing the metrics’ reliance on word- or sentence-level similarity rather than semantic equivalence.Source: §7.2.1 · pp. 71–73, Figure 7.3 · p. 72, Figure 7.4 · p. 72
• GPT-4 often produced more plausible answers and outperformed GPT-3 for people-and-places questions, but showed only slight gains on Confusion questions.Some Confusion errors suggest that parametric knowledge alone may be insufficient, motivating fact-checking with verified external sources such as Retrieval Augmented Generation.Source: §7.2.1 · p. 73

7.2.2 GPT-4 as a Judge

• Judge GPT-4 selected the GPT-4-generated answer 87.76% of the time, versus 11.01% for GPT-3 and 1.23% for neither.Its explanations used semantic and conceptual similarity regardless of answer length, but human overlap was only 50.8%, indicating that its reasoning did not necessarily mirror human judgments.Source: §7.2.2 · pp. 73, 75

8.1 Warm-up with two basic examples

• The autoregressive architecture lacks an inner dialogue or scratchpad for storing intermediate results, although prompting it to externalize steps can sometimes recover correct answers.It answers the prime-counting question correctly after listing the primes, whereas a single-pass response is wrong; the authors argue that the architecture cannot implement a for loop or human-like scratchpad reasoning.Source: §8.1 · pp. 76–77

8.2 Lack of planning in arithmetic/reasoning problems

• GPT-4 often fails at simple arithmetic and ahead-planning, with accuracy falling from 58% to 16%, 12%, and 0% as operand ranges increase, while step-by-step prompting raises accuracy to 100% or 90%.The authors argue that the deeper problem is sequential autoregression and inability to plan ahead, not calculation alone; in 100 constrained equation samples, accuracy remains low across ranges, from 32/100 to 18/100.Source: §8.2 · pp. 77–78

8.3 Lack of planning in text generation

• The model handles local text constraints well through greedy generation, but fails when a global constraint requires planning the ending while generating the beginning.It produced a coherent acrostic story satisfying the requested plot twist, but a poem requiring the final sentence to reverse the first became grammatically incorrect; proposed planning prompts also failed.Source: §8.3 · pp. 79–80

8 Limitations of autoregressive architecture highlighted by GPT-4

• Overall, next-word prediction is suited to incremental tasks but struggles with discontinuous problems requiring global understanding, creative reframing, working memory, or slow deliberate oversight.The paper interprets this gap as impressive fast thinking without a slow-thinking component that organizes thought and uses working memory, echoing Kahneman and LeCun.Source: §8 · p. 76, §8.3 · pp. 80–81

9 Societal influences

• GPT-4 may improve healthcare, education, engineering, and the arts and sciences, but poorly designed applications could introduce reliability failures and broader societal costs.The effects depend on use cases, application design, established practices, and efforts to mitigate downsides.Source: §9 · p. 82

9.1 Challenges of erroneous generations

• Hallucinations can be persuasive and difficult to detect, requiring intensive verification—especially in medicine, journalism, transportation, and attribution—because unrecognized errors may propagate downstream.Closed-domain consistency checks can help, while open-domain errors require broader research; reported procedures include labeling LLM-generated content and assigning human editors responsibility for fact-checking.Source: §9.1 · pp. 82–83

9.2 Misinformation and manipulation

• GPT-4 can enable efficient, scalable disinformation and personalized manipulation, while simple prompting lowers the expertise barrier; these risks motivate technical, policy, and regulatory safeguards.The examples came from an early pre-alignment model, whereas deployed GPT-4 reportedly refused or produced unlikely-to-harm responses in testing [Ope23].Source: §9.2 · pp. 83, 85, 86

9.3 Bias

• GPT-4’s pronoun choices reflected occupations’ gender representation, but adding “in an inclusive way” shifted choices to they/their, illustrating both bias and a possible mitigation strategy.The model also generated potentially biased analogies while explaining their offensiveness, and reflected on implicit stereotypes in the surgeon riddle, suggesting uses for bias-awareness assistants.Source: §9.3 · pp. 87–89

9.4 Human expertise, jobs, and economics

• GPT-4 may automate more expert tasks and disrupt or devalue occupations, while human-AI collaboration could instead extend human abilities, creativity, problem-solving, and decision-making.Its performance across professional and scholarly domains may reshape education, training, reskilling, and career choices [NKMM+23].Source: §9.4 · pp. 89–90

9.5 Constellation of influences and considerations

• Limited access to powerful LLMs could create an AI divide, amplifying inequalities across demographic groups, countries, and sectors unless investments and incentives expand opportunity while managing risks.The consequences may affect health, education, science, and other domains where advanced-model applications could be valuable.Source: §9.5 · pp. 90–91
• More general AI systems may require stronger confidentiality and privacy protections, including private model instances, because detailed interactions, inferential capabilities, memorization, and generalization could expose sensitive information.People and organizations may seek protection against logging or leakage of personal or organizational information and preferences.Source: §9.5 · p. 91
• The demonstrations may increase demand for provenance markers that distinguish human, machine, or mixed contributions to content and reasoning, helping mitigate confusion, deception, and harm.Widespread AI-generated information may also become training data despite questionable accuracy, reliability, and truthfulness.Source: §9.5 · p. 91
• People may seek the ability and right to decide which contributions are crawled for training and which receive provenance information describing contributors and data roles.This reflects concerns about controlling contributions to large-scale general AI systems and the downstream use of generated information.Source: §9.5 · p. 91

10 Directions and Conclusions

• GPT-4 exhibits sparks of artificial general intelligence through reasoning, creativity, deduction, broad expertise, and diverse task performance, although substantial work remains toward complete AGI.The exploration finds capabilities comparable to human-level performance across many tasks, while treating this assessment as preliminary and informal.Source: §10 · p. 92

10.1 Definitions of intelligence, AI, and AGI

• Existing definitions of intelligence and AGI remain incomplete or controversial, motivating more formal approaches that account for agency, priors, learning efficiency, and artificial systems’ distinct constraints.The authors do not adopt goal-oriented, human-equivalence, or other reviewed definitions, and note that agency and autonomous action raise alignment and safety concerns.Source: §10.1 · p. 92

10.2 On the path to more general artificial intelligence

• GPT-4’s path toward more general intelligence is limited by hallucinations, weak calibration, limited memory, absent continual learning and personalization, planning difficulties, inconsistent explanations, irrationality, and prompt sensitivity.These limitations can produce costly errors, outdated knowledge, poor adaptation, non-robust results, and reduced trust, especially in high-stakes settings.Source: §10.2 · p. 93
• The authors cannot distinguish which limitations arise from RLHF versus the underlying architecture, recommending comparisons across RL stages to clarify causes and potential remedies.They specifically identify hallucination and cognitive biases as problems whose sources may reflect either training artifacts or deeper capability constraints.Source: §10.2 · p. 94
• Potential extensions beyond next-word prediction include external tools, slow-thinking oversight, integrated long-term memory, and hierarchical generation of higher-level concepts.These mechanisms could support planning, verification, working memory, correction, and richer representations, but the authors question whether scaling the current paradigm alone will suffice.Source: §10.2 · p. 94

10.3 What is actually happening?

• The study is phenomenological and does not explain how GPT-4 reasons, plans, creates, or develops general intelligence; emergence and neural-circuit hypotheses remain difficult to verify at scale.The authors argue that understanding these mechanisms has become urgent, while existing evidence for emergence is largely limited to toy models.Source: §10.3 · pp. 94–95

A GPT-4 has common sense grounding

• The common-sense tests use puzzles requiring basic understanding of the world rather than specialized prompting.The authors frame such puzzles as one way to test whether an AI system possesses common-sense knowledge.Source: A GPT-4 has common sense grounding · p. 101

B Appendix for multimodal and interdisciplinary composition

• The passage presents examples intended to demonstrate GPT-4’s common-sense grounding, comparing it with ChatGPT and using puzzles requiring basic understanding of the world.Source: A GPT-4 has common sense grounding · p. 101

A GPT-4 has common sense grounding

• GPT-4 correctly reasons that the hunter’s bear is white, whereas ChatGPT says “I don’t know.”The solution depends on recognizing that the scenario occurs at the North Pole, where polar bears live.Source: A GPT-4 has common sense grounding · p. 101
• GPT-4’s performance may partly reflect exposure to familiar puzzles during training, motivating new puzzles with similar structure but different knowledge requirements.The equator puzzle is introduced specifically to challenge possible memorization of the well-known bear puzzle.Source: A GPT-4 has common sense grounding · p. 101
• GPT-4 solves a novel equator puzzle by identifying that the tiger could be a Bengal or Sumatran species native to the Equator, while ChatGPT gives up.The reasoning requires knowing the Equator’s 24,901-mile circumference, the relevant travel geometry, and tiger habitats.Source: A GPT-4 has common sense grounding · p. 101
• These examples indicate that GPT-4 has superior common-sense grounding and reasoning abilities compared with ChatGPT.The paper presents additional examples involving physical events, geography, employment history, and river-crossing planning.Source: A GPT-4 has common sense grounding · pp. 102–104

B.1 Further details on integrative ability results

• GPT-4 generated code for diverse integrative tasks, including Kandinsky-style image generation, BMI calculation, and diabetes-risk assessment.The examples span JavaScript and Python outputs and combine programming with artistic or medical objectives.Source: Figure B.1 · p. 105, Figure B.2 · p. 106, Figure B.3 · p. 107

B.2 Further details on vision results

• GPT-4 was prompted to create SVG and 3D visualizations from natural-language descriptions without external images or textures.Examples included a frog-themed bank scene, a floating-island fantasy landscape, and other shape-based graphics.Source: §B.1 · p. 108, §B.2 · p. 108, Figure B.4 · p. 108
• Additional vision prompts asked GPT-4 to compose people and objects from alphabet letters and generate corresponding SVG illustrations.The examples included a letter-based stick figure, dressed variants, and alphabet-object combinations, alongside SVGs for automobile, truck, cat, and dog classes.Source: §2.2.1 · p. 17, §B.2 · p. 109, Figure B.6 · p. 109

B.3 Graphic novel design example

• For a graphic novel aimed at little girls, GPT-4 proposed five one-page story concepts, including a girl whose magic crayon brings a unicorn to life.The concepts also covered detective, fear-of-darkness, baking, and girls’ soccer stories.Source: §B.3 · p. 110
• After selecting the magic-crayon idea, GPT-4 outlined a workflow covering panel layout, dialogue, drawing, coloring, and lettering.The proposed process included concrete panel structure, sample dialogue and sound effects, character details, color choices, and typography.Source: §B.3 · p. 110
• GPT-4 was then asked to produce SVG code for the one-page graphic novel and supplied a possible SVG example.The passage identifies the output as SVG code but does not provide its full contents.Source: §B.3 · p. 110

C.1 Measuring human performance on LeetCode

• LeetCode acceptance rate may be an improper benchmark because it aggregates historical submissions and may include copied solutions released after questions appeared.The paper notes that Hard questions can have higher acceptance rates than Medium questions, motivating the contest-based alternative.Source: §C.1 · p. 111
• Human LeetCode performance was estimated separately for Easy, Medium, and Hard problems from contest statistics focused on users with nonzero scores.The estimates were presented in Table 9 based on the contest statistics in Table 8.Source: Table 9 · p. 111, §C.1 · p. 111, Table 8 · p. 111

C.2 Example of GPT-4 visualizing IMDb data.

• GPT-4 creates an interactive IMDb network graph linking movie titles, writers, and directors, spontaneously applying community-based coloring and supporting zooming and hover labels.The graph enables users to inspect regions of interest and node labels interactively.Source: §C.2 · p. 112
• GPT-4 simplifies the large IMDb network by filtering irrelevant data, aggregating related nodes, and using force-directed or hierarchical layouts with community detection.The proposed implementation filters adventure movies released from 2010 onward and organizes collaborations into network subgroups.Source: §C.2 · p. 113

C.3 More examples on visualization

• GPT-4 produces a visualization matching vague instructions, including a gradually changing rainbow color pattern and the specified 0.2-second timing.The result is presented as an impressive visualization created from vague human instructions.Source: Figure C.1 · p. 115

C.4 Example for 2D HTML game development

• GPT-4 generates a fully functioning 2D HTML tank game from vague specifications, implementing movement, firing, collisions, random walls and enemies, win/loss conditions, and unspecified shell disappearance after wall impacts.It also edits the game according to user requests, whereas ChatGPT refuses and produces nonfunctional movement code.Source: §C.4 · p. 116

C.5 Example for graphical user interface programming

• GPT-4 demonstrates GUI-programming expertise by extracting layouts, widgets, labels, and actions, while handling complicated input events and creating accurate interfaces.The paper presents a GUI for a query model generated from a prompt.Source: §C.5 · p. 116, Figure C.2 · p. 117
• GPT-4 adds robust GUI behavior by running the query model in a separate cancellable thread, highlighting $-delimited expressions, summarizing text, and inferring file-browsing behavior for the load button.These functions are inferred or implemented beyond the prompt’s explicitly stated requirements.Source: §C.5 · p. 117
• GPT-4 faithfully implements a drawing panel’s described interactions, including temporary red highlighting, object-list consistency after deletion, and intermediate objects during mouse dragging, although curved-arrow drawing still needs improvement.The drawing-panel task tests GPT-4’s zero-shot GUI programming ability on a challenging specification.Source: §C.5 · pp. 117–118, Figure C.3 · p. 118

C.6 Example for reverse engineering

• GPT-4 progressively guided reverse engineering by identifying the executable, inspecting strings, launching LLDB, setting a printf breakpoint, and reading registers.These steps led to assembly-based analysis of the input checks and password-printing path.Source: §C.6 · p. 119
• The program accepted any 10-character input matching only the password’s first byte, exposing a weak verification scheme that enabled brute-force recovery.GPT-4 inferred this from assembly showing length and first-byte checks, while the remaining bytes were printed as the password.Source: §C.6 · p. 120
• Password found: aaaaaaaaaa after GPT-4 proposed a Python subprocess script to brute-force the first byte and capture successful output.The reported runtime was 0.15888214111328125 seconds.Source: §C.6 · p. 120

C.7 Testing GPT-4’s ability to execute (pseudo) code

• The pseudo-code experiment fixed one input array at length 4, randomly sampled its elements from 1−9, and varied the other array’s length and update steps.The tested length/step settings were 4/32, 6/48, 8/64, 10/80, and 12/96.Source: §C.7 · p. 121
• At 96 steps, GPT-4 tracked the pseudo-code array state with 54% exact-match accuracy, despite outputs approaching its 8129-token limit.Accuracy was measured as the percentage of input instances whose outputs exactly matched the target.Source: §C.7 · p. 121
• The pseudo-code task tested whether GPT-4 could understand and execute programming logic, not merely generate equivalent code from natural-language instructions.The authors characterize this execution ability as a significant step toward an AGI compiler, while noting it is not yet an acceptable compiler.Source: §C.7 · p. 121

D.1 Limitations

• GPT-4 is more prone to arithmetic mistakes when it skips calculation steps, whereas prompting it to break calculations into smaller steps can produce a correct solution.The section identifies arithmetic and attention errors as recurring localized failure categories.Source: §D.1 · p. 122
• Even explicit instructions to think step by step do not reliably prevent GPT-4 from skipping intermediate algebraic manipulations, motivating research on training models to decompose calculations.The authors note that online mathematics sources often omit calculation steps, potentially encouraging this autoregressive behavior.Source: §D.1 · pp. 122, 124
• As expression length increases, GPT-4 becomes more likely to make algebraic calculation mistakes; at least 90% of 100 inspected errors arose from skipping steps when merging similar terms.GPT-4 nevertheless outperforms the previous model on this simple algebra task.Source: §D.1 · pp. 123–124
• GPT-4’s counting ability improves substantially over the previous model on short sequences, but accuracy drops significantly as sequence length grows from 5 to 10, remaining below human counting capacity.The authors suggest integrating a counting component into the architecture because counting is basic to many applications.Source: §D.1 · p. 124
• When GPT-4 begins mathematical reasoning with a wrong conclusion, it often produces incoherent justifications instead of verifying local consistency; bottom-up reasoning improves performance significantly.The authors relate this tendency to mathematical writing styles that state conclusions before reasoning and to training data dominated by straightforward solutions.Source: §D.1 · p. 124, Figure D.1 · p. 125

D.2 Further examples

• Across specially composed, likely unseen mathematics questions, GPT-4 demonstrates capabilities beyond ChatGPT, including strong geometric handling and meaningful discussion of some advanced topics.The examples are intended to illustrate capability range rather than provide comprehensive or representative evaluation.Source: §D.2 · p. 126
• Compared with ChatGPT, GPT-4’s answers are generally more coherent, accurate, relevant, and logically justified, while ChatGPT often relies on template matching, incorrect reasoning, or error-prone computations.The questions were collected before testing both models without modification, and the authors did not select them to favor GPT-4.Source: §D.2 · p. 126

D.2.1 Algebra

• GPT-4 correctly solves the function-inversion problem, whereas ChatGPT produces an incorrect solution reflecting a misunderstanding of function inversion.GPT-4 uses f(g(x)) = x to derive f(6) = 9, f(9) = 7, and f(7) = 4.Source: §D.2.1 · p. 127
• GPT-4 and ChatGPT both err on an iterative multiply-by-2-and-subtract-1 problem, although GPT-4’s reasoning contains the correct answer while ChatGPT’s argument is mostly incoherent.GPT-4 states that every starting number eventually exceeds any finite target, while ChatGPT gives the condition x > 2^n−1 and the example x > 512 for n = 10.Source: §D.2.1 · pp. 127–128
• GPT-4 correctly solves the polynomial root problem by substituting u = x^4, while ChatGPT gives an incorrect solution.GPT-4 identifies that the second substituted equation has no real solutions, leaving only the real roots stated in the solution.Source: §D.2.1 · p. 129
• Both models answer the trigonometry problem incorrectly: GPT-4 uses broadly correct reasoning but makes calculation mistakes, whereas ChatGPT’s reasoning is mostly invalid.The problem assumes sin(a), sin(b), cos(a), and cos(b) are positive and asks for sin(a − b) in terms of x and y.Source: §D.2.1 · p. 130

D.2.2 Geometry

• GPT-4 correctly solves the vector-geometry problem, while ChatGPT makes an incoherent comparison between vectors and scalars.GPT-4 associates the geometric description with vector notation and manipulates the vectors to obtain the final result.Source: §D.2.2 · p. 131
• GPT-4 correctly applies the Pythagorean theorem to the square-area problem but reports an approximate area of 114.92 square units because of unnecessary numerical estimation.ChatGPT instead gives the incorrect area 72, and its output is described as incoherent.Source: §D.2.2 · p. 132
• GPT-4 correctly identifies exactly two configurations of the equilateral-triangle problem up to rotation and dilation.The configurations correspond to D lying inside or outside the triangle, represented using complex-number coordinates.Source: §D.2.2 · p. 133
• GPT-4 identifies exactly two configurations in the geometry problem, whereas ChatGPT’s final answer is correct but its argument is invalid and lacks geometric comprehension.Source: §D.2.2 · p. 133

D.2.3 Calculus

• In calculus, GPT-4 correctly solves the integral using integration by parts, correctly applies implicit differentiation, and gives a sound Lagrange-multiplier argument, while ChatGPT makes mistakes or uses irrelevant reasoning.Source: §D.2.3 · pp. 134–136

D.2.4 Mathematical modeling

• GPT-4 models the gold-egg problem using weight and bag strength rather than volume alone, but its final numerical answer is wrong; realistic estimates require factors such as bag durability and safety margins.Source: §D.2.4 · p. 137
• For the iron-ball problem, GPT-4 constructs a physical model combining motion, solar radiation, heating, melting, and multiple explicit simplifying assumptions.Source: §D.2.4 · p. 138
• GPT-4 models the iron ball’s motion by equating kinetic energy with the heat required to raise it to its melting point, then solving for velocity.It uses KE = 1/2 ∗m ∗v2 and Q = mc∆T.Source: §D.2.4 · p. 138

D.3 Generating math problems with GPT-4

• GPT-4 generates new math problems through content modification, which preserves structure and logic while changing names, objects, or values, and difficulty enhancement, which adds complexity or constraints.Examples include rewriting an elementary counting problem and adding discount conditions to a multistep profit problem.Source: §D.3 · pp. 138–139

D.4 Mitigating calculation errors via external code execution

• External code execution is proposed to mitigate calculation errors by having GPT-4 generate code for a calculation and appending the executed result to the prompt, but this approach was not systematically evaluated.The example uses Python and SymPy to compute a weighted two-dice probability.Source: §D.4 · pp. 139–140

E.1 Explanation Agent Mismatches

• Explanation quality depends on matching the model’s explaining persona and generative process: mismatches produce poor explanations, whereas GPT-4 can explain another model when it can emulate both roles.The examples contrast a useless one-word response with richer explanations and a useful explanation of a linear model by a statistician.Source: Figure E.1 · p. 141, Figure E.2 · p. 141, Figure E.3 · p. 142

F.1 Interact with tools

• GPT-4 is shown interacting with tools by diagnosing a cold house, listing required calendar API functions, and supporting a penetration-testing scenario.The calendar example illustrates that it can identify the tools it needs and then use them.Source: Figure F.1 · p. 144, Figure F.2 · p. 145, Figure F.3 · p. 146

F.1.1 The zoo game in Section 5.1.1

• In the zoo-game task, GPT-4 receives a multi-step todo list requiring file deletion, renaming, descriptions, relocation, directory creation, and email drafting, and begins executing these commands.The transcript includes correcting a failed directory path and deleting files from the Tropical Rain Forest folder.Source: §F.1.1 · p. 147

F.1.2 The zoo game in Section 5.1.1 with more guidelines

• With additional guidelines, the zoo-game task requires GPT-4 to inspect each email, search related folders, and write both internally referenced and externally clean replies.The instructions require references in the internal answer using [ref: ...txt] while excluding them from the external reply.Source: §F.1.2 · p. 148
• The zoo-game log records weekday hours of 9:30 a.m.–4:00 p.m. on Monday and 9:30 a.m.–3:00 p.m. Tuesday–Sunday.Source: §F.1.2 · p. 148
• The system’s internal answer directs the response to Hours.txt and Special Hours.txt for the zoo’s opening hours.Source: §F.1.2 · p. 148
• The generated reply begins “Dear Alice,” expresses hope to see the recipient at the zoo, and closes “Sincerely,”.Source: §F.1.2 · p. 148

F.2.1 The first game in Section 5.2.2

• The agent unlocked the Canadian chest by finding and using its latchkey, scoring 1/1 in 32 turns.The successful sequence involved navigating rooms, locating the Canadian chest and latchkey, and issuing the unlock command.Source: §F.2.1 · p. 150

F.2.3 The second game in Section 5.2.2 with 0-shot

• In the zero-shot cooking game, the agent followed the recipe far enough to chop the red hot pepper and slice the green apple, but could not find a frying pan to continue.The game awarded one point for each completed preparation action, while repeated searches failed to locate the frying pan.Source: §F.2.3 · pp. 152–153

F.2.4 The second game in Section 5.2.2 with 1-shot

• The supplied 1-shot example begins with the cooking objective and a kitchen containing the ingredients, knife, cookbook, and cooking appliances, but no subsequent actions or result are shown.The cookbook and relevant kitchen objects are presented as the initial state for the example.Source: §F.2.4 · p. 154
• GPT-4 executed the cooking sequence by chopping and frying the red hot pepper, slicing and frying the green apple, then preparing and eating the meal.Each required action increased the score by one point.Source: §F.2.4 · p. 154
• When GPT-4 initially used the unrecognized command “fry red hot pepper with stove,” it corrected to “cook red hot pepper with stove” and completed the frying action.The corrected command produced the intended result and awarded a point.Source: §F.2.4 · p. 154
• GPT-4 scored 8 out of a possible 8 points in 11 turns, completing the game successfully.The game ended after GPT-4 prepared and ate the meal.Source: §F.2.4 · p. 154

• The section reports detailed results for the analysis presented in Section 7.2.Source: §G.1 · p. 155
• Judge GPT-4 often selects GPT-4’s own answer as better than GPT-3’s answer across most categories.Source: Table 10 · p. 155