Emergent Analogical Reasoning in Large Language Models

1 Introduction

• Analogical reasoning helps people solve unfamiliar problems through structured comparison and supports everyday problem-solving2, creative thought and scientific innovation3; its tests measure fluid intelligence4,5.The introduction describes fluid intelligence as the capacity to reason about novel problems.Source: §1 · p. 1
• Prior deep-learning work trained networks on very large problem sets7,8, but did not test whether analogical reasoning could emerge zero-shot, without direct training.The introduction identifies zero-shot reasoning as especially central to human thought.Source: §1 · p. 1
• Large language models trained to predict next tokens from massive text corpora may solve problems from instructions with little direct training, raising the question of human-like zero-shot analogy13.The described models use generic neural network architectures and may require no parameter updates when given a task instruction.Source: §1 · p. 1
• The evaluation compared GPT-3 with humans on text-only Raven-structured Digit Matrices, letter-string and verbal analogies, and story analogies, spanning generative and multiple-choice tasks.Digit Matrices tested abstract rule induction, not reasoning directly from visual inputs.Source: §1 · pp. 1–2, Figure 1 · p. 2
• GPT-3 performed as well as or better than college students in most conditions without direct training on Digit Matrices, and showed strong zero-shot performance across three other analogy tasks.The tasks included letter-string analogies, four-term verbal analogies, and story analogies.Source: Figure 1 · p. 2, §1 · p. 2

2.1 Matrix reasoning problems

• GPT-3’s matrix accuracy patterns correlated with humans’ (r(30) = 0.39), with progression, more unique rules, and spatial permutation making problems harder for both.The text-based matrices provide pseudo-symbolic inputs rather than testing visual parsing; nevertheless, error rates and patterns were similar to those for image-based SPM, suggesting overlap in core rule-induction processes.Source: §2.1 · pp. 3–5
• OR = 1.88 for direct generation and OR = 6.27 for answer-choice selection: GPT-3 surpassed average human performance across all Digit Matrix problem types.The task used novel, text-based matrices with Raven’s rule structure and no prompt or in-context examples; individual human performance varied, and the best participant answered every problem correctly.Source: §2.1 · p. 3, Figure 3 · p. 5

2.2 Letter string analogies

• OR = 1.76: GPT-3 outperformed humans on letter-string analogies overall, chiefly on zero-generalization problems, and its subtype error patterns correlated with theirs (r(39) = 0.7).More generalizations reduced accuracy for both groups. GPT-3 also generalized successorship from letters to real-world concepts, though performance varied by transformation and was sensitive to prompt and formatting.Source: §2.2 · pp. 6–7, Figure 6 · p. 8

2.3 Four-term verbal analogies

• GPT-3 performed as well as or better than humans on the UCLA and Sternberg–Nigro verbal analogy sets, exceeded estimated SAT performance, and was numerically weaker but in-range on Jones et al.’s set.On Jones et al.’s problems, semantic distance also affected GPT-3’s performance (OR = 3.24, p = 0.0165, CI = [1.24, 8.5]), as it did human performance.Source: Figure 7 · p. 9, §2.3 · pp. 9–10

2.4 Story analogies

• OR = 0.37: humans outperformed GPT-3 on story analogies, although both groups preferred stories sharing the source’s causal relations overall.GPT-3 and humans showed that preference in near analogies, but only humans did so significantly in far analogies; initial GPT-4 testing showed stronger performance, including nearly perfect near-analogy performance.Source: Figure 8 · p. 10, §2.4 · p. 11

2.5 Analogical problem-solving

• GPT-3 failed to find the radiation solution unaided but found it after hearing an analogous source story, and could explain the story-to-problem correspondences.Distractor stories disrupted transfer unless GPT-3 was prompted to consider the preceding stories; with that prompt it selected the relevant analogy and produced the convergence solution. In a separate gumball task, it also identified a solution analogous to a source story.Source: §2.5 · pp. 11–12
• GPT-3 mapped the major correspondences but proposed mechanically nonsensical solutions without a clear way to transfer gumballs, suggesting difficulty with physical reasoning.The original study presented the task to children with real objects, which may have aided physical reasoning compared with GPT-3's text-only input; overall, its analogical problem-solving ability was constrained by the content it could reason about.Source: §2.5 · p. 12

3 Discussion

• GPT-3 outperformed human participants on the novel, zero-shot Digit Matrices task and matched or surpassed human performance across a broad range of text-based analogy problems.It also captured specific signatures of human behavior and solved four-term verbal analogies and analogies between stories about complex real-world events.Source: §3 · p. 12
• GPT-3’s analogical reasoning is limited by its lack of episodic long-term memory and physical understanding, and by difficulty evaluating causal relations across stories.It reverts to a non-analogical solution after its context is emptied, struggled with a tool-transfer problem relative to children, and was particularly limited on far analogies.Source: §3 · p. 12
• Transformer self-attention and key–value indirection may support analogical mapping and variable binding, but whether GPT-3 uses human-like relational representations remains unclear.The discussion links pairwise similarity in self-attention to similarity-based analogy models and suggests that variable binding could support generalization across contexts; these are proposed possibilities, not established mechanisms.Source: §3 · pp. 12–13
• GPT-3 may solve complex analogies holistically through computational scale, whereas humans often decompose them into simpler problems [59–61].The authors present this as a possibility, including for Raven’s Matrices.Source: §3 · p. 13
• LLMs acquire analogical abilities through prediction of human-generated text, so their capacity is parasitic on human intelligence rather than a model of human development or evolution [59,62].The discussion notes that human language contains many analogies, while there is no reason to expect the same system to develop analogical thinking without human-generated inputs.Source: §3 · p. 13

4 Methods

• Human behavioral experiments were approved by the UCLA Institutional Review Board under protocol #22-000841 on May 17, 2022.Source: §4 · p. 13

4.1 Code

• Most code used Python v3.9.6 with NumPy, SciPy, statsmodels, Matplotlib, and pandas; logistic regression analyses used R v4.2.2.Source: §4.1 · p. 13
• Human-experiment stimuli were written in JavaScript using jsPsych v7.2.1.Source: §4.1 · p. 13

4.2 GPT-3

• Main-text simulations used text-davinci-003 through the OpenAI API, with temperature set to 0 and all other parameters left at their defaults.Supplementary simulations also used davinci, code-davinci-002, and text-davinci-002.Source: §4.2 · p. 13
• Maximum token limits were 10 for Digit Matrices, 40 for letter string analogies, 10 for four-term verbal analogies, and 256 for story analogies and analogical problem-solving.The maximum-token setting controls the maximum number of generated tokens per prompt.Source: §4.2 · p. 13
• For Digit Matrices and four-term verbal analogies, GPT-3 scored each answer choice by its average log probability and selected the choice with the highest score.The problem and choice were concatenated, and the average log probability assigned to the choice tokens served as the score.Source: §4.2 · p. 14

4.3.1 Dataset

• Digit Matrices combined transformation problems using constant, distribution-of-3, and progression rules with single-rule logic problems using OR, XOR, or AND.Logic problems could also spatially permute corresponding elements, making the underlying rule less intuitive to grasp.Source: §4.3.1 · p. 14
• Subtypes covered one- through five-rule and logic problems; most had 100 generated instances, and each problem had seven distractors plus the correct answer.Logic distractors were sampled from possible element subsets excluding the correct answer, with element order aligned or randomly permuted to match problem type.Source: §4.3.1 · p. 15

4.3.2 Human behavioral experiments

• 40 participants were analyzed in experiment 1 after three of 43 were excluded; experiment 2 included 47 participants and tested problems in increasing complexity order.Participants generated free responses before choosing from eight answer options; experiment 1 covered one- through three-rule and logic problems, while experiment 2 covered one- through five-rule problems.Source: §4.3.2 · p. 16

4.3.3 Evaluating GPT-3

• In the primary evaluation, GPT-3 was presented zero-shot with 40 problem instances from each of the 32 subtypes used in the first human behavioral experiment, without fine-tuning or in-context learning.For multiple-choice items, GPT-3’s selection was the answer choice with the highest average log probability of its tokens, excluding brackets.Source: §4.3.3 · p. 16

4.3.4 Statistical analyses

• Analyses used trial-level logistic regressions of response accuracy and correlated GPT-3 and human accuracy across all 32 problem subtypes.Regression models compared GPT-3 with humans by problem type and examined selected subtype features, including progression rules, unique-rule counts, and spatial alignment.Source: §4.3.4 · p. 17

4.4.1 Problem set

• The problem set combined six letter-string transformations with zero to three generalizations, including changes to representation, ordering, sequence length, distractors, and interval size.The study generated 600 problems at each generalization level; real-world concept problems used four transformations and no additional generalizations.Source: §4.4.1 · pp. 17–18

4.4.2 Evaluating GPT-3

• GPT-3 received bracketed, line-broken analogy prompts, and evaluation covered 300 problems at each alphanumeric generalization level plus 50 real-world concept problems per transformation.Alternative formats omitted the prompt or presented the analogy as a sentence; responses were truncated at a closing bracket or period, respectively.Source: §4.4.2 · p. 19

4.4.3 Human behavioral experiment

• Fifty-seven UCLA undergraduates completed 28 randomly sampled, randomly ordered problems and provided free responses.Each participant received problems spanning zero to three generalizations and real-world concept generalization.Source: §4.4.3 · p. 19

4.4.4 Statistical analyses

• Analyses used trial-level logistic regression for correctness and subtype-accuracy correlations, retaining only subtypes with at least five human and five GPT-3 trials.Of 252 possible subtypes, 41 met the inclusion criterion; regression analyses treated alphanumeric and real-world concept problems separately.Source: §4.4.4 · p. 19

4.5 Four-term verbal analogies

• On UCLA-VAT, GPT-3 chose between two answers across 80 categorical, functional, antonym, and synonym problems, compared with 57 UCLA undergraduates.Problems and selected answers were recursively appended to the prompt; problems were shuffled. The dataset and human behavioral data are from 17.Source: §4.5 · p. 20
• The Sternberg and Nigro dataset comprised 200 problems across five relation types, including linear order, and GPT-3 was compared with 20 Yale undergraduates.GPT-3 was evaluated as for UCLA-VAT; the human behavioral data are from 19.Source: §4.5 · p. 20
• For 374 SAT analogies spanning varied relation types, GPT-3 selected among five possible C and D terms and was compared with estimated high-school SAT performance.The dataset is from Turney et al. 20; candidate analogies were selected by the highest log probability assigned to their C and D terms, with choices recursively appended to the prompt.Source: §4.5 · p. 20
• The Jones et al. dataset contained 120 two-choice problems across categorical, causal, and compositional relations, half semantically near and half far, compared with 241 Wayne State undergraduates.GPT-3 was evaluated as for UCLA-VAT; the human behavioral data are from 21.Source: §4.5 · p. 20

4.6.1 Materials

• The task used 18 source stories with near and far target pairs that differed in whether entities and causal relations preserved the source analogy [Gentner et al.18; 41].Each pair contained a correct and incorrect target; near analogies shared entities, while far analogies used distinct entities. In both, the alternatives differed in causal relations.Source: §4.6.1 · p. 21

4.6.2 Human behavioral experiment

• Human participants chose the better analogy, the equally analogous option, or neither, across 18 trials; accuracy was the proportion of trials with a correct choice.Each trial presented one source story and two target stories.Source: §4.6.2 · p. 22

4.6.3 Evaluating GPT-3

• GPT-3 completed 72 comparisons covering near and far pairs in both target-story orders, with only its forced-choice response used in analysis.Stories were entered directly into the OpenAI playground, and GPT-3’s context window was cleared after each comparison.Source: §4.6.3 · p. 22

4.6.4 Evaluating GPT-4

• GPT-4 was tested on the same 72 problems and in the same format as GPT-3, with its context window cleared after each comparison.GPT-4 was evaluated through the ChatGPT web interface.Source: §4.6.4 · p. 22

4.6.5 Statistical analyses

• Near and far performance was tested against 50% chance accuracy, and trial-level logistic regression compared GPT-3 and human responses.The chance-level tests used binomial tests for GPT-3 and subject-level one-sample t-tests for humans; the regression predicted response correctness from whether the response came from GPT-3 or a human.Source: §4.6.5 · p. 22

4.7 Analogical problem-solving

• Problems were entered directly into the OpenAI playground, with each subsection reporting one continuous session and responses shown without truncation or curation.All prompts and responses are provided in Supplementary Section S5.Source: §4.7 · p. 23

Supplementary Results

• Supplementary Figure 1 compares four GPT-3 variants’ generative accuracy on Digit Matrices with average results from 40 human participants across transformation and logic problems.Transformation problems involve one to three rules; the figure includes standard-error bars for human results and 95% binomial confidence intervals for model results.Source: Figure 1 · p. 28
• Supplementary Figure 2 compares four GPT-3 variants’ generative accuracy on letter-string analogies by the number of generalizations, alongside average results from 57 human participants.The figure shows standard-error bars for human results and 95% binomial confidence intervals for model results.Source: Figure 2 · p. 28
• Supplementary Figure 3 compares four GPT-3 variants’ multiple-choice accuracy on UCLA Verbal Analogy Test problems across relation categories with chance performance and average results from 57 human participants.The figure includes standard-error bars for human results and 95% binomial confidence intervals for model results.Source: Figure 3 · p. 29

• The visual matrix solution is option 5, using a constant rule for shape count and distribution-of-3 rules for color and shape.Source: S1 Solutions to example matrix… · p. 29
• The Digit Matrix solution is option 7, using a constant rule for center digits and distribution-of-3 rules for left-side and right-side digits.Source: S1 Solutions to example matrix… · p. 29

S2 GPT-3 model variants

• Davinci used web-text next-token prediction, code-davinci-002 added GitHub-code prediction, text-davinci-002 received supervised prompt tuning, and text-davinci-003 received RLHF [30, 31].Text-davinci-002 was initialized from code-davinci-002, and text-davinci-003 from text-davinci-002; RLHF uses human ratings to train a reward model that guides reinforcement-learning fine-tuning.Source: S2 GPT-3 model variants · p. 29
• Text-davinci-003 performed best overall across Digit Matrices, letter-string analogies, and four-term verbal analogies; code-davinci-002 also performed well on the first two.These were the three task types used to evaluate all four GPT-3 variants.Source: S2 GPT-3 model variants · p. 29
• Code-davinci-002 performed near chance on verbal analogies while davinci did relatively well; text-davinci-002 and -003 appeared to combine both abilities.The authors suggest that code training may explain code-davinci-002’s strength on alphanumeric tasks but may have come at the cost of processing real-world relational concepts.Source: S2 GPT-3 model variants · p. 30
• Prompt training may have improved text-davinci-002 and -003’s ability to solve tasks without few-shot demonstrations, enabling zero-shot evaluation of their latent capabilities.Source: S2 GPT-3 model variants · p. 30
• On Gentner et al.’s story analogies [18], GPT-4 more reliably identified higher-order relational matches and gave more precise explanations than GPT-3.The authors could not test GPT-4 on the other analogy tasks because they lacked API access; they suggest increased model and training-set scale may partly explain the improvement [44].Source: S2 GPT-3 model variants · p. 30

• Study-created digit matrices and letter-string sets were absent from GPT-3’s training data; the digit format was novel, while similar letter problems may have appeared online.The authors found no evidence that GPT-3 knew about the similar letter-string problems.Source: S3 Presence of test materials in… · pp. 30–31
• The UCLA Verbal Analogy Test,17, four-term sets from Sternberg and Nigro19 and Jones et al.21, and SAT problems from Turney et al.20 might have been deliberately included, but probing found no evidence of memorization.The verbal sets were available in a downloadable supplement to a published paper,41, while the SAT dataset was not publicly available.Source: S3 Presence of test materials in… · p. 31
• Story analogy materials from Gentner et al.18 appeared in a downloadable supplement,41, but the authors found no evidence GPT-3 had memorized the stories.Source: S3 Presence of test materials in… · p. 31
• GPT-3 knew publication details of the Gick and Holyoak study,22, but the authors found no evidence it memorized the analogy; it solved the radiation problem using lesser-known source stories, though results warrant caution.The authors also found no evidence GPT-3 knew of or memorized other materials discussed in the associated publication,23.Source: S3 Presence of test materials in… · pp. 31–32

S4.1 Near analogy

• In the near-analogy example, GPT-3 chose Story A because Story B ends with the hunter killing the eagle using her own feathers.Story 1 instead ends with the grateful hunter promising not to shoot at hawks again.Source: S4.1 Near analogy · pp. 32–33

S4.2 Far analogy

• In the far-analogy example, GPT-3 chose Story A because both stories offer something valuable in exchange for a promise of peace.The compared stories pair Karla’s feathers and Zerdia’s computers with promises not to attack again.Source: S4.2 Far analogy · p. 33

S4.3 GPT-4

• In the far-analogy example, GPT-4 chose Story A, mapping William’s and Karen’s failed efforts and incentives while noting Story B changes the cause of failure.Both characters fail because they are too far behind, whereas Karen in Story B fails after spending time daydreaming and preparing for the trip.Source: S4.3 GPT-4 · p. 34

S5.1 Solution to radiation problem in isolation

• In isolation, GPT-3 proposed brachytherapy—placing a radioactive source in or near the tumor—to destroy it while sparing surrounding healthy tissue.It suggested placing the source in the tumor with a catheter or in a sealed container near it.Source: S5.1 Solution to radiation problem in… · p. 35

S5.2 Solution to radiation problem following general story

• After the general story, GPT-3 proposed splitting rays into low-intensity paths that converge on the tumor, and mapped doctor/general, tumor/dictator, and rays/army.It also suggested imaging to ensure the rays reached the tumor simultaneously.Source: S5.2 Solution to radiation problem… · pp. 35–36

S5.3 Solution to radiation problem following alternative source analogs

• With laser and lower-surface-similarity ultrasound source stories, GPT-3 identified convergence: multiple low-intensity rays converge on the tumor while sparing healthy tissue.The alternative source stories used convergence solutions that differed from the original general story.Source: S5.3 Solution to radiation problem… · pp. 36–38

S5.4 Solution to radiation problem with distracting stories

• With distracting stories, GPT-3 missed the general-story convergence analogy and instead proposed brachytherapy and focused stereotactic radiotherapy.This was the same solution it gave when the radiation problem was presented in isolation.Source: S5.4 Solution to radiation problem… · pp. 38–39

S5.5 Solution to radiation problem with distracting stories and prompt

• When prompted to use a prior story as a hint, GPT-3 named the general story and restored the low-intensity convergence solution.The prompt was the same as in the original human behavioral study, where it increased the likelihood of identifying the convergence solution.Source: S5.5 Solution to radiation problem… · p. 39

S5.6 Solution to gumball problem in isolation

• In isolation, GPT-3 proposed attaching posterboard to a tube and using string and a cane to pull the tube and gumballs toward the solver.Its steps also proposed securing the tube with paper clips and rubber bands.Source: S5.6 Solution to gumball problem in… · p. 40

S5.7 Solution to gumball problem following magic staff story

• After the magic-staff story, GPT-3 recognized the broad transfer analogy and suggested rope and cane, but failed to produce the children’s bowl-pulling solution or map staff to cane.In follow-up responses, it described the analogy as transferring items between containers and suggested using a rope and walking cane to move the gumballs.Source: S5.7 Solution to gumball problem… · pp. 40–41
• In the magic-staff analogy, GPT-3 matched the filled bowl to the genie’s old home, the empty bowl to the new home, gumballs to jewels, and the rope and cane to the staff.The mapped homes contain and receive valuable items, while the corresponding tools transfer them between locations.Source: S5.7 Solution to gumball problem… · p. 41

S5.8 Solution to gumball problem following magic carpet story

• GPT-3 proposed rolling posterboard into a tube to transfer gumballs, matching children’s solution but adding unnecessary steps, including cutting the posterboard in half.Children often chose the rolled posterboard despite having an already formed cardboard tube, likely because it matched the rolled-up magic carpet.Source: S5.8 Solution to gumball problem… · p. 42
• The magic-carpet analogy mapped the genie’s transfer of jewels between bottles to moving gumballs between bowls with a posterboard tube, suggesting the target solution.The compared transfers both keep the person or genie at the original location.Source: S5.8 Solution to gumball problem… · pp. 42–43