PaperlayerView exact revision on arXiv

Source-linked AI summary

Can Large Language Models generalize analogy solving like children can?

Claire E. Stevenson, Alexandra Pafford, Han L. J. van der Maas, Melanie Mitchell

arXiv:2411.02348v3cs.AIcs.CLcs.HC

TL;DR

The paper asks whether LLMs can generalize analogy solving to unfamiliar domains as people do. It compares children, adults, and LLMs on letter-string analogies across Latin, Greek, and Symbol alphabets, finding that LLMs do not generalize as people do.

  • Problem

    Whether LLMs generalize analogy solving to novel domains as people do remains unclear, despite analogical transfer being a fundamental part of human thought and learning.

    Source: §1 · pp. 1–2

  • Method

    The study compared children, adults, and LLMs solving letter-string analogies in Latin, Greek, and an invented Symbol alphabet.

    Source: §1 · p. 2, §2.1.1 · p. 3

  • Results

    LLMs performed at or above children’s level in Latin, but performance declined in Greek and deteriorated greatly in the Symbol domain, unlike human performance.

    Source: §5 · p. 9

  • Takeaways & Limitations

    The findings suggest that LLMs’ representation of an alphabet as an ordered sequence does not flexibly transfer to less familiar domains as it does in people.

    Source: §5 · p. 9

    LLMs’ difficulty identifying predecessors in the Next-Previous Letter task means those results do not fully explain their performance decline across domains.

    Source: §5 · p. 9

Abstract

from arXiv · show

In people, the ability to solve analogies such as "body : feet :: table : ?" emerges in childhood, and appears to transfer easily to other domains, such as the visual domain "( : ) :: < : ?". Recent research shows that large language models (LLMs) can solve various forms of analogies. However, can LLMs generalize analogy solving to new domains like people can? To investigate this, we had children, adults, and LLMs solve a series of letter-string analogies (e.g., a b : a c :: j k : ?) in the Latin alphabet, in a near transfer domain (Greek alphabet), and a far transfer domain (list of symbols). Children and adults easily generalized their knowledge to unfamiliar domains, whereas LLMs did not. This key difference between human and AI performance is evidence that these LLMs still struggle with robust human-like analogical transfer.

1 Introduction

The study asks whether LLMs can generalize analogy solving to unfamiliar domains as people do. It compares adults, children, and LLMs using letter-string analogies across Latin, Greek, and Symbol domains.

1 Introduction

  • The central question is whether LLMs generalize analogy solving to new domains as children do, a question motivated by conflicting evidence about LLM analogical reasoning.Prior studies found LLM performance could degrade on unfamiliar alphabets, while people generalized to novel domains.Source: §1 · pp. 1–2
  • Analogical reasoning lets people apply known concepts through relational similarity, and transfer can involve either concrete similarities or abstract relations.Children can solve familiar, causally framed analogies early, but may use associative strategies on more challenging or far analogies.Source: §1 · pp. 1–2
  • Letter-string analogies offer a text-based way to study transfer with little domain knowledge, using transformations from one string to solve an analogous item.Other analogy tasks may be unsuitable for children or LLMs because of unfamiliar words or difficulty with visual problems.Source: §1 · p. 2
  • Adults, children, and LLMs are compared on letter-string analogies in Latin, Greek, and Symbol domains to test near and far transfer.Greek is the near-transfer domain and the invented Symbol alphabet is the far-transfer domain.Source: §1 · p. 2

2 Method

The study compared adults, children, and LLMs on parallel letter-string analogy items across Latin, Greek, and Symbol alphabets. Human and model procedures were adapted to make performance comparisons possible.

2 Method

  • The study compared 42 children, 62 adults, and 55 runs of each of four LLMs across Latin, Greek, and Symbol conditions.The LLMs were Claude-3.5, Gemma-2 27B, GPT-4o, and Llama-3.1 405B.Source: §2 · p. 2

2.1.1 Letter-String Analogy Task

  • Items used successor, predecessor, or repetition transformations, with Latin items adapted to Greek and an invented Symbol alphabet for near and far transfer.The Symbol list was unfamiliar to both people and LLMs, while the same transformations and generalizations were retained across alphabet versions.Source: §2.1.1 · pp. 2–3, Figure 2 · p. 3

2.1.2 Human Data Collection

  • Adults and children completed browser-based tasks with practice items and five items per alphabet; the human samples were recruited online and at a local school, respectively.Six adults and two children were excluded under the reported criteria.Source: §2.1.2 · p. 3

2.1.3 LLM Data Collection

  • LLMs received alphabet-specific instructions and item prompts, with five prompt templates tested and the best-performing template reported.The models were run in chat-completion mode at temperature zero.Source: §2.1.3 · p. 4
  • LLM testlets used shifted item variants, and prior conversation was included so models could learn across items as participants did.The study reported selected high-performing models for its main comparisons; LLMs received zero-shot instructions, with a worked example not given to humans.Source: §2.1.3 · p. 4

3 Results

Adults and children generalized letter-string analogy solving across alphabet domains, whereas LLM performance declined as domains became less familiar. Follow-up tasks indicate that some model difficulties align with trouble applying particular letter-sequence rules.

3 Results

  • The study compared adults, children, and LLMs using mixed ANOVAs for Latin-alphabet performance and within-group performance across alphabets.Plots show means and standard errors.Source: §3 · p. 4

3.1 RQ1: How well do LLMs solve letter-string analogy problems in the Latin alphabet compared to adults and children?

  • In the Latin alphabet, adults and some LLMs outperformed children; GPT-4o performed similarly to adults, followed closely by Llama-3.1 405B.Gemma-2 27B and Claude-3.5 were exceptions to the LLMs outperforming children.Source: §3.1 · p. 5, Table 2 · p. 6

3.2 RQ2: How well do adults, children and LLMs generalize letter-string analogy solving from Latin to Greek (near) and Symbol (far) alphabets?

  • Adults and children performed similarly across alphabets, while every tested LLM declined significantly from Latin to Greek and again from Greek to the Symbol list.The Latin-to-Greek decline was significant for all models (all p<.001 except Llama-3.1 405B, p=0.012); the Greek-to-Symbol decline was significant for all models (all p<.001).Source: §3.2 · p. 5

3.3.1 Performance by Item

  • Across groups, item 1, requiring only the first successor transformation, was easiest, while item 5, combining predecessor and repetition, was hardest.The predecessor rule appears to be the most difficult rule for both humans and LLMs; item 3, involving the second successor rule, also caused difficulty.Source: §3.3.1 · p. 5, Table 4 · p. 6

3.3.2 Next-Previous Letter Task

  • On the Next-Previous Letter Task, models generally did better on one-step than two-step positions, but task-specific transfer patterns differed from analogy performance.Claude-3.5 performed similarly across domains on this task despite declining on analogies; GPT-4o also did well in Latin and Greek here, unlike its decline from Latin to Greek on analogies. These results may explain difficulty with the second-successor item, but not the predecessor item.Source: §3.3.2 · p. 7, Figure 3 · p. 7

3.3.3 Rule Check Task

  • In the Rule Check Task, models solved all rules in Latin and repetition rules in Greek and the Symbol domain, but all struggled with successor_2 and predecessor in the Symbol domain.In Greek, successor and predecessor performance varied, with Claude-3.5 performing best followed by GPT-4o.Source: §3.3.3 · p. 7, Figure 4 · p. 8

3.3.4 Error Analysis

  • Adults and children did not use the Literal rule in their errors, whereas every model used it sometimes; Other rules were most common for all except Claude-3.5.Claude-3.5 most often made One rule errors; Incorrect and Copy rules were uncommon among people and models.Source: §3.3.4 · p. 8

4 RQ4: What is the effect of model size on letter-string analogy performance?

The paper reports that typical scaling patterns appear to hold for LLM letter-string analogy generalization, with a marked performance increase from smaller to larger models in the Symbol domain.

4 RQ4: What is the effect of model size on letter-string analogy performance?

  • A marked performance increase from smaller to larger models appeared in the Symbol domain, where typical scaling patterns generally held for letter-string analogy generalization.The authors relate this pattern to prior reports that larger LLMs perform better on reasoning tasks (Wei et al., 2022; Huang and Chang, 2022).Source: §4 · p. 8
  • Typical scaling laws generally appeared to hold for LLM generalization on letter-string analogies.The reported pattern concerns performance as model size increases.Source: §4 · p. 8

5 Discussion

The tested LLMs matched or exceeded children on familiar Latin analogies but transferred less successfully to Greek and Symbol alphabets. The authors connect this gap to difficulty representing unfamiliar alphabets flexibly, while noting that this account does not explain all performance declines.

5 Discussion

  • LLMs performed at or above children on Latin analogies, declined somewhat on Greek, and deteriorated greatly on the Symbol alphabet.Adults and children generalized across the domains more successfully than the tested LLMs.Source: §5 · p. 9
  • Next-Previous results explain some difficulty with second-successor analogies, but do not fully explain the decline from Latin to Greek and Symbol domains.LLMs had little trouble identifying predecessors in that task.Source: §5 · p. 9
  • Repetition rules transferred readily, but predecessor and second-successor rules were harder because they require representing an alphabet as an ordered sequence.Ordering the Symbol alphabet by Unicode value produced only some improvement for Claude and Gemma, and the main findings still held.Source: §5 · p. 9
  • Models used a literal response rule 5–21% of the time, unlike people, while children’s incorrect answers were generally farther from the expected answer.The authors suggest children may have used nonanalogical strategies not captured by their error coding; the task also permits multiple solutions.Source: §5 · pp. 9–10
  • LLMs appeared unable to create on-the-fly representations of novel alphabets despite receiving ordered lists, although relative performance scaled with model size.The authors suggest an in-weight linear representation may be needed; next-token prediction supported simple repetition and successor rules, but not more complex rules at children’s level.Source: §5 · p. 10
  • Children’s analogy solving was not influenced by familiarity with letters or symbols, and the authors call for uncontaminated far-generalization tasks to probe human and artificial general intelligence.The authors concur with the view that robust analogical transfer may be uniquely human, while framing this as an open question.Source: §5 · p. 10

A Open Science Practices

The study reports ethics approval, preregistration with a later methods update, and public availability of its materials and data.

A Open Science Practices

  • The University of Amsterdam Social and Behavioural Sciences ethics board approved the study on May 24, 2023 (ID: FMG-2495).Source: A Open Science Practices · p. 14
  • The hypotheses, methods, and analyses were preregistered before ethical approval, then updated to accommodate new LLM data collection; the specified LLMs changed from the preregistration.The change in models was the main deviation reported.Source: A Open Science Practices · p. 14
  • The preregistration, materials, data, and code are publicly available through the project’s OSF repository.Source: A Open Science Practices · p. 14

B LLM prompt engineering results

The authors compared five prompt templates and report results using the best-performing template. Template 1 worked best overall, while template 4 performed less well on the Symbol list.

B LLM prompt engineering results

C LLM results without previous messages

The authors compared administering items individually with including earlier items and responses in the message history. History was generally advantageous, though two models had a Symbol-domain exception.

C LLM results without previous messages

D Ordered Symbol Task

Ordering symbols by Unicode value improved performance for Claude 3.5 and Gemma 2, but the overall decline from Latin to Greek and Symbol alphabets remained.

D Ordered Symbol Task

  • Performance still degraded from Latin to Greek and Symbol alphabets after Unicode ordering.The table compares LLM mean accuracy with and without Unicode ordering of symbols.Source: D Ordered Symbol Task · p. 15, Table 10 · p. 15
  • Unicode ordering improved Claude 3.5 and Gemma 2, with Claude 3.5 reaching its Greek-alphabet performance level.GPT-4o and Llama 3.1 showed no significant improvement from reordering.Source: D Ordered Symbol Task · p. 15
  • For future experiments using a Symbol domain, ordering symbols by Unicode value may benefit LLM performance.Source: D Ordered Symbol Task · p. 15
Loading 2411.02348v3…