PaperlayerView exact revision on arXiv

Source-linked AI summary

The Generative AI Paradox: "What It Can Create, It May Not Understand"

Peter West, Ximing Lu, Nouha Dziri, Faeze Brahman, Linjie Li, Jena D. Hwang, Liwei Jiang, Jillian Fisher, Abhilasha Ravichander, Khyathi Chandu, Benjamin Newman, Pang Wei Koh, Allyson Ettinger, Yejin Choi

arXiv:2311.00059v1cs.AIcs.CLcs.CVcs.LG

TL;DR

The paper asks how generative models can produce expert-challenging outputs while still making basic understanding errors. It tests the Generative AI Paradox across language and vision, finding that generation can exceed human performance while understanding lags, and urging caution when interpreting AI by analogy to human intelligence.

  • Problem

    Generative models produce outputs that challenge experts yet make basic understanding errors, raising whether these capabilities align as they do in humans.

    Source: §1 · p. 1

  • Method

    Controlled experiments in language and vision compare generation with understanding through selective and interrogative evaluations.

    Source: §1 · p. 2

  • Results

    Across tasks and modalities, models can outperform humans in generation but fall short in discrimination and in answering questions about generated content; human discrimination is more tightly linked to generation and more robust to challenging inputs.

    Source: §5 · p. 8

  • Takeaways & Limitations

    The findings support caution in applying human-derived concepts of intelligence to AI and in inferring human-like mechanisms from expert-like outputs.

    Source: §1 · p. 2

    The study covers a small set of popular models and investigates only one divergence between human and artificial intelligence.

    Source: §5 · pp. 8–9

Abstract

from arXiv · show

The recent wave of generative AI has sparked unprecedented global attention, with both excitement and concern over potentially superhuman levels of artificial intelligence: models now take only seconds to produce outputs that would challenge or exceed the capabilities even of expert humans. At the same time, models still show basic errors in understanding that would not be expected even in non-expert humans. This presents us with an apparent paradox: how do we reconcile seemingly superhuman capabilities with the persistence of errors that few humans would make? In this work, we posit that this tension reflects a divergence in the configuration of intelligence in today's generative models relative to intelligence in humans. Specifically, we propose and test the Generative AI Paradox hypothesis: generative models, having been trained directly to reproduce expert-like outputs, acquire generative capabilities that are not contingent upon -- and can therefore exceed -- their ability to understand those same types of outputs. This contrasts with humans, for whom basic understanding almost always precedes the ability to generate expert-level outputs. We test this hypothesis through controlled experiments analyzing generation vs. understanding in generative models, across both language and image modalities. Our results show that although models can outperform humans in generation, they consistently fall short of human capabilities in measures of understanding, as well as weaker correlation between generation and understanding performance, and more brittleness to adversarial inputs. Our findings support the hypothesis that models' generative capability may not be contingent upon understanding capability, and call for caution in interpreting artificial intelligence by analogy to human intelligence.

1 INTRODUCTION

Generative models can produce expert-challenging outputs while making basic understanding errors, prompting the paper to test whether generation and understanding diverge in models compared with humans. Across language and vision, the reported pattern supports that divergence and cautions against treating AI intelligence as analogous to human intelligence.

1 INTRODUCTION

  • The central puzzle is that models produce outputs that challenge experts while also making basic errors that even non-experts would avoid.The contrast appears in both language and visual domains.Source: §1 · p. 1
  • The paper proposes that generative models can acquire expert-like generation without commensurate understanding, unlike humans, for whom understanding nearly always precedes expert-level generation.This is the Generative AI Paradox hypothesis.Source: §1 · p. 2
  • The experiments assess generation and understanding in language and vision through selective and interrogative evaluations.Selective evaluation tests choosing correct responses in a generative task; interrogative evaluation tests questions about a generated response’s content and appropriateness.Source: §1 · p. 2
  • Models can match or outperform humans in generation yet fall short in selective discrimination and in answering questions about their own generated content.Human discrimination is more tightly linked to generation and more robust to challenging inputs; the gap grows with task difficulty.Source: §1 · p. 2
  • The findings caution that human-derived concepts of intelligence may not transfer to AI, and that human-like outputs need not indicate human-like mechanisms.The paper frames models as a counterpoint to human intelligence rather than a parallel.Source: §1 · p. 2

2 THE GENERATIVE AI PARADOX

The section defines the Generative AI Paradox as a gap between generative performance and understanding, then lays out tests across modalities. Its hypotheses compare model understanding with human performance at comparable generation levels.

2 THE GENERATIVE AI PARADOX

  • The paradox is illustrated by strong generations paired with failures to inspect them: GPT-4 misstates a story detail, while a vision model cannot select its correct image against plausible alternatives.Both examples show generation meeting or exceeding human ability while understanding lags.Source: §2.1 · p. 3

2.1 OPERATIONAL DEFINITIONS

  • The hypothesis compares generation and understanding: at equal generation performance, it predicts human understanding will exceed the model’s by a meaningful margin.Generation is observable content produced for a task input; understanding is assessed through selective choice or questions about generated content.Source: §2.1 · p. 3

2.2.1 HYPOTHESES

  • Models outperform humans in generation but underperform them in discrimination in most Figure 2 cases; humans show a larger positive relationship between discrimination and generation than GPT-4.The scatter plot’s points represent different tasks.Source: Figure 2 · p. 4
  • The two sub-hypotheses predict that models can meet or exceed humans in generation while lagging in discrimination, and struggle with simple questions about generated content.For interrogative evaluation, the human baseline assumes near-perfect accuracy on questions about one’s own generations.Source: §2.2.1 · p. 4

2.2.2 MODELS AND EXPERIMENTS

  • Experiments cover selective and interrogative evaluation in both language and vision, focusing on prominent models including GPT-4, GPT-3.5, Midjourney, CLIP, OpenCLIP, BLIP-2, BingChat, and Bard.The vision experiments use separate generators and understanding models.Source: §2.2.2 · p. 4

3 CAN MODELS DISCRIMINATE WHEN THEY CAN GENERATE?

The section compares model and human generation and discrimination across language and vision, then examines how harder answer choices affect discrimination and how people prefer model-generated responses. Models often show a generation advantage alongside weaker discrimination, but the pattern varies across tasks and modalities.

3.1 GENERATIVE AND DISCRIMINATIVE CAPABILITIES IN MODELS VS. HUMANS

  • Figure 3 reports that vision models fall short of human discriminative accuracy, while their generative accuracy is lower than their discriminative accuracy.Source: Figure 3 · p. 5

3 CAN MODELS DISCRIMINATE WHEN THEY CAN GENERATE?

  • The study compares generative responses with candidate selection across language and vision, using human correctness judgments for generation and accuracy against ground truth for discrimination.Human workers perform the discriminative tasks to provide a performance baseline.Source: §3 · pp. 4–5

3.1 GENERATIVE AND DISCRIMINATIVE CAPABILITIES IN MODELS VS. HUMANS

  • In language, the generation-over-human and discrimination-below-human pattern appears for at least one model on 10 of 13 datasets and for both models on 7 of 13.For vision, CLIP and OpenCLIP fall short of human discrimination; the authors assume average human generation is lower rather than measuring it directly.Source: §3.1 · pp. 5–6

3.2 MODELS FALL FURTHER SHORT OF HUMAN PERFORMANCE WITH HARDER DISCRIMINATION TASKS

  • GPT4’s XSUM accuracy falls from 100% with easy negatives to 19% with hard negatives, while OpenCLIP drops from 100% to ∼69%; humans remain consistently accurate across difficulty levels.Hard negatives are subtly wrong and challenging, whereas easy negatives are semantically distant from the question topic.Source: §3.2 · p. 6

3.3 MODEL GENERATIONS ARE PREFERRED OVER HUMAN GENERATIONS

  • Workers often prefer GPT4-generated answers for their length, elegant style, and informativeness, while human answers are preferred for brevity and concision.Source: §3.3 · p. 6

4 CAN MODELS UNDERSTAND WHAT MODELS GENERATE?

The paper tests whether models can understand their own generated text and images by asking questions about those outputs. Models make frequent errors on this task, while humans answer more accurately in both modalities.

Language experimental setup

  • For language, models generate text from task-specific inputs and then answer multiple-choice questions about their generated content; evaluation uses zero-shot GPT3.5 and GPT4.Selection is evaluated only on generations verified as correct by humans.Source: §4 · p. 7

Vision experimental setup

  • For vision, Midjourney generates images from benchmark prompts, then image-understanding models answer open-ended questions about image content, with human responses also collected and judged.The questions concern prompt-specified content such as object existence, attributes, and relative positions.Source: §4 · p. 7

Results

  • Humans achieve higher accuracy than models when answering questions about model-generated text and images, although image generators can produce images beyond most average humans’ generation abilities.Source: §4 · pp. 7–8
  • Advanced multimodal LLMs such as Bard and BingChat still struggle with simple questions about generated images, and their model-human performance gap is larger than for simpler models.Source: §4 · p. 8

5 DISCUSSION

The discussion reports evidence for the Generative AI Paradox while considering possible explanations and boundaries on the findings. It cautions that current results cover a limited set of models, human comparisons, and dimensions of divergence.

5 DISCUSSION

  • Models can outperform humans in text and image generation yet fall short in discriminative and question-answering measures of understanding.Generation and discrimination are more tightly linked in humans than in GPT-4, and humans are more robust to challenging inputs.Source: §5 · p. 8
  • The generative learning objective explicitly trains reconstruction or generation, while encouraging understanding only when it advances that goal.The passage contrasts this with human learning, which likely encourages behavior beyond pure reconstruction of stimuli.Source: §5 · p. 8
  • Models may favor distributionally correct outputs over details needed for understanding because they model a medium across many authors and encounter vast, diverse training data.The passage suggests that previously encountered solutions may help models answer questions that require humans to reason.Source: §5 · p. 8
  • Evolutionary and economic pressures may favor architectures and training choices that prioritize generation over understanding.The passage gives the greater human difficulty and utility of generation as a possible reason for this preference.Source: §5 · p. 8
  • The study covers a small set of popular models, and potential proprietary-model benchmark contamination is discussed as a limitation.The authors say contamination should affect generation and discriminative evaluations similarly; their later evaluation uses novel generations not seen during training.Source: §5 · p. 8
  • Human-performance evaluation is focused, and the paper investigates only one divergence between human and artificial intelligence.The authors call for wider model–human comparisons and study of other divergences.Source: §5 · pp. 8–9

6 RELATED WORK

Related work covers model self-critique, inconsistencies in language-model behavior, and comparisons between model competencies and human cognition.

6 RELATED WORK

  • Prior work has used language models to critique or improve their own generations and identify hallucinations or omissions in generated content.Examples include self-critique and improvement, identifying hallucinated content, and finding omitted elements in clinical extraction data.Source: §6 · p. 9
  • Studies report inconsistent language-model predictions, contradictory answers, and failures to infer from facts that models can recognize.The cited work also finds errors on basic tasks and cases where models produce both correct facts and their negations.Source: §6 · p. 9
  • Prior comparisons find gaps between model and human competencies in commonsense psychology, Theory of Mind, and physical reasoning.The passage describes findings involving infants' reasoning about agents' actions and models' physical commonsense capacities.Source: §6 · p. 9

7 CONCLUSIONS

The paper proposes that generative ability in models may outstrip understanding, tests this hypothesis in language and vision, and finds robust support. It concludes that models may be a counterpoint to, rather than a parallel of, human intelligence.

7 CONCLUSIONS

  • The Generative AI Paradox hypothesis holds that models' impressive generation may not depend on commensurate understanding, unlike human abilities as characterized by the paper.The authors report robust support across controlled experiments in language and vision, with variation by task and modality.Source: §7 · p. 9
  • The findings suggest that human-derived conceptualizations of intelligence may not apply to AI because capability patterns may diverge from human expectations.Source: §7 · p. 9
  • The paper frames studying models as an intriguing counterpoint to human intelligence rather than a parallel.Source: §7 · p. 9

REPRODUCIBILITY

The paper provides brief experiment-specific descriptions in the main text and directs readers to appendices for extensive details and evaluation materials.

REPRODUCIBILITY

  • Overall details appear in section 2, while experiment-specific datasets and evaluation setups are briefly described at the beginnings of sections 3 and C.Source: REPRODUCIBILITY · p. 9
  • Appendix A provides additional information on models, model settings, and datasets, while appendices B and C provide experimental details and further experiments.Source: REPRODUCIBILITY · p. 9
  • Appendix D contains more extensive human-evaluation templates, and the datasets and models are public or accessible through public interfaces.Source: REPRODUCIBILITY · p. 9

ETHICS STATEMENT

This section describes the models, datasets, evaluation setups, and negative-example construction used in the study, alongside its ethics and scope. Language and vision negatives range from unrelated examples to semantically close, subtly incorrect ones.

ETHICS STATEMENT

  • The study uses existing benchmarks and models, reports IRB-exempt human studies, pays workers at least $15 per hour, and does not specifically measure bias or toxicity.The authors state that they introduce no new data, methodology, or models.Source: ETHICS STATEMENT · p. 10

A.1 MODELS

  • Language evaluation uses GPT4 and GPT3.5, while vision generation uses Midjourney and vision understanding uses CLIP-family and image-question-answering models.The vision models include CLIP, OpenCLIP, BLIP, BLIP-2, Instruct-BLIP, Bard, and BingChat.Source: §A.1 · p. 15

A.2 DATASETS

  • Vision datasets cover real-image captions, object and spatial relations, challenging prompts, compositional generation, and image-question answering.The selective evaluation draws on COCO, PaintSkill, DrawBench, and T2ICompBench; interrogative evaluation uses TIFAv1.0.Source: §A.2 · p. 15

B.1 SETUP

  • For vision evaluation, Midjourney generates images from prompts and humans judge prompt matching, while understanding models choose between a successful image and a negative candidate.The study samples 100 prompts per dataset and reports generation success per prompt and discriminative accuracy.Source: §B.1 · p. 15

B.2 NEGATIVE EXAMPLE CONSTRUCTION

  • Language negatives include unrelated responses, answers with up to two replaced entities, and GPT4-generated hallucinations prompted by subtle changes to knowledge spans.For hard negatives, replaced entities come from the same document and share the original entity type.Source: §B.2 · pp. 15–16
  • Figure 7 reports a larger positive correlation between discrimination and generation abilities for humans than for GPT3.5.The section also includes Figure 8 on human-generated and GPT3.5 response quality scores.Source: Figure 7 · p. 16, Figure 8 · p. 16
  • Vision hard negatives are prompt-matched images with subtle mistakes, whereas easy negatives come from another prompt; workers verify retained examples.The resulting set contains 345 hard-negative and 372 easy-negative instances.Source: §B.2 · p. 16

B.3 ADDITIONAL RESULTS

The additional results report language and vision comparisons, while the experimental setup also describes constrained language generation and vision interrogative evaluation. The results include a stronger human correlation between discrimination and generation and a growing vision performance gap as negative-example difficulty rises.

B.3 ADDITIONAL RESULTS

  • Humans show a larger positive correlation between discrimination and generation than GPT3.5, and workers often favor GPT3.5 responses over human-generated ones.Figure 9 presents rationales for preferences between GPT4 and ground-truth human responses.Source: §B.3 · p. 16, Figure 9 · p. 18
  • Vision models consistently fall short of human discriminative accuracy, and the model–human gap grows as task difficulty increases.The additional results cover model variants and easy versus hard negatives.Source: §B.3 · p. 16, Table 1 · p. 17, Table 2 · p. 17

C.1 EXPERIMENTAL SETUP.

  • Constrained language generation uses COLLIE-v1, spanning word, sentence, paragraph, and passage tasks, with 50 examples generated per task.Models are asked whether their outputs satisfy the given constraints.Source: §C.1 · pp. 16, 19, Table 3 · p. 17
  • Figure 10 shows GPT4 constraint-satisfaction results on Collie alongside discriminative accuracy on its generations.The figure caption identifies the two measures but provides no specific values.Source: §C.1 · pp. 16, 19, Figure 10 · p. 19
  • Vision interrogative evaluation samples 100 TIFAv1.0 prompts and assembles 1,871 answerable image-question pairs, including images that do not strictly match their prompts.Human performance is measured by comparing the majority of three responses with a fourth response.Source: §C.1 · p. 19

C.2 ADDITIONAL RESULTS

The additional results show that constrained language generation becomes difficult for models when constraints require precise composition, even as models may answer questions about their outputs more successfully. Vision understanding models perform below humans across the reported variants, and a small qualitative study probes errors about generated stories.

C.2 ADDITIONAL RESULTS

  • GPT4 performs better on simple Collie constraints than on arbitrary-position and strict-counting constraints, and is often better at answering questions about its generations than generating under constraints.The authors suggest precise constraints are easier to judge than to satisfy through flexible generation.Source: §C.2 · p. 19, Table 4 · p. 20
  • Vision understanding performance is consistently lower than human performance across the reported model variants and TIFAv1.0 subsets.Bard and BingChat results use only cases where the models provide a reasonable answer, since they can sometimes refuse questions about images containing people.Source: §C.2 · p. 19

C.3 QUALITATIVE EXPERIMENTS

  • A small-scale GPT-4 study uses author-written prompts and hand-written questions to probe simple questions about details in generated stories.The questions target content details that do not specifically depend on the prompt.Source: §C.3 · p. 20

D HUMAN ANNOTATION ON AMT

The annotation procedures use multiple AMT workers to assess discrimination, generation quality, preferences, and image-based question answering. The section also reports annotation agreement and notes that label skew makes kappa unsuitable for the data.

D HUMAN ANNOTATION ON AMT

  • The study qualified 130 AMT workers, paid median wages of $15–25 per hour, and reports pairwise agreement rates of 90–97% across datasets.Pay varied with task difficulty.Source: D HUMAN ANNOTATION ON AMT · p. 20
  • For discrimination evaluation, three workers select the correct response or best-matching image, and the majority vote determines the reported result.The language task provides context and a question; the vision task asks workers to choose the image that best matches the caption.Source: D HUMAN ANNOTATION ON AMT · p. 20, Figure 11 · p. 22
  • For generation evaluation, three workers judge whether a model response is acceptable or a generated image matches its caption, with majority vote used for results.The same template is also used to evaluate ground-truth responses.Source: D HUMAN ANNOTATION ON AMT · p. 20, Figure 12 · p. 22
  • In language comparative evaluation, workers choose between a model generation and a ground-truth answer and select checklist rationales for their preference.Example rationales concern response length or writing style.Source: D HUMAN ANNOTATION ON AMT · pp. 20–21, Figure 13 · p. 23
  • Because the data are skewed to a single label, the authors consider kappa unsuitable as an agreement measure for this work.The section includes qualitative examples of GPT-4 answers that contradict story details or miss an answer inferable from the story.Source: D HUMAN ANNOTATION ON AMT · pp. 20, 25
  • For vision human writing, one worker establishes ground-truth answers for generated images and another provides answers to measure human performance.Each example is annotated by two workers, with expert adjudication used if they disagree.Source: D HUMAN ANNOTATION ON AMT · p. 22, Figure 14 · p. 26
Loading 2311.00059v1…