PaperlayerView exact revision on arXiv

Source-linked AI summary

ChatGPT as Research Scientist: Probing GPT's Capabilities as a Research Librarian, Research Ethicist, Data Generator and Data Predictor

Steven A. Lehr, Aylin Caliskan, Suneragiri Liyanage, Mahzarin R. Banaji

arXiv:2406.14765v1cs.AIcs.CLcs.CYcs.IRcs.LG

TL;DR

The paper examines whether general-purpose AI can assist scientific research, an ability that had received little substantive testing. Across studies of GPT-3.5 and GPT-4, the models could replicate known effects but generally failed to predict unfamiliar data, while GPT-4 showed progress in research curation and ethics assessment.

  • Problem

    Despite widespread adoption, general-purpose AI’s ability to perform tasks critical to scientific discovery had not been substantively tested.

    Source: SIGNIFICANCE · p. 2

  • Method

    The paper probes GPT-3.5 and GPT-4 across scientific research tasks, including literature curation, research ethics, and replicating known versus predicting novel data patterns.

    Source: SIGNIFICANCE · p. 2, Section S1: GPT as Research Librarian… · p. 15

  • Results

    Both models were relatively successful at replicating known effects but generally failed to predict novel, unfamiliar data; GPT-4 also reduced fictional references and showed stronger research-methods performance.

    Source: General Discussion · pp. 9, 11

  • Takeaways & Limitations

    Successful replication of known results may simplify research piloting, while the models’ limited performance on highly novel data marks a boundary on their use for prediction.

    Source: General Discussion · p. 11

    The authors lacked insight into how other elements of GPT’s training, such as reinforcement learning and fine-tuning, affected its responses in the corpus-exploration task.

    Source: Results · p. 6

Abstract

from arXiv · show

How good a research scientist is ChatGPT? We systematically probed the capabilities of GPT-3.5 and GPT-4 across four central components of the scientific process: as a Research Librarian, Research Ethicist, Data Generator, and Novel Data Predictor, using psychological science as a testing field. In Study 1 (Research Librarian), unlike human researchers, GPT-3.5 and GPT-4 hallucinated, authoritatively generating fictional references 36.0% and 5.4% of the time, respectively, although GPT-4 exhibited an evolving capacity to acknowledge its fictions. In Study 2 (Research Ethicist), GPT-4 (though not GPT-3.5) proved capable of detecting violations like p-hacking in fictional research protocols, correcting 88.6% of blatantly presented issues, and 72.6% of subtly presented issues. In Study 3 (Data Generator), both models consistently replicated patterns of cultural bias previously discovered in large language corpora, indicating that ChatGPT can simulate known results, an antecedent to usefulness for both data generation and skills like hypothesis generation. Contrastingly, in Study 4 (Novel Data Predictor), neither model was successful at predicting new results absent in their training data, and neither appeared to leverage substantially new information when predicting more versus less novel outcomes. Together, these results suggest that GPT is a flawed but rapidly improving librarian, a decent research ethicist already, capable of data generation in simple domains with known characteristics but poor at predicting novel patterns of empirical data to aid future experimentation.

SIGNIFICANCE

The paper tests whether GPT-3.5 and GPT-4 can support scientific research, focusing here on literature curation. GPT-4 generated fewer fictional references and more often acknowledged fiction, but smaller citation errors persisted.

SIGNIFICANCE

  • Testing scientific AI assistance addresses a gap: despite widespread adoption, general-purpose AI’s ability to perform tasks critical to scientific discovery had not been substantively tested.Study 1 assessed whether GPT could gather relevant, comprehensive scientific content through literature reviews.Source: SIGNIFICANCE · p. 2, Study 1: GPT as Research Librarian · p. 2

Study 1: GPT as Research Librarian

  • The study asked both models to find and summarize 20 influential articles across 25 psychology topics, then assessed the references’ correctness, completeness, relevance, and citation counts.Topics ranged from broad questions about bias and discrimination to narrow applications of the Implicit Association Test.Source: Study 1: GPT as Research Librarian · pp. 2–3

Results

  • 5.40% of GPT-4 references versus 36.00% of GPT-3.5 references were hallucinations, while smaller error rates were similar at 4.60% and 4.20%, respectively.GPT-4 improved substantially in avoiding fabricated references, but not in smaller errors such as an incorrect year or journal.Source: Results · p. 3
  • 62.41% of incomplete references were hallucinations, compared with 14.30% of complete references, a pattern observed for both models.This relationship was not significant for smaller errors.Source: Results · p. 3
  • GPT-4 acknowledged 84.30% of its fictional references, compared with 12.20% for GPT-3.5; it also generated fewer fictional references overall, 23.12% versus 39.05%.The reported acknowledgment difference did not extend to smaller errors.Source: Results · p. 3
  • As topics narrowed, hallucinations first increased and then decreased as the models increasingly acknowledged not knowing narrow-topic articles; the linear pattern was robust for GPT-3.5 but not GPT-4.The curvilinear pattern arose because acknowledgments of not knowing were counted as correct rather than hallucinations.Source: Results · p. 3
  • GPT-4 found relevant articles more often overall, but the difference among real references was not significant; its clearest advantage appeared on moderately broad topics.Among real references, relevance was 53.70% for GPT-4 and 48.13% for GPT-3.5 (P = 0.124); at moderate breadth, the rates were 75.00% and 32.14%, respectively.Source: Results · pp. 3–4
  • GPT-4’s relevant articles averaged 2,936.7 citations, compared with 3,105.2 for GPT-3.5, with no significant difference.The comparison was not explained by GPT-3.5 drawing more from broader domains or GPT-4 citing newer articles.Source: Results · p. 4

Study 2: GPT as Research Ethicist*

Study 2 tested whether GPT-3.5 and GPT-4 would identify flawed practices in fictional research protocols, including blatant and subtle cases. GPT-4 outperformed GPT-3.5, and responses improved when prompts evoked data ethics.

Study 2: GPT as Research Ethicist*

  • Researchers tested the models with fictional protocols containing blatant or subtle flaws and varied prompts to probe whether framing affected feedback.The subtle vignettes used more realistic descriptions, such as contrasting sample sizes that might signal significance-driven stopping; “Ethicist” refers to advice aligned with accepted markers of ethical and replicable research, not human-like morality.Source: Study 2: GPT as Research Ethicist* · p. 4

Results

  • In blatant vignettes, GPT-4 scored 8.86/10 versus GPT-3.5’s 5.39; in subtle vignettes, GPT-4 scored 7.26 versus GPT-3.5’s 4.05.GPT-4 noticed and offered advice to correct most poor practices even when presented subtly.Source: Results · p. 5
  • None of nine basic matched-prompt contrasts was significant; the p-hacking/open-science contrast favored ethical prompting but missed the corrected threshold.The jailbreak comparison also failed to reach significance after correction.Source: Results · p. 5
  • Responses following prompts that evoked data ethics scored 7.35 on average versus 5.78 after prompts without such cues (P < 0.0001; d = 0.625).The effect also appeared when ethics primes encouraged unethical behavior, though that robustness check did not survive Bonferroni correction.Source: Results · p. 5
  • Both models identified accepted good research practices at similar rates: GPT-4 recognized 92.67% and GPT-3.5 recognized 90.42%, a non-significant difference.GPT-4 was substantially better at identifying bad practices.Source: Results · pp. 5–6

Study 3: GPT as Data Generator

Study 3 found that ChatGPT reproduced established cultural associations in word-association estimates, while Study 4 found little success predicting country-level patterns beyond explicit sexuality attitudes. Study 4 also found little evidence that the models used substantially different information for more novel implicit predictions.

Study 3: GPT as Data Generator

  • GPT-3.5 and GPT-4 estimated cultural associations by responding to randomly ordered word dyads, with responses analyzed using a WEAT D-score procedure.The study adapted stimuli from prior work and treated estimates as analogous to cosine similarities in word-embedding research.Source: Study 3: GPT as Data Generator · p. 6

Results

  • GPT-3.5 and GPT-4 estimates reproduced prior cultural preferences and gender associations across the four examined constructs.The constructs were Gender Attitudes, Gender Art/Science, Gender Home/Work, and Gender Math/Reading; Table 1 compares GPT estimates with real-corpus WEAT D-scores.Source: Results · p. 6, Table 1 · p. 7
  • Interpretation is bounded by unknown effects of reinforcement learning and fine-tuning, while cross-model agreement on dyad estimates was variable and modest.The reported inter-item correlations ranged from r = 0.382 to r = 0.666; the authors say these may indicate differing approaches or reliability constraints.Source: Results · pp. 6–7
  • GPT estimates were often stronger than prior findings, inconsistently, and were not generally smaller for GPT-4; GPT results diverged in being driven mainly by Female associations with stereotypically female categories.The authors note that stronger effects may reflect amplification of training-data biases; the divergence from prior findings concerns the Single-Category WEAT analysis.Source: Results · p. 7

Study 4: GPT as Novel Data Predictor

  • Study 4 elicited 60 predictions of cross-country attitudes and stereotypes using a previously unpublished dataset posted after the models' training cutoff.The predictions covered Explicit and Implicit Sexuality Attitudes, Age Attitudes, and Gender Science/Liberal Arts stereotypes.Source: Study 4: GPT as Novel Data Predictor · p. 7

Results

  • For Sexuality and Age Attitudes, implicit predictions showed little evidence of substantially new information use compared with explicit predictions.GPT-4's implicit and explicit Sexuality predictions were equally correlated; its Age predictions showed similar correlations across and between those prediction types.Source: Results · p. 8
  • GPT-3.5's Age and Gender predictions were uncorrelated across prompts, whereas GPT-4's Age predictions were more reliable but negatively correlated with explicit age results.GPT-3.5's collective predictions were also uncorrelated with actual country-level Age and Gender patterns.Source: Results · p. 8
  • Neither model predicted novel Age Attitudes or Gender Science/Liberal Arts patterns; both predicted Explicit but not Implicit Sexuality Attitudes.GPT-3.5's age and gender predictions were uncorrelated with actual patterns, and GPT-4's age predictions were negatively correlated with explicit results.Source: Results · pp. 8–9

General Discussion

Across four studies focused on psychological science, ChatGPT showed mixed potential to support scientific work, with GPT-4 notably improving in some areas. The discussion presents simulation of familiar results as promising but distinguishes it from predicting empirical novelty, which the authors argue remains dependent on real-world data.

Scope

  • Because the studies tested psychological science, the authors say further research should establish how well the findings generalize to other disciplines.Their psychological-science focus reflects the authors’ expertise in judging GPT’s output.Source: General Discussion · p. 9

Study 1: Research Librarian

  • GPT-4 produced fewer fictional references and more often acknowledged fiction, but it did not meaningfully improve on smaller citation errors.Acknowledgment often applied to a chat generally rather than identifying which references were false; those smaller errors may still propagate through citation systems.Source: General Discussion · p. 9
  • Both models hallucinated more with incomplete references, while GPT-4 was more successful than GPT-3.5 at finding references on moderately broad topics.The discussion treats the completeness pattern as exploratory and calls for testing whether request formality explains it.Source: General Discussion · pp. 9–10

Study 2: Research Ethicist

  • GPT-4 gave strong feedback on flawed research protocols, including subtly framed issues, whereas GPT-3.5 sometimes praised poor practices.GPT-4’s advice was aligned with modern research practices; the authors suggest it could help well-intentioned researchers improve protocols.Source: General Discussion · p. 10

Study 3: Data Generation

  • Although Study 3’s known-effect simulations suggest potential for piloting and hypothesis generation, the authors caution that its results are not decisive.They raise the possibility that GPT may self-censor socially undesirable associations, making it unreliable for data on such effects.Source: General Discussion · pp. 10–11

Implications for Novelty

  • The authors argue GPT may combine existing knowledge in novel ways to reveal discoveries about historical-data patterns, but true empirical novelty likely requires real-world data absent from its training.Source: General Discussion · p. 11

Studies 3 and 4: Data Generation and Novel Prediction

  • GPT-3.5 and GPT-4 replicated familiar word-embedding effects relatively successfully but generally failed to predict novel, unfamiliar data.The authors therefore suggest that data simulation may be useful in relatively simple domains with known or predictable results, while true empirical novelty remains beyond these models’ apparent reach.Source: General Discussion · p. 11

Study 4: Novel Data Prediction

  • Both models predicted explicit Sexuality Attitudes relatively well, but their nearly identical Implicit Sexuality Attitudes predictions suggested little additional information for the more novel task.Their explicit-attitude success was limited to Sexuality Attitudes; both failed on Age Attitudes and Gender Liberal Arts/Science stereotypes, which receive less media coverage.Source: General Discussion · p. 11

Supporting Information for:

The supporting information details how GPT-3.5 and GPT-4 were asked to locate and summarize psychology articles across 25 topics. Topic breadth was varied to examine how research availability and specificity related to discovery and hallucination.

Section S1: GPT as Research Librarian – Detailed Methods

  • For each of 25 psychology topics, both models were asked to find 20 influential peer-reviewed papers and provide citations and summaries.Prompts continued until a model supplied 20 references it claimed were real or indicated it lacked relevant knowledge; acknowledged fictional references were preserved for further analysis.Source: Section S1: GPT as Research Librarian… · p. 15
  • Topics varied in breadth and reflected the authors’ expertise in bias, discrimination, and implicit social cognition; Google Scholar searches informed selection and updates.Searches were conducted on July 12, 2023, to roughly identify how many references appeared for each topic.Source: Section S1: GPT as Research Librarian… · p. 15

2. Unconscious Biases or Stereotypes (“Unconscious Bias”: 35,300; “Unconscious Stereotypes”: 1,850)

  • The unconscious-bias and stereotype topics included automatic or unconscious processes, bias and discrimination, implicit social cognition, and IAT methodology and predictive validity.The listed search counts ranged from 450 for implicit gender stereotypes and malleability to 279,000 for psychology of bias and discrimination.Source: §2 · pp. 15–16

3. Cross-Cultural Results on Implicit Association Tests (“Implicit Association Test” AND “Cross-Cultural”: 7,160)

The supporting information lists research topics and documents coding decisions for the Research Librarian study. It defines how acknowledgments of fictional references and reference quality were handled, including why hallucinations and admissions of defeat counted as irrelevant.

3. Cross-Cultural Results on Implicit Association Tests (“Implicit Association Test” AND “Cross-Cultural”: 7,160)

  • The cross-cultural IAT topic list included implicit attitudes in law and business or workplace settings.The search phrases covered cross-cultural IAT work, implicit attitudes and the law, and implicit attitudes in business and the workplace.Source: §3 · p. 16

1. Malleability of Implicit Attitudes (“Implicit Attitudes” AND “Malleability”: 3,700)

  • The malleability topic list included IAT applications to mental illness, gender stereotypes, suicide or self-harm, marketing, and cross-cultural predictive validity.The listed topics also included the malleability of implicit gender stereotypes.Source: §1 · p. 16

4. Implicit Association Test and Intersectionality (“Implicit Association Test” AND “Intersectionality”: 1,870)

  • Intersectionality-related topics covered IAT research on medical treatment, county-level validity, dentists, pilot risk-taking, patent law, and soft-drink marketing..Source: §4 · p. 16
  • Acknowledgment of fictional references was coded at the chat level, since GPT generally did not identify particular articles as fictional.The researchers counted acknowledged chats and their hallucinated references in a further analysis.Source: §4 · p. 17
  • Two coders assessed correctness, completeness, topic relevance, and citation counts; correctness distinguished minor errors from fabricated references or major errors.Incomplete references were treated as a separate variable, not automatically as errors or hallucinations; hallucinations and admissions of defeat were automatically labeled irrelevant.Source: §4 · pp. 17–18
  • Hallucinations and admissions of defeat were coded as irrelevant because neither outcome meant GPT had located a helpful article.The coding treated these as different outcomes but assigned them the same relevance label.Source: §4 · p. 18

Section S2: GPT as Research Librarian – Supporting Analyses

GPT-4 hallucinated fewer references than GPT-3.5, while the models had similar rates of smaller citation errors. Supporting analyses also show that hallucination rates varied with citation completeness and topic breadth, and that GPT-4’s broader relevance advantage was largely tied to finding more real references.

Overall Hallucinations & Errors

  • 5.40% of GPT-4’s claimed-real references were hallucinations, versus 36.00% for GPT-3.5; smaller-error rates were similar at 4.60% and 4.20%, respectively.The hallucination difference was significant, whereas the error-rate difference was not.Source: Section S2: GPT as Research Librarian… · p. 18

Completeness as Moderator

  • 62.41% of incomplete references were hallucinations, compared with 14.30% of complete references; this pattern was significant for both models, unlike smaller errors.GPT-3.5’s incomplete-reference hallucination rate was 75.29% versus 27.95% for complete references; GPT-4’s was 39.58% versus 1.77%.Source: Section S2: GPT as Research Librarian… · p. 18

Incomplete References by Model

  • GPT-3.5 gave more incomplete references than GPT-4, and both incompleteness and model version independently predicted hallucination.For GPT-3.5, incomplete citations were less frequent on broader topics; GPT-4 showed an exploratory curvilinear pattern, while incomplete formatting often began with a chat’s first citation.Source: Section S2: GPT as Research Librarian… · p. 19

Acknowledgment of Hallucination

  • GPT-4 acknowledged 84.30% of its fictional references, compared with 12.20% for GPT-3.5, and generated fewer fictional references overall.Across 1,269 references, GPT-3.5’s fictional-reference rate was 39.05%, compared with 23.12% for GPT-4.Source: Section S2: GPT as Research Librarian… · p. 20

Effects of Topic Broadness on Hallucination

  • Topic breadth related to hallucination nonlinearly in analyses including admissions of defeat, but excluding them strengthened a linear relationship across models and for GPT-3.5.For GPT-4, the exploratory curvilinear pattern persisted after excluding admissions of defeat, but may have been driven by one chat/topic.Source: Section S2: GPT as Research Librarian… · pp. 20–21

Topic Broadness and Reference Relevance

  • GPT-4 identified more relevant articles overall than GPT-3.5, but their difference was not significant when analyses included only references labeled real.The overall rates were 50.80% versus 30.80%; among references labeled real, they were 53.70% versus 48.13%.Source: Section S2: GPT as Research Librarian… · p. 22
  • After excluding hallucinations, GPT-3.5’s success at finding relevant articles declined faster than GPT-4’s as topics narrowed, robustly across modeling choices.The model difference was significant for moderate-breadth topics after correction for multiple comparisons, but not for other breadth levels.Source: Section S2: GPT as Research Librarian… · pp. 22–23

Relevant Citation Counts

  • GPT-4 and GPT-3.5 selected relevant articles with similar citation counts, including after controlling for topic breadth and article year.A small exploratory recency difference for GPT-4 disappeared after controlling for topic breadth.Source: Section S2: GPT as Research Librarian… · p. 23

Section S3: GPT as Research Ethicist – Detailed Methods

The study tested whether GPT-3.5 and GPT-4 could identify methodological and ethical problems in fictional research protocols. The methods contrasted blatant and subtle descriptions and scored responses against a rubric of problem recognition and proposed remedies.

Vignettes and practices

  • Six fictional vignettes described four questionable practices, including significance-based stopping, post-hoc exclusions, HARKing, and reporting only selected exploratory results.Other examples included suppressing null outcomes, selecting data subsets or covariates, and using potentially collinear predictors.Source: Section S3: GPT as Research Ethicist –… · pp. 24, 26

Blatant and subtle versions

  • Three vignette pairs varied whether problematic decisions were stated blatantly or implied more subtly, with the latter designed to resemble real-world descriptions.One subtle example paired interim significance with a later sample-size increase, without explicitly saying the interim result determined collection.Source: Section S3: GPT as Research Ethicist –… · p. 24

Presentation design

Scoring rubric

  • Responses were scored with a 10-point rubric covering recognition of four problematic practices, remedies, preregistration or other firm pre-research decisions, and replication.A sufficiently clear statement of a problem could also earn credit for its remedy.Source: Section S3: GPT as Research Ethicist –… · p. 24

7. Did it draw attention to the problem of suppressing null results in reporting?

The rubric asked whether GPT recommended reporting all results as a remedy for suppressing null findings.

Null-result reporting

  • The rubric credited advice to report all results as a way to mitigate suppression of null results.This is one scoring criterion, not a reported model outcome.Source: §7 · p. 27

10. Did it suggest pre-registration or other forms of commitment to pre-decided methodological strategies?

The supplied passage labels a rubric for Vignettes 2a and 2b but gives no scoring criteria.

Vignette 2a & 2b rubric

  • The appendix identifies a separate rubric for Vignettes 2a and 2b.The passage does not state what that rubric assessed.Source: §10 · p. 27

1. Did it draw attention to the issue of ignoring or suppressing pilot studies?

The study assessed ethical feedback through rubric-coded responses to research vignettes, varying both the prompts and the methodological issues presented.

3. Did it draw attention to the issue of making inappropriate post hoc exclusions of participants?

  • The rubric asked whether GPT noticed and advised against suppressing pilot findings, making post hoc exclusions, adding subjects after nonsignificant results, or presenting marginal results as significant.Advice included reporting all outcomes, deciding exclusions in advance, using power analysis, and reporting exact p-values.Source: §1 · p. 27, §3 · p. 27

10. Did it suggest pre-registration or other forms of commitment to pre-decided methodological strategies?

  • The rubric also covered preregistration and advance commitments, including reporting all outcomes, retaining data, addressing multiple comparisons, and checking statistical assumptions.It also asked about replication and caution around generalizability.Source: §10 · p. 27
  • Two coders rated responses while blinded to model and vignette conditions; after discussion, their ratings had high reliability (α = 0.9827), with disagreement on 34 of 2,160 possible points.They clarified in advance how to credit preregistration and other advance methodological commitments.Source: §10 · pp. 27–28
  • The prompt experiment paired 18 initial prompts with six vignettes for GPT-3.5 and GPT-4, producing 216 responses scored on a 10-point rubric.Prompt variants manipulated feedback requests, researcher characteristics, and appeals to ethical or unethical responses.Source: §10 · pp. 28–29

Section S4: GPT as Research Ethicist – Supporting Analyses

GPT-4 outperformed GPT-3.5 on blatant and subtle research-ethics vignettes. Prompt analyses found stronger responses after ethics-evoking prompts, while individual prompt contrasts were generally inconclusive after correction.

Overall results

Prompt analyses

  • No individual prompt contrast reached significance after correction; a p-hacking concern prompt versus a request not to mention p-hacking was only a tentative trend.The authors caution that this contrast had a small sample and did not meet the Bonferroni threshold.Source: Section S4: GPT as Research Ethicist –… · pp. 29–30
  • Responses following prompts that negatively evoked data ethics were higher quality than those following prompts without ethics cues, but this robustness contrast did not survive Bonferroni correction.The authors describe the comparison as exploratory and interpret it cautiously.Source: Section S4: GPT as Research Ethicist –… · pp. 29–30

Non-parametric tests

  • GPT-4 also outscored GPT-3.5 on blatant vignettes (median 9 vs. 5) and subtle vignettes (median 8 vs. 4) in non-parametric tests.GPT-4’s subtle-vignette median also exceeded GPT-3.5’s blatant-vignette median (8 vs. 5).Source: Section S4: GPT as Research Ethicist –… · p. 30

Prompt analyses

  • Anti-jailbreaking prompts scored descriptively higher than jailbreaking prompts, but the contrast was not significant after multiple-comparison correction.The reported medians were 9 and 8, respectively.Source: Section S4: GPT as Research Ethicist –… · p. 31
  • Responses were stronger after prompts evoking data ethics (median 8) than after prompts without those cues (median 6), a difference significant after multiple-comparison correction.This aggregate comparison grouped varied prompts that evoked data ethics.Source: Section S4: GPT as Research Ethicist –… · p. 31

Section S5: Good Research Vignette Study

Both models identified good practices in fictional research protocols, with similarly high rubric scores. The study used two deliberately rigorous vignettes and a blinded, rubric-based coding procedure.

Study design

  • The two fictional vignettes described unusually rigorous research, including preregistration, replication, public materials, and correction for multiple comparisons.The protocols also included practices such as rival researchers replicating results and statistical consultants checking analyses.Source: Section S5: Good Research Vignette Study · pp. 31–32

Coding and analysis

  • Two coders blinded to model version scored each response on a 10-point rubric; their final coding showed high reliability (Cronbach’s α = 0.9695).Responses were aggregated by chat, and Wilcoxon Rank-Sum tests were used because results were not normally distributed.Source: Section S5: Good Research Vignette Study · p. 32

Results

  • GPT-4 identified 92.67% of rubric points for good research practices, and GPT-3.5 identified 90.42%; their performance difference was not statistically significant.Both models had a median score of 9.Source: Section S5: Good Research Vignette Study · p. 32

Interpretation

  • The authors note that the contrast with the main study could reflect easier, non-subtle vignettes, model differences in criticality, or the prompts used.They present these as competing possibilities for future research to distinguish.Source: Section S5: Good Research Vignette Study · p. 32

Section S6: GPT as Data Generator – Detailed Methods

The supporting analyses examine how GPT’s generated cultural associations align with known stereotypes and which associations drive the scores. The analyses found a divergence from prior work in the single-category patterns underlying the overall effects.

Section S6: GPT as Data Generator – Detailed Methods

  • The study prompted GPT to rate every Male/Female-target and attribute-word pairing on a 1–10 association scale, then treated responses as cosine-similarity analogues for WEAT D-score analyses.The four tasks covered gender attitudes and gender associations with Art/Science, Home/Work, and Reading/Math.Source: Section S6: GPT as Data Generator –… · pp. 32–33

Section S7: GPT as Data Generator – Supporting Analyses

  • GPT’s overall stereotype results generally aligned with prior work, but its single-category results diverged: effects were usually driven by stronger Female associations with stereotypically female attributes.Only the Art–Science effect was clearly jointly driven by Female–Art and Male–Science associations.Source: Section S7: GPT as Data Generator –… · pp. 33–34
  • Single-category WEAT D-scores can appear smaller for Female comparisons because larger within-category variance in difference scores increases the standard deviation used to standardize them.The passage illustrates this with larger Good-minus-Bad differences for some Female words than for Male words.Source: Section S7: GPT as Data Generator –… · p. 34

Section S8: GPT as Novel Data Predictor – Detailed Methods

The detailed methods describe how GPT-3.5 and GPT-4 predicted cross-country attitudes, while supporting analyses show weak or inconsistent performance on novel outcomes and probe possible reasons for better Sexuality predictions.

Section S8: GPT as Novel Data Predictor – Detailed Methods

  • The study tested three attitudes using explicit survey items and implicit tests, with five runs for each model, attitude, and measure.The source dataset covered over 2.3 million participants across 34 countries and was published after ChatGPT’s training cutoff at data collection.Source: Section S8: GPT as Novel Data… · p. 35

Section S9: GPT as Novel Data Predictor – Supporting Analyses

  • Real country-level implicit–explicit correlations differed by attitude: Sexuality was positive, Age was essentially uncorrelated, and Gender Science/Liberal Arts showed a negative trend.The analyses used country-level summaries; ChatGPT was not asked to predict feeling-thermometer items.Source: Section S9: GPT as Novel Data… · p. 36
  • For Sexuality, the authors reverse-scored one GPT-3.5 chat after follow-up clarified its opposite scoring; Age and Gender answers lacked a comparable interpretable pattern.GPT-3.5 sometimes appeared confused about its own scoring direction, including for explicit attitudes.Source: Section S9: GPT as Novel Data… · p. 36
  • GPT-3.5’s Age and Gender Science/Liberal Arts predictions were uncorrelated across runs and with real country-level patterns.The reported mean correlations with real patterns were near zero for both explicit and implicit predictions in both attitude domains.Source: Section S9: GPT as Novel Data… · p. 36
  • Implicit Sexuality predictions correlated nearly as strongly with explicit predictions as within-type predictions, despite poorer implicit accuracy.GPT-3.5’s mean correlations were 0.602 for explicit and -0.014 for implicit predictions; GPT-4’s were 0.714 and 0.152, respectively.Source: Section S9: GPT as Novel Data… · p. 37
  • Exploratory removal of East Asian countries improved both models’ Sexuality predictions, especially for implicit attitudes.The authors relate this pattern to prior reports of limitations in ChatGPT’s training data in East Asian languages.Source: Section S9: GPT as Novel Data… · p. 37
  • Three convergent checks indicated greater global coverage of Sexuality Attitudes than Age Attitudes or Gender-Science stereotypes.The checks used Bing searches, Google Ngrams, and GPT-4’s responses in 10 chats.Source: Section S9: GPT as Novel Data… · p. 38
  • GPT-4 stated that implicit and explicit attitudes often operate independently, yet its predictions showed no meaningful differences between Implicit and Explicit Sexuality Attitudes, which the authors describe as socially sensitive.Source: Section S9: GPT as Novel Data… · p. 39

SI References

The references cover language-corpus biases, social attitudes and stereotypes, implicit measures, and an international dataset of attitudes.

SI References

  • Reference 1 examines gender stereotypes across child and adult language corpora of more than 65 million words.Source: SI References · p. 39
  • References 2 and 3 address human-like bias in corpus semantics and evidence for system justification theory.Source: SI References · p. 39
  • References 4 and 6 concern gender attitudes, stereotypes, competence judgments, and automatic in-group bias.Source: SI References · p. 39
  • Reference 5 introduces the go/no-go association task, while Reference 7 describes implicit and explicit attitudes measured across 34 countries.Source: SI References · pp. 39–40
  • Reference 8 is a paper on language models trained through few-shot learning.Source: SI References · p. 40
Loading 2406.14765v1…