PaperlayerView exact revision on arXiv

Source-linked AI summary

Automating psychological hypothesis generation with AI: when large language models meet causal graph

Song Tong, Kai Mao, Zhen Huang, Yukun Zhao, Kaiping Peng

arXiv:2402.14424v3cs.AIcs.CY

TL;DR

Psychological hypothesis generation with causal graphs can be labor-intensive, and LLM-based hypothesis generation offers a different route. This study combines LLMs, a psychology causal graph, and link prediction to generate well-being hypotheses; its approach matched doctoral scholars in novelty and exceeded LLM-only hypotheses.

  • Problem

    Causal-graph methods for psychological hypothesis generation are labor-intensive and require multidisciplinary expertise, while LLMs offer a route to extracting causal knowledge from text.

    Source: Introduction · p. 2

  • Method

    The authors combine LLMs with causal graphs and link prediction to generate well-being hypotheses from psychological research.

    Source: Introduction · p. 2, General discussion · p. 10

  • Results

    The approach matched doctoral scholars in novelty and surpassed LLM-only hypotheses, with deep semantic analysis also finding broader conceptual incorporation and semantic range.

    Source: Introduction · p. 2

  • Takeaways & Limitations

    The findings support using causal graphs alongside LLMs to generate psychological hypotheses with novelty comparable to doctoral scholars.

    Source: Introduction · p. 2, General discussion · p. 10

    About 13% of extracted relationship pairs did not align with human expert estimates, and validation covered only 130 hypotheses.

    Source: General discussion · p. 12

Abstract

from arXiv · show

Leveraging the synergy between causal knowledge graphs and a large language model (LLM), our study introduces a groundbreaking approach for computational hypothesis generation in psychology. We analyzed 43,312 psychology articles using a LLM to extract causal relation pairs. This analysis produced a specialized causal graph for psychology. Applying link prediction algorithms, we generated 130 potential psychological hypotheses focusing on `well-being', then compared them against research ideas conceived by doctoral scholars and those produced solely by the LLM. Interestingly, our combined approach of a LLM and causal graphs mirrored the expert-level insights in terms of novelty, clearly surpassing the LLM-only hypotheses (t(59) = 3.34, p=0.007 and t(59) = 4.32, p<0.001, respectively). This alignment was further corroborated using deep semantic analysis. Our results show that combining LLM with machine learning techniques such as causal knowledge graphs can revolutionize automated discovery in psychology, extracting novel insights from the extensive literature. This work stands at the crossroads of psychology and artificial intelligence, championing a new enriched paradigm for data-driven hypothesis generation in psychological research.

Introduction

Psychological hypothesis generation and causal-graph methods can be labor-intensive, while LLMs offer a way to extract causal knowledge from large text collections. The study combines LLM extraction with causal graphs and link prediction, drawing on a filtered psychology literature corpus.

Introduction

  • Psychological causal-graph methods require substantial labor and multidisciplinary expertise, while LLMs offer a route to extracting causal knowledge from large text collections.The proposed combination pairs LLM semantic and conceptual analysis with the graph's global view of causality.Source: Introduction · p. 2

Methodological framework for hypothesis generation

  • The framework combines LLM-based causal concept extraction with a graph and link prediction to identify potential causal relationships and generate hypotheses.The framework includes literature retrieval, causal-pair extraction, and hypothesis generation.Source: Introduction · p. 2, Figure 1 · p. 3

Introduction

  • On 130 well-being hypotheses, the combined approach matched doctoral scholars in novelty and surpassed LLM-only hypotheses; semantic analysis found broader conceptual incorporation and range.The study also presents the approach as a bridge between theory-driven and data-centric research in psychology.Source: Introduction · p. 2

Step 1: Literature retrieval.

  • The authors curated approximately 140,000 psychology- and neuroscience-related articles using title or abstract keywords and metadata aligned with those fields.Their source was the PMC Open Access Subset, and the final article selection required journals to include “Psychol” in their titles.Source: Step 1: Literature retrieval · pp. 2–3

Step 2: Causal pair extraction.

The authors extracted and standardized causal relations from psychology articles, using GPT-4 with filtering and expert review. An exploratory evaluation found that most sampled relations were validated by graduate students.

Step 2: Causal pair extraction.

Text extraction and cleaning.

  • The authors chose GPT-4 for its contextual performance on complex scientific texts and its ability to generate coherent, relevant responses for the project.They cite its capabilities and training on diverse data as reasons for the choice.Source: Causal knowledge extraction method · pp. 3–4

Causal knowledge extraction method.

  • GPT-4 processed article segments of at most 4,000 tokens, returned causal or correlational pairs in a specified JSON format, and its outputs were filtered and checked for relevance, relation type, and direction.The prompt excluded relationships attributed to or cited from other studies; the authors also periodically sought domain-expert feedback on random samples.Source: Causal knowledge extraction method · p. 4
  • 87.54% of 289 GPT-4-identified relationships were validated; 65.55% (156/238) of relationships classified as causal were agreed to be causal by reviewers.Four graduate students evaluated relationship pairs from psychology articles they knew.Source: Causal knowledge extraction method · p. 4

Graph database storage.

The graph database captures a broad range of concepts from psychological research, spanning mental illness, well-being, neuroscience, and self-perception.

Graph database storage.

  • The extracted literature gives prominence to both mental-illness concepts, including depression and anxiety, and positive concepts such as life satisfaction and happiness.The authors describe this as a balance between health and illness in psychological research.Source: Graph database storage · p. 4
  • The concept landscape also includes neuroscience topics, self-efficacy, positive emotions, self-esteem, age, resilience, and creativity.The authors interpret these topics as evidence of psychology's broad range of inquiry.Source: Graph database storage · p. 5
  • The authors characterize the resulting view of psychological research as multidimensional, encompassing challenges and positive aspects of human experience.This characterization follows their review of the graph's recurring concepts.Source: Graph database storage · p. 5

Step 3: Hypothesis generation using link prediction.

The authors use link prediction to propose causal links missing from the graph, then ask GPT-4 to turn high-probability concept pairs into hypotheses.

Step 3: Hypothesis generation using link prediction.

  • Link prediction embeds concepts with node2vec, scores unconnected pairs, filters them against a threshold, ranks candidates using neighboring-concept Jaccard similarity, and selects high-probability pairs.The method orders candidate relationships by derived probability.Source: Step 3: Hypothesis generation using… · p. 5
  • A proposed BIS–interference link illustrates how shared connections to BAS and BAS reward response can suggest a plausible causal pair for further study.The authors present such high-probability pairs as starting points for experimental designs or research hypotheses.Source: Step 3: Hypothesis generation using… · p. 5
  • GPT-4 generates novel causal hypotheses by bridging the high-probability concept pairs identified by the method.Table 4 is described as providing example hypotheses, including an example the authors characterize as innovative.Source: Step 3: Hypothesis generation using… · p. 5

Hypotheses evaluation and results

The study evaluates hypothesis quality across AI-generated, LLM-generated, and doctoral-student ideas, using well-being as a complex, familiar focus. It compares randomly selected LLMCG hypotheses with other contributor groups.

Hypotheses evaluation and results

  • Hypothesis quality was assessed by comparing LLMCG ideas with doctoral students’ ideas, LLM output, and ideas refined by psychological researchers.The evaluation focused on novelty and utility as dimensions of inventive research ideas.Source: Hypotheses evaluation and results · p. 5

Selection of the focus area for hypothesis generation.

  • Well-being was selected for its multidimensional emotional, psychological, and social scope, its importance in positive psychology, and access to specialized Ph.D. students.The focus-area criteria also called for a complex topic familiar to human contributors without requiring expertise equivalent to the AI’s corpus knowledge.Source: Selection of the focus area for… · p. 6

Hypotheses comparison.

  • The comparison included two LLMCG groups, doctoral students, and representative LLMs as experimental and control groups.The 130 well-being hypotheses generated by LLMCG were based on likely causal relationships not previously documented in the research-literature datasets; 30 were randomly selected.Source: Hypotheses comparison · p. 6, LLMCG algorithm output… · p. 6

LLMCG algorithm output (Random-selected LLMCG).

The study constructs comparison groups from expert-selected LLMCG hypotheses, doctoral students’ ideas, and Claude-2 output. It assesses the hypotheses through expert ratings and semantic analysis.

LLMCG expert-vetted hypotheses (Expert-selected LLMCG).

  • Two experienced positive-psychology researchers selected 30 promising LLMCG hypotheses, emphasizing novelty and practical relevance.They excluded hypotheses in the random-selected LLMCG group and considered relevance, coherence, and potential academic value.Source: LLMCG expert-vetted hypotheses… · p. 6
  • Thirteen of 16 invited doctoral students returned 41 well-being hypotheses; 30 were randomly selected for comparison.Each participant was asked to develop 3–4 hypotheses, with four working days to complete the questionnaire.Source: LLMCG expert-vetted hypotheses… · p. 6

Claude model output (Control-Claude).

  • Claude-2 produced 50 hypotheses, and GPT-4 selected 30 judged most innovative, relevant, and academically valuable.Claude-2 was chosen as the LLM control; the refinement involved minimal human intervention.Source: Claude model output (Control-Claude) · pp. 6–7

Hypotheses assessment.

  • Three psychology professors blindly reviewed 120 hypotheses for novelty and utility, while BERT embeddings and t-SNE supported semantic comparison.BERT represented each hypothesis as a high-dimensional vector; t-SNE reduced dimensions while preserving relative pairwise distances for visualization.Source: Hypotheses assessment · p. 7

Results

Qualitative analyses distinguish the topics emphasized by doctoral students and LLMCG groups, while the human-evaluation analysis reports reviewer agreement and differences in perspective.

Qualitative analysis by topic analysis.

  • The Control-Human word cloud emphasized “robot” and “AI,” whereas LLMCG groups, especially Expert-selected LLMCG, emphasized community and collective experience.The authors interpret the LLMCG themes as reflecting higher-order social-psychological concepts.Source: Qualitative analysis by topic analysis · p. 7
  • Connection graphs linked Control-Human’s “Robot Companionship” theme to heart rate variability, while Random-selected LLMCG produced more societal and structural themes.The latter themes were described as pointing toward broader determinants of individual well-being.Source: Qualitative analysis by topic analysis · p. 7

Analysis of human evaluations.

  • Reviewer agreement varied: novelty ratings between reviewers 1 and 2 correlated at Spearman r = 0.387, while usefulness ratings between reviewers 2 and 3 correlated at r = 0.376.Both correlations had p < 0.0001.Source: Analysis of human evaluations · p. 7
  • Reviewer perspectives diverged on specific hypotheses: reviewers 1 and 3 rated “Virtual Resilience” higher than reviewer 2, while reviewer 3 favored the social-neuroscience idea.The passage relates the differences to reviewers’ varied research backgrounds and argues for considering multiple expert opinions.Source: Analysis of human evaluations · pp. 7–8
  • Novelty and usefulness were analyzed separately; novelty ratings were standardized by reviewer with z-scores before averaging, and Fig. 2 displays both score distributions.Its box plots show novelty and usefulness, while smoothed line plots order scores descending and apply a moving average with window size 2.Source: Novelty analysis · p. 8, Figure 2 · p. 8

Novelty analysis.

Novelty comparisons favored Random-selected LLMCG and Control-Human over Control-Claude on mean scores. Median and maximum-score analyses broadly reinforced that pattern, while mean usefulness did not differ significantly by group.

Novelty analysis.

  • Novelty differed by group (F(3116) = 6.92, p = 0.0002); Random-selected LLMCG and Control-Human exceeded Control-Claude (t(59) = 3.34, p = 0.007; t(59) = 4.32, p < 0.001).The reported Cohen’s d values were 0.8809 and 1.1192, respectively.Source: Novelty analysis · p. 8, Table 5 · p. 9
  • Median novelty scores again favored Control-Human and Random-selected LLMCG over Control-Claude, while other pairings, including Expert-selected LLMCG versus Control-Claude, were not significant.The reported comparisons were t(59) = 4.01, p = 0.001 and t(59) = 3.40, p = 0.006; median scoring was used to reduce the influence of reviewer inclinations.Source: Novelty analysis · p. 9
  • Maximum novelty scores were higher for Control-Human, Random-selected LLMCG, and Expert-selected LLMCG than Control-Claude (t(59) = 4.36, p < 0.001; t(59) = 3.47, p = 0.004; t(59) = 3.12, p = 0.014).The passage reports Cohen’s d values of 1.1637, 1.0457, and 0.6987, respectively.Source: Novelty analysis · p. 9
  • Mean usefulness did not differ significantly by group (F(3,116) = 5.25, p = 0.553), though Expert-selected LLMCG tended toward higher usefulness scores.Control-Human scores had a narrow interquartile range, while Control-Claude scores showed a wider spread and outliers.Source: Usefulness analysis · p. 9

Usefulness analysis.

LLMCG hypotheses scored higher in novelty, but not usefulness, than GPT-4-only hypotheses. Deep semantic analyses found broader topic dispersion in LLMCG and distinct semantic orientations among groups.

Comparison between the LLMCG and GPT-4.

  • t(119) = 6.60, p < 0.0001 for novelty and t(119) = 1.31, p = 0.1937 for usefulness distinguish LLMCG from GPT-4.The authors report higher novelty without a significant usefulness difference; all Cohen’s d values for novelty exceeded 1.1.Source: Comparison between the LLMCG and GPT-4 · pp. 9–10

Deep semantic analysis.

  • LLMCG groups showed broader topic dispersion than the distinct clustering among PhD students, and the authors report broader thematic coverage than human input.Their semantic analysis also describes LLMCG hypotheses as having broader complementary domains.Source: Deep semantic analysis · p. 10
  • F(3,1652) = 84.1611, p < 0.00001 for semantic distances across groups; Control-Human distances exceeded Control-Claude distances (t = 16.41, adjusted p < 0.0001; Cohen’s d = 1.1630).Control-Human also differed from Random-selected LLMCG (p = 0.0036), while its difference from Expert-selected LLMCG was a trend (p = 0.0687); the two LLMCG groups did not differ (p = 0.4362).Source: Deep semantic analysis · p. 10

General discussion

The discussion presents LLMCG as a way to combine literature-derived causal graphs with LLMs for hypothesis generation, especially around well-being. It also notes limits in graph accuracy, validation coverage, and expert evaluation.

General discussion

  • The authors report that AI-augmented methods exceed LLM-only techniques in novelty and show quality on par with human expertise, with broader semantic coverage.They center the investigation on well-being and describe the assessment as combining topic analysis, human evaluation, and deep semantic analysis.Source: General discussion · p. 10
  • LLM-based causal extraction automates causal-pair derivation and minimizes the need for extensive expert input in building psychological causal graphs.The authors describe the graphs as a platform for synthesizing psychological domains and hypothesizing across them.Source: General discussion · p. 10
  • The authors attribute LLMCG’s greater novelty than LLM-only hypotheses to a causal network curated from 43k research papers, which focuses LLM knowledge on scientific psychology.They also argue that the graph provides structure that can clarify the logic of LLM outputs.Source: General discussion · p. 12
  • About 13% of extracted relationship pairs did not align with human expert estimates, and validation was limited to 130 hypotheses.The authors also report inconsistent evaluations by senior expert panels and identify evaluation as an ongoing need for scarce expert resources.Source: General discussion · pp. 12–13

Data availability

Study data are partially available in the supplementary materials, with additional supporting data available from the corresponding author upon reasonable request.

Data availability

  • Study data are partially available in the supplementary materials; additional supporting data may be requested from the corresponding author.The article lists receipt on 8 November 2023 and acceptance on 25 June 2024.Source: Data availability · p. 13

Competing interests

The authors report no potential conflicts of interest. The article also records ethics approval and participant consent, and provides supplementary-material and licensing information.

Competing interests

  • The authors declared no potential conflicts of interest regarding the research, authorship, or publication.Source: Competing interests · p. 14
  • The study received approval from Tsinghua University’s Department of Psychology IRB, and all participants gave informed consent before participating.The approval was documented as IRB202306.Source: Competing interests · p. 14

Additional information

  • Supplementary material is available online, and the article is licensed under Creative Commons Attribution 4.0, subject to its stated terms.The article directs correspondence and material requests to Yukun Zhao or Kaiping Peng.Source: Additional information · p. 14
Loading 2402.14424v3…