PaperlayerView exact revision on arXiv

Source-linked AI summary

EmpiriGraph-Psy: A Dataset and LLM Pipeline for Extracting Empirical Relation Graphs from Psychology Abstracts

Danqin Zhao, Yicun Liu, Xingwei Tan, Thomas T. Hills

arXiv:2606.08362v2cs.IR

TL;DR

Existing scientific relation-extraction benchmarks do not adequately represent variable-oriented findings in psychology. The paper introduces a normalized, typed graph task, a 210-abstract benchmark, and a staged LLM pipeline; the best pipeline achieves macro-F1 0.74, while psychology-only scope and omitted scientific components limit generalization and completeness.

  • Problem

    Existing benchmarks focus largely on computer-science entities, leaving variable-oriented empirical fields such as psychology underrepresented.

  • Method

    The paper constructs EmpiriGraph-Psy and uses a staged pipeline for variable extraction, normalization, hierarchy construction, evidence selection, relation extraction, and validation.

  • Results

    The best pipeline achieves macro-F1 0.74 for graph-level extraction, substantially improving over direct prompting and approaching human agreement.

  • Takeaways & Limitations

    EmpiriGraph-Psy provides a benchmark and extraction framework for constructing empirical knowledge graphs in psychology and broader variable-oriented fields.

  • Takeaways & Limitations

    The dataset is limited to psychology abstracts, and the annotation scheme excludes samples, methods, statistical procedures, and tasks.

Abstract

from arXiv · show

Existing scientific relation extraction benchmarks mainly target domains such as computer science, where entities are tasks, methods, datasets, materials, or metrics. This leaves a gap in variable-oriented empirical fields such as psychology, where findings are expressed as relations among constructs, measurements, interventions, and outcomes. We introduce variable-centered empirical graph extraction, the task of mapping scientific abstracts to typed graphs whose nodes are normalized variables and whose edges represent empirical and hierarchical relations. To support this task, we construct EmpiriGraph-Psy, a benchmark of 210 psychology abstracts annotated by domain-trained annotators with normalized variables, concept hierarchies, empirical relation types, and validation states. We evaluate frontier and open-weight LLMs using both direct extraction and a staged graph-construction pipeline that separates variable extraction, normalization, hierarchy construction, evidence selection, relation extraction, and edge validation. The staged pipeline substantially outperforms direct extraction, with the best configuration achieving a macro-F1 of 0.74. Error analysis shows that moderation relations and concept hierarchies remain the most challenging cases, highlighting the difficulty of extracting higher-order empirical claims and implicit abstraction structure from scientific abstracts.

1 Introduction

The paper addresses the mismatch between existing scientific relation-extraction schemas and variable-oriented empirical fields by introducing a psychology benchmark and staged graph-extraction pipeline.

  • Existing benchmarks emphasize computer-science entities such as tasks, datasets, models, and metrics, whereas psychology organizes findings around variables and their empirical relations.
  • Variable-centered extraction must normalize synonymous variable mentions and preserve relations across broad constructs and finer-grained dimensions.Flattening abstraction levels loses theoretical structure.
  • The multi-stage LLM pipeline decomposes graph construction into variable extraction, normalization, hierarchy construction, evidence selection, relation extraction, and edge validation.
  • The structure-first evaluation framework aligns predicted and gold graphs under partial node matching and measures typed edge recovery.It separates structural graph recovery from surface variation in variable naming.
  • Empirical Research Knowledge Graph Extraction maps research abstracts into typed graphs over normalized variables.
  • EmpiriGraph-Psy contains 210 psychology abstracts annotated with normalized variables, validation states, and associational, mechanistic, moderational, and hierarchical relations.
  • The staged pipeline substantially outperforms direct prompting across multiple LLMs.

2 Related Work

Related work has expanded scientific information extraction across domains and schemas, while relation discovery remains a central challenge and LLMs offer a newer extraction approach.

  • Scientific relation extraction has been used to identify and organize structured relations across large academic corpora.
  • Early benchmarks such as SemEval-2017 tagged scientific keyphrases across computer science, materials science, and physics.
  • Later work developed richer domain-specific schemas for biomedical relations, chemical reactions, and materials-synthesis procedures.
  • Existing studies report that entity recognition and type classification are comparatively tractable, whereas relation discovery remains the main bottleneck.
  • Instruction-tuned LLMs have shown competitive performance on relation extraction and scientific information extraction across materials science and biomedicine.

3 Background

The paper defines empirical relation graphs over variables and constructs, distinguishing substantive empirical edges from abstraction hierarchies and validation states.

  • The task maps a scientific document to a graph whose vertices are variables or constructs and whose edges represent their relationships.
  • Empirical edges represent associational, mechanistic, and conditional relations among variables.
  • Associational edges encode covariance or correlation without claiming a causal or mechanistic effect.
  • Mechanistic edges encode directional effects in which one variable affects, predicts, influences, or enables another.
  • Conditional relations capture moderation or interaction by encoding a third variable’s conditioning of an established relationship.
  • Each empirical edge receives a validation state: validated, null, or hypothesized.
  • The annotation scheme distinguishes higher-level constructs from specific behavioral outcomes when abstracts report relations at multiple abstraction levels.
  • Hierarchy edges connect higher-level constructs to lower-level dimensions, indicators, or measurements while retaining specific lower-level relationships.Higher-level graphs collapse these variables into more abstract theoretical relationships.

4 Dataset and Human Annotation

The paper builds a psychology-abstract corpus with broad temporal and subfield coverage, then uses trained annotators and review to produce finalized gold graphs.

  • Data Collection: The corpus samples psychology abstracts from six journals spanning different subfields and historical periods.The sampling design targets variation in relation frequency, density, and linguistic realization over time.
  • Human Annotation: Three psychology students at PhD or undergraduate level annotated the corpus after training and qualification.Coders used a customized Label Studio platform, with guidelines refined through team discussion.
  • Human Annotation: The annotation covers the full corpus of 210 abstracts, with 50 abstracts independently annotated by all three coders for reliability assessment.The remaining gold graphs were jointly covered, while the shared subset supported agreement analysis.
  • Validation: Annotator disagreements over variable boundaries, normalization, hierarchy edges, relation types, and validation labels were discussed and resolved.The reviewed annotations formed the finalized gold graphs used for evaluation.

5 Methodology

The methodology decomposes variable-centered graph extraction into staged prediction and evaluates outputs through structure-first alignment that tolerates surface variation in node names.

  • Pipeline: The pipeline sequentially performs variable extraction, normalization and hierarchy construction, evidence sentence extraction, graph construction, and edge validation.Each stage produces structured intermediate output passed to the next stage as context.
  • Pipeline: Variable extraction identifies candidate variables and normalized spans, while normalization merges near-duplicates and links higher- and lower-level variables.These steps establish a consistent vocabulary and preserve abstraction structure for later relation extraction.
  • Pipeline: Evidence sentence extraction selects relational sentences using the variable list, reducing distracting context and grounding subsequent edge decisions in textual evidence.The stage functions as an information bottleneck intended to improve precision.
  • Pipeline: Graph construction classifies associational, mechanistic, and moderational relations, followed by edge validation to correct likely false positives and relation attributes.Validation serves as a final consistency and quality-control pass.
  • Evaluation: Structure-first evaluation aligns predicted and gold graphs with partial node matching and measures recovery of directed, typed edges.The procedure separates graph structure from surface variation in variable naming and reports typed, higher-level, and type-agnostic views.
  • Evaluation: Aligned node pairs had mean embedding cosine similarity of 0.735, and 87 of 100 manually inspected pairs referred to the same variable or construct.The authors therefore describe most structure-aligned pairs as semantically valid, while acknowledging some noisy alignments.

6 Experiment

Experiments compare direct prompting with staged pipelines across multiple LLMs using structural graph evaluation. The staged approach performs better overall, while hierarchy and moderation remain difficult.

  • Experimental Design: The experiment compares direct one-step prompting, a collapsed five-stage prompt, and a full staged pipeline using GPT-5.4.The full staged pipeline executes the five subtasks separately.
  • Experimental Design: The full staged pipeline is evaluated across GPT-5.2, GPT-5.4, Claude Sonnet 4.6, Claude Opus 4.7, DeepSeek V4 Pro, Gemini 3 Flash, and GPT-4o.All outputs use the same structural graph evaluation protocol.
  • Results: 0.72 micro F1 and 0.74 macro F1 are achieved by the best GPT-5.4/GPT-5.2 configuration.The configuration uses GPT-5.4 for Steps 1 and 5 and GPT-5.2 for the remaining stages.
  • Results: 0.736 F1 is achieved by the combined GPT-5.2 + GPT-5.4 pipeline, compared with 0.530 for direct prompting.The combined configuration reaches 0.767 precision and 0.771 recall, while GPT-5.4 reaches 0.694 F1 among single models.
  • Results: Explicit decomposition improves graph extraction quality from direct prompting through chain-of-thought prompting to the staged pipeline.The reported comparison indicates improvement beyond model choice alone.
  • Error Analysis: Hierarchy has the highest false-negative rate at 27.4%, while moderation has the highest type-confusion rate at 15.3%.Hierarchy errors require implicit component-of inference, whereas moderation errors often simplify three-way interactions.
  • Error Analysis: False-negative rates are uniformly higher than false-positive rates across edge types, indicating a conservative tendency to miss rather than hallucinate relationships.The authors characterize this bias as desirable for downstream knowledge-graph construction.

7 Conclusion

EmpiriGraph-Psy introduces a psychology dataset and LLM pipeline for extracting empirical relation graphs from abstracts. Decomposing graph construction improves performance, with the best pipeline achieving F1 = 0.74.

  • EmpiriGraph-Psy is a dataset and LLM pipeline for extracting empirical relation graphs from psychology abstracts.
  • The framework represents variable-centered scientific findings as graphs for psychology and broader variable-oriented empirical fields.
  • Decomposing graph construction into variable extraction, normalization, evidence selection, relation extraction, and validation substantially improves performance over direct prompting.
  • F1 = 0.74 was achieved by the best pipeline configuration, with performance approaching human agreement and remaining stable across publication periods.

Limitations

The dataset and annotation scheme have defined scope boundaries. The corpus is limited to psychology abstracts, and the scheme excludes several scientific components needed for more complete knowledge graphs.

  • The dataset is limited to psychology abstracts, so generalization to health science, biology, and other disciplines remains unclear.
  • Future work could extend the workflow across domains and integrate it with prior scientific information extraction methods.
  • The annotation scheme focuses on empirical and conceptual relationships rather than samples, methods, statistical procedures, or tasks.

A.1 Dataset and Human Annotations

The dataset comprises selected empirical psychology abstracts annotated by trained psychology students using a customized graph-labeling interface and detailed guidelines. Agreement was assessed on a 50-article overlap set, and the released resource contains derived annotations rather than copyrighted abstract text.

  • Dataset: The final dataset includes empirical, peer-reviewed abstracts from six psychology journals, excluding reviews, letters, meta-analyses, and other non-empirical studies.
  • Human Annotations: Three psychology student annotators used a customized Label Studio interface to highlight variables, normalize names, and assign relation types.
  • Human Annotations: Annotators followed guidelines to locate variables and empirically examined relationships, including hierarchical relationships between variables.
  • Human Annotations: Coder A and C reached pairwise F1 of .82 and Cohen’s κ of .60 on the overlap set, while comparisons involving coder B were slightly lower but broadly comparable.
  • Dataset: The released resource provides metadata identifiers and derived annotation layers, not redistributed copyrighted abstract text.

A.2 Model Performance Across Journals and Periods

Model performance is stable across publication periods but varies more across journals. Journal-level F1 ranges from 0.67 to 0.81, with JCCP highest and BRT lowest.

  • Across Periods: F1 scores exceed 0.71 across publication periods, indicating stable extraction performance over time.
  • Across Journals: Journal-level F1 ranges from 0.67 to 0.81, showing greater variation across journals than across publication periods.
  • Across Journals: JCCP achieves the strongest performance with F1 = 0.807, precision 0.863, and recall 0.806.
  • Across Journals: BRT and JEP:G have the lowest reported F1 scores, at 0.669 and 0.694 respectively.

A.3 Graph Evaluation Details

The evaluator aligns predicted and gold typed directed graphs through an injective partial node mapping, then scores recovered edges under complementary structural views. A safe branch-and-bound search optimizes typed edge overlap, while preprocessing and semantic checks support consistent graph-level evaluation.

  • Preprocessing: The evaluator preprocesses completed graphs by propagating lower-level relationships to higher-level nodes and removing duplicate relationships before scoring.The stated purpose is to improve consistency across annotators and models and reduce errors from manual annotation complexity.
  • Graph representation and alignment: Gold and predicted graphs are represented as directed, typed graphs whose nodes are aligned through an injective partial mapping.The mapping permits unmatched gold nodes and prevents two gold nodes from sharing one predicted node.
  • Graph representation and alignment: The alignment objective maximizes the number of gold edges whose mapped endpoints and relation types appear in the predicted graph.A matched edge requires both endpoints to be mapped and the corresponding typed predicted edge to exist.
  • Search algorithm: Safe branch-and-bound search prunes only when an admissible upper bound cannot exceed the incumbent, preserving any alignment that could achieve a higher score.The bound may overestimate achievable matches because it ignores some joint compatibility constraints, but it does not underestimate them.
  • Evaluation views: Three evaluation views report typed directed scores, higher-level scores after removing lower-level hierarchy structure, and agnostic scores after collapsing edge types.These views separate full typed recovery from higher-level structure and type-agnostic connectivity.
  • Node validation: Semantic validation found a mean cosine similarity of 0.735 across aligned node pairs, with 87 of 100 sampled pairs judged to represent the same variable or construct.Pairs with cosine similarity at least 0.5 were almost always judged to refer to the same construct, while all sampled pairs below 0.3 were incorrectly paired.
Loading 2606.08362v2…