PaperlayerView exact revision on arXiv

Source-linked AI summary

Artificial intelligences and human scientists exhibit complementary strengths in theory building

Ke Li, Spyros I. Zoumpoulis, Phanish Puranam, Philip Parker, Matthew Eshbaugh-Soha, Izzy Gainsburg, Michael Gilead, Igor Grossmann, Britt Hadar, Yoel Inbar, Almog Simchon, Robb Willer, Rui Ai, Ruicheng Ao, Gavin J. Bala, Matthew Bidwell, Shuang Cai, Kai Chang, Skyler Y. Chen, Cory J. Clark, Irmak Dai, Abhinandan Dalal, Connor Douglas, Alexis Du, Zhehang Du, Leyun Feng, Isabel Fernandez-Mateo, Linnea Gandhi, Cyrille Grumbach, Anmol Gupta, Vansh Gupta, Maria Hademer, Jay H. Hardy III, Chen Kai Huang, Jacob Xiangyu Jin, Ufuk Keskin, Na Hyun Kim, Mert Kobaş, Byounghoon Koh, Gabrielle Lamont-Dobbin, Gregory Lanzalotto, Sun Young Lee, Dingzhe Leng, Chenjun Li, Weiyuan Li, Zeyuan Li, Zhongyuan Liang, Ning Liu, Peihong Liu, Yuhan Liu, Jiuyao Lu, Wanteng Ma, Nicolas Martinet, Natnael Mulat, Christina A. Nguyen, Khai Nguyen, Quang Minh Nguyen, Naja Pape, Chanwoo Park, Stefanos Poulidis, Jeffrey Sanchez-Burks, Michael Schaerer, Isabelle Solal, Yanbo Song, Junghyo Sun, Qingyao Sun, Rui Sun, Roderick Swaab, Kevin Tan, Dequn Teng, Michelle A. Vaccaro, Robin Vigerbaeck, Xiaomeng Wang, Randol H. Yao, Duygu Yilmaz, Shun Yiu, Ecem Yucesoy, Allen Zang, Ruijia Zhang, Xilan Zhang, Yichi Zhang, Zhanhao Zhang, Eric Luis Uhlmann

arXiv:2609.32562v1cs.AIcs.CYcs.HC

TL;DR

The paper examines whether large language models can perform high-level social-science theory tasks alongside human scientists, amid uncertainty about AI’s ability to reason about complex social processes. It compares human and AI theorists on inequality-discourse predictions and theory formulation, then finds contrasting strengths in AI elaboration and human diversity.

  • Problem

    The study addresses how AI compares with human scientists at theory building and prediction in complex social domains, where LLM performance on creative explanations of human behavior remains uncertain.

    Source: Introduction · pp. 4–5

  • Method

    The study compares 13 senior scientists, 60 doctoral students, and 25 LLMs predicting and explaining machine-learning patterns in race- and gender-inequality discourse, then examines theories and revisions after evidence exposure.

    Source: Introduction · p. 5

  • Results

    GenAI theories were more structurally complex than human theories, while human theories were more semantically diverse; complexity was associated with perceived quality but generally not prediction accuracy.

    Source: §R.2 · p. 10, §R.5 · p. 15

  • Takeaways & Limitations

    The paper proposes hybrid teams that combine human diversity and error-sensitive revision with GenAI’s complexity handling and willingness to abandon prior theories.

    Source: Limitations and future directions · p. 18

    The ML benchmark captures correlations rather than causal relationships, and the sample, though large, is not comprehensive.

    Source: Introduction · p. 5

Abstract

from arXiv · show

We investigate the effectiveness of artificial intelligences (AI)-specifically large language models (LLMs)-relative to human scientists at high-level cognitive tasks in social science such as theory formulation, predictions of novel empirical results, and theory revision in response to new evidence. The research domain was academic discourse regarding gender and race inequality. Our findings, comparing 25 LLMs with 13 senior researchers and 60 doctoral scholars, reveal that the AIs outperformed most humans individually on most of the present tasks, while human theories were more diverse and exhibited greater gains in predictive accuracy from aggregation. AI-generated theories were more extensively elaborated, involving additional theoretical paths and latent variables, and were rated as higher quality than human theories by independent raters blinded to source. However, this theoretical complexity was in part ornamental, in that it was not associated with more accurate predictions about empirical patterns in data; in contrast, human scientists achieved greater predictive efficiency with simpler theories. The AIs were significantly more likely than human scientists to revise their theories to incorporate new evidence; human scientists updated their beliefs in a selective way that is sensitive to prior prediction errors. We speculate that the superior processing capacity of artificial intelligences makes them especially well-suited to tasks requiring grappling with complexity, but that the greater diversity of human ideas is essential to wise crowds and collective creativity.

Introduction

The study asks how LLMs compare with human scientists in building and revising social-science theories, addressing skepticism about AI’s ability to reason about complex human behavior. It tests predictions and theories against empirical patterns in academic discourse on race and gender inequality.

Introduction

  • The study tests whether AI can match or exceed human social scientists at creative theory-building tasks in the complex social domain.The motivation includes concerns that LLMs may produce impressive but false theories and may struggle with opaque human processes and interacting variables.Source: Introduction · pp. 4–5
  • The crowdsourced study compares 13 senior scientists, 60 doctoral students, and 25 LLMs on race- and gender-inequality discourse.Theorists predicted and explained machine-learning benchmarks, then revised their accounts after seeing the results.Source: Introduction · p. 5
  • The approach evaluates fully articulated theories through their structure, independent ratings, predictive alignment, and revision after new evidence.The benchmark identifies robust correlations, not causal relationships; the sample is large but not comprehensive, and disagreement with the benchmark does not by itself make a theory scientifically valueless.Source: Introduction · p. 5
  • The paper frames processing capacity, emotional detachment, and intellectual diversity as potential sources of complementary human and AI strengths.It suggests AI may handle complexity and conflicting evidence, while human diversity can support atypical insights and wise crowds when predictions are aggregated.Source: Introduction · p. 6
  • The study directly compares human theorists and LLMs on predicting and explaining correlational patterns in inequality discourse.It aims to identify their comparative advantages for social-science theory building and future human–AI collaboration.Source: Introduction · pp. 4, 6

R.1 Humans and generative AI exhibit complementary strengths in making predictions about correlational patterns

Against machine-learning benchmarks of correlational patterns, generative AI matched or exceeded individual humans on some prediction tasks, while human crowds generally gained more from aggregation. Human and AI predictions also showed distinct error patterns, and expertise mattered most on a difficult interaction task.

R.1 Humans and generative AI exhibit complementary strengths in making predictions about correlational patterns

  • For main effects, GenAI matched humans on racial inequality (0.416 vs. 0.411, P = 0.820) and outperformed them on gender inequality (0.624 vs. 0.578, P = 0.026).Fig. 1 plots individual contributors’ cosine similarity to the ML benchmark across main effects and interactions for both inequality domains.Source: §R.1 · p. 7, Figure 1 · p. 8
  • GenAI scored 0.400 versus 0.128 for humans on racial-inequality interaction predictions, while both groups performed near chance on gender-inequality interactions.The interaction task required selecting combinations among 78 possible feature pairs and predicting their joint-effect directions.Source: §R.1 · p. 7, Figure 1 · p. 8
  • Aggregated human predictions outperformed GenAI on racial-inequality main effects (0.584 vs. 0.475, P = 0.023), and human crowds generally gained more from aggregation across equal-sized resamples.Human prediction and error diversity were greater across all four task-by-effect comparisons; human crowds were on average 86% more diverse, and greater heterogeneity was associated with larger aggregation gains.Source: §R.1 · p. 8, Figure 2 · p. 9
  • Both humans and GenAI often missed cross-domain predictors and favored salient relationships unsupported by the benchmark; they also sometimes assigned the wrong direction to a selected interaction.These error patterns are descriptive and exploratory; for example, 82% of selectors in the racial task and 100% in the gender task gave social_science ∗ female the opposite sign from the ML benchmark.Source: §R.1 · p. 9
  • Expertise and seniority showed little difference on main effects but some advantage on interactions, with topic experts significantly outperforming non-experts on gender-inequality interactions.The authors caution that this pattern cannot support a general conclusion from the small set of tasks and calls for confirmatory studies.Source: §R.1 · p. 10

R.2 GenAI theories were more complex, whereas human theories were more diverse

GenAI theories were structurally more elaborate than human theories, while human explanations were more diverse across tasks and phases. Human explanations occupied a broader semantic space and formed a longer tail beyond a concentrated AI core.

R.2 GenAI theories were more complex, whereas human theories were more diverse

  • GenAI theories contained more causal paths, latent variables, and longer chains, and their explanations were roughly two to three times as long as human explanations.Before ML evidence, GenAI diagrams typically had about seven paths and five latent variables, versus about five paths and two latent variables for humans; GenAI theory lengths were approximately 270–300 versus 120–130 words.Source: §R.2 · p. 10
  • Human theories were more semantically diverse across all four tasks and both phases, with greater within-group dispersion than GenAI explanations.Human explanations were about 2.4 times farther from their group centroids and had about 2.2 times greater pairwise distances; PhD students and senior scientists did not significantly differ in dispersion.Source: §R.2 · p. 11
  • HDBSCAN classified only 9.6–31.5% of human explanations in the semantic core, compared with 92–100% of GenAI explanations.Human explanations formed a sparse core with a long tail, while GenAI explanations concentrated in a dense core; the core-fraction difference was qualitatively robust across minimum cluster sizes of 2–10.Source: §R.2 · p. 11

Revision after exposure to evidence reduced human theory diversity, but human theories

Exposure to machine-learning evidence reduced human theories’ semantic diversity, but did not erase their greater dispersion or long-tail structure relative to GenAI. The figures depict contributors’ theory locations and changes, alongside measures of within-group dispersion before and after evidence exposure.

Revision after exposure to evidence reduced human theory diversity, but human theories

  • After ML evidence, human explanations had lower within-group dispersion, yet remained more dispersed than GenAI theories, with 68.5–78.1% still outside the semantic core.Human centroid distances declined by an average of 12.6% and pairwise distances by 11.5%; GenAI dispersion remained largely unchanged.Source: Revision after exposure to evidence… · p. 11
  • The PCA panels show theory embeddings by contributor group across pre- and post-ML phases; open circles mark pre-ML positions, filled circles post-ML positions, and lines connect each contributor’s trajectory.The projections cover race- and gender-inequality tasks.Source: Figure 3 · p. 12
  • Fig. 4 compares centroid and pairwise cosine distances before and after evidence exposure, with separate panels for all humans versus GenAI and for PhD students versus senior scientists.Higher values indicate greater within-group semantic dispersion; bars show means and error bars 95% confidence intervals.Source: Figure 4 · p. 12, Revision after exposure to evidence… · p. 13

R.3 Human updating was more sensitive to prior errors, whereas GenAI revised more extensively

GenAI systems revised their theories more extensively after seeing ML evidence, while human revisions were more closely tied to prior prediction errors. Figure 5 compares revision magnitude and its relationship to prior accuracy.

R.3 Human updating was more sensitive to prior errors, whereas GenAI revised more extensively

  • GenAI systems showed 42% larger pre- to post-ML semantic revisions than human contributors (0.145 vs. 0.102; P < 0.001).The measured revision was cosine distance between pre- and post-ML explanation embeddings.Source: §R.3 · p. 13
  • Among humans, lower pre-ML accuracy predicted larger revisions, whereas no comparable relationship appeared for GenAI systems.The human OLS slope was −0.098 and Spearman ρ was −0.31; both Ps < 0.001, compared with GenAI OLS β = 0.008 and Spearman ρ = 0.08, both Ps ≥ 0.05.Source: §R.3 · p. 13
  • Figure 5a reports group mean revision as cosine distance with bootstrap 95% confidence intervals; panels b–c plot accuracy against revision, and panel d models the error–revision relationship.Panel d includes task-by-effect fixed effects and contributor-level random intercepts; its human slope is 0.105 (P < 0.001), while the interaction is −0.091 (P = 0.003).Source: §R.3 · p. 13, Figure 5 · p. 14

R.4 Human and LLM independent raters agreed in rating GenAI theories more highly

Independent evaluators agreed in their assessments, and GenAI theories received higher quality ratings than human theories. After ML evidence exposure, human-theory ratings rose while GenAI ratings remained largely unchanged.

R.4 Human and LLM independent raters agreed in rating GenAI theories more highly

  • Human and LLM theory-quality ratings were strongly correlated (Pearson r = 0.82; Spearman ρ = 0.82; both Ps < 0.001).The ratings were blinded to theory provenance and whether theories preceded or followed ML evidence exposure.Source: §R.4 · p. 14
  • GenAI theories were rated significantly higher than theories by PhD students, senior scientists, and topic experts across human and LLM evaluation (all Ps < 0.001).Human-evaluator ratings covered a random subset, while the LLM evaluator rated the full theory set.Source: §R.4 · p. 14
  • After ML evidence exposure, ratings of human-authored theories increased (human evaluators P = 0.019; LLM evaluation P = 0.020), while GenAI-theory ratings remained largely unchanged.Source: §R.4 · p. 15

R.5 Theory complexity is associated with perceived quality but not predictive accuracy

Theory complexity was associated with perceived quality but generally not with prediction accuracy; humans achieved higher prediction accuracy per unit of complexity. The paper situates these findings in a crowdsourced comparison of scientists and generative LLMs.

R.5 Theory complexity is associated with perceived quality but not predictive accuracy

  • Across four complexity measures, complexity–accuracy associations were small and generally non-significant for humans and GenAI, with confidence intervals spanning zero.The exception was the complexity–GenAI interaction for maximum path length (β = −0.04, P = 0.025); more causal paths and words predicted higher LLM-assessed quality.Source: §R.5 · p. 15
  • Humans achieved significantly higher prediction accuracy per unit of complexity than GenAI on all four measures (Ps ≤ 0.006) and higher quality per word (0.051 vs. 0.032, P < 0.001).Source: §R.5 · p. 15

Discussion

  • The crowdsourced study asked human scientists and GenAI systems to build a priori theories and predict empirical relationships, then explain observed patterns after receiving results.The crowd approach enabled comparisons at individual and collective levels.Source: Discussion · p. 15

Possible AI advantages in theoretical reasoning over complexity

Individual LLMs sometimes outperformed human scientists, but their more elaborate theories were not necessarily more accurate, and human contributions brought greater diversity. The discussion highlights both the limits of the task evidence and the possible value of diverse human crowds.

Possible AI advantages in theoretical reasoning over complexity

  • Individual LLMs matched or outperformed the average human scientist on several challenges, and GenAI accuracy for racial-inequality interactions was approximately three times that of humans.AI theories were preferred by blind expert raters and more elaborated, but this complexity was not accuracy-enhancing.Source: Possible AI advantages in theoretical… · p. 16
  • The four race- and gender-inequality tasks cannot establish a general complexity gradient, and the largest AI advantage appeared in only one of two interaction tasks.The authors leave possible moderating effects of task difficulty and expertise for confirmatory investigation.Source: Possible AI advantages in theoretical… · p. 16
  • GenAI revised more readily, but revision magnitude was largely unrelated to initial accuracy; the authors speculate this may reflect contextual regeneration rather than error-calibrated belief updating.They also note that LLMs may lack an explicit belief state and instead follow a stateless, prompt-driven revision process.Source: Possible AI advantages in theoretical… · p. 16
  • Humans were more predictively efficient across four complexity measures, and cognitive limits may regularize complexity that does not improve accuracy.Source: Possible AI advantages in theoretical… · p. 17
  • The updating comparison was a conservative test because contributors had only recently formulated their theories; active scientists may find abandoning career-invested published theories harder.The authors caution that calling AI comparatively open-minded may anthropomorphize text generation.Source: Possible AI advantages in theoretical… · p. 17

The benefits of human intellectual diversity

  • Human theories were more semantically diverse, and that heterogeneity helped human predictions benefit more from aggregation than AI predictions.For race-task main effects, aggregated human predictions were more accurate than aggregated LLM predictions.Source: The benefits of human intellectual… · p. 17
  • Human crowds gained more from aggregation even at equal crowd size; the authors suggest overlapping pretraining knowledge may leave less independent information for each additional LLM.They identify diversity in the scientific community, rather than any individual theorist, as a durable human strength.Source: The benefits of human intellectual… · p. 18
  • The value of diversity may depend on the problem: social science may have multiple high-quality answers, while some formal and natural sciences may have a single correct answer.The authors call for empirical study of diversity in theory tasks with multi-peaked theoretical landscapes.Source: The benefits of human intellectual… · p. 18

Limitations and future directions

The discussion proposes human–AI teams that combine AI’s capacity for complexity with human diversity and error-sensitive revision. It also cautions that the findings may be specific to this task environment and calls for broader, incentivized comparisons.

Limitations and future directions

  • Human–AI teams could route simple predictions to non-experts and GenAI, reserving scarce expert attention for harder problems that neither reliably solves.Because complex explanations may build on mechanisms for simple effects, the authors also suggest iterative collaborations with experts involved throughout.Source: Limitations and future directions · p. 18
  • Hybrid teams could combine human tail exploration and error-sensitive revision with GenAI’s complexity handling and willingness to abandon prior theories.The authors propose maximizing cognitive diversity to strengthen aggregation of scientific predictions and judgments.Source: Limitations and future directions · p. 18
  • A “one AI, many humans” team is a testable future prediction, but its benefits must be weighed against coordination costs.The authors also suggest testing whether collaboration preserves the distinct strengths of human and AI contributions.Source: Limitations and future directions · p. 19
  • Human effort may partly explain apparent AI advantages, despite remaining scientists spending an average of 2.69 hours on the survey.The authors recommend incentivized comparisons that manipulate rewards for predictive accuracy and independent quality ratings.Source: Limitations and future directions · p. 19
  • This is an N = 1 case study, and its fixed-variable text-data task and structured format may favor LLMs over other theory-building settings.Only diversity among the candidate mechanisms was directly measured; none was independently manipulated.Source: Limitations and future directions · p. 19
  • Future meta-scientific RCTs should randomly assign workflow stages to AI or humans and assess productivity, quality, and intellectual diversity.The authors encourage further studies with more advanced AI and additional research domains.Source: Limitations and future directions · p. 20

Extended Data

The Extended Data documents the survey sequence, machine-learning benchmarks, prediction comparisons, and measures of theory structure and quality. Its figures and tables also describe robustness checks and analyses of contributor characteristics and aggregation.

Extended Data

  • The survey had contributors generate theories and predictions before ML evidence, then revise or retain theories; race and gender task order was randomized.Human and GenAI contributors completed identical questions.Source: Figure 1 · p. 21
  • Extended Data Tables 1 and 2 report group prediction accuracy and aggregation, alongside prediction and error diversity and their associations with aggregation gains.Table 1 covers main effects and second-order interactions; Table 2 uses separate models by task and effect type.Source: Table 1 · p. 22, Table 2 · p. 25
  • The racial- and gender-discourse benchmarks rank predictive features, estimate logistic-regression coefficients, and assess classification metrics on held-out data.For each benchmark, panels a–b aggregate 1,000 model runs; panel c reports threshold-dependent test-set metrics.Source: Figure 2 · p. 23, Figure 3 · p. 24
  • Extended Data Fig. 4 shows how often theorists selected main effects and interactions and, for benchmark-selected items, the share predicting the benchmark direction.Panels distinguish racial versus gender discourse and main effects versus second-order interactions.Source: Figure 4 · p. 26
  • Extended Data Table 3 estimates associations of gender, expertise, and seniority with predictive accuracy across task–effect cells for human scientists.It reports OLS coefficients with HC1 confidence intervals for N = 73.Source: Table 3 · p. 27
  • Extended Data Table 4 describes theory-complexity comparisons, complexity–accuracy and complexity–quality models, and performance-per-complexity ratios.The caption specifies phase-matched complexity for quality ratings and pre-ML complexity for prediction accuracy.Source: Table 4 · p. 28
  • Extended Data Fig. 5 estimates core fractions by group, topic, and effect condition and tests sensitivity to HDBSCAN minimum cluster size.The caption identifies min_cluster_size = 2 as the primary-analysis setting.Source: Figure 5 · p. 29
  • Extended Data Fig. 6 displays agreement between human and LLM quality ratings, pre- and post-ML comparisons, and quality-score distributions.It specifies bootstrap confidence intervals and Welch’s t-tests for the comparisons.Source: Figure 6 · p. 30

Methods

The study compared human scientists and generative AI systems on predicting empirical patterns, constructing theories, and revising explanations after machine-learning evidence. Its methods combined crowdsourced tasks with machine-learning benchmarks and analyses of prediction accuracy, theory structure, and semantic change.

M.1 Study design

  • Contributors predicted effects for race and gender inequality, explained their predictions, drew causal diagrams, and could revise both after seeing machine-learning evidence.The study included 13 senior scientists, 60 PhD students, and 25 generative AI models; task order was randomized and instructions were identical.Source: §M.1 · p. 31

M.2 Discovering empirical patterns using machine-learning algorithms

  • The benchmark corpus used keyword dictionaries to identify inequality papers and a shared negative set, then ranked predictors across repeated random-forest models.Separate racial- and gender-inequality datasets comprised approximately 80,000 and 95,000 papers, respectively; logistic regression estimated directions for selected predictors.Source: §M.2 · p. 32

M.3 Analyses of scientific predictions

  • Prediction accuracy was cosine similarity between each contributor’s signed feature predictions and the machine-learning benchmark, with scores from −1 to 1.Main-effect vectors represented 13 features; interaction predictions used 78 pairwise combinations. Aggregation gain was aggregated accuracy minus mean individual accuracy, and diversity analyses examined prediction and error heterogeneity.Source: §M.3 · pp. 33–34

M.4 Analyses of theories

  • Theory analyses measured diagram complexity, embedding-based semantic diversity, core–tail structure, and pre-to-post evidence revision.Complexity included causal paths, maximum path length, latent variables, and word count; revision magnitude was cosine distance between explanation embeddings, with associations tested against prior prediction accuracy.Source: §M.4 · pp. 34–35

M.5 Theory evaluation

The study assessed theory quality through blind ratings by human experts and an LLM, using a shared multidimensional rubric. Analysis code and reproduction instructions were made publicly available.

M.5 Theory evaluation

  • Two expert human raters and an LLM evaluator rated anonymized, randomly ordered theories blind to contributor type and study phase.Overall quality averaged scores for clarity and coherence, causal reasoning, theoretical depth, creativity, and persuasiveness.Source: §M.5 · pp. 35–36

M.7 Code availability

  • The study’s public repository contains analysis code for survey data and machine-learning benchmarks, plus instructions for reproducing the analyses.The repository is identified at https://github.com/KeLiChloe/human-ai-theory-building.Source: §M.7 · p. 36

S.1 Deviations from the preregistered analysis plan

The final analyses refined preregistered constructs and expanded some planned analyses, while changing the operational measures of prediction accuracy and mechanistic complexity. One preregistered analysis of theory–test pairs was not reported and was reserved for a follow-up paper.

S.1 Deviations from the preregistered analysis plan

  • The preregistered plan covered prediction performance, theory complexity and structure, blind quality evaluation, and moderator analyses; the final manuscript also added analyses and refinements.The study was preregistered before data collection.Source: §S.1 · p. 42
  • Final measures operationalized prediction accuracy with signed cosine similarity and theory complexity with diagram features and explanation word count.Theory diversity used embedding-based distances and HDBSCAN core–tail structure; quality evaluation expanded to pre- and post-ML theories, PhD-student theories, and an LLM evaluator.Source: §S.1 · p. 42
  • Additional analyses examined crowd aggregation, diversity and aggregation gain, prediction errors, revision in relation to prior accuracy, and links between complexity, accuracy, and quality ratings.These analyses were not specified in the preregistration.Source: §S.1 · p. 42
  • The final prediction measure replaced separate feature-overlap, rank-correspondence, and sign-agreement analyses with signed cosine similarity.For mechanistic complexity, the manuscript reports diagram-based measures and word count rather than a separate coding of narrative explanations.Source: §S.1 · p. 42
  • The planned analysis of empirical support for contributor-generated theory–test pairs was not reported because it will be addressed in a follow-up paper.Source: §S.1 · p. 43

S.2 Machine-learning empirical benchmarks

The empirical benchmarks classified papers using expert-generated keyword dictionaries and compared inequality papers with a shared negative set. Thirteen preregistered features represented publication domains, author characteristics, and lagged attention to inequality discourse.

S.2.1 Dictionaries of keywords for labeling papers

  • The racial- and gender-inequality dictionaries supplied keyword lists for labeling papers, including terms related to discrimination, disparities, and equality.The gender dictionary contained 63 words and the racial-inequality dictionary contained 57 words.Source: §S.2.1 · p. 43

S.2.2 List of features

  • The 13 features covered publication domain, author-team composition, and recent inequality attention in news and academic records.Author features included estimated gender and racial composition and diversity; time-lagged attention measures summarized the preceding three years.Source: §S.2.2 · pp. 44–45

S.2.3 Model specifications

The section specifies how random-forest feature rankings and logistic-regression predictor directions and confidence intervals were produced. Both model types used cross-validated hyperparameter tuning.

S.2.3 Model specifications

  • Random forests were tuned by three-fold cross-validation for macro-averaged F1 and fitted on ten reshuffled, approximately equal subsets in each of 100 repetitions.The grid varied tree count, depth, split and leaf sample minima, and features considered per split.Source: §S.2.3 · p. 45
  • Random-forest predictors were ranked by mean decrease in impurity, with votes for appearing among each run’s five highest-ranked features and ties broken by mean MDI.Predictors were ordered by total votes.Source: §S.2.3 · p. 46
  • Logistic regression was tuned by stratified five-fold cross-validation for ROC-AUC; predictor directions used mean coefficient signs, with 95% confidence intervals based on across-run standard errors and t critical values.The grid varied solver, maximum iterations, and inverse regularization strength C; models and tuning used scikit-learn.Source: §S.2.3 · p. 46

S.3.1 Human sample demographics

This section documents the human and GenAI samples, survey tasks, and methods used to study theories and predictions about inequality. It covers participants’ responses to machine-learning evidence, theory revision and proposed tests, as well as demographic and professional background information.

S.3.1 Human sample demographics

  • The final human sample included 13 senior and 60 junior academics after one participant from each group was excluded for completing the survey unusually quickly and providing an extremely brief theory.The excluded participants completed the survey in 10–15 minutes or less, whereas other contributors took hours.Source: §S.3.1 · p. 46

S.3.2 GenAI model contributors

  • The GenAI group comprised 25 large language model configurations that completed the same survey tasks as the human contributors.The listed configurations spanned Claude, ChatGPT, Gemini, Qwen, DeepSeek, Grok, and Mistral; Theorista was a custom GPT designed as a social-science theory-building partner.Source: §S.3.2 · p. 49

S.3.3 Potential exposure of GenAI systems to the underlying literature

  • Although models may have encountered some source literature during pretraining, they received no benchmark data, browsing, or external tools for prediction.They predicted the most predictive main effects and pairwise interactions, and their expected directions, from candidate feature definitions; the benchmark was independently constructed, though indirect familiarity with scholarly regularities cannot be ruled out.Source: §S.3.3 · p. 49

S.4.1 Extraction of theory-complexity measures for diagrams

  • Diagram coding counted explicit causal paths, maximum arrows in any path, and unique non-feature, non-outcome concepts as latent variables.The coding instructions distinguished the 13 official observed features and outcome from additional mediators, mechanisms, and theoretical constructs.Source: §S.4.1 · pp. 50–51
  • After adjudication, human and LLM diagram codes agreed completely for diagrams judged codable by the human coder.Initial exact agreement was 18/20 for paths, 19/20 for maximum path length, and 17/20 for latent variables; disagreements involved an unreadable diagram, omitted mediated paths, and latent-variable counts.Source: §S.4.1 · p. 52

S.4.2 LLM theory evaluation

  • An LLM rated each theory independently on five 1–10 dimensions, with overall quality defined as their unweighted mean.The dimensions were clarity and coherence, causal reasoning, theoretical depth, creativity, and persuasiveness; theories were anonymized, shuffled, and scored in batches, with malformed identifier responses retried.Source: §S.4.2 · pp. 52–53

Dataset Description

  • The survey used a broad academic-paper corpus and expert-generated inequality keywords to label papers positive when titles or abstracts contained a keyword, and negative otherwise.The dataset was sampled from the Semantic Scholar Academic Corpus, with approximately 150,000 papers.Source: Dataset Description · p. 56

Overview of Your Tasks

  • Contributors forecast racial- and gender-inequality mentions, selecting five predictive features, ranking them, and assigning each a positive or negative effect sign.The survey framed the task as predicting whether academic papers discuss inequality; respondents were asked not to use AI tools, and eligible contributors were offered co-authorship on the first resulting publication.Source: Instructions · p. 55, Authorship · p. 55, Overview of Your Tasks · p. 56

Q Race.4–Q Race.5

  • For racial-inequality theories, respondents explained feature importance and represented causal paths with arrows and signed effects.They were asked to explain their assumptions in a narrative paragraph and indicate the direction and sign of causal relationships.Source: Q Race.4 · p. 58, Q Race.5 · p. 58

Q Race.6–Q Race.10

  • For racial-inequality predictions, respondents selected and ranked three feature pairs as important interactions, assigned each a sign, and explained their reasoning.Each interaction was defined by two features; respondents indicated whether the predictors reinforced or weakened one another’s influence.Source: Q Race.6 · p. 59, Q Race.7 · p. 59, Q Race.8 · p. 60

Predicting Mentions of Gender Inequality - Phase 1: Human Forecasting

  • Human forecasting began without access to machine-learning patterns and asked participants to predict whether papers discussed gender inequality.A paper counted as discussing gender inequality when its title or abstract contained at least one gender-inequality keyword.Source: Predicting Mentions of Gender… · p. 61

Q Gender.1–Q Gender.4

  • For gender-inequality predictions, respondents selected five features, ranked them, indicated effect signs, and explained their beliefs.A positive sign meant higher feature values were associated with a higher probability of a paper discussing gender inequality; a negative sign meant a lower probability.Source: Q Gender.1 · p. 61, Q Gender.2 · p. 61, Q Gender.3 · p. 62

Q Gender.5–Q Gender.9

  • For gender-inequality theories, respondents diagrammed signed causal paths and selected three important feature-pair interactions, signing and explaining them.The interaction questions asked respondents to rank the first, second, and third most important pairs.Source: Q Gender.5 · p. 62, Q Gender.6 · p. 64, Q Gender.7 · p. 64

Predicting Mentions of Racial Inequality - Phase 2: Integrating Machine Learning Evidence

Q Race.16

  • For racial inequality, participants proposed tests of revised theories using the 13 available features, with publication year spanning 2000–2020 available for time-based events, and could suggest additional variables.They were asked to describe results that would support their theories.Source: Q Race.16 · pp. 70–71

Machine Learning Evidence (1) - Main Effects

Q Gender.16

  • For gender inequality, participants proposed tests of revised theories using the 13 available features, with publication year spanning 2000–2020 available for time-based events, and could suggest additional variables.They were asked to describe results that would support their theories.Source: Q Gender.16 · pp. 77–78

Final Step

  • The final survey questions asked participants to predict which human or GenAI theory sets would receive the highest blinded quality ratings and strongest empirical support.The listed sets included human-expert theories after reviewing machine-learning results, GenAI theories after reviewing those results, and GenAI theories after also reviewing revised human theories.Source: Final Step · p. 79

Demographic Information

  • The survey collected age, gender, birth and residence regions, English experience, doctoral-degree timing, academic field, job rank, inequality-related publication counts, and methods-course teaching experience.Source: Demographic Information · p. 80

Co-authorship

  • Completers could opt to be listed as manuscript co-authors and, if they opted in, provide their preferred name and institutional affiliation.Source: Co-authorship · p. 80
Loading 2609.32562v1…