Source-linked AI summary
Research Design Tracking and Assessment for the Social Sciences
Marco Rovera, Sergiu Burlacu, Dominique Cappelletti, Alessio Tomelleri, Sonia Marzadro, Martina Bazzoli, Annalisa Tassi, Jessica Gagete-Miranda
TL;DR
Reliable assessment of causal research designs matters for evidence-based policy but has traditionally depended on manual expert analysis. ARDTrA formalizes this task, builds an expert-annotated dataset, and evaluates a multi-turn RAG pipeline, finding that passage length is the main performance driver while human and machine difficulty diverge.
Problem
Assessing causal research designs and the quality of their application in social science papers is important for policy evidence but has largely remained a manual expert task.
Method
The paper defines ARDTrA, creates a 140-paper expert-annotated dataset covering six counterfactual-design families, and benchmarks a multi-turn RAG conversational pipeline.
Results
Passage length explains 52–66% of performance variance across retrieval strategies, LLMs, and embedding models, while human and machine difficulty do not align.
Takeaways & Limitations
ARDTrA provides a framework and benchmark for estimating the feasibility and current performance of automated research-design assessment.
Takeaways & Limitations
The dataset is small because its detailed annotation schema requires substantial expert effort, and its English-only coverage limits generalizability to other languages.
Abstract
from arXiv · showhide
Reliable assessment of causal research designs in the social sciences is critical for evidence-based policy-making, yet has so far relied entirely on manual expert analysis. We introduce Automated Research Design Tracking and Assessment (ARDTrA), a task that involves detecting the research design used in a paper and assessing the quality of its application. We create an expert-annotated dataset of papers covering six families of counterfactual research designs and evaluate the task using a multi-turn RAG-based conversational pipeline. Across four retrieval strategies, four LLMs and six embedding models, we find that passage length is the main driver of performance, explaining 52-66% of the variance. A per-research-design analysis also shows that human and machine difficulty do not align: the designs that prove hardest for the system are not those on which expert annotators disagree most, pointing to two independent sources of task difficulty.
1 Introduction
ARDTrA formalizes the underexplored task of identifying causal research designs in social science papers and assessing how credibly they are applied. It contributes an expert-designed framework, a 140-paper annotated corpus, and systematic RAG-based benchmarking.
- ARDTrA addresses the underexplored NLP task of identifying and assessing causal research designs in social science papers.The task matters because policy advisors need to locate studies supporting credible causal claims, while methodological quality varies across applications.
- The task requires detecting a paper’s research design and assessing the credibility, validity, and robustness of its application.Both steps involve locating and interpreting fine-grained methodological details distributed throughout long papers.
- The authors create an expert-annotated corpus of 140 scientific papers spanning applied economics, sociology, political science, and related fields.The corpus covers six families of research designs identified in the causal-inference literature.
- The paper benchmarks retrieval strategies in a multi-turn conversational RAG pipeline across chunking methods, embedding models, and LLMs.This evaluates document analysis requiring synthesis of fine-grained methodological information from long scientific texts.
2 Related Work
Prior work has used text mining and language models to track causal-inference methods in social science literature. These efforts motivate broader automated analysis of research designs across fields.
- Earlier studies tracked causal-inference methods using keyword-based text mining in economics and language models across additional social science fields.The cited work includes NBER papers, leading economics journals, finance, macroeconomics, and working-paper series.
3 Counterfactual Research Designs
Counterfactual research designs support causal effect estimation, but their credibility depends on how each design is applied in its empirical setting. ARDTrA organizes the relevant designs into six families.
- The credibility of a causal claim depends on both the method employed and the specific empirical setting in which it is applied.The paper illustrates this with a Difference-in-Differences study using New Jersey and Pennsylvania as a context-specific comparison.
- Experimental Designs include field, survey, and laboratory experiments.
- Threshold-based Designs include regression discontinuity, regression kink, and bunching designs.
- Instrumental Variables and Selection-on-Observables form two additional families, including matching, weighting, regression adjustment, and doubly robust methods.
- Difference-in-Differences and Synthetic Control Methods comprise the remaining families, including event studies, triple differences, and augmented synthetic control.
- The analytical framework is not specific to economics because its causal-inference elements are shared across empirical sciences and applied fields.Examples include psychology, medicine, epidemiology, public health, and health policy.
4 Dataset
The dataset contains 140 English-language papers sampled across social-science fields and research-design categories, annotated by domain experts using a detailed analytical schema. Expert agreement is high for design identification but lower for assessing application quality.
- Dataset construction: The dataset contains 140 English-language papers sampled from 6,554 articles across multiple economics subfields and quality tiers.The sampling spans labour, health, education, environmental, urban, and regional economics.
- Dataset construction: The corpus balances counterfactual and non-counterfactual designs and maintains coverage across research-design categories.The distribution of research designs is shown in Figure 1.
- Annotation framework: Eight domain experts annotated each paper by answering methodological questions from the ARDTrA analytical schema.The schema provides instructions and finite answer options for the full annotation task.
- Annotation framework: RD Identification uses six progressively specific questions to characterize each paper’s type, data, analysis, and research design.
- Annotation framework: Counterfactual papers proceed to RD Assessment, which contains 76 design-specific questions across six research-design families.The family-specific question sets range from 9 questions for Selection-on-Observables to 16 for Instrumental Variables and Synthetic Control Methods.
- Inter-annotator agreement: Full-task expert agreement is α=.81, with higher agreement for RD Identification (α=.92) than RD Assessment (α=.75).Per-design agreement ranges from .83 for Experimental Designs and .82 for Synthetic Control Methods to .64 for Selection-on-Observables.
5 Methodology
ARDTrA frames research-design assessment as a multi-turn RAG classification task, comparing retrieval, embedding, and language-model configurations on 140 annotated papers. The study evaluates both sparse and dense retrieval, alongside a Long-Context baseline.
- RAG-based task: ARDTrA processes one closed-ended methodological question per conversational turn using the question, answer options, and dynamically retrieved passages.Question and answer options jointly form the retrieval query because answer choices contain method-specific terminology.
- Retrieval strategies: The benchmark compares fixed and variable chunking with sparse BM25, dense, Adaptive, and Propositional retrieval strategies.Adaptive uses hierarchical indexing, while Propositional retrieval rewrites paragraphs into atomic propositions before forming topic-aware chunks.
- Experimental design: Across strategy, hyperparameter, embedding, and LLM combinations, the study conducts 164 experimental rounds over the full dataset of 140 documents.There are 82 rounds for each of the two main LLMs, distributed across BM25, Dense, Adaptive, and Propositional retrieval.
6 Evaluation and Results
Performance depends strongly on passage length and varies across retrieval strategies, embeddings, LLMs, and research designs. BM25 is most reliable after averaging over embeddings, while human and machine difficulty show no significant alignment.
- Retrieval Strategies: Adaptive reaches Full Task F1 scores of .563 for Llama-8B and .660 for Qwen-32B at its best configuration, slightly ahead of BM25.The comparison selects the top-performing embedding and hyperparameter combination for each strategy.
- Retrieval Strategies: Pearson r values between average passage length and performance are 0.779 for Llama-8B, 0.815 for Qwen-32B, and 0.727 for Llama-70B.The correlations are higher for Assessment than Identification, while gpt-5.1 shows no significant correlation with passage length.
- Retrieval Strategies: Across 30 matched-length comparisons, the average strategy difference is 1.0 F1-points, with no strategy showing a consistent advantage.The worst observed difference is 3.1 points, supporting passage length as more consequential than strategy-specific properties in these comparisons.
- Text Embeddings: bge is the best embedding model in 41.5% of cases, followed by SFR at 20.7% and mxbai at 13.4%.The broader analysis reports smaller best-versus-worst embedding gaps with larger LLMs, especially for RD Assessment.
- Research Design Analysis: TbD are consistently the easiest research designs for models to identify and assess, while BM25 shows the highest variability across research designs.BM25’s lexical matching may be more sensitive to how explicitly each design’s methodology is described.
- Research Design Analysis: None of the correlations between human agreement and model performance across research designs is statistically significant.DiD is comparatively easier for models despite lower human agreement, whereas ED shows the opposite pattern; IV is challenging for both.
- Long Context vs. RAG: Most retrieval configurations fail to outperform the Long-Context baseline across both subtasks and all four LLMs.Retrieval’s supported value is focusing the model on inspectable passages rather than consistently improving competitive performance.
7 Conclusions and Future Work
ARDTrA introduces an expert-designed framework and dataset for assessing research designs, then benchmarks retrieval strategies, LLMs, and embedding models. Passage length explains 52–66% of performance variance, while retrieval strategies perform comparably at matched lengths and stronger LLMs become less dependent on passage length.
- ARDTrA provides an expert-designed analytical framework and an annotated dataset of 140 social-science papers.
- The benchmark evaluates four retrieval strategies with four LLMs, six embedding models, and a Long Context setting.
- 52–66% of performance variance is explained by passage length rather than retrieval strategy across configurations.
- At matched passage lengths, different retrieval strategies perform comparably, although this effect diminishes with stronger LLMs such as gpt-5.1.
- Sparse BM25-based retrieval matches and often outperforms dense embedding-based strategies.
8 Limitations
The study’s main limitation is its small dataset, reflecting the intensive annotation required for long papers and subtle methodological judgments. The dataset is English-only, limiting generalizability to other languages.
- The dataset is small because annotators must closely read papers of up to 50 pages and answer subtle methodological questions.
- RD estimates should be interpreted with caution given the dataset’s limited size.
- The dataset focuses on English-language papers, limiting the generalizability of findings to other languages.
9 Use of AI Assistants
AI Assistants supported manuscript revision and the creation of complex plots and tables. The authors retain responsibility for the scientific content, experimental design, data analysis, and interpretation.
- AI Assistants supported manuscript revision and the creation of complex plots and tables.
- The authors state that all scientific content, experimental design, data analysis, and interpretation of results remain their responsibility.
10 Ethics Statement
The study analyzes published papers without human subjects, personal data, or sensitive content. Its dataset contains bibliographic metadata because most papers are copyrighted, and the authors report no ethical concerns.
- The study analyzes published scientific papers and does not involve human subjects, personal data, or sensitive content.
- The released dataset contains only bibliographic metadata because most papers are copyrighted.
- The authors report that the work raises no ethical concerns to the best of their knowledge.
A Retrieval Strategies
The retrieval-strategy analysis compares F1 scores across strategies and configurations, with Table 7 identifying the best-performing strategy for each row.
- Analysis scope: The section reports results from the retrieval-strategy analysis described in Section 6.1.
- Reported results: Full Task F1 is .640 in one reported configuration and .694 in another.
- Strategy comparison: Table 7 compares retrieval strategies using F1 scores across task and configuration rows.The table distinguishes best-embedding results from averages across six embedding models; Llama-70B and gpt-5.1 use bge-large-en-v1.5 only.
B Text Embeddings
The embedding analysis aggregates F1 performance across configurations, tasks, strategies, and language models to measure both average performance and sensitivity to hyperparameter choices.
- Reported results: Full Task variability is reported as .028 and .011 in one comparison and .024 and .010 in another.
- Variability: Embedding impact is summarized by F1-score range and standard deviation across six embedding models for each task and LLM.
- Best-result frequency: The analysis counts how often each embedding model achieves the highest F1 across 78 combinations.The combinations span 13 hyperparameter configurations, two LLMs, and three tasks; ties can prevent row totals from summing to the same number.
- Embedding comparison: Embedding models are compared using mean F1, standard deviation, and range across 13 hyperparameter configurations, two LLMs, and two tasks.
C Research Design Analysis
ARDTrA analyzes causal research designs through research-design identification and assessment, using expert-defined questions and benchmarking retrieval-based systems on the full task and its subtasks.
- Experimental analysis: The full-task experiments report F1 scores for complete runs across retrieval strategies, hyperparameters, embedding models, and LLMs.Tables 11–13 separately cover the Full Task, RD Identification, and RD Assessment.
- Benchmark reporting: The benchmark tables report configurations that outperform the Long Context baseline and identify best results by LLM and overall task performance.The final row also reports average passage length generated by each hyperparameter.
- Framework: The analytical framework separates Research Design Identification from Research Design Assessment.The framework is presented as two main tasks, RD-I and RD-A.
D.2.1 Experimental Designs (ED)
The experimental-design assessment framework records assignment, protocol, balance, attrition, estimation, clustering, robustness, and potential implementation issues, while also including questions for other counterfactual designs.
- Assignment: The framework records randomization strategy and treatment-assignment level, including individual, cluster, multiple-level, and unclear assignment.
- Protocol and sample quality: It assesses protocol registration or ethics approval, balance checks, covariate balance, and general and differential attrition.
- Estimation and inference: It records estimation methods and whether inference appropriately accounts for clustering.Listed estimation approaches include regression, matching or weighting, Bayesian, permutation-based, multilevel, and machine-learning estimators.
- Robustness: Robustness questions cover identification-assumption violations, alternative estimators, covariate choices, inference approaches, and sample exclusions.
- Potential issues: Potential issues include analyzing nonrandomized entrants, ignoring differing assignment probabilities, mismatching assigned and analyzed treatment, and post-assignment exclusions.
- Threshold-based designs: Threshold-based designs are classified by continuity framework, local randomization, running-variable structure, cutoffs, and density continuity evidence.
- Instrumental variables: Instrumental-variable assessment asks how authors support the instrument’s independence from unobserved outcome determinants.Possible support includes theoretical arguments, institutional details, balance tests, placebo tests, and robustness checks.