Research Design Tracking and Assessment for the Social Sciences

1 Introduction

• Automating design identification and assessment could help policy advisors find studies with credible causal claims, a process social scientists have so far performed manually.Differences in how a design is applied can affect the quality and robustness of the evidence.Source: §1 · p. 1
• ARDTrA detects the research design used in a paper and assesses the credibility, validity, and robustness of its application in context.Both steps require locating and interpreting fine-grained methodological details scattered throughout a paper.Source: §1 · p. 1
• The authors create a 140-paper corpus annotated by domain experts across six research-design families and diverse social science fields.The expert evaluation framework includes an analytical scheme for each research-design family.Source: §1 · pp. 1–2
• The paper benchmarks retrieval strategies in a multi-turn RAG agent, comparing chunking methods, embedding models, and LLMs on fine-grained methodological analysis.The study evaluates the task as a domain-specific document-analysis challenge and estimates current system performance for social scientists.Source: §1 · pp. 1–2

2 Related Work

• Earlier studies tracked causal-method keywords or extracted research designs in social-science papers (Currie et al., 2020; Goldsmith-Pinkham, 2024; Garg and Fetzer, 2025).Garg and Fetzer reported 70%–93% accuracy across designs using GPT-4o-mini, validated against Brodeur et al. (2024), whose dataset covers four designs across 1,106 economics articles.Source: §2 · p. 2
• Garg and Fetzer (2025) used a single LLM without systematic retrieval evaluation and detected designs without assessing application quality.Source: §2 · p. 2
• This work evaluates not only which research design a paper uses but also how credibly it is applied, across retrieval strategies, embedding models, and LLMs.Source: §2 · p. 2
• Other related work combines NLP with causal mapping to extract policy evidence, examines causal-inference methodology, or uses RAG for scientific synthesis.OpenScholar used RAG on 45 million open-access papers and found retrieval-augmented systems substantially outperformed parametric LLMs on scientific synthesis tasks.Source: §2 · p. 2

Rationale

• A design’s credibility depends on its empirical setting: Card and Krueger’s minimum-wage Difference-in-Differences study used New Jersey and neighboring Pennsylvania as a context-specific comparison.Causal designs aim to isolate effects by exploiting variation and estimating outcomes in the absence of an intervention.Source: §3 · p. 2

Design families

• The remaining families are Selection-on-Observables, Difference-in-Differences and related designs, and Synthetic Control Methods and related designs.Examples include matching and weighting; event studies and triple differences; and augmented synthetic control and synthetic difference-in-differences.Source: §3 · pp. 2–3
• The six families include Experimental Designs, Threshold-based Designs, and Instrumental Variables, spanning experiments, regression discontinuity, and instrument-based approaches.Examples include field, survey, and lab experiments; regression kink and bunching designs; and Instrumental Variables.Source: §3 · p. 2

Cross-disciplinary scope

• The framework is not economics-specific: its causal-inference toolkit is shared across empirical sciences and used in fields including psychology, medicine, epidemiology, public health, and industry.Terminology varies across disciplines; the annotation scheme accounts for terms such as “unconfoundedness,” “ignorability,” and “conditional independence assumption” for Selection-on-Observables.Source: §3 · p. 3

4 Dataset

• The dataset contains 140 English-language papers sampled from 6,554 articles, spanning multiple quality tiers and economics subfields with a reasonable balance across research-design categories.Because most papers are not open access, the released supplementary materials include metadata, annotations, and the analytical framework rather than full texts.Source: §4 · p. 3, Figure 1 · p. 3

4.1 Analytical Framework

• Eight causal-inference experts used a closed-ended analytical schema to annotate papers through two subtasks: RD Identification and RD Assessment.The schema provides methodological questions, instructions, and predefined answer options.Source: §4.1 · p. 3
• RD Identification uses six questions to classify a paper’s type, data and design, analysis, counterfactual status, primary design family, and any additional counterfactual designs.A negative counterfactual decision ends the analysis; papers identified as counterfactual proceed to RD Assessment.Source: §4.1 · p. 3

4.2 Inter-Annotator Agreement

• Full-task agreement was α=.81; RD Identification scored .92 versus .75 for RD Assessment, while per-design agreement ranged from .64 (SoO) to .83 (ED).Pairwise agreement was .92–.94 for RD Identification, while RD Assessment was lower and more variable.Source: Table 1 · p. 4, §4.2 · p. 4, Table 2 · p. 4

4.1 Analytical Framework

• RD Assessment applies design-specific questions across 76 total items, covering study context, data, estimation and inference, and falsification or sensitivity analyses.Each of the six design families has 9–16 questions; all are closed-ended, and most permit multiple selections.Source: §4.1 · p. 4

5 Methodology

• The system answers the same ordered, closed-ended questions as human annotators, retrieving passages per turn and carrying prior answers forward within a 20K-token context.Questions and answer options are combined as the retrieval query; passage-level retrieval makes answers traceable to source spans.Source: §5 · pp. 4–5
• BM25 retrieves fixed-length chunks with sparse Okapi scoring, while dense retrieval ranks fixed-length chunk embeddings by cosine similarity.BM25 is included as a low-cost retrieval baseline.Source: §5 · p. 5
• Adaptive auto-merging retrieves embedded leaf chunks, then recursively replaces retrieved children with parents when their retrieved-child ratio exceeds threshold t.The hierarchy uses 1024-, 512-, 256-, and 128-token chunks; tested thresholds were 0.3, 0.5, and 0.7, with lower values yielding larger merged chunks.Source: §5 · p. 5
• Propositional topic-aware retrieval rewrites paragraphs into atomic propositions, then groups them into variable-length topical chunks using similarity and sliding-window settings.The method uses Llama-3.1-8B-Instruct to create propositions (Chen et al., 2024b); tested similarity thresholds were 0.3, 0.5, and 0.8, and window sizes were 2 and 5.Source: §5 · p. 5
• The Long-Context baseline supplies the full paper, truncated to 32K tokens, without retrieval while retaining the same multi-turn closed-ended questions.This setting tests whether retrieval helps compared with direct access to the document.Source: §5 · p. 5

5.1 Experimental Setup

• Across 140 documents, the main phase ran 164 experimental rounds using six embedding models, two LLMs, and the top five retrieved passages per turn.The best-performing embedding model was then used to test Llama-3.3-70B-Instruct and gpt-5.1 in further experiments.Source: §5.1 · pp. 5–6

6.1 Retrieval Strategies

• Passage length explained 52%–66% of task variance, and matched-length comparisons found no retrieval strategy with a consistent advantage.For Llama-8B, Qwen-32B, and Llama-70B, performance correlated with average passage length at Pearson r = 0.779, 0.815, and 0.727, respectively; gpt-5.1 showed no significant correlation. At each strategy’s best configuration, Adaptive scored .563 (Llama-8B) and .660 (Qwen-32B) on the Full Task, slightly above BM25 (.555 and .645), but averaged across embeddings BM25 matched or outperformed dense strategies across both LLMs and all subtasks. Across 30 matched-length comparisons, the average difference was 1.0 F1-points (worst case: 3.1).Source: §6.1 · pp. 6–7

6.2 Text Embeddings

• bge was the best embedding model in 41.5% of cases, while embedding choice mattered less for larger LLMs and more for the harder RD Assessment task.SFR was best in 20.7% of cases and mxbai in 13.4%; bge had the top mean F1 across subtasks and the best performance-stability trade-off. The best-to-worst embedding gap fell from 2.5 to 2.1 F1 points in RD-I and from 4.7 to 3.3 in RD-A as LLM size increased.Source: §6.2 · pp. 7–8

6.3 Research Design Analysis

• TbD was consistently easiest for models to identify and assess, followed by DiD, while IV was consistently among the most difficult designs.SCM and ED varied more by LLM, SoO had a more consistent ranking across models, and BM25 showed the highest variability across research designs, possibly reflecting sensitivity to how explicitly each methodology is described.Source: §6.3 · p. 8
• Human and machine difficulty showed no statistically significant correlation across research designs; IV was the only design where their difficulty aligned.Some designs showed the inverse pattern: DiD had lower human agreement yet was comparatively easier for models, while ED had higher human agreement yet was comparatively harder. The results suggest human judgment difficulty and machine retrieval difficulty are largely independent dimensions.Source: §6.3 · pp. 8–9, Table 6 · p. 9

6.4 Long Context vs. RAG

• Across both subtasks and all four LLMs, most retrieval configurations did not outperform Long-Context prompting, with only BM25 and Adaptive beating it at specific RD-A configurations.On RD-I, the baseline was only locally surpassed, mainly by smaller models; the authors argue retrieval’s value is interpretability through focused passages rather than a general need for competitive performance.Source: Figure 2 · p. 7, §6.4 · p. 9

7 Conclusions and Future Work

• ARDTrA provides an expert-designed analytical framework and an annotated dataset of 140 social-science papers for assessing research-design application.The task is intended to help social scientists and policy advisors estimate paper robustness quickly and reliably.Source: §7 · p. 9
• Passage length explained 52%–66% of performance variance across configurations, while gpt-5.1 performance was largely independent of passage length.At matched passage lengths, retrieval strategies performed comparably, and the passage-length effect diminished with stronger LLMs.Source: §7 · p. 9
• BM25-based sparse retrieval matched and often outperformed dense embedding strategies, while Long-Context document prompting was a strong baseline against RAG.These findings were based on evaluations spanning four retrieval strategies, four LLMs, six embedding models, and a Long Context setting.Source: §7 · p. 9
• Embedding-model choice mattered less than retrieval strategy or hyperparameter configuration, and bge-large-en-v1.5 was the most robust across settings.Source: §7 · p. 9
• The research designs experts disagreed on most were not those where the automated system struggled most, indicating two independent dimensions of task difficulty.The authors identify subjective judgment and information retrievability as distinct factors.Source: §7 · p. 9
• Future work will test whether linguistic expression and the location of relevant information explain residual performance variance, and whether human-machine cue differences generalize to other expert tasks.Source: §7 · p. 9

• The dataset is limited in size because its deep annotation schema requires annotators to closely read papers up to 50 pages and answer subtle methodological questions.Source: §8 · p. 10
• Per-RD estimates should be interpreted cautiously; the authors counterbalanced the limited scale by selecting representative papers across each RD and a range of application domains.Source: §8 · p. 10
• Focusing on papers in English limits the generalizability of the findings to other languages.Source: §8 · p. 10

• AI assistants supported manuscript revision and the creation of complex plots and tables.Source: §9 · p. 10
• The authors retained sole responsibility for all scientific content, experimental design, data analysis, and interpretation of results.Source: §9 · p. 10

10 Ethics Statement

• Annotators were co-authors acting in their regular institutional duties, and the dataset release contains only bibliographic metadata because most papers are copyrighted.The authors state that the work does not raise ethical concerns to the best of their knowledge.Source: §10 · p. 10

A Retrieval Strategies

• Table 7 compares strategies by F1-score, marking the best strategy per row and reporting best-embedding and six-embedding-average configurations; Llama-70B and gpt-5.1 use bge-large-en-v1.5 only.“Best embedding” uses the best hyperparameter configuration with the best-performing embedding model; “Embedding avg.” uses the best hyperparameter configuration averaged over all six embedding models.Source: Table 7 · p. 13

B Text Embeddings

• Table 8 reports the range and standard deviation of F1 across six embedding models by task and LLM, after averaging model scores across strategies.The range and standard deviation are calculated from the averaged F1 scores for each embedding model.Source: Table 8 · p. 13, B Text Embeddings · p. 13
• Table 9 counts each embedding model’s highest-F1 outcomes across 78 combinations of 13 hyperparameter configurations, two LLMs, and three tasks.Some rows do not sum to the same total because a few combinations produce ties; the underlying counts appear in Appendix C, Tables 11–13.Source: Table 9 · p. 13
• Table 10 summarizes mean F1 across 13 hyperparameter configurations and two LLMs, with variability measured across configurations, LLMs, and tasks.Mean F1 averages performance across configurations and LLMs; standard deviation and range summarize variability across configurations, averaged over two LLMs and two tasks.Source: Table 10 · p. 13, B Text Embeddings · p. 13

C Research Design Analysis

• Tables 11–13 report raw F1 scores from complete whole-dataset runs for the full task, research design identification, and research design assessment.The supplied passages identify the tables’ scopes but do not provide their score values.Source: C Research Design Analysis · p. 13, Table 11 · p. 14, Table 12 · p. 14

D ARDTrA Analytical Framework

• The analytical framework has two main tasks: Research Design Identification (RD-I) and Research Design Assessment (RD-A).Source: D ARDTrA Analytical Framework · p. 16

D.1 Research Design Identification (6 questions)

• RD-I first asks what type of paper is being analyzed, distinguishing empirical, theoretical, methodological, review, and opinion/comment/reply papers.Source: §D.1 · p. 16
• RD-I also classifies the research design and data as quantitative, qualitative, or other.The passages characterize quantitative work as primarily numeric or categorical and qualitative work as primarily non-numerical.Source: §D.1 · p. 16
• The questionnaire distinguishes causal analysis from descriptive analysis and predictive or simulation analysis.Source: §D.1 · p. 16
• RD-I asks whether a paper uses experimental, selection-on-observables, instrumental-variable, threshold-based, difference-in-differences, or synthetic-control designs.Source: §D.1 · p. 16
• It asks respondents to identify the paper’s main or preferred design from the same causal-design families.The listed families include experiments, selection-on-observables, instrumental variables, threshold-based designs, difference-in-differences, and synthetic control.Source: §D.1 · p. 16
• The final identification question asks whether the paper uses additional design-based causal-inference methods beyond its selected design.The listed additional methods cover experimental, selection-on-observables, instrumental-variable, threshold-based, difference-in-differences, and synthetic-control approaches.Source: §D.1 · pp. 16–17

D.2.1 Experimental Designs (ED)

• Experimental-design assessment distinguishes complete, matched, stratified, covariate-adaptive, and re-randomization strategies.It also provides options for studies where the strategy is not reported or is another approach.Source: §D.2.1 · p. 17
• The experimental-design rubric checks protocol registration and ethics approval, balance testing and evidence, and attrition.Balance checks may use predetermined covariates or outcomes, or placebo outcomes not expected to respond to treatment.Source: §D.2.1 · p. 17
• The rubric records experimental treatment-effect estimators and whether inference appropriately accounts for assignment-level clustering.Estimator options include difference-in-means, regression, matching or weighting, ANOVA/ANCOVA, and other approaches.Source: §D.2.1 · p. 18
• Experimental-design checks cover sensitivity to assumptions and analytic choices, alongside potential problems such as nonrandom inclusion or post-assignment exclusions.The rubric also flags failure to account for differing assignment probabilities and mismatches between assigned and analyzed treatment status.Source: §D.2.1 · pp. 18–19

D.2.2 Threshold-based designs (e.g., regression discontinuity, regression kink or bunching)

• Threshold-based-design assessment distinguishes continuity-based from local-randomization RDD frameworks and records running-variable and cutoff configurations.The listed configurations include single or multiple running variables and cutoffs.Source: §D.2.2 · p. 19
• RDD validity checks examine running-variable density continuity near the cutoff and whether tests support the no-manipulation assumption.The listed evidence includes graphical analysis, binomial tests, and density-estimation approaches.Source: §D.2.2 · p. 19
• The RDD rubric records whether point estimates use global or local polynomials, randomization inference, or other approaches.It distinguishes methods using the full running-variable support from those using observations near the cutoff.Source: §D.2.2 · p. 19
• Threshold-based designs are assessed on how the RDD bandwidth or window was selected, with options including MSE-optimal, covariate-based, manual, cross-validation, or IK methods.Source: §D.2.2 · pp. 19–20
• Threshold-design inference is checked for clustering at the assignment level when analysis occurs below that level, with exceptions when levels match.The checklist also notes clustering for repeated observations and panel data, and flags interference or spillovers as an additional reason to cluster.Source: §D.2.2 · p. 20
• Threshold-design falsification checks include predetermined-covariate balance, placebo outcomes, placebo populations, and placebo treatments or cutoffs.Source: §D.2.2 · p. 20
• Threshold-design robustness checks cover identification-assumption violations, donut-hole exclusions, bandwidth or framework choices, kernels or polynomial orders, covariates, and inference approaches.Source: §D.2.2 · p. 20

D.2.3 Instrumental Variables

• The IV checklist asks how instrument independence is justified, including quasi-random variation, eligibility rules, substantive reasoning, and empirical tests.It cautions assessors to consider the authors’ claims and the variation used, without inferring justification not discussed or implied.Source: §D.2.3 · p. 21
• The IV checklist records instrument type and number, along with the first-stage F-statistic in preferred specifications.Instrument examples include randomized assignment, policy rules, examiner designs, geographic or historical variation, treatment diffusion, and shift-share instruments.Source: §D.2.3 · pp. 21–22
• The IV assessment also records the F-test type, causal-effect estimator, and inference approach, including robust or weak-instrument procedures.Options include the Kleibergen-Paap, Effective, Cragg-Donald, and Sanderson-Windmeijer F-statistics; 2SLS, LIML, and GMM estimators; and bootstrap, Anderson-Rubin, CLR, or tF inference.Source: §D.2.3 · p. 22
• The IV checklist asks whether second-stage clustering is appropriate, first-stage coefficients match theory, and IV estimates are compared with OLS.It notes clustering is recommended at the assignment level when analysis is at a lower level, but may be unnecessary when analysis and assignment levels match.Source: §D.2.3 · pp. 22–23
• The IV rubric checks whether IV and OLS estimates are compared and how any differences are justified, including selection bias, complier differences, or measurement error.It also asks whether OLS estimates are absent or IV estimates are similar to OLS.Source: §D.2.3 · p. 23
• The IV rubric records falsification tests, sensitivity analyses, overidentification tests, and explicit discussion or empirical support for exclusion and monotonicity assumptions.Examples include placebo outcomes, treatments or populations; assumption-violation bounds, alternative estimators or inference, weak-instrument procedures, and outlier exclusion.Source: §D.2.3 · p. 23

D.2.4 Selection-on-observables (e.g., regression adjustment, matching, weighting doubly robust

• For selection-on-observables designs, the rubric assesses identification arguments, from quasi-random policy or contextual variation and eligibility rules to theory, prior studies, or empirical tests.It also classifies covariates used for unconfoundedness, including demographic measures, pre-treatment outcomes, and potentially problematic post-treatment variables.Source: §D.2.4 · p. 24
• The selection-on-observables rubric evaluates common-support evidence, trimming or sample restrictions, and whether adjustment achieves covariate balance.Possible trimming includes restricting to common support or dropping units based on propensity-score thresholds or data-driven criteria; balance may be high, partial, inadequate, or unclear.Source: §D.2.4 · pp. 24–25
• The rubric records selection-on-observables estimation methods—including regression adjustment, matching, weighting, and doubly robust approaches—and whether inference appropriately accounts for clustering.Clustering is assessed relative to treatment-assignment and analysis levels, including repeated observations within units.Source: §D.2.4 · p. 25
• For selection-on-observables designs, the rubric also catalogs placebo tests and sensitivity analyses, including tests using placebo outcomes, treatments, populations, or pre-treatment measures.Sensitivity checks include assumption-violation analyses, alternative estimators, covariate selection, and trimming strategies.Source: §D.2.4 · pp. 25–26
• For selection-on-observables, the supplied rubric includes sensitivity to alternative inference approaches, such as different methods or clustering levels.The excerpt provides this criterion across two continuation fragments.Source: §D.2.4 · p. 26

D.2.5 Difference in Differences (DiD) and related designs (e.g., event studies, changes-in-changes, triple differences)

• For DiD-family designs, the rubric centers identification credibility on parallel trends and asks what variation and justification the paper offers, without inferring unreported rationale.It recognizes that pre-treatment parallel trends do not ensure post-treatment trends would remain parallel absent treatment, and that anticipated outcome changes can undermine credibility.Source: §D.2.5 · p. 26
• The rubric distinguishes credibility arguments based on policy or contextual quasi-random variation, eligibility rules, substantive reasoning, prior studies, and empirical tests.It includes administrative quirks and historical or contextual forces as possible sources of plausibly exogenous variation.Source: §D.2.5 · p. 26
• The assessment records whether treatment is blocked or staggered, whether no-anticipation is addressed, and the available pre- and post-treatment periods.If anticipation is argued, the rubric says to back-date treatment when counting periods.Source: §D.2.5 · p. 27
• The rubric checks whether pre-treatment trends are analyzed and whether treatment and control groups appear to follow parallel trends before intervention.It recognizes group-level outcome plots, event-study plots or estimates, and reported arguments for parallel pre-trends as forms of assessment.Source: §D.2.5 · pp. 27–28
• The assessment identifies the main estimator and checks whether inference appropriately accounts for clustering at the treatment-assignment level.Estimator options include standard DiD/TWFE, event studies, and alternative estimators for staggered adoption.Source: §D.2.5 · p. 28
• The rubric catalogs placebo and sensitivity analyses and flags estimation issues such as including periods when all units are treated or assuming homogeneous effects.Examples include tests using placebo outcomes, populations, or treatments; sensitivity to parallel-trends assumptions and alternative estimators; and inference choices.Source: §D.2.5 · pp. 28–29
• The rubric checks whether papers address no-anticipation and defend identification empirically through checks such as balance, placebo, or sensitivity tests.Addressing anticipation may include backdating treatment or showing no early effects.Source: §D.2.5 · pp. 29–30
• The coding records available pre- and post-treatment periods and counts periods relative to a backdated start when anticipation effects are argued.It also records the time unit of observation and treated and control unit counts.Source: §D.2.5 · p. 30
• Synthetic-control construction criteria cover pre-treatment outcomes and covariates, as well as donor-pool trimming to exclude exposed or substantially different units.The rubric also identifies the estimation approach, including classical, augmented or penalized, synthetic DiD, generalized, and Bayesian synthetic control.Source: §D.2.5 · pp. 30–31
• Synthetic-control assessment checks pre-treatment outcome volatility, which can raise overfitting risk, and evaluates treated-versus-synthetic fit using reported statistics or graphs.Filtering or smoothing volatile outcomes is advised, particularly in comparative case studies with one or few treated units; averaging across many units can reduce noise.Source: §D.2.5 · pp. 30–31
• The rubric classifies inference methods, checks whether clustering is appropriate to the assignment and analysis levels, and records placebo falsification tests.Listed inference approaches include permutation or placebo tests, jackknife, bootstrap, and conformal inference; placebo tests may use outcomes, treatment timing, or populations.Source: §D.2.5 · pp. 31–32
• Sensitivity-analysis criteria include violations of identification assumptions, covariate or donor-pool choices, alternative estimators, and alternative inference procedures.The rubric includes bounding or partial identification, robustness to estimator choice, and sensitivity to inference methods or clustering levels.Source: §D.2.5 · p. 32