PaperlayerView exact revision on arXiv

Source-linked AI summary

HLER: Human-in-the-Loop Economic Research via Multi-Agent Pipelines for Empirical Discovery

Chen Zhu, Xiaolu Wang

arXiv:2603.07444v1cs.AIecon.GNq-fin.EC

TL;DR

Text-centric LLM workflows struggle with the data validation, identification, and human judgment required for credible empirical research. HLER combines specialized agents, dataset-aware questions, human decision gates, and iterative review; dataset-aware generation yields feasible questions in 87% of cases versus 41% under unconstrained ideation.

  • Problem

    Text-centric LLM workflows struggle to reliably perform the end-to-end steps of empirical research, which requires dataset validation, identification choices, and scrutiny.

    Source: §1 · p. 1

  • Method

    HLER coordinates specialized agents for empirical research and uses dataset-aware question generation, human selection, and review-triggered re-analysis and revision.

    Source: §1 · p. 2

  • Results

    87% of dataset-aware research questions are feasible, compared with 41% under unconstrained LLM ideation.

    Source: §6 · p. 9

  • Takeaways & Limitations

    HLER is positioned as a collaborative tool that keeps domain expertise and scientific judgment central, rather than a replacement for researchers.

    Source: §6 · p. 9

    Generated questions depend on available datasets and variables and are not guaranteed to be theoretically meaningful or novel; supported econometric designs are limited.

    Source: §5.4 · p. 9

Abstract

from arXiv · show

Large language models (LLMs) have enabled agent-based systems that aim to automate scientific research workflows. Most existing approaches focus on fully autonomous discovery, where AI systems generate research ideas, conduct analyses, and produce manuscripts with minimal human involvement. However, empirical research in economics and the social sciences poses additional constraints: research questions must be grounded in available datasets, identification strategies require careful design, and human judgment remains essential for evaluating economic significance. We introduce HLER (Human-in-the-Loop Economic Research), a multi-agent architecture that supports empirical research automation while preserving critical human oversight. The system orchestrates specialized agents for data auditing, data profiling, hypothesis generation, econometric analysis, manuscript drafting, and automated review. A key design principle is dataset-aware hypothesis generation, where candidate research questions are constrained by dataset structure, variable availability, and distributional diagnostics, reducing infeasible or hallucinated hypotheses. HLER further implements a two-loop architecture: a question quality loop that screens and selects feasible hypotheses, and a research revision loop where automated review triggers re-analysis and manuscript revision. Human decision gates are embedded at key stages, allowing researchers to guide the automated pipeline. Experiments on three empirical datasets show that dataset-aware hypothesis generation produces feasible research questions in 87% of cases (versus 41% under unconstrained generation), while complete empirical manuscripts can be produced at an average API cost of $0.8-$1.5 per run. These results suggest that Human-AI collaborative pipelines may provide a practical path toward scalable empirical research.

1 Introduction

Text-centric LLM workflows struggle to complete credible empirical research, where datasets, identification, and human judgment matter. HLER combines dataset-aware question generation with human decision gates and iterative review to support empirical research automation.

1 Introduction

  • Credible empirical work requires data validation, transparent estimation, and robustness, while analytic choices can shift estimates even with the same data and question.Replication audits also find published economics results often fail to reproduce or require substantial author assistance.Source: §1 · p. 2
  • HLER orchestrates agents for data auditing and profiling, question generation, econometric analysis, manuscript drafting, and automated review.Candidate questions are constrained by dataset structure, variable availability, missingness, and distributional diagnostics.Source: §1 · p. 2
  • Infeasible hypotheses fall from 59% to 13% with dataset-aware generation, compared with unconstrained LLM ideation.The mechanism conditions questions on dataset audits and statistical profiles.Source: §1 · p. 2
  • The two-loop architecture combines hypothesis generation, feasibility screening, and human selection with review-triggered re-analysis and manuscript revision.Reviewer scores improve across revision iterations.Source: §1 · p. 2
  • HLER produces complete empirical manuscripts at an average API cost of $0.8-$1.5 per run.The evaluation covered three empirical datasets and 14 pipeline runs.Source: §1 · p. 2

2 Related Work

Related work spans autonomous scientific agents and economics-focused research automation, alongside approaches emphasizing human-AI collaboration. HLER combines agent execution with human decision gates, dataset-informed question generation, and iterative review.

2.1 LLM Agents for Scientific Discovery

  • Multi-agent systems generate hypotheses, implement experiments, and produce manuscripts, while biomedical and chemistry systems extend automation to domain-specific workflows.Many such systems focus on computational or experimental settings where experiments run in simulation or laboratories.Source: §2.1 · pp. 2–3

2.2 Autonomous Research Systems in Economics

  • APE generates empirical economics papers end-to-end, but its full-automation approach highlights challenges with hallucinated hypotheses and limited feasibility checks.Its workflow includes question identification, public-data retrieval, econometric code, figures and tables, and manuscript drafting.Source: §2.2 · p. 3

2.3 Human-in-the-Loop Research Automation

  • HLER pairs autonomous execution with human decision gates, dataset-audit-based question generation, and loops for feasibility screening and manuscript revision.The motivation is that economic research design requires judgment about identification strategies and economic significance.Source: §2.3 · p. 3

3 System Design and Implementation

HLER decomposes empirical research into specialized agent stages coordinated by an orchestrator, with human decision gates and two feedback loops. Its implementation supports dataset inspection, econometric analysis, manuscript drafting, and iterative review.

3 System Design and Implementation

  • HLER is a multi-agent pipeline that decomposes empirical research into structured stages coordinated by a central orchestrator.Source: §3 · p. 4

3.1 Architecture Overview

  • The PI selects research questions and approves publication, while the orchestrator dispatches agents sequentially and stores their outputs in shared state.Source: §3.1 · p. 4

3.2 Agent Roles

  • Dataset auditing inventories available variables, profiling checks distributions and risks, and question generation conditions candidate questions on those diagnostics.Source: §3.2 · p. 4
  • Data agents retrieve and prepare data; econometrics agents estimate models, paper agents draft manuscripts, and reviewer agents score drafts and request revisions.Source: §3.2 · p. 4

3.3 Control Loops

  • The question quality loop screens candidates before human selection, while the revision loop cycles through additional analyses, manuscript updates, and reviewer critique.Source: §3.3 · p. 5

3.4 Human Oversight

  • Researchers retain control over question selection and final publication decisions, while routine data processing and manuscript drafting are automated.Source: §3.4 · p. 5

3.5 Implementation Details

  • HLER uses Python agents and structured shared message objects; its prototype supports local and external data sources.Source: §3.5 · pp. 5–6
  • Econometric analysis combines LLM-generated plans with programmatic estimation, supporting OLS, fixed-effects, difference-in-differences, and event-study specifications.Source: §3.5 · p. 6

4 Experimental Evaluation

Across three datasets and 14 pipeline runs, the evaluation tests question feasibility, end-to-end completion, revision dynamics, and cost. Dataset-aware generation improves feasibility, and completed runs produce manuscripts with reviewer scores rising over revisions.

4.1 Experimental Setup

  • Experiments used three empirical datasets and 14 complete pipeline runs, with researchers selecting one question per run.Source: §4.1 · p. 6

4.2 Research Question Feasibility

  • Across 14 pipeline runs on three datasets, dataset-aware generation produced 79 candidate questions, 69 (87%) fully feasible, versus 34 of 82 (41%) under unconstrained generation.Source: §4.2 · p. 6, Table 2 · p. 7

4.3 End-to-End Pipeline Completion

  • 12 of 14 runs (86%) completed the workflow without manual intervention beyond the two planned decision gates.The two incomplete runs halted during fixed-effects estimation because of convergence issues on sparse subsamples.Source: §4.3 · pp. 6–7

4.4 Revision Loop Dynamics

  • Mean reviewer scores rose from 4.8 initially to 6.3 in final drafts, with the largest gains in clarity (+2.1) and identification credibility (+1.4).The most frequent first-round requests were robustness checks (10 of 12 runs), clearer identification discussion (8 of 12), and variable descriptions (7 of 12).Source: §4.4 · p. 7, Table 3 · p. 7

4.5 Runtime and Cost

  • A typical run took approximately 20–25 minutes and cost $0.8–$1.5 in LLM API calls.Source: §4.5 · p. 7

4.6 Case Study: Rural Women’s Labor Outcomes in China

  • In the CHNS case study, seven of eight candidate questions met feasibility criteria, and the researcher selected a question about education and occupational gender gaps.Source: §4.6 · p. 8
  • The CHNS case study used fixed-effects estimation on 19,466 observations; after three revisions, reviewer scores rose from 4.6 to 6.5.The final run produced a manuscript, replication files, regression outputs, and figures.Source: §4.6 · p. 8

5 Discussion

HLER combines human decision gates with automation to support collaborative empirical research, and its potential to scale comes with methodological and evaluation limits. The discussion emphasizes that human judgment remains central and that broader evaluation and responsible reporting are still needed.

5.1 Human-AI Co-Production of Empirical Research

  • Human decision gates let researchers select hypotheses, assess intermediate results, and decide whether to pursue manuscripts, keeping domain expertise and scientific judgment central.HLER is framed as a collaborative tool, not a replacement for researchers.Source: §5.1 · p. 8

5.2 Scaling Empirical Research

  • Automating repetitive empirical tasks may let researchers explore more questions, with potential value in data-rich fields such as health, labor, and development economics.The discussion identifies dataset inspection, variable screening, robustness checks, and manuscript drafting as time-consuming tasks.Source: §5.2 · p. 8

5.3 Researcher Degrees of Freedom and Ethical Considerations

  • Generating and testing many hypotheses raises selective-reporting and multiple-testing risks, including automated p-hacking when only significant results are reported.Pipeline logs record unselected questions and earlier analyses, supporting pre-registration-like transparency; researchers must still use appropriate corrections and report all questions considered.Source: §5.3 · pp. 8–9

5.4 Limitations

  • HLER's generated questions depend on available data, its econometric toolkit is limited, and its evaluation has potential reviewer circularity and limited scale.The evaluation covers 14 runs across three datasets; broader benchmarking and independent human or different-LLM review would strengthen evidence about generalizability and manuscript quality.Source: §5.4 · p. 9

5.5 Implications for Scientific Workflows

  • HLER is presented as a tool that can extend researchers' capabilities to explore larger hypothesis spaces and conduct more systematic robustness checks.The discussion cautions against viewing such systems as replacements for researchers.Source: §5.5 · p. 9

6 Conclusion

HLER is a human-in-the-loop multi-agent pipeline for empirical research, preserving human oversight across key scientific decisions. Across three datasets, it outperformed unconstrained ideation on hypothesis feasibility, while the paper identifies evaluation and toolkit extensions as future work.

6 Conclusion

  • HLER structures empirical research into auditing, profiling, hypothesis generation, econometric analysis, manuscript drafting, and automated review while retaining human decision gates.Researchers retain control over key scientific decisions, making HLER a collaborative tool rather than a replacement.Source: §6 · p. 9
  • 87% of dataset-aware hypotheses were feasible, compared with 41% under unconstrained LLM ideation, across three datasets and 14 pipeline runs.The system completed end-to-end pipelines in 86% of runs; revision improved average reviewer scores from 4.8 to 6.3 over 2–3 iterations.Source: §6 · p. 9
  • Future work includes expanding the econometric toolkit, benchmarking across more datasets and domains, independently evaluating manuscripts, and developing transparency standards for AI-assisted research.Source: §6 · p. 9

Related papers

Loading 2603.07444v1…