PaperlayerView exact revision on arXiv

Source-linked AI summary

HybridQuestion: Human-AI Collaboration for Identifying High-Impact Research Questions

Keyu Zhao, Fengli Xu, Yong Li, Tie-Yan Liu

arXiv:2602.03849v1cs.HCcs.AIcs.CLcs.LG

TL;DR

The paper asks whether LLMs can identify meaningful research questions beyond task-specific ideation. It tests a human–AI framework that synthesizes literature and context, generates candidates with an LLM ensemble, and progressively involves human judgment in selection. AI aligns highly with experts on established breakthroughs but diverges more when forecasting future questions.

  • Problem

    Whether LLMs can extend beyond task-specific ideation to identify major past breakthroughs and forecast important future scientific questions remains largely unexplored.

    Source: §1 · p. 3

  • Method

    A three-phase human–AI framework combines literature and web-context synthesis, six-model candidate generation and voting, and multi-stage selection with increasing human oversight.

    Source: §1 · pp. 3–4

  • Results

    AI agents align highly with human experts in recognizing established breakthroughs but diverge more in forecasting prospective questions.

    Source: §1 · pp. 3–4, §5 · p. 14

  • Takeaways & Limitations

    Human judgment remains crucial for evaluating subjective, forward-looking challenges.

    Source: §1 · p. 4

Abstract

from arXiv · show

The "AI Scientist" paradigm is transforming scientific research by automating key stages of the research process, from idea generation to scholarly writing. This shift is expected to accelerate discovery and expand the scope of scientific inquiry. However, a key question remains unclear: can AI scientists identify meaningful research questions? While Large Language Models (LLMs) have been applied successfully to task-specific ideation, their potential to conduct strategic, long-term assessments of past breakthroughs and future questions remains largely unexplored. To address this gap, we explore a human-AI hybrid solution that integrates the scalable data processing capabilities of AI with the value judgment of human experts. Our methodology is structured in three phases. The first phase, AI-Accelerated Information Gathering, leverages AI's advantage in processing vast amounts of literature to generate a hybrid information base. The second phase, Candidate Question Proposing, utilizes this synthesized data to prompt an ensemble of six diverse LLMs to propose an initial candidate pool, filtered via a cross-model voting mechanism. The third phase, Hybrid Question Selection, refines this pool through a multi-stage filtering process that progressively increases human oversight. To validate this system, we conducted an experiment aiming to identify the Top 10 Scientific Breakthroughs of 2025 and the Top 10 Scientific Questions for 2026 across five major disciplines. Our analysis reveals that while AI agents demonstrate high alignment with human experts in recognizing established breakthroughs, they exhibit greater divergence in forecasting prospective questions, suggesting that human judgment remains crucial for evaluating subjective, forward-looking challenges.

1 Introduction

The paper asks whether LLMs can move beyond task-specific ideation to identify meaningful breakthroughs and future questions, and tests a three-phase human–AI workflow. Across five fields, AI aligns strongly with experts on established breakthroughs but diverges more on prospective questions, where human judgment remains crucial.

1 Introduction

  • A three-phase human–AI framework combines information gathering, six-model candidate proposing with cross-model voting, and selection that progressively increases human oversight.The final expert-driven refinement uses weighted voting of human:AI = 7:1.Source: §1 · pp. 3–4
  • AI agents align highly with human experts on established breakthroughs but diverge more on prospective questions, where humans favor practical infrastructure and AI favors theoretical capabilities.The study evaluates these tasks across Artificial Intelligence, Physics, Chemistry, Biology, and Economics.Source: §1 · pp. 3–4
  • The work extends LLM ideation from solving specified problems toward strategic review of past breakthroughs and forecasting of future scientific questions.The paper situates this gap against prior task-specific ideation.Source: §1 · pp. 3–4
  • The methodology produces curated lists of the Top 10 Scientific Breakthroughs of 2025 and Top 10 Scientific Questions for 2026.The lists are presented as strategic compass points for researchers and policymakers.Source: §1 · p. 4

2.1 Overview

The methodology has three phases: information gathering, candidate question proposing, and hybrid selection that screens and finalizes the lists.

2.1 Overview

  • The workflow combines computational literature analysis and web-based deep research, ensemble candidate generation and filtering, then multi-stage hybrid screening.The overview is illustrated in Figure 1.Source: §2.1 · p. 4

2.2 AI-Accelerated Information Gathering

AI-Accelerated Information Gathering combines computational analysis of publication data with web research to supplement literature-derived themes with broader context.

2.2.1 Embedding-based Keyword Analysis

  • OpenAlex [24] publications from 2015–2025 form annual keyword graphs embedded with node2vec [27], then hotness-scored and greedily clustered around high-hotness seeds.Hotness weights keyword frequency by semantic proximity; clustering uses dynamically determined distance thresholds.Source: §2.2.1 · p. 5
  • Breakthrough keywords combine 2024 prominence with rising 2025 hotness, while question keywords capture high importance or accelerating hotness within high-impact clusters.The keyword sets guide selection of themes for each list.Source: §2.2.1 · p. 5

2.2.2 Deep Research for Supplementary Context

  • Deep research supplements academic literature with media, industry, and public-discourse context, using separate US- and China-based models whose outputs a third LLM consolidates.The prompts seek solved or advanced challenges for breakthroughs and emerging or accelerating directions for future questions.Source: §2.2.2 · pp. 5–6

2.3 Candidate Question Proposing

Candidate Question Proposing uses synthesized information to generate and refine initial candidate pools through a multi-LLM ensemble.

2.3 Candidate Question Proposing

  • The proposing phase draws on both embedding analysis and supplementary context to generate initial candidate lists.The lists cover breakthroughs and grand scientific questions.Source: §2.3 · p. 6, §2.3.1 · p. 6

2.3.1 Hybrid Proposing of Initial Candidate Lists

  • Breakthrough proposals combine 2025 high-citation publications with media and industry context; question proposals also use post-2015 foundational papers for long-term context.The dual-source approach grounds candidates in academic trends and real-world context.Source: §2.3.1 · p. 6

2.3.2 Ensemble Voting for Bias Mitigation

  • Six LLMs—three from the United States and three from China—generate 600 candidates per list, then cross-model voting selects the 100 highest-voted candidates.Each model casts 100 approval votes on the combined candidate pool.Source: §2.3.2 · p. 6

2.4 Hybrid Question Selection

The candidate pool is narrowed in two stages, with expert human judgment given greater weight in the final refinement.

2.4 Hybrid Question Selection

  • The 100 candidates are reduced to 30 through broad screening, then to 10 through expert-driven refinement.The two stages use Approval Voting and Limited Voting, respectively.Source: §2.4 · p. 6

2.4.1 Stage1: Initial Screening, from 100 to 30 candidates

  • 30 graduate-level human voters and 70 AI voters cast unlimited approval votes; equally weighted totals determine which 30 advance.AI voters were assigned distinct specializations and balanced research backgrounds.Source: §2.4.1 · pp. 6–7

2.4.2 Stage2: Expert Refinement, from 30 to 10 candidates

  • 10 professor-level humans and 30 professor-level AI voters each allocate 10 votes, with human votes weighted 7 to AI’s 1.The ten highest-scoring candidates are shortlisted, and the top two are inducted into the relevant final list.Source: §2.4.2 · p. 7

3 Results

The implementation draws publication data from OpenAlex across five disciplines, applies node2vec and threshold-based keyword selection, and uses configured research and language models.

3.1.1 General Scope and Data Source

  • The analysis covers Artificial Intelligence, Physics, Chemistry, Biology, and Economics, using OpenAlex topic fields to identify disciplines and concepts fields to extract keywords.Source: §3.1.1 · p. 7

3.1.2 Data Synthesis Settings

  • Keyword analysis uses node2vec with 128-dimensional embeddings, 20-step walks, 10 walks, a window of 3, and 25 epochs.The implementation also sets p=1, q=1, and five negative samples.Source: §3.1.2 · p. 7
  • Selection settings include sigma_perc_1 and sigma_perc_2 at 0.0005, a hotness percentile threshold of 0.05, and absolute-rank thresholds of 50, 5, and 5.The absolute-rank thresholds apply to breakthrough keywords, cluster hotness, and question keywords, respectively.Source: §3.1.2 · p. 7

3.1.3 LLM Configuration

  • Deep research used OpenAI o3-deep-research and Qwen qwen-deep-research, with context consolidated by gpt-5; six models proposed candidates, and gpt-5 instantiated the voting agents.The six proposal models comprised three US and three Chinese models; generation and voting temperatures were set to 0.6.Source: §3.1.3 · pp. 7–8

3.2 Quantitative Keyword Presentation

The keyword presentation reports hotness rankings, semantic groupings, and the cluster and keyword selections used to generate breakthroughs and future questions.

3.2.1 Annual Hotness Analysis

  • For 2025, Artificial intelligence, Machine learning, and Deep learning occupy the highest ranks among the top 50 AI-domain keywords, alongside methodological, theoretical, and application-oriented topics.Source: §3.2.1 · p. 8

3.2.2 Semantic Embedding Visualization

  • The 2D map groups semantically related keywords, including neural-network methods apart from application-focused and theoretical clusters.The map projects 128-dimensional node2vec embeddings of the 100 hottest 2025 keywords; the reported clusters include Deep learning with Artificial neural network, and Algorithm in a theoretical cluster.Source: §3.2.2 · pp. 8–9, Figure 3 · p. 9

3.2.3 Thematic Clustering and Keyword Selection

  • The five hottest-seed clusters divide AI themes into methodological, theoretical, security, and optimization areas, while selected breakthrough keywords combine prior prominence with 2025 growth.The selected sets also distinguish high-absolute-hotness question keywords from high-acceleration keywords, targeting enduring problems and emerging trends.Source: §3.2.3 · pp. 9–10, Table 1 · p. 10

3.3 Vote Result Presentation

The results show a diverse initial candidate pool and trace how AI question proposals were filtered toward final selections.

3.3.1 Initial Candidate Pool

  • The AI-domain sample contrasts retrospective breakthroughs, including DeepSeek-R1-Zero’s “Pure Reinforcement Learning,” with prospective questions about causal reasoning and actionable 3D world models.The authors characterize this contrast as separating retrospective validation from prospective inquiry.Source: §3.3.1 · p. 11

3.3.2 A Case of Question Selection

  • The continuous-learning question rose from 40 initial votes to 81 Stage 1 votes and won the final selection, while the verifiable-reasoning-traces question placed second.Proposals about multi-agent protocols and scaling laws were filtered out early.Source: §3.3.2 · p. 11

4 Analysis

In the AI domain, human and AI voting align more closely on established breakthroughs than on prospective questions. The divergence on future questions reflects their speculative, subjective nature.

4 Analysis

  • The AI-domain analysis tracks voting as simulated expertise rises from graduate-level Stage 1 to professor-level Stage 2.Source: §4 · p. 11

4.1 Distribution Visualization

  • In Stage 2, Candidate 1 received the highest vote count from both groups, while Candidate 14 received 7 human votes and none from AI agents.Candidate 14 concerns unified, domain-agnostic evaluation standards for real-world reliability.Source: §4.1 · pp. 12–13, Figure 6 · p. 13

4.2 Quantitative Alignment Analysis

  • 0.394 to 0.209: Stage 1-to-Stage 2 Breakthrough JS Distance fell, and Stage 2 Breakthrough JS Distance was 0.209 vs. 0.352 for Questions.Table 4 compares human and AI voting distributions across both stages and categories.Source: Table 4 · p. 13, §4.2 · p. 13
  • Breakthrough JS Distance is consistently lower than Questions; prospective, speculative questions invite more subjective judgments and likely greater opinion variance, even among experts.Source: §4.2 · pp. 13–14

5 Conclusion

The paper presents a human-AI framework for identifying breakthroughs and future scientific questions. Its experiment finds stronger alignment on retrospective achievements than prospective challenges, for which human judgment remains essential.

  • The framework combines AI-accelerated data synthesis with multi-stage collaborative voting to identify scientific breakthroughs and future questions.Source: §5 · p. 14
  • The experiment identifies the Top 10 Major Scientific Breakthroughs of 2025 and Top 10 Grand Scientific Questions for 2026.Source: §5 · p. 14
  • AI agents align highly with human experts on retrospective achievements but diverge on prospective challenges, where human judgment remains essential.Source: §5 · p. 14

Related papers

Loading 2602.03849v1…