PaperlayerView exact revision on arXiv

Source-linked AI summary

Into the Unknown Unknowns: Engaged Human Learning through Participation in Language Model Agent Conversations

Yucheng Jiang, Yijia Shao, Dekun Ma, Sina J. Semnani, Monica S. Lam

arXiv:2408.15232v2cs.CLcs.AIcs.IR

TL;DR

Complex information-seeking systems can leave users struggling to discover information beyond their known queries. Co-STORM lets users observe and steer LM-agent conversations, organizes them into a mind map, and generates cited reports. Evaluations report improved discovery and mental effort, with most human-evaluation participants preferring it to search and RAG alternatives.

  • Problem

    In complex information seeking, users need to discover unknown unknowns, while existing systems often react to queries that users with limited prior knowledge may struggle to formulate.

    Source: §1 · p. 1

  • Method

    Co-STORM lets users observe and steer discourse among multiple LM agents, organizes collected information in a dynamic mind map, and can generate a cited report.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    70% preferred Co-STORM over a search engine and 78% over a RAG chatbot; evaluations also report improved unknown-unknown discovery and reduced mental effort.

    Source: §1 · p. 2, §8 · p. 9

  • Takeaways & Limitations

    Co-STORM can help humans discover unknown unknowns for learning while reducing mental effort.

    Source: §8 · p. 9

    The current work considers generating and retrieving information only from English sources.

    Source: Ethics Statement · p. 10

Abstract

from arXiv · show

While language model (LM)-powered chatbots and generative search engines excel at answering concrete queries, discovering information in the terrain of unknown unknowns remains challenging for users. To emulate the common educational scenario where children/students learn by listening to and participating in conversations of their parents/teachers, we create Collaborative STORM (Co-STORM). Unlike QA systems that require users to ask all the questions, Co-STORM lets users observe and occasionally steer the discourse among several LM agents. The agents ask questions on the user's behalf, allowing the user to discover unknown unknowns serendipitously. To facilitate user interaction, Co-STORM assists users in tracking the discourse by organizing the uncovered information into a dynamic mind map, ultimately generating a comprehensive report as takeaways. For automatic evaluation, we construct the WildSeek dataset by collecting real information-seeking records with user goals. Co-STORM outperforms baseline methods on both discourse trace and report quality. In a further human evaluation, 70% of participants prefer Co-STORM over a search engine, and 78% favor it over a RAG chatbot.

1 Introduction

Complex information seeking often requires discovery beyond known queries, but existing systems offer limited support for evolving user goals. Co-STORM lets users observe and steer LM-agent discourse, organizes the discussion, and generates cited reports.

1 Introduction

  • Complex information-seeking can involve unknown unknowns, while traditional search and RAG systems react to user queries that users with limited prior knowledge may struggle to formulate.Unlike static reports, these tasks can involve dynamically evolving queries toward a goal.Source: §1 · pp. 1–2
  • Co-STORM combines collaborative discourse among multiple LM agents with user participation, a dynamic hierarchical mind map, and the option to generate a cited report.Users can observe and occasionally steer the discourse, while experts offer different perspectives and a moderator guides discussion.Source: §1 · p. 2
  • Co-STORM significantly outperforms RAG chatbots in surfacing in-depth and serendipitous information while providing a more engaging learning experience.Source: §1 · p. 2
  • 70% preferred Co-STORM over a search engine and 78% over a RAG chatbot for the overall information-seeking experience.In a human evaluation with 20 users, participants also found it facilitated serendipitous discovery and required less mental effort.Source: §1 · p. 2
  • The authors construct WildSeek from real-world human information-seeking records to evaluate information-seeking assistance tools.Source: §1 · p. 2

2 Complex Information Seeking

The paper frames complex information seeking as an interactive, multi-source task that produces a curated report, then introduces WildSeek as a dataset of real users’ topics and goals.

2.1 Problem Formulation

  • Complex information seeking involves multiple sources, ongoing user interaction, and a report-like product; the paper states that none of the compared systems fully supports the task.Source: §2.1 · p. 2
  • The task starts from a user’s topic and goal and produces a custom long-form report whose sentences cite retrieved information.Source: §2.1 · pp. 2–3

2.2 WildSeek: An In-the-Wild Information Seeking Dataset

  • WildSeek uses real STORM-web-application topic-and-goal records, filters for well-motivated examples, assigns topic taxonomies with manual review, and downsamples to 100 examples across 24 domains.Source: §2.2 · p. 3

3 Method

Co-STORM emulates collaborative discourse among users, perspective-guided LM experts, and a moderator, while a dynamic mind map tracks shared information. Users can interject, and the system can steer discussion and turn the map into a cited report.

3.1 Collaborative Discourse Protocol

  • Users may take a turn at any time; otherwise, experts alternate until L consecutive answer or detail turns trigger moderator intervention.The mixed-initiative protocol lets the discourse respond to user questions or arguments, or continue automatically when the user does not intervene.Source: §3.1 · pp. 3–4

3.2 Tracking the Discourse with a Mind Map

  • The mind map organizes retrieved information into a concept hierarchy, inserting items under suitable concepts and reorganizing concepts that exceed K items.A bottom-up cleanup removes unsupported concepts and collapses concepts with only one subtopic.Source: §3.2 · p. 4

3.3 User Participation

  • When users contribute, their utterance becomes a retrieval query; once satisfied, they can generate a report using the mind map as its outline.Retrieved information associated with each concept supplies the corresponding report section.Source: §3.3 · p. 5

3.4 Simulating the Roundtable Participant

  • Experts are assigned different topic expertise, choose an utterance intent from discourse history, retrieve and cite information for answers, and polish utterances for engagement.Questions are generated directly from discourse history, while potential answers and further details use search.Source: §3.4 · p. 5

3.5 Simulating the Moderator

  • The moderator counters repetitive expert discussion by using unused retrieved information, reranked for topic relevance and dissimilarity to its associated question, to generate new questions.The combined context also includes concept names to avoid repetitive concepts.Source: §3.5 · p. 5

4 Co-STORM Implementation

Co-STORM uses zero-shot prompting with GPT-4o and Internet search, with specified hyperparameters and text embeddings.

4 Co-STORM Implementation

  • The system uses zero-shot prompting through DSPy with gpt-4o-2024-05-13 and grounds responses using the You.com search API.The authors note compatibility with other search engines or information-retrieval systems.Source: §4 · p. 5
  • Hyperparameters N, K, L, and α are set to 3, 10, 2, and 0.5, respectively.Text embeddings use text-embedding-3-small; LM temperature is 1.0 and top_p is 0.9.Source: §4 · p. 5
  • For human evaluation, the authors developed a web application for real-time interaction with Co-STORM.Source: §4 · p. 5

5 Automatic Evaluation

Automatic evaluation assesses reports and discourse turns using simulated users on WildSeek. Co-STORM outperforms baselines on reported quality measures, and ablations indicate that moderator steering contributes more than multiple experts.

5.1 Evaluation Setup

  • Evaluation uses WildSeek topic-goal pairs, simulates users with an LM, and ends sessions at 30 search queries for each method.Final reports are generated from interaction histories using STORM's two-stage outline and section-generation approach.Source: §5.1 · p. 6

5.2 Automatic Metrics

  • Reports are graded for Relevance, Breadth, Depth, and Novelty; discourse turns receive intent-specific rubrics and unique cited URLs indicate information diversity.The report rubric uses a 5-point scale, and Information Diversity is the average pairwise dissimilarity of collected information.Source: §5.2 · p. 6

5.3 Automatic Evaluation Results

  • Co-STORM surpasses STORM + QA particularly in report Depth and Novelty, and its question-answering turns outperform both baselines in Consistency and Engagement.STORM + QA improves on RAG Chatbot across all four report-grading dimensions; the authors attribute Co-STORM's stronger question-answering turns to its collaborative setup and utterance polishing.Source: §5.3 · pp. 6–7, Table 3 · p. 6

5.4 Ablation Studies

  • Both ablations score below full Co-STORM across report and question-answering metrics, with moderator removal causing a larger decline than reducing experts.One expert plus a moderator retains most of the multi-agent benefits; the moderator asks about unused topic information to steer discovery.Source: §5.4 · p. 7
  • The authors also report controlled experiments on mind-map quality in Appendix B.Source: §5.4 · p. 7

6 Human Evaluation

The human evaluation compared Co-STORM with a search engine and a RAG chatbot across information-seeking tasks, ratings, preferences, and participant feedback. Participants reported broader and deeper information, less effort, and a better overall experience with Co-STORM.

6.1 Evaluation Setup

  • The 20-person evaluation split participants between Google Search and the RAG Chatbot, with paired topic tasks, counterbalanced system order, and ratings plus preference questions.One RAG-group rating without a usage record was excluded from aggregated results.Source: §6.1 · p. 7, §6 · p. 7, §6.2 · p. 7

6.2 Human Evaluation Results

  • Participants rated Co-STORM higher for Breadth and Depth than the search engine (50% and 60%) and RAG Chatbot (67% and 56%), while also rating its Relevance higher.Ratings used a 1–5 scale; Table 4 reports win rates and paired t-test p-values.Source: §6.2 · pp. 7–8, Table 4 · p. 8
  • Participants found Co-STORM required less effort, better mitigated echo chambers, and provided a better overall experience.They also described it as providing more serendipitous information.Source: §6.2 · p. 8
  • 89% of evaluated Co-STORM utterances were rated as steering discourse in an interesting new direction, and the mind map tracked discourse accurately in 71% of 80 snapshots.Participants evaluated 32% of the system's utterances.Source: §6.2 · p. 8
  • Four of 19 participants requested more customization; feedback also said the RAG Chatbot better follows clear-target instructions and Co-STORM should be more concise in those cases.The authors identify adapting discourse to users' evolving mental states and preferences as future work.Source: §6.2 · p. 8

7 Related Works

Related work spans question answering, long-form synthesis, conversational search, multi-agent systems, and collaborative learning. The paper positions Co-STORM as a human-in-the-loop approach for complex, evolving information needs.

Information-Seeking Support in NLP

  • QA work often assumes answers fit within one document or users can formulate complex queries, assumptions that may not hold in complex information seeking.Source: §7 · p. 8
  • Long-form QA, expository writing, and conversational search synthesize information or answer questions, but typically omit human interaction or passively respond to users.Source: §7 · p. 8

Multi-Agent Systems; Collaborative Discourse for Human Learning

  • Prior multi-agent work studies automated tasks and human oversight, while classroom research stresses critical discussion across perspectives and facilitation through questions and complementary information.Source: §7 · p. 9

8 Conclusion

The conclusion presents Co-STORM as an interactive system that emulates collaborative discourse and supports complex information seeking. The authors report that it outperforms search engines and RAG chatbots in surfacing unknown unknowns and reducing mental effort.

8 Conclusion

  • Co-STORM emulates collaborative discourse among users and multiple LM agents, and the authors report better unknown-unknown discovery and lower mental effort than search engines and RAG chatbots.Source: §8 · p. 9

Limitations

The authors identify limits in adapting discourse to users, providing control and multilingual access, and reducing latency.

Limitations

  • Co-STORM does not yet tailor discourse to users' prior knowledge, such as skipping basics for knowledgeable users or introducing concepts progressively for novices.Source: Limitations · p. 9
  • Users sometimes want more control over expert perspectives and utterance length; multilingual deployment would also require diverse-language retrieval, moderation, and conflict identification.Source: Limitations · p. 9
  • Co-STORM has higher latency than the RAG Chatbot because it decides utterance intent and updates the mind map, though the authors say current latency is acceptable for real-time interaction.They identify efficiency improvements as a way to provide a smoother experience.Source: Limitations · p. 9

Ethics Statement

The ethics statement describes protections for dataset participants and grounding practices for generated content, while identifying internet bias and English-only information processing as limitations.

Ethics Statement

  • WildSeek data collection required users’ explicit approval, removed personally identifiable information, and included checks for privacy compliance.The dataset was manually reviewed, and topics involving harmful, illegal, racist, sexual, non-English, or personal content were rejected.Source: Ethics Statement · p. 9
  • Generated content is grounded in publicly accessible internet sources with citations, and the authors report that evaluations reduce hallucinations.They avoid publishing generated content without careful accuracy examination.Source: Ethics Statement · pp. 9–10
  • Internet bias is the primary risk; the system currently processes English sources, with stronger filtering, accuracy checks, and multilingual support identified as possible additions.The authors describe rule-based filtering and multiple sources as mitigations for internet bias.Source: Ethics Statement · p. 10

A Dataset Details

WildSeek is curated from real user information-seeking records, and controlled tests find that Co-STORM’s mind-map insertion outperforms two baselines. The appendix also describes the prompting setup and taxonomy.

A Dataset Details

  • WildSeek collection used explicit login consent, collected no personally identifiable information, and excluded topics involving personal information or experience.The authors also rejected illegal, harmful, racist, sexual, and non-English topics.Source: A Dataset Details · p. 13
  • WildSeek contains 100 manually labeled topic-purpose pairs, downsampled from 6,608 unique pairs and spanning 24 fine-grained categories across six domains.The source application had 8,777 users; topic classification combined gpt-4o-2024-05-13 with human inspection, and rule-based filtering removed trivial purposes.Source: A Dataset Details · p. 13

B Mind Map Insert Operation

  • The insertion benchmark uses 111 FreshWiki-derived tasks, treating each citation’s original outline placement as ground truth.It compares Co-STORM’s insert operation with embedding-only and language-model-only placement; the concept hierarchy evolves as information is added.Source: B Mind Map Insert Operation · p. 13
  • Co-STORM’s insert operation consistently outperformed embedding-only and language-model-only baselines in insertion accuracy.For second- and third-level placements, the evaluation also counts insertion into a ground-truth ancestor as partially correct.Source: B Mind Map Insert Operation · p. 13

C Full Prompts in Co-STORM

  • The appendix provides zero-shot gpt-4o prompts for the expert and moderator roles and identifies the WildSeek taxonomy’s category counts.Parenthetical taxonomy counts include data points in a category and its descendants.Source: C Full Prompts in Co-STORM · p. 13, Figure 5 · p. 14

D Automatic Evaluation Details

Automatic evaluations use Prometheus with shortened inputs to fit its context window. A human-scored sample shows positive correlations for rubric grading, with statistical significance on four of five items.

D Automatic Evaluation Details

  • Report evaluation omits references and trims inputs below 2,000 words, while discourse evaluation uses the last 2,000 words of history.The Prometheus model’s limited context window motivated these reductions.Source: D Automatic Evaluation Details · p. 14
  • Automatic rubric grading positively correlated with average human scores, with statistical significance for four of five rubric items.Two independent evaluators graded a random sample of 50 discourse-quality items, with 10 items for each rubric criterion.Source: D Automatic Evaluation Details · p. 14
  • Figures 6 and 7 show the age and education-level distributions of human-evaluation participants.The supplied captions do not state distribution values.Source: Figure 6 · p. 15, Figure 7 · p. 15

E Human Evaluation Details

This section describes the human-evaluation interface and rating procedure, and illustrates moderator-steered agent discourse through examples focused on AlphaFold 3. The examples cover its modeling capabilities, research and therapeutic applications, practical limitations—including access to downloadable code—and industry use in drug discovery with potential long-term financial benefits.

E Human Evaluation Details

  • Participants rated Google Search, a RAG chatbot, and Co-STORM on four aspects using a 1–5 Likert scale, then gave pairwise preferences after two tasks.The online evaluation used a web application with real-time Co-STORM and RAG-chatbot modes; participants consented to feedback collection, and personally identifiable information was not stored.Source: E Human Evaluation Details · p. 15

F Case Study

  • In the case studies, the moderator steers an NMN discussion toward personalized supplementation and genetic profiling, and an artificial-superintelligence discussion toward quantum digital twins.The appendix also provides a full AlphaFold 3 discourse and associated report, showing moderator- and user-directed shifts among its topics.Source: F Case Study · pp. 15–16

G Full Collaborative Discourse Example

  • The AlphaFold 3 transcript presents a broader biomolecular prediction scope and improved protein-complex predictions, while noting that DeepMind had not released downloadable code.It describes the AlphaFold Server as a free, user-friendly way to generate molecular complexes, and highlights prediction improvements for protein-protein complexes and antibody-protein interfaces.Source: G Full Collaborative Discourse Example · p. 25
  • The conversation links AF3’s interaction predictions to drug discovery, citing a hepatocellular carcinoma drug candidate and a potent hit molecule as examples.The examples were described as involving AlphaFold alongside other AI tools.Source: G Full Collaborative Discourse Example · p. 26
  • AF3’s diffusion-based architecture is described as jointly modeling complexes of proteins, DNA, RNA, small molecules, ions, and modified residues.The conversation contrasts this broad modeling capability with AlphaFold 2’s focus on individual protein structures.Source: G Full Collaborative Discourse Example · p. 26
  • AlphaFold 3 is reported to be 50% more accurate than the best traditional methods on PoseBusters without structural information input, outperforming physics-based tools in this domain.These are claims made in the sample discourse.Source: G Full Collaborative Discourse Example · p. 26
  • The dialogue presents AF3 as useful for understanding genetic regulation and developing targeted therapies, while noting that restricted access may slow wider application.It also cautions that AF3’s transformative impact on drug discovery may remain incremental at this stage.Source: G Full Collaborative Discourse Example · pp. 26–27
  • The dialogue reports at least 50% higher prediction accuracy for protein-DNA and protein-RNA interactions, with some categories doubling, to support molecular insights into gene regulation.The accuracy figures are attributed to cited reports in the conversation.Source: G Full Collaborative Discourse Example · pp. 26–27
  • The discussion proposes combining AF3 with self-driving laboratories to streamline drug-candidate testing, while noting that scaling AF3 requires IT infrastructure and reliable data interoperability.The conversation frames the integration as promising, not a cure-all.Source: G Full Collaborative Discourse Example · p. 28
  • Both large and small firms are using AlphaFold 3 to boost efficiency and innovation in drug discovery, with potentially significant long-term financial benefits.Source: G Full Collaborative Discourse Example · p. 28

H Final Report Example

This report example presents AlphaFold 3’s prediction of interactions across diverse biomolecules, including reported accuracy gains. It also outlines potential research applications and notes limitations around access and the impact on drug development.

Advancements in Biomolecular Structure Prediction

  • Enhanced prediction of protein–DNA and protein–RNA interactions may advance research on gene regulation and personalized medicine [5].The report describes these interaction predictions as especially accurate.Source: H Final Report Example · p. 29
  • AlphaFold 3’s diffusion architecture jointly predicts proteins, nucleic acids, small molecules, ions, and modified residues, improving protein–ligand accuracy over specialized tools [8][38].The reported accuracy improvement is particularly noted for protein–ligand interactions.Source: H Final Report Example · p. 29
  • Some AlphaFold 3 predictions remain inaccurate, and the model does not fully reveal the mechanisms of protein folding [42].Source: H Final Report Example · p. 30

Accessibility Enhancements for Researchers

  • The free, user-friendly AlphaFold Server is powered by AlphaFold 3, but downloadable AlphaFold 3 code was not released; the open-access AlphaFold Database houses over 200 million protein structure predictions [14][13].Source: H Final Report Example · p. 30
  • Accuracy gains also extend to protein–protein complexes and antibody–protein interfaces, offering researchers more reliable data [18].The broader predictive scope also includes nucleic acids, small molecules, ions, and modified residues [12][16][17].Source: H Final Report Example · p. 31

New Features and Advancements in AlphaFold 3

  • AlphaFold 3 uses a diffusion-based model to predict raw atom coordinates and covers biomolecular interactions involving proteins, DNA, RNA, and ligands[29].The report attributes broader prediction capability to the diffusion architecture.Source: H Final Report Example · p. 31

ProteinRNA Interaction Prediction

  • At least 50% improvement in accuracy for interactions between proteins and other molecule types, with accuracy doubled in certain categories compared with existing methods[30].A separate passage reports the same gains for protein interactions with other molecule types[33].Source: H Final Report Example · p. 31

New Features and Advancements in AlphaFold 3

  • 50% more accurate than the best traditional methods on PoseBusters, AlphaFold 3 requires no structural input and is the first AI system reported to outperform physics-based tools[26].The benchmark examples include protein–ligand binding and antibody–target binding.Source: H Final Report Example · p. 32
  • The cited passage identifies antibody binding as an example of a prediction problem relevant to drug discovery[28].Source: H Final Report Example · p. 32

H Final Report Example

  • AlphaFold 3 reports approximately 70% success in accurately predicting protein–protein interactions, and models interactions involving proteins, DNA, RNA, ligands, and ions.Its diffusion-based architecture predicts raw atom coordinates, enabling predictions across a broad range of biomolecular systems.Source: H Final Report Example · pp. 33–34

Efficiency and Accuracy Improvements in Drug Discovery

  • AI-powered compound generation yielded ISM042-2-048, which inhibited CDK20, a crucial HCC protein, with an IC50 of 33.4 ± 22.6 nM and showed selective anti-proliferation in an HCC cell line, marking AlphaFold’s first hit-identification use [22].Source: H Final Report Example · p. 34
  • AlphaFold 3 is described as improving antibody–antigen interaction predictions through bioinformatics and physics, surpassing previous computational methods [24].The report presents this as relevant to immunology and therapeutic antibody development.Source: H Final Report Example · p. 34

Biotechnology Applications

  • AlphaFold 3 predicts structures and interactions across proteins, DNA, RNA, and small-molecule ligands, extending beyond protein-only prediction [1].The report links this broader biomolecular scope to potential biotechnology applications.Source: H Final Report Example · p. 34

Genetic Regulation and Personalized Medicine

  • AlphaFold 3 is reported to improve protein–DNA and protein–RNA interaction prediction by up to 50% or more, with implications for understanding gene regulation [30].The report also connects DNA–protein modeling to targeted gene therapies [50][52].Source: H Final Report Example · p. 35
  • AlphaFold 3 access is described as restricted, potentially limiting its near-term widespread application [51].The report says its implications for genetic research and personalized medicine remain promising as access evolves.Source: H Final Report Example · p. 35

Economic Impact and Market Implications

  • The report says AlphaFold 3’s exact role in accelerating drug discovery remains uncertain, noting that unforeseen interactions cause many drugs to fail before reaching market [71].It nevertheless describes the technology as holding promise for overcoming some traditional hurdles.Source: H Final Report Example · p. 35

Related papers

Loading 2408.15232v2…