PaperlayerView exact revision on arXiv

Source-linked AI summary

Asterism: Exploring and Synthesizing Scattered Observations into Literature-Grounded Hypotheses and Theories

Joseph Chee Chang, Michael D'Arcy, Amy X. Zhang, Pao Siangliulue, Sangho Suh, Aakanksha Naik, Jena D. Hwang, Javier Ramos Benitez, Stella Wroblewski, Matt Latzke, Michael Cuoco, Ruben Lozano-Aguilera, Kris Ganjam, Joel Chan, Doug Downey, Peter Jansen, Kyle J. Travaglini, Daniel S. Weld

arXiv:2610.02673v1cs.HCcs.CLcs.DLcs.IR

TL;DR

Researchers face a costly task synthesizing scattered observations that use varied terminology and abstraction levels, while the useful focus depends on their questions and preferences. Asterism extracts observations into ontology-organized triples and lets researchers curate evidence graphs and steer hypothesis formation; across its studies, researchers retained project-relevant ideas and domain scientists identified mechanisms beyond their standard analyses.

  • Problem

    Theory development is costly because observations are scattered across papers with varied terms and abstraction levels, and useful concepts depend on researchers’ questions and preferences.

    Source: §1 · p. 2, §9 · p. 18

  • Method

    Asterism extracts observations as concept-relation triples organized in an ontology and lets researchers curate evidence graphs to explore literature and guide hypothesis generation.

    Source: §1 · p. 2, §4 · p. 4

  • Results

    Across the studies, participants curated evidence around their projects, chose abstraction levels, and kept hypotheses that surprised them or made sense of conflicting evidence; domain scientists found mechanisms and hypotheses potentially worth testing.

    Source: §9 · p. 18

  • Takeaways & Limitations

    Researchers used Asterism to develop provisional, literature-grounded theories while retaining control over which evidence and hypotheses fit their projects.

    Source: §9 · p. 18

    The base extraction graph remains static during interaction, so users cannot enrich it with additional evidence or custom ontology concepts while synthesizing.

    Source: §8.3.1 · p. 17

Abstract

from arXiv · show

A theory draws many independent observations into one framework with novel hypotheses. A researcher building such a theory must synthesize observations scattered across many papers, each describing related concepts but often in different terms. Which concepts matter most also depends on their preferences and research questions. Recent approaches scale theory synthesis with LLMs, but automate away choices and intuitions from researchers. We present Asterism, which extracts observations from hundreds of papers as concept-relation triples, with concepts unified in a hierarchical ontology. Researchers curate an evidence graph using the ontology and aggregate observations at different levels of granularity to focus theory formation on specific phenomena of interest. In a field deployment (n=10), researchers worked from observations to theories, and kept concepts and hypotheses fitting their preferences. In two case studies, teams of immunology and agriculture researchers discovered mechanisms outside their standard analyses and constructed hypotheses worth follow-up experiments.

1 Introduction

Asterism supports researcher-led theory development from literature by helping researchers explore observations, curate concepts and relations at preferred abstraction levels, and form provisional theories. Its field deployment and case studies found that researchers retained hypotheses fitting their projects and that domain scientists identified mechanisms beyond their usual analyses.

1 Introduction

  • Theory development is costly because relevant observations are spread across papers that use varied phrasing and abstraction levels, while useful levels of generality depend on researchers’ questions and preferences.Researchers may also need to explore beyond concepts they already know.Source: §1 · p. 2
  • Asterism extracts observations from hundreds of papers as concept-relation triples and organizes concepts in an ontology so researchers can aggregate evidence at different abstraction levels.Researchers can zoom concepts up or down to inspect where a relation holds consistently and where papers disagree.Source: §1 · pp. 2–3
  • Researchers curate an evidence graph that guides literature exploration and anchors hypothesis generation, then review candidate hypotheses and synthesize those they keep into an evidence-grounded theory.The researcher and system share concepts, relations, and the distinction between observations and authors’ claims.Source: §1 · p. 3
  • In one example, Example-based Explanations increased decision-making accuracy across five papers while also increasing both AI overreliance and underreliance.A generated hypothesis proposed that this pattern applies when AI accuracy is roughly 50–80%; its advantage diminishes as accuracy approaches 100%.Source: §1 · p. 3
  • In a field deployment and two domain case studies, researchers curated evidence around their projects and preferences, while domain scientists found mechanisms and hypotheses for potential experiments.Researchers also attended to abstraction levels and conflicting evidence.Source: §1 · p. 3

2 Related Work

Existing literature tools support synthesis, ideation, and computational discovery, but few support interactive theory building from broad literature. Asterism builds on autonomous theory-generation work by adding researcher input to steer exploration and synthesis.

2.1 Interactive Literature Synthesis Tools

  • Literature-synthesis tools organize related work and deep-research reports, while ideation tools ground or pivot proposals against literature; these approaches generally target a single future study rather than broad theory development.Interactive theory-development tools remain scarce.Source: §2.1 · p. 3

2.2 Computational Scientific Discovery

  • Computational scientific discovery distinguishes taxonomies of concepts, laws describing relations, and theories that holistically explain laws and their observations.The passage illustrates these categories with biological taxonomies, Fitts’s Law, and Information Foraging Theory.Source: §2.2 · pp. 3–4
  • Recent LLM discovery systems either search datasets autonomously or generate theories from literature, while interactive hypothesis exploration can use shared representations and AI hints to support backtracking.Theorizer generates theories from hundreds of papers but offers no researcher input to steer concepts of focus.Source: §2.2 · p. 4

3 Design Goals

The design goals make theory development an interactive process in which researchers explore literature, choose the abstraction level of evidence, and steer synthesis. A shared, visual evidence graph is intended to keep generated theories grounded in observations.

3 Design Goals

  • DG1 calls for extracting experimental observations across many papers as concept-relation triples connected by an ontology.This structures observations across concepts and abstraction levels.Source: §3 · p. 4
  • DG2 lets users explore from prior knowledge and curate an evidence graph using concepts at their preferred abstraction level.The goal is to support exploration for new insights while allowing researchers to choose what to focus on.Source: §3 · p. 4
  • DG3 uses the evidence graph as a shared representation to steer hypothesis and theory generation, while DG4 calls for interactive visualization that grounds generated theories in concrete observations.Source: §3 · p. 4

4 Asterism: User Scenario & Synthesis Pipeline

Asterism turns research papers into an ontology-linked evidence graph that researchers curate and explore, then use to select hypotheses for an evidence-linked theory.

4 Asterism: User Scenario & Synthesis Pipeline

  • Starting from 100–200 relevant papers, users curate concepts and relations, triage hypotheses from graph regions, then connect selected hypotheses in a theory that flags unaccounted-for or conflicting evidence.The theory retains the selected hypotheses verbatim and identifies evidence that does not fit or is not addressed.Source: §4 · p. 4

4.1 Building an Evidence Graph from Papers

  • From around 100 papers, Asterism extracted roughly 1,700 relations over 500 concepts and inferred an ontology linking concepts across papers and abstraction levels.The scenario corpus yielded about 1,700 relations over 500 concepts.Source: §4.1 · p. 4

4.2 User-driven Concept-relations Discovery

  • Researchers—not the model—choose which concepts to focus on; relation ranking favors more numerous, agreeing papers while penalizing abstraction broad enough to hide distinct relations.When the same relation appears at multiple ontology levels, Asterism lists only the highest-scoring level.Source: §4.2 · pp. 4–5, §4.2.2 · p. 5

4.2.2 Expanding from a selected node.

  • Selecting a node exposes related findings, ontology neighbors, and hypotheses; chat searches the canvas and corpus, letting researchers add or reject proposed concepts and relations.The agent can propose connections between disconnected canvas clusters and return best-supported relations pre-checked.Source: §4.2.2 · p. 5, §4.2.3 · pp. 5–6

4.3 Drilling into the Evidence

  • Evidence panels show paper counts and agreement, then unfold normalized relations into raw phrasing, observations, experimental setups, and highlighted source sentences for verification.The example edge is supported by six papers, five reporting an increase.Source: §4.3 · pp. 6–7, Figure 8 · p. 9

4.4.1 Generating and triaging hypotheses.

  • A generated hypothesis contrasted explanation types; Dora selected five hypotheses for a theory and flagged one with far fewer observations as a candidate for her lab’s next experiment.The example-based explanation hypothesis proposed a condition under which accuracy rises even as both overreliance and underreliance increase.Source: §4.4.1 · p. 7, §4.4.2 · p. 7

4.5.1 Phase 1: Initial Extraction.

  • Backend extraction separates experimental observations from author claims and retains paragraph evidence; ontology-constrained triples unify terminology, while relation normalization maps findings to at most twelve labels.The ontology is inferred from experiment designs, observations, and claims; seven relation labels are fixed and five remain query-specific.Source: §4.5.1 · p. 8, §4.5.2 · p. 8, §4.5.3 · p. 8

4.5.5 Phase 5: Hypothesis Generation.

  • Hypotheses use the researcher’s chosen abstraction and per-paper evidence for conditions; theory synthesis keeps selected hypotheses verbatim, connects them under one lens, and identifies unaccounted-for or conflicting evidence.Generated content references evidence edges so users can trace it back through the graph to highlighted PDF snippets.Source: §4.5.5 · p. 8, §4.5.6 · pp. 8–9

5 Generation Pipeline Technical Evaluation

Asterism’s ontology is compared with survey outlines by matching concepts individually, rather than requiring both systems to reproduce the same outline. On this reference comparison, Asterism’s ontology matched survey concepts more closely overall, though the authors caution that the pipelines target different properties.

5 Generation Pipeline Technical Evaluation

  • Asterism’s ontology supports more specific concepts than thematic survey outlines, such as outcomes that papers measure but are rarely about.The comparison is a reference point, not a claim that one method outperforms the other, because the pipelines are rewarded for different properties.Source: §5 · pp. 9–10
  • The evaluation matched each survey concept to the output concept with the highest paper-set overlap, using Jaccard similarity.It excluded 51 survey concepts that covered either all or a single paper.Source: §5 · p. 10
  • Asterism generated concepts beyond survey themes, including hallucination in a math-language-model survey, where it is typically reported as a limitation.Whether researchers could find such concepts in practice was left to the field study.Source: §5 · p. 10
  • 71% of 288 non-trivial survey concepts had a closer match in Asterism’s ontology than in Zhu et al.’s method (19%; 55 concepts).The within-ontology win rate averaged 74.0% across 12 ontologies and exceeded 50% in each; mean best-match Jaccard was 0.508 for Asterism and 0.375 for Zhu et al.Source: §5 · p. 10, Table 1 · p. 11

6 Study 1: Field Deployment w/ CS Researchers

In a field deployment with 10 researchers, participants built theories around their own research questions and reported learning from Asterism. They selected concepts and hypotheses according to personal relevance and prior knowledge, while also engaging with abstraction, breadth, and conflicting evidence.

6 Study 1: Field Deployment w/ CS Researchers

  • The deployment asked researchers to use Asterism on graphs built from their own questions, at their own pace, and submit at least one theory.The 10 completers were graduate students in computer science or adjacent fields, recruited through online research communities; 21 researchers signed up.Source: §6 · p. 11, §6.1 · p. 11

6.2 Procedure; 6.3 Data Collection and Analysis

  • Participants spent 1–2 weeks using the system for one to two hours, then submitted a theory and took part in an interview; analysis combined logs, artifacts, and interviews.The first author used multiple rounds of thematic coding of interview transcripts.Source: §6.2 · p. 11, §6.3 · p. 11

6.4 Findings

  • All 10 participants reported learning something; they averaged 117.8 active minutes, and 6/10 voluntarily built additional graphs or theories.Most participants worked on an already-underway project; one planned to test hypotheses in fieldwork the following week.Source: §6.4 · p. 11
  • P0 added 60 concepts and removed 16, while P7 added 51 and removed none; both read evidence while building graphs, but differed in how they revisited it.P7 returned to graph evidence after theory creation; P0 mostly read evidence in the theory report.Source: §6.4 · p. 12, Figure 10 · p. 12

6.4.1 Participants valued personal relevance and prior knowledge

  • Participants prioritized project relevance and prior knowledge over paper counts; 66% of added concepts came from agent proposals.Nine of 10 judged concepts by relevance or familiarity, and only one participant reported removing a concept because it had little paper support.Source: §6.4.1 · p. 12
  • Participants used nested ontology levels to find a useful abstraction, but canvas overload prompted cleanup requests and hypotheses captured context-specific or contradictory relations.Nine of 10 considered how general or specific hypotheses were; five of 10 kept, exported, or named a hypothesis centered on contradicting or context-specific relations.Source: §6.4.2 · pp. 12–13, §6.4.1 · p. 12

6.4.3 Participants kept hypotheses that were surprising and covered

  • Participants favored personally relevant, surprising hypotheses that covered more canvas relations; hypotheses built mostly from leaf concepts were kept less often.Personal relevance was the most common interview reason for keeping or dismissing a hypothesis (9/10); breadth predicted keeping in the regression, while LeafConcepts had a negative coefficient.Source: §6.4.3 · p. 13
  • 81% to 33% when the papers behind a relation agreed, versus 43% to 49% when they disagreed, as AnchorPapers moved from −1 to +1 SD.Participants attended to opposing evidence, and two participants who doubted a generated paradox were convinced by its extracted evidence.Source: §6.4.3 · pp. 13–14

7 Study 2: Case Studies with Domain Scientists

The two case studies ran for 2–4 months with agricultural and immunology groups exploring active research questions; the agriculture researchers also built graphs and theories themselves. Researchers surfaced unfamiliar mechanisms, challenged assumptions, and identified hypotheses relevant to follow-up experiments.

7.2.3 Experiments, novelty, and feasibility.

  • At mean property values, Figure 12 gives a 53% modeled keep probability; triage also reflected personal preferences, novelty, evidence counts, and lab feasibility.I2 treated abundant evidence as a reason a hypothesis might already be known, while feasibility and resources affected experiment choices.Source: Figure 12 · p. 14, §7 · p. 14, §7.2.3 · p. 16

7.1.1 Direct comparison to Theorizer on the same query.

  • A1 preferred Asterism to Theorizer on the same query because concept-relations clarified hypothesis reasoning and made exceptions visible.He highlighted that a relationship held for one plant group but not another.Source: §7.1.1 · pp. 14–15

7.1.2 A1 checked the theory against the literature and against his

  • A1 checked the theory against public protein databases, found its content correct, and valued its scope, novelty, and special-case information.He also found the report useful for writing a paper’s discussion section.Source: §7.1.2 · p. 15

7.1.3 A1 valued parts of theories that pointed at experiments.

  • Unaccounted-for mechanisms prompted A1 to notice links to heat-shock response and propose a roughly $10,000 growth-chamber temperature-gradient experiment.He estimated that one or two such experiments could support roughly six new papers.Source: §7.1.3 · p. 15

7.1.4 Interacting with Asterism.

  • Self-directed use surfaced hifiasm to A1 and led A2 to question additive landrace-conservation scores when one household remains the sole custodian.A2’s theory described the four conservation dimensions as non-additive, threshold-dependent, and asymmetrically compensatory.Source: §7.1.4 · p. 15

7.1.5 Reading the canvas as ideas instead of papers.

  • A1 treated the canvas as ideation, using shared assembly-quality links to favor multiple sequencing platforms; paper-only search misses curated coffee-genome records.He noted that neither platform alone produced a gap-free kiwifruit assembly.Source: §7.1.5 · p. 15, §7.1.6 · p. 15

7.2.1 Interactive canvas versus static theory reports.

  • With only 10 input genes, adding known context shifted predictions toward the immunology team’s expectations; the team also considered testing TLR4 feasible.I2 said the initial prediction was close to the opposite of what her team expected and attributed this to the limited query.Source: §7.2.1 · p. 16, §7.2.2 · p. 16

7 Study 2: Case Studies with Domain Scientists

  • The immunology team found TLR4 signaling outside its standard pathway analysis, while agriculture researchers surfaced overlooked mechanisms and experiment-relevant hypotheses.A2’s hypothesis challenged additive conservation scoring when only one household maintains a landrace.Source: §7.1.3 · p. 15, §7.1.4 · p. 15, §7.2.2 · p. 16

8 Discussion

The discussion describes design trade-offs, study and system boundaries, and future directions for extending Asterism. Researchers valued unexpected relationships, but reported overload and limits in how the system represents and supports evidence-based theorizing.

8.1.2 Users were more selective of concepts than of relations.

  • Participants still reported being overwhelmed by relations, even as many discovered unexpected, useful relationships between known concepts.The design shifted to a blank canvas and added ways to explore concepts; edges were de-emphasized and auto-accepted above a user-adjustable unique-source threshold.Source: §8.1.2 · pp. 16–17, §8.1.1 · p. 16

8.2 Study Limitations

  • The field deployment favored depth over a controlled baseline comparison, giving participants multiple hours over one to two weeks to work at their own pace.Two unincentivized case studies involved agricultural and immunology researchers using Asterism on their own questions.Source: §8.2.1 · p. 17

8.2.1 User study.

  • The study did not rerun Theorizer’s knowledge-cutoff benchmarks; Asterism’s evidence structure is designed for interactivity, not fully autonomous theory generation.The authors instead evaluated extracted evidence against 12 expert-curated survey outlines and studied the interactive system with researchers.Source: §8.2.2 · p. 17

8.3 System Limitations

  • The base graph remains static during interaction, so users cannot add evidence or reorganize the ontology within the synthesis workflow.The authors identify fluid transitions between evidence gathering and synthesis as an open research question.Source: §8.3.1 · p. 17

8.3.2 Visualization: Depth and Context.

  • Graph visualization is less useful for domains with shallow concept-relation structures, such as methods research organized around shared benchmarks.One participant said a list of concepts might have served their breadth-oriented work better.Source: §8.3.2 · p. 17
  • Relations cannot be split visually by context, although context remains in evidence snippets and experiment-design fields that chat and hypothesis generation can synthesize.The authors note that context is sparse and difficult to normalize, complicating its visualization.Source: §8.3.2 · p. 17

8.4 Future Work and Opportunities

  • In natural sciences with curated vocabularies, an inferred ontology may miss expected concepts when open-access literature is thin; unique-source counts also do not capture domain-specific uncertainty.Future work includes community-curated ontologies and extracting detailed statistical analyses for meta-analysis.Source: §8.4.1 · p. 17

8.4.3 From hypotheses to experiments to new evidence.

  • Future work could let researchers develop one hypothesis in detail while retaining the broader graph, and connect theorizing to experiment design and new evidence.The authors propose ranking experiments by feasibility and incorporating unpublished results as new concept-relation triples.Source: §8.4.2 · p. 17, §8.4.3 · pp. 17–18

9 Conclusion

Asterism turns observations from many papers into an ontology-organized evidence graph that researchers curate while forming provisional theories. In evaluation, researchers kept hypotheses that fit their preferences, and domain scientists identified mechanisms beyond standard analyses and hypotheses they considered potentially worth experiments.

9 Conclusion

  • Asterism extracts thousands of experimental observations from up to hundreds of papers into normalized concept-relation triples organized by a generated ontology.Researchers curate an evidence graph and triage hypothesis candidates to form a provisional theory.Source: §9 · p. 18
  • Researchers curated graphs for relevance to their projects, chose abstraction levels, and kept hypotheses that surprised them, covered more of the graph, and made sense of disagreement.In two case studies, immunology and agriculture researchers found mechanisms outside standard analyses and hypotheses potentially worth an experiment.Source: §9 · p. 18
  • The authors position Asterism between ideation tools grounded in a few papers and fully autonomous scientific discovery, aiming to retain researchers’ intuition and preferences.They envision theory synthesis as an ideation probe that researchers can reframe or discard before deciding what merits an experiment.Source: §9 · p. 18

A Example Generated Hypotheses

The appendix presents two system-generated hypotheses, with one discussed in the introduction and the other in the agriculture case study.

A Example Generated Hypotheses

A.1 Asymmetric Error Correction Advantage

The appendix proposes that example-based explanations can improve team performance by helping people detect AI errors, despite also increasing overreliance and underreliance. The advantage depends on there being errors users can detect and examples that make them recognizable.

A.1 Asymmetric Error Correction Advantage

  • The proposed advantage applies where AI accuracy is below 100% and examples accompany predictions; it diminishes near-perfect accuracy and disappears when adversarial examples are indistinguishable to lay users.The authors say it is strongest when example quality and diversity reveal AI errors without causing excessive general skepticism.Source: §A.1 · p. 20
  • On CUB, CHM-Corr users correctly rejected 53.45% of misclassified images versus 41.22% for no-explanation users, a +12.23pp gain on errors.The passage reports an approximately 4pp loss on correct predictions, producing positive team performance.Source: §A.1 · p. 20
  • Example-based explanations yielded complementary performance in income prediction (71.1% vs. 61.1% human-alone vs. 62.5% AI) and biography classification (71.2% vs. 60.0% vs. 62.5%).Feature-based explanations did not improve over the no-AI baseline; 3-NN was the only explanation type with complementary performance on fine-grained dog classification (82.88% vs. 81.14% AI-only).Source: §A.1 · p. 20
  • On adversarial ImageNet images where AI was always wrong, 3-NN achieved 75.07% error-detection accuracy, a consistent +4% over confidence-only.Example-based explanations did not decrease participant accuracy when AI was wrong in income or biography classification, while feature-based explanations caused significant decreases.Source: §A.1 · p. 20

A.2 Stewardship Singularity Threshold

The section describes a stewardship threshold: when a landrace depends on a single custodian household, losing that custodian can leave no other conservation measure able to compensate. It situates this risk in farmer-maintained landraces and identifies exceptions and sharper vulnerabilities.

A.2 Stewardship Singularity Threshold

  • When only one household or family maintains a landrace, losing that custodian can eliminate it from active cultivation, regardless of geographic dispersion, altitudinal breadth, or cultivation abundance.The section describes persistence as increasingly fragile below approximately seven maintaining households per community.Source: §A.2 · p. 20
  • The analysis applies to farmer-maintained landraces in smallholder systems, especially where functional ex situ collections are unavailable; it does not apply when viable, accessible genebank collections enable reintroduction..Source: §A.2 · p. 20
  • Cultural motivations may sustain landraces at low household-stewardship levels, but erosion of cultural practices can still bring them to an extinction threshold; clonally propagated crops and post-disaster settings face sharper risks.Clonally propagated crops are harder to store or exchange informally, while disasters can lower stewardship across multiple communities simultaneously.Source: §A.2 · p. 20
  • A participatory analysis used seven of fifteen households as the threshold between “many” and “few” maintainers, classifying landraces below it as vulnerable.Farmer migration can reduce active maintainers below viable levels.Source: §A.2 · p. 21

B Chat Agent Tools

The chat agent’s tools are grouped by their role in the interface and execution location. Most operate in the browser, and none directly writes to the canvas.

B Chat Agent Tools

  • Table 6’s tools are ordered from finding graph concepts and relations, to proposing canvas changes, reading evidence, and tracking session and theory state.The interface also exposes each tool’s system prompt and the canvas snapshot it receives.Source: B Chat Agent Tools · p. 21
  • Only ask_paper runs on the server; the other ten tools are dispatched in the browser against live interface state.The ranked triple pool, canvas, and candidate hypothesis list remain client state.Source: B Chat Agent Tools · p. 21
  • No tool writes directly to the canvas; tools that propose changes render cards for the researcher to accept or dismiss.Source: B Chat Agent Tools · p. 21

C Triple Ranking and the Default Zoom Level

Asterism lifts extracted triples through the ontology, combines evidence for concept pairs, and filters the candidates before ranking them. It removes redundant ancestor–descendant pairs so each relation appears at the abstraction level where it scores best.

C Triple Ranking and the Default Zoom Level

D Evidence Sampling for Hypothesis Pre-generation

Asterism samples aggregated graph evidence across multiple hypothesis-generation rounds because the full graph may exceed a prompt’s capacity. Its sampling and run structure aim to expose different evidence while placing stronger-supported edges first.

D Evidence Sampling for Hypothesis Pre-generation

E Sanity Checks under Zhu et al.’s Protocol

The sanity check adapts Zhu et al.’s partition-based protocol to Asterism’s multi-concept ontology, and a perfectly guided user scores on par with Zhu et al. across all three metrics.

E Sanity Checks under Zhu et al.’s Protocol

  • Because Asterism attaches each paper to multiple concepts rather than one, the comparison scores survey concepts individually instead of comparing a single partition.Zhu et al.’s protocol assigns each paper to its most-immediate concept and compares partitions using NMI, ARI, and Purity.Source: E Sanity Checks under Zhu et al.’s… · p. 22
  • Using the same models, the authors ran Zhu et al.’s best-performing method with k-means, while a Gaussian-mixture ablation scored lower on all three metrics.Their run scored within ±0.04 of published TaxoBench-CS results on the 12-ontology subset: NMI 0.622, ARI 0.185, and Purity 0.585.Source: E Sanity Checks under Zhu et al.’s… · p. 22
  • A perfectly guided user selecting the closest ontology concept for each survey concept produced partition scores on par with Zhu et al. across all three metrics.Table 7 reports partition-level scores under Zhu et al.’s protocol, with 95% confidence intervals over 12 ontologies.Source: E Sanity Checks under Zhu et al.’s… · p. 22, Table 7 · p. 22

F Hypothesis Triage Model: Fit and Checks

The full hypothesis-triage model fit better than a random-effects-only model. The reported checks found no added variance from two further fixed effects, while alternative specifications tested coefficient stability and adjusted agreement estimates for sparse evidence.

F Hypothesis Triage Model: Fit and Checks

  • Adding hypothesis rank and the share of relations backed by a single paper explained no further variance (χ2=0.74, p=0.39).Quadratic and spline terms for AnchorPapers were also null.Source: F Hypothesis Triage Model: Fit and… · p. 22
  • Every coefficient kept its sign under batch-stratified conditional logistic regression, leave-one-participant-out refits, and stricter negative coding, but only LeafConcepts remained significant under the strict coding.AnchorPapers’ quadratic and spline terms were null.Source: F Hypothesis Triage Model: Fit and… · p. 22
  • AnchorAgreement was shrunk toward the corpus base rate according to evidence count, using an empirical-Bayes Beta(7.69, 2.16) prior fitted to 387 cited relations with at least two directional findings.Thus, 50% agreement among 12 sources counts as a bigger split than 50% agreement among 2.Source: F Hypothesis Triage Model: Fit and… · p. 23

G Prompts

Appendix G documents Asterism’s prompts for extracting evidence-backed observations, organizing concepts in a hierarchical ontology, and normalizing relation labels. It also covers prompts for adapting hypothesis and theory generation to researchers’ curated inputs and guiding an evidence-aware chat agent through interactive concept-graph exploration.

G.1 Finding Extraction

  • Asterism uses one prompt for both extraction passes: the first has an empty ontology block, while the second renders the ontology and requests endpoint links.The linked fields identify matching ontology node IDs for each triplet’s source and target concepts.Source: §G.1 · pp. 23, 25
  • The prompt requests experiment designs, specific observations, authors’ claims, and exact supporting evidence excerpts, prioritizing results, discussion, and conclusions.It calls for 3–15 observations depending on the paper and 1–3 evidence excerpts per observation and experiment.Source: §G.1 · p. 23
  • Triplet endpoints must be specific, nameable concepts; experimental conditions belong in context rather than being hidden in a concept or relation.The prompt favors named entities and short relation phrases, and says to split long or multi-clause concepts into simpler triplets.Source: §G.1 · pp. 23–24
  • The prompt distinguishes reproducible, experiment-specific observations from broader, generalizable claims and asks for comprehensive coverage, including relevant weaker results and limitations.Claims should be understandable without the paper; weaker results and limitations count when authors make them as findings or future work.Source: §G.1 · pp. 23–24
  • The prompt requires triplets for each observation and claim, separate triplets for distinct entities and causal relations, and exhaustive coverage of conjunctions.It advises using the most specific available concept name rather than adding generic restatements, since generalization is handled downstream by the ontology hierarchy.Source: §G.1 · p. 24
  • In reflection mode, triplets must be grounded in paper observations and reuse exact ontology labels, choosing the most specific matching concept.When an observation names multiple ontology concepts, the prompt calls for a separate triplet for each.Source: §G.1 · pp. 24–25
  • For ontology links, the prompt specifies the exact node when available, the closest parent for an unlisted specific variant, or null when no concept matches.The prompt says this linking lets the downstream causal graph map triplet concepts to the ontology hierarchy.Source: §G.1 · p. 25
  • The finding-extraction prompt asks for detailed experiments and their observations from the paper.Source: §G.1 · p. 25

G.2 Ontology Construction

  • The ontology organizes concepts through hierarchical taxonomy, while a separate causal graph represents causal relationships and supports interactive zooming.Users can collapse concepts into ontology parents or expand them to inspect more specific concepts and causal links.Source: §G.2 · p. 25
  • Meaningful sibling groupings determine which concepts collapse together and which causal edges aggregate, revealing cross-paper patterns at different abstraction levels.Multiple parents provide alternative zoom paths for examining the same concept.Source: §G.2 · p. 26
  • Every concept in an observation triplet must map to an ontology node; normalize terminology and merge only exact semantic equivalents, preserving distinct specific concepts.The prompt explicitly keeps different models, methods, and learning settings as separate leaf nodes.Source: §G.2 · p. 26
  • The ontology is a directed acyclic taxonomy using is_a and part_of relations, with multiple parents allowed and intermediate nodes bridging specific concepts to abstract roots.Most branches should have 3–5 levels, with intermediate nodes providing useful zoom stops.Source: §G.2 · p. 26
  • Prompts recommend using claims and experiment designs, as well as triplets, to infer useful intermediate categories that organize specific concepts.Categories expressed in claims, such as thinking models or parameter-efficient methods, should become ontology nodes.Source: §G.2 · pp. 26–27
  • For outcome-, metric-, or phenomenon-like concepts, add beneficial, adverse, or neutral intermediate groups only when valence is unambiguous and siblings mix valences.Skip the grouping when valence is context-dependent or a valence label would be artificial.Source: §G.2 · p. 27
  • The ontology uses specific, intermediate, and abstract levels, with domain-informed intermediate concepts bridging observed leaves to broader roots.The example hierarchy has three to four levels; intermediate nodes need not appear in the input triplets.Source: §G.2 · pp. 27–29
  • Nodes receive consistent categories, such as method, metric, phenomenon, outcome, system, task, or dataset, with a recommended limit of 8–10 categories.The prompt also includes variable, intervention, and property as common categories.Source: §G.2 · pp. 27–28
  • Specific concepts remain separate leaf nodes, while normalization merges only exact variants at the leaves and meaning-equivalent concepts higher in the hierarchy.Every triplet source and target must map to an ontology node, and merges are documented in normalization decisions.Source: §G.2 · p. 29
  • For outcome-, metric-, or phenomenon-like concepts with clearly positive, negative, or neutral siblings, the hierarchy inserts valence groups and favors domain-natural names.Examples include “Beneficial Clinical Outcome” and “Adverse Clinical Outcome.”Source: §G.2 · pp. 27, 29

G.3 Relation Normalization

  • Relation normalization collapses synonyms so zoomed-out edges trigger “mixed effects” warnings only when studies genuinely disagree about causal direction.Leaving synonyms such as “improves” and “enhances” separate can create false warnings.Source: §G.3 · p. 29
  • The prompt requires seven canonical labels: increases, decreases, no effect, correlates with, outperforms, underperforms, and matches.The labels distinguish causal changes, undirected associations, and direct comparisons.Source: §G.3 · p. 29
  • The prompt allows up to five extra labels only when at least five input relations form a meaningful cluster, and says to ignore modifiers and mechanism phrases.Numbers, adverbs, hedges, and mechanism descriptions do not affect label assignment.Source: §G.3 · pp. 29–30
  • Relation labels follow the verb’s direction for the target, not whether the outcome is desirable: reductions are decreases, while causes and introduces are increases.Thus “reduces errors” is decreases, while “causes errors” is increases.Source: §G.3 · pp. 29–30
  • Downward verbs applied to inherently declining or loss-related objects map to increases, but mechanism phrases are ignored and neutral targets retain the downward label.For example, “reduces loss” maps to increases, whereas “mitigates hallucination” maps to decreases.Source: §G.3 · p. 30
  • Relation normalization assigns each input exactly one label based on the main verb’s effect on its target, with increases and decreases as the dominant categories.Direct comparison labels apply only to A-versus-B comparisons, while “correlates with” is reserved for undirected associations.Source: §G.3 · p. 30

G.4 Hypothesis and Theory Generation

  • The hypothesis and theory prompts, adapted from Jansen et al., use the researcher’s curated canvas or a weighted graph sample, and build theory around selected hypotheses.These replace a fixed retrieved corpus and independently generated hypotheses, respectively.Source: §G.4 · p. 30

G.5 Paper Question Answering

  • Paper question answering retrieves the full text first and requests a concise, comprehensive answer supported by exact paper excerpts, with at most five spans and 500 words.The prompt requires objectively supported answers in JSON fields for the answer and excerpts.Source: §G.5 · p. 30

G.6 Chat Agent System Prompt

  • The agent presents candidate concepts or relations in interactive cards, proposing additions without asking permission; users review them before the canvas changes.For orientation, it offers concepts; for a chosen target or requested links, it offers relations, and when unsure it defaults to concepts.Source: §G.6 · p. 31
  • For scientific-content answers, the agent must read findings and escalate to the paper when needed; canvas summaries alone suffice for topology questions.It can suggest 1–3 short, user-voiced clickable next steps when useful, matching explicit choices already presented.Source: §G.6 · pp. 31–32
  • The agent updates a conversation-specific goal after the first user message and whenever the user narrows or pivots, while keeping it distinct from the immutable corpus query.It should leave the goal unchanged for chit-chat or one-off questions without sustained intent.Source: §G.6 · p. 32
  • For existing canvas entities, use relation chips without repeating their displayed labels, and resolve user concept mentions to human-readable labels rather than echoing input markers.Newly proposed triples should not be chipped because their proposal card already displays them in full.Source: §G.6 · pp. 32–33
  • Canvas context distinguishes new neighboring triples, missing relations between existing concepts, ontology links for abstraction, and the immutable corpus question that frames recommendations.The prompt cautions against zooming out to very high-level concepts and treats concepts without ontology context as paper-local entities.Source: §G.6 · p. 33
  • The graph-building loop plans missing structural roles, checks canvas context, explores from existing concepts, critiques weak candidates, and proposes one focused batch of 3–8 triples per turn.The batch should address one role or sub-mechanism rather than overwhelm the user with multiple cards.Source: §G.6 · p. 33
  • Each proposal batch includes a short story about its structural role and weak evidence; the widget displays each triple’s rationale and evidence, so the accompanying message is usually empty.Order proposals strongest-first because single relations clearing the evidence floor start pre-checked.Source: §G.6 · pp. 33–34
  • For orientation, propose up to eight new ontology concepts in best-first order; each launcher adds one concept and opens a relations widget, and concepts already on the canvas must not be proposed.The concept label and paper count are rendered from the data, so the gloss should not supply a paper count.Source: §G.6 · p. 34
  • The theory workflow treats candidate hypotheses separately from canvas relations: consult the user’s theory stage and annotations, and use filtering only while hypotheses are being selected.Name hypotheses with human-readable chips rather than raw IDs, and do not confuse them with suggested triples.Source: §G.6 · pp. 33–34

Related papers

Loading 2610.02673v1…