Human-LLM Compound System for Scientific Ideation through Facet Recombination and Novelty Evaluation

Introduction

• Scideator combines three LLM-powered modules—Analogous Paper Facet Finder, Faceted Idea Generator, and Idea Novelty Checker—using purposes, mechanisms, and evaluations throughout.Users can select, modify, or build on system outputs at each stage.Source: §1 · pp. 2–3
• Users select candidate facets for recombination, steer modules through facet-level actions, and retain judgment over which ideas to pursue and novelty signals to weigh.Scideator retrieves analogous papers, generates ideas from promising facet analogies, and suggests replacing a facet to improve an idea’s novelty.Source: §1 · p. 2
• The novelty checker defines novelty by difference in a core facet, a unique facet combination, or application to a new domain, and provides explanations and related papers.Participants tended to lower novelty assessments when Scideator classified ideas as not novel and could verify this using the papers and explanations.Source: §1 · p. 2
• In a within-subjects study with 22 computer-science researchers, Scideator provided significantly more creativity support than a same-LLM baseline, especially for exploring ideas.Participants cited Scideator’s facets or ideas as sources of new concepts, while baseline participants most often cited input papers.Source: §1 · pp. 2–3
• Novelty-checker ablations found facet-based re-ranking surfaced more relevant prior work than general-relevance re-ranking or embedding-only retrieval, and facet-grounded classification outperformed whole-idea-and-paper classifiers.Together with the user study, these findings provide the first systematic evaluation of novelty assessment in a human-AI ideation system.Source: §1 · pp. 2–3

• Scideator’s building blocks include facets, analogous facet generation and paper retrieval, idea generation, and novelty assessment, which Section 3 shows working together in a human-directed workflow.Source: §2 · p. 3

Shared Representation: Paper Facets

• Scideator adds an evaluation facet to prior work’s purpose and mechanism facets [26] [28], extracting all three as phrases of no more than seven words from paper titles and abstracts.Purpose captures the problem, mechanism the proposed solution, and evaluation how to determine whether the solution works.Source: Shared Representation: Paper Facets · p. 3
• Purpose and mechanism facets drive analogous-paper retrieval, while all three facets describe retrieved papers, compose ideas, assess novelty, and suggest improvements.Users select, combine, and revise the same facets across stages, making their interactions precise signals that shape subsequent module outputs.Source: Shared Representation: Paper Facets · p. 3

Module 1: Analogous Paper Facet Finder

• The module retrieves analogous papers using purpose-mechanism pairs at controlled conceptual distances and extracts all three facets, including evaluation.Source: Module 1: Analogous Paper Facet Finder · p. 3
• Its near, far, and very far levels span same-topic alternatives, different topics in the same computer science subarea, and high-level analogies from other subareas.Near pairs address a similar problem with a different approach; far pairs share a more abstract structural parallel.Source: Module 1: Analogous Paper Facet Finder · p. 3
• For each of three distances, the module generates four facet-pairs and grounds them through LLM-generated Semantic Scholar queries, shortening queries iteratively when needed.The first retrieved paper represents each pair, while three others provide additional context; four very-near papers are retrieved separately via Semantic Scholar’s similarity endpoint [33].Source: Module 1: Analogous Paper Facet Finder · p. 3
• The module extracts purpose, mechanism, and evaluation facets from 16 retrieved papers and summarizes relevant input and very-near works for the idea generator to distinguish new ideas from existing work.Source: Module 1: Analogous Paper Facet Finder · p. 3

Module 2: Faceted Idea Generator

• Users can select or add facets, or rely on automatic selection, then evaluate ideas, review novelty classifications with paper-based rationales, and iterate with facet-swapped alternatives when needed.When an idea is classified as not novel, Scideator suggests alternatives by swapping one facet; users can adopt a suggestion and return to facet selection.Source: Module 2: Faceted Idea Generator · p. 4
• The generator proposes six analogies between purpose–mechanism pairs from two distance groups, selects the two strongest, then converts them into reciprocal purpose/mechanism combinations.Analogy quality is judged by understandability, relevance, feasibility, specificity, and novelty.Source: Module 2: Faceted Idea Generator · p. 4
• It adapts recombination to user intent: no selections maximize diversity across distance groups, one selected facet is paired with complementary facets, and two are combined directly.Near-paper combinations tend to stay within a research area but use a different approach, while far or very-far combinations cross domain boundaries.Source: Module 2: Faceted Idea Generator · p. 4
• Before finalizing each idea, the system differentiates it from existing work in a relevant-works summary and self-critiques it against quality criteria.Source: Module 2: Faceted Idea Generator · p. 4

Module 3: Idea Novelty Verification

• The checker uses a four-step retrieve-then-rerank pipeline to find candidate papers, select the most relevant, assess novelty, and suggest more novel ideas.Its stages are described in Figure 12 in the Appendix.Source: Module 3: Idea Novelty Verification · p. 4
• Candidate papers come from prior modules and related works, keyword queries, and snippet-text search, broadening retrieval beyond facet-insensitive simple methods [33] [43] [63].Keyword queries use LLM-extracted keywords and potential titles; snippet-text search matches the full idea against paper passages to find contextual work keyword queries may miss.Source: Module 3: Idea Novelty Verification · p. 4
• SPECTER filtering selects the top 100 candidates by default, then RankGPT reranks papers by overlap with the idea’s application domain, purpose, mechanism, and evaluation [15] [61].The top 10 papers by default proceed to an LLM novelty classification with paper-specific reasoning; the rubric treats facet-level difference, a new facet combination, or transfer to a new domain as novel [26] [28].Source: Module 3: Idea Novelty Verification · p. 5
• For ideas classified as not novel, the checker generates three alternatives, each replacing the original idea’s purpose, mechanism, or evaluation with another available facet.Each suggestion changes a different facet to aim for greater novelty relative to the retrieved papers.Source: Module 3: Idea Novelty Verification · p. 5

Expert Annotation Study

• Cohen’s κ rose from 0.64 in the initial three-category round to 0.68 in the revised binary round, which judged novelty solely against retrieved papers.The study assessed 51 ideas and collected 67 consensus-labeled examples: 39 novel and 28 not novel; the examples informed the checker and held-out ablation tests.Source: Module 3: Idea Novelty Verification · p. 5

3 End-to-End Workflow

• Scientists provide a topic and input papers; Scideator extracts facets and retrieves analogous papers at four conceptual distances.Extracted facets are organized by distance for review.Source: §3 · p. 5
• To clarify short or distant-domain facets, users can trace them to source papers, view longer descriptions, add custom facets, or request query-guided retrieval.Facets are phrases of up to seven words.Source: §3 · p. 5
• Users select facets or let the system choose them for idea generation, then expand generated ideas or add their own ideas for decomposition into the facet pool.The selected facet combination determines which papers and distances the generator draws from; generated ideas display their constituent facets by type.Source: §3 · p. 5
• For novelty evaluation, the checker retrieves and re-ranks papers, classifies an idea, and, if it is not novel, suggests three alternatives that each replace a different facet.Users can adjust the classification, adopt a suggestion, or return to facet selection.Source: §3 · p. 5
• The loop lets users check an idea, adopt a facet-swapping suggestion, and regenerate; suggestions and user-added facets can expand the pool for later rounds.Facets selected in one round may instead be discarded after evaluation.Source: §3 · p. 5
• Facet-level actions give the system structured signals about what to change, making user intent explicit and system responses predictable rather than requiring interpretation of free-text instructions.For example, adopting a suggestion that replaces an overlapping mechanism signals which dimension to change.Source: §3 · p. 5

4.1 User Study

• In the within-subjects study, 22 computer-science researchers completed randomized 20-minute Scideator and baseline sessions using the same LLM, contrasting facet-level interaction with paper selection and free-text prompting.Participants worked on assigned HCI or NLP topics and starting papers.Source: §4.1 · p. 6

4.1.1 RQ1: Does Faceted Interaction Lead to More Creativity Support for Idea Generation?

• Scideator yielded significantly higher CSI scores than the baseline (medians 70.5 vs. 61.0; Wilcoxon p<.01), with greatest gains in exploration and expressiveness.CSI scores were measured on a 0–100 scale.Source: §4.1.1 · p. 6
• Participants more often attributed newly found concepts to Scideator’s facets or generated ideas (6 of 6) than to the baseline’s input papers (6 of 8), and valued facet transparency, context-switching, and control.Favorite-idea newness ratings showed only a slight Scideator advantage (median difference: 0.5 points).Source: §4.1.1 · p. 6
• Although 14 participants found facet-level interaction useful and participants more often selected their own facets—especially purposes—they also used input and near facets more than far ones, while four found far facets useful for discovery.Five participants appreciated paper-level interaction, including its closer connection to the literature.Source: §4.1.1 · p. 7, Figure 3 · p. 7

4.1.3 RQ3: Does the Idea Novelty Verification Module Influence Participants’ Confidence in Novelty Assessments?

• The novelty checker did not significantly improve overall confidence, but participants changed their assessments most when Scideator labeled an idea “not novel,” which they found more verifiable against retrieved papers.The intended-setup analysis included 17 participants; participants also described retrieved papers as more useful than the checker’s reasoning alone.Source: Figure 4 · p. 7, §4.1.3 · p. 7

4.2.1 Retrieval Ablation.

• The complete retrieval system substantially outperformed the retrieval variants; facet-based re-ranking combined with embedding filtering was critical to surfacing overlapping prior work for novelty assessment.Keyword- or snippet-only retrieval and alternative re-ranking strategies failed to consistently bring overlapping work into the top 10.Source: Table 1 · p. 8, §4.2.1 · p. 8

4.2.2 Classifier Ablation.

• With the same retrieved papers, the facet-grounded classifier outperformed zero-shot, AI Scientist, and OpenReview-example configurations; expert-labeled facet-level reasoning also beat OpenReview examples and whole-idea/paper reasoning.These ablations indicate that facets benefit classification as well as retrieval.Source: Table 2 · p. 8, §4.2.2 · p. 8

5.1 Facet-Based Ideation

• Purpose and mechanism facets support combinatorial creativity through analogies across contexts, with prior applications spanning papers, product ideas, biological designs, and authors [26] [32] [26] [28] [31] [49].Source: §5.1 · p. 8
• Scideator carries purposes, mechanisms, and evaluations across retrieval, generation, novelty assessment, and suggestion, so facet choices shape each stage [32] [49] [58] [31] [13] [50].Earlier tools use structured representations for individual stages; IdeaSynth refines existing ideas but does not support early ideation from a few papers, multiple directions, cross-paper recombination, or novelty assessment [50].Source: §5.1 · p. 8

5.2 Scientific Idea Novelty Evaluation

• Prior novelty systems either compare ideas with papers without structured facet overlap or assess externally provided ideas; related-paper displays lack explicit novelty judgments [39] [45] [59] [38] [50].Existing novelty evaluations have not systematically tested the novelty component or used controlled comparisons against expert judgments.Source: §5.2 · pp. 8–9
• Scideator closes the evaluation-generation loop by grounding “not novel” rationales in retrieved papers and suggesting facet replacements, evaluated through expert-label ablation and a user study [17].The user study examines how researchers interpret and act on novelty assessments in practice.Source: §5.2 · p. 9

5.3 Human-LLM Scientific Ideation Systems

• Prior human-LLM ideation systems steer users through free-text prompts or LLM-output selection, leaving choices unstructured across modules; unconstrained generation can reduce idea diversity [38] [37].This can require users to re-articulate intent at each stage and leaves the influence of inputs on outputs difficult to trace.Source: §5.3 · p. 9
• Scideator propagates facet-level user actions as structured signals across modules, enabling users to steer generation or let the system select facets while preserving traceability.Source: §5.3 · p. 9

6 Conclusion and Discussion

• Scideator maintains one facet representation across retrieval, generation, and novelty evaluation, with users steering decisions; transparency mitigates but does not eliminate risks of overreliance.Users used facets to steer modules directly, and generated ideas were immediately interpretable. Future work could support unfamiliar distant facets, combine facet- and paper-level interaction, broaden novelty retrieval, and give modules more autonomy.Source: §6 · p. 9
• Participants experienced significantly more creativity support with Scideator than the baseline, particularly in exploration, and attributed new concepts to its facets and generated ideas rather than input papers.The conclusion also reports improved expressiveness and new concept discovery through facets.Source: §6 · p. 9

Appendix Overview

• The appendix overview lists supplementary materials including extended user-study results, module annotation, novelty-checker ablations, generated ideas, implementation details, and system figures.Source: Appendix Overview · p. 11

A.1 Participants

• The study recruited 22 computer-science researchers: 12 HCI and 10 NLP researchers, mostly PhD students, who reported frequent LLM use.Participants received a $60 Amazon gift card.Source: §A.1 · p. 11

A.2 Study Design

• In a within-subjects design, participants completed treatment and baseline tasks in randomized order, with randomized topics and no difference in topic familiarity ratings.HCI and NLP participants received discipline-specific preset topics, each with three starting papers.Source: §A.2 · p. 11

A.3 Baseline Tool

• The baseline enabled paper-level selection and custom instructions with the same LLM as most Scideator functions, but omitted facet-based generation criteria and novelty-checker outputs.It generated six candidate ideas for each pair presented to participants.Source: §A.3 · pp. 11–12

A.4 Treatment Tool Modifications for Study

• For study control, Scideator’s treatment version separated generation from evaluation and disabled on-demand novelty assessment, manual idea addition, and facet generation without a query.Novelty checking was confined to a separate tab; participants could also prompt the LLM directly and take notes, but could not adjust or iterate on novelty assessments.Source: §A.4 · p. 12

A.5 Procedure

• Each 105-minute session included idea-generation and novelty-evaluation tasks in both conditions, with participants using the tool, input-paper abstracts, and Semantic Scholar.Participants spent 20 minutes generating ideas, rated and selected favorites, completed surveys and an interview, then spent five minutes assessing two ideas for novelty.Source: §A.5 · p. 12

B.1 Additional CSI Factor Results

• Participants generally found results more worth the effort with Scideator, although immersion did not improve, enjoyment gains were small, and baseline ratings slightly favored idea impact.Scideator averaged 22.04 seconds to generate two ideas versus 15.21 seconds for the baseline; participants also cited cognitive load, while favorite-idea ratings were similar for feasibility, specificity, and imaginativeness.Source: §B.1 · pp. 12–13

B.2 Evaluation Facet Utility

• Participants often found evaluation facets less helpful than purpose and mechanism facets, with four describing evaluation as unimportant because they could infer how to evaluate a solution.The authors suggest future work should investigate evaluation facets in mixed-initiative facet-based idea generation.Source: §B.2 · p. 13

B.3 Paper-Level Interaction Preferences

• Participants sometimes preferred paper-level interactions as a more natural map of literature, while some saw the baseline as supporting convergence and Scideator as supporting divergence.Participants also avoided distant facets when unfamiliarity or unclear links between mechanisms and purposes made them hard to apply, potentially missing unexpected ideas.Source: §B.3 · p. 13

B.4 Desire for More Papers in Baseline

• Participants wanted more input papers in the baseline: four asked for a way to add papers and five more felt limited by its three-paper cap.The baseline allowed custom instructions but had no feature to add input papers; future work may compare against a version with unrestricted paper additions.Source: §B.4 · p. 13

B.5 Novelty Checker Usage Patterns

• Participants used the novelty checker an average of six times per session and trusted “not novel” judgments more because retrieved papers let them verify them directly.“Novel” judgments were less persuasive; concerns included too-frequent novel classifications and missed relevant papers, potentially reflecting the study’s limit of 10 retrieved papers per idea.Source: §B.5 · pp. 13–14

C.1 Facet Distance Consistency

• Annotators judged most near and far purpose-mechanism labels consistently with the system, suggesting meaningful conceptual-distance variation despite subjectivity in judging analogy distance.Source: §C.1 · p. 14

C.2 Novelty Checker: Expert-Labeled Examples

• The novelty checker uses expert-labeled examples containing facet-marked ideas, top-10 reranked papers, and grounded reasoning to teach the LLM to cite retrieved papers in its judgments.The examples include both novel and not-novel cases.Source: §C.2 · p. 14

D Novelty Checker: Additional Ablation Results

• Replacing facet-based reranking changed about 20–30% of top-10 papers for embedding filtering or general-relevance RankGPT; snippet-only and keyword-only retrieval had minimal overlap with the complete system.These comparisons highlight the importance of the reranker stage.Source: D Novelty Checker: Additional Ablation… · p. 14

E Sample Generated Ideas

• Table 4 presents six ideas saved as favorites during the user study: four generated with Scideator and two with the baseline.The table indicates the generation method triggered by facet selections, selected facets and their conceptual distances, and participant-provided instructions for treatment ideas.Source: E Sample Generated Ideas · p. 14, Table 4 · pp. 16–17
• Scideator favorites drew on facets from papers at varying distances, often with participants selecting the purpose and using a retrieved paper’s mechanism; baseline favorites relied more on custom instructions.The six saved favorites comprised four Scideator ideas and two baseline ideas.Source: E Sample Generated Ideas · p. 14, Table 4 · pp. 16–17
• For treatment ideas, Table 4 records the generation method, selected facets and distances, and custom instructions; baseline entries show only custom instructions.Baseline interaction operates at the paper level rather than through facets.Source: E Sample Generated Ideas · p. 14, Table 4 · pp. 16–17

F Implementation Details

• Scideator’s frontend uses React and TypeScript and its backend uses Python; the interfaces support facet-driven ideation and novelty review.Unless noted otherwise, the system uses gpt-4o-2024-08-06 at temperature zero; facet-idea generation uses temperature 0.75, novelty re-ranking uses gpt-4o, and novelty assessment uses o3-mini. The novelty interface shows an idea, its facets, related papers, and adjustable novelty classification and rationale; the ideation interface supports facet selection or generation, custom instructions, and idea browsing.Source: F Implementation Details · p. 18, G System Figures · p. 18

H.1.1 Extract Facets from Paper.

• Paper facet extraction produces purpose, mechanism, and evaluation facets with definitions, constrained to short, specific phrases that avoid paper-specific detail.Definitions are self-contained and replace proper nouns and jargon; evaluation facets must not reference the purpose.Source: §H.1.1 · p. 24

H.1.2 Generate Analogy Queries at Three Distances.

• Analogy queries preserve the purpose–mechanism relationship across same-topic, same-subarea, and different-subarea distances; overly specific queries are shortened when retrieval returns fewer than four relevant papers.Titles and abstracts of input and very-near papers are summarized as grounding context for idea-generation prompts.Source: §H.1.2 · p. 24, §H.1.3 · p. 24, §H.1.4 · p. 24

H.2.1 Shared Chain-of-Thought Structure.

• Idea-generation prompts ground ideas in prior work, avoid duplicates, brainstorm and select analogies, then elaborate selected ideas through self-critique and revision.Each selected analogy receives an imaginative twist, topic-relevance justification, initial idea, self-identified issues, revision plan, revised idea, and expanded version. Quality requirements cover understandability, relevance, specificity, feasibility, and novelty; optional user instructions must not contradict those requirements.Source: §H.2.1 · p. 25

H.2.2 No Facets Selected by User.

• With either purpose or mechanism—but not both—selected, ideas combine facets from papers at different distances; if a Set-1 paper lacks a facet, the LLM creates one for the analogy.Source: §H.2.3 · p. 25

H.3.1 Extract Keywords and Titles for Retrieval.

• For novelty retrieval and reranking, the prompts extract idea keywords, titles, and structured facets, then prioritize candidate papers by their matches to those facets.The prompts specify 3–6 keyword phrases and four short titles for retrieval, extract application domain, purpose, mechanism, and evaluation metrics, and rank papers from matches across all key facets to partial domain or topic matches.Source: §H.3.1 · p. 26, §H.3.2 · p. 26, §H.3.3 · p. 26

H.3.4 Assess Idea Novelty.

• The novelty assessment prompt compares an idea with top-ranked papers in a 60–100 word review and classifies it as novel or not novel.An idea is not novel if it closely replicates existing work with minimal contributions; novelty can include new concepts, unique combinations, or applying an approach to a genuinely new domain. A simulated multi-turn dialogue presents each retrieved paper separately, with expert-labeled examples included in context.Source: §H.3.4 · p. 27

H.3.5 Generate More-Novel Idea Suggestions.

• When an idea is classified as not novel, Scideator generates three alternatives, each replacing exactly one purpose, mechanism, or evaluation facet.The suggestions use the original idea, overlapping papers, the novelty review, and available workspace facets, and follow the shared idea-quality requirements.Source: §H.3.5 · p. 27
• Each option reports the removed and added facets with IDs, a 100–150-word revised idea, and justifications of novelty and usefulness.Source: §H.3.5 · p. 27
• Suggestions swap one facet at a time, keeping revisions close to the user’s intent while systematically exploring the facet space for novelty.Source: §H.3.5 · p. 27