PaperlayerView exact revision on arXiv

Source-linked AI summary

GraphEval: A Lightweight Graph-Based LLM Framework for Idea Evaluation

Tao Feng, Yihang Sun, Jiaxuan You

arXiv:2503.12600v2cs.LG

TL;DR

LLM-based idea evaluation is unstable and struggles with complex semantic content and novelty. GraphEval decomposes ideas into viewpoint-graphs and applies lightweight label propagation or GNN methods, reporting improved evaluation performance with low resource costs and plagiarism detection.

  • Problem

    Existing LLM-based and lightweight-model evaluators are unstable and struggle to faithfully assess complex research ideas and their novelty.

    Source: §1 · p. 1, §2 · p. 4, §5 · p. 7

  • Method

    GraphEval extracts fine-grained viewpoints, connects them into viewpoint-graphs, and evaluates ideas with training-free label propagation or minimally trained GNNs.

    Source: §1 · p. 2, §4 · p. 5, §5 · p. 6

  • Results

    GraphEval achieves at least a 14% F1 improvement on two datasets with low computation and API costs, while detecting plagiarized ideas.

    Source: §1 · p. 3, §7.1 · p. 8

  • Takeaways & Limitations

    Graph representations can support more robust research-idea evaluation by combining viewpoint-level information with lightweight propagation or GNN prediction.

    Source: §8 · p. 10

Abstract

from arXiv · show

The powerful capabilities of Large Language Models (LLMs) have led to their growing use in evaluating human-generated content, particularly in evaluating research ideas within academic settings. Existing solutions primarily rely on prompt-based LLM methods or fine-tuned lightweight language models for idea evaluation. However, these methods are often unstable and struggle to comprehend the complex semantic information embedded in the ideas, impeding their ability to perform high-quality evaluations. To address the above challenges, we propose GraphEval, a lightweight graph-based LLM framework for idea evaluation. Our insight is that a complex idea can be broken down into comprehensible viewpoint nodes using prompts from small LLMs. These viewpoint nodes can then be linked together through edges created from LLM-based relation extraction and/or BERT similarity scores. The created viewpoint-graph can be used to conveniently propagate scores across view-nodes to improve the robustness of the idea evaluations. In particular, we propose two lightweight graph-based methods for idea evaluation: (1) GraphEval-LP: a training-free label propagation algorithm that propagates evaluation scores from known view-nodes to unknown nodes; (2) GraphEval-GNN: a Graph Neural Networks (GNN) that is trained to predict the evaluation scores given the observed graph with minimal computation resources. Moreover, to overcome LLM's limitation in objectively assessing the novelty of ideas, we further propose a novelty detection model to GraphEval-GNN to enhance its capability in judging idea novelty. Experiments on two datasets show GraphEval improves F1 scores by at least 14% with low computation and API costs. Additionally, GraphEval can effectively detect plagiarized ideas.

1 INTRODUCTION

LLM-based idea evaluation is fast but unreliable because prompt sensitivity, friendliness bias, limited comprehension of complex ideas, and overlooked factual errors undermine fidelity. GraphEval addresses these problems by decomposing ideas into viewpoints, connecting them in graphs, and evaluating them with lightweight propagation or GNN methods.

Motivation

  • LLM evaluators are highly prompt-sensitive and tend to produce friendly Accept judgments rather than negative Reject judgments.Minor prompt variations for the same idea can drastically change the final evaluation.Source: §1 · p. 1, Figure 1 · p. 2
  • Existing methods can hallucinate, require advanced capabilities to understand complex research ideas, and overlook factual inaccuracies embedded within them.Direct whole-idea analysis may miss errors interspersed throughout an idea, producing biased evaluations.Source: §1 · p. 2
  • Breaking ideas into simpler viewpoints and modeling their connections is motivated by findings that both strategies improve understanding of abstract ideas.GraphEval also incorporates temporal information and negative plagiarism samples into GNN training to lower scores for plagiarized ideas.Source: §1 · pp. 2–3

Contributions

  • GraphEval decomposes complex ideas into viewpoints, connects them into viewpoint-graphs, and evaluates them with GraphEval-LP or GraphEval-GNN.GraphEval-LP propagates labels without training, while GraphEval-GNN predicts evaluation scores with minimal training and includes plagiarism detection for novelty assessment.Source: §1 · p. 2
  • At least 14% higher F1 scores on two datasets, with low computation and API costs, are reported for GraphEval over other baselines.The authors also report effective detection of plagiarized ideas and fair evaluation.Source: §1 · p. 3

2 RELATED WORKS

Prior work combines LLMs, lightweight language models, and graph structures for evaluation, indexing, knowledge representation, and reasoning. GraphEval focuses these ideas on extracting and connecting diverse viewpoints from research ideas for evaluation, a direction identified as underexplored.

Automatic Idea Evaluation

  • Existing automated idea-evaluation methods use CNNs, fine-tuned lightweight language models, or prompt-based and representation-based LLM evaluators.These approaches address academic innovation, complex texts, reviewer-style assessment, preference alignment, and decision consistency.Source: §2 · p. 3
  • Fine-tuned lightweight models often require extensive large-scale training and face generalizability limitations, while LLM evaluations remain inconsistent with human reviews for complex, subjective ideas.Recent long-form evaluation methods that divide text into subsets motivate GraphEval’s viewpoint decomposition.Source: §2 · p. 3

Graph for LLMs

  • Graph-LLM research spans unified graph representations, graph-based text indexes, knowledge-graph construction and completion, graph reasoning benchmarks, structured-data tokens, and recommendation systems.LangChain and LlamaIndex also support graph databases as open-source software.Source: §2 · p. 3

Research Gap

  • Extracting diverse viewpoints from research ideas, structuring them as graphs, and using those graphs for idea evaluation remains an underexplored direction.This gap directly motivates GraphEval’s graph-based evaluation framework.Source: §2 · p. 4

3 VIEWPOINT-GRAPH EXTRACTION: A GRAPH STRUCTURE FOR DIVERSE RESEARCH VIEWPOINTS AND THEIR RELATIONSHIPS

GraphEval converts each complex research idea into fine-grained viewpoint-subgraphs and links them into a larger graph. Prompted LLM extraction supplies viewpoints and relations, while BERT similarity provides denser edges and scalable cross-idea connectivity.

Framework Overview

  • The resulting viewpoint-graph supports label propagation or GNN evaluation using quality information from labeled training ideas.The graph is treated as an extensible database of research viewpoints and their semantic interrelations.Source: §3 · p. 4

Viewpoint Extraction through Prompted LLMs

  • A viewpoint is a semantically independent, maximally granular idea, argument, or fact extracted from research content by a prompted LLM.The extraction relies primarily on summarization and abstraction, enabling implementation with smaller LLMs such as 7B-parameter models.Source: §3 · p. 4

Viewpoint-subgraph Construction through Prompted LLMs

  • Prompted LLMs connect related viewpoint-nodes into viewpoint-subgraphs, but experiments on 300 ICLR submissions found their extracted relations too sparse.Sparse edges create isolated nodes, omit relational information, and incur additional resource costs.Source: §3 · p. 5

Viewpoint-subgraph Construction

  • Embedding-similarity relation extraction is proposed to address the sparse edges and extra costs of LLM-based relation extraction.This provides a denser alternative for representing relationships among viewpoints.Source: §3 · p. 5

Viewpoint-subgraph Construction through BERT-based Encoder

  • BERT embeddings connect each viewpoint-node to its top-k most similar nodes with weighted undirected edges, and k controls subgraph edge density.Cosine similarity supplies the edge weights.Source: §3 · p. 5

Viewpoint-graph Construction through Connecting Viewpoint-subgraphs

  • Viewpoint-subgraphs from training and test ideas are connected through top-m cross-subgraph similarity edges to form a graph integrating viewpoints across ideas.The resulting graph can incorporate new viewpoint-subgraphs in linear time.Source: §3 · p. 5

4 GRAGHEVAL-LP: A SIMPLIFIED AND LIGHTWEIGHT IMPLEMENTATION

GraphEval-LP applies label propagation to the viewpoint-graph, initializing training nodes with labels and test nodes without labels, then iteratively spreading weighted information to predict idea evaluations.

GraphEval-LP

  • GraphEval-LP uses label propagation on the viewpoint-graph as a simple, lightweight evaluation framework.The method propagates evaluation information through the graph without requiring model training.Source: §4 · p. 5

Initialization and Regularization

  • Training viewpoints receive one-hot label vectors, while test viewpoints start with zero vectors and edge weights are normalized over each node’s neighbors.The label-vector dimensionality equals the number of quality labels.Source: §4 · pp. 5–6

Label Propagation

  • Each node repeatedly updates its vector by adding weighted neighbor vectors until the labels no longer change.The update is normalized so the resulting vector remains properly scaled.Source: §4 · p. 6

Label Prediction

  • After propagation, the method sums viewpoint vectors for each test idea and selects the dimension with the highest summed value as its predicted label.The number of summed vectors corresponds to the viewpoints extracted from the idea.Source: §4 · p. 6

5 GRAGHEVAL-GNN: A GRAPH NEURAL NETWORK-BASED SOLUTION

GraphEval-GNN learns to predict viewpoint evaluation scores from graph structure and features, while extending the model with temporal and synthetic negative-sample signals for novelty assessment.

Motivation

  • GraphEval-GNN learns how to propagate evaluation scores from known to unknown viewpoints instead of relying on fixed label propagation.It predicts the evaluation score for each viewpoint node using a learned model.Source: §5 · p. 6

Method Overview

  • The model represents viewpoints as nodes, relationships as edge features, and applies a weighted GNN to embed both for prediction.BERT initializes node embeddings, while BERT-derived similarity relations initialize edge features.Source: §5 · p. 6

Predict via a weighted GNN

  • Weighted GraphConv layers iteratively aggregate local neighborhoods using learnable parameters to produce expressive node embeddings.Each node update uses its direct neighbors and the current layer’s learnable parameters.Source: §5 · p. 6

Subgraph Prediction

  • GraphEval-GNN treats idea evaluation as subgraph prediction by combining viewpoint embeddings with mean and max pooling for global and local information.The resulting subgraph embedding is converted into an idea-level probability distribution.Source: §5 · p. 6

Training and Testing

  • During testing, GraphEval selects the category with the highest output probability as the final evaluation.The training procedure is summarized in Algorithm 1.Source: §5 · pp. 6–7

Novelty Assessment

  • For novelty assessment, the model adds temporal features and trains on duplicated or combined viewpoints labeled with lower evaluation scores.This targets later ideas that closely resemble earlier research and helps the model assign them lower scores.Source: §5 · p. 7

6 EXPERIMENTAL SETUP

The experiments evaluate idea-review prediction on two academic-paper datasets against prompt-based, agentic, and fine-tuned baselines using classification and resource metrics.

Task

  • The task predicts review decisions from paper abstracts and titles across Reject, Accept (Poster), Accept (Oral), and Accept (Spotlight) categories.Abstracts serve as research-idea representations because they summarize the problem, methods, experiments, and contributions.Source: §6 · p. 7

Datasets

  • The study uses 300 ICLR papers for training and 50 for testing, plus an AI Researcher test set focused on novel prompting methods.The latter combines Accept (Oral) and Accept (Spotlight) into one class because those labels are scarce.Source: §6 · p. 7

Baselines

  • Baselines include prompted LLMs, chain-of-thought variants, Tree of Thoughts, Research Agent, and fine-tuned DistilBERT.The prompt-based methods use criteria, demonstrations, self-consistency, or multi-step reasoning, while DistilBERT provides a learning-based comparison.Source: §6 · pp. 7–8

Implementation Details

  • LLM-based baselines use Mistral 7B Instruct v0.3 or Qwen 2 Instruct 72B, while GraphEval-GNN uses a two-layer weighted GNN trained on one NVIDIA A100 GPU.The GNN uses hidden dimension 64, batch size 64, and up to 1000 epochs.Source: §6 · p. 8

Evaluation Metrics

  • Evaluation measures accuracy, macro precision, macro recall, macro F1, average token cost, and normalized cost.Normalized cost is scaled so the most expensive method equals 1, with smaller values indicating lower resource expenditure.Source: §6 · p. 8

7 EXPERIMENT RESULTS

GraphEval-GNN outperforms the reported baselines across both datasets while using resources near the minimum expenditure level, and novelty assessment improves detection of plagiarized or derivative ideas.

7.1 COMPARISON WITH EXISTING BASELINES.

  • 10%-72% accuracy and 18%-42% macro F1 advantages on ICLR Papers, plus 13%-53% accuracy and 14%-48% macro F1 advantages on AI Researcher, establish GraphEval-GNN as the strongest reported method.Its normed cost on both datasets remains comparable to the minimum expenditure level.Source: §7.1 · p. 8, Table 2 · p. 9, Table 3 · p. 9
  • Prompt-based baselines generally achieve lower accuracy and macro F1, while more complex prompting techniques such as ToT and Research Agent show no significant advantage over CoT-SC.The authors also report that larger LLMs do not consistently improve evaluation performance.Source: §7.1 · pp. 8–9
  • Fine-tuned BERT performs better than other prompt-based baselines but remains below GraphEval because the viewpoint-graph captures local viewpoint quality and global interrelations.GraphEval’s graph construction supplies information that prompt-based methods and fine-tuned language models do not effectively leverage.Source: §7.1 · p. 9
  • GraphEval-LP consistently ranks second-best across both datasets without training, demonstrating a lightweight alternative that exploits correlations among research-idea viewpoints.These implicit correlations are not effectively leveraged by prompt-based methods or fine-tuned language models.Source: §7.1 · pp. 9–10

7.2 GRAPHEVAL FOR NOVELTY ASSESSMENT.

  • Novelty assessment significantly improves GraphEval’s performance when detecting plagiarized or derivative ideas on the ICLR Papers dataset across four metrics.The evaluation uses 80 artificially constructed plagiarized ideas and compares GraphEval variants with and without novelty assessment.Source: Figure 4 · p. 10, §7.2 · p. 10

7.1 COMPARISON WITH EXISTING BASELINES.

  • GraphEval-GNN improves performance over GraphEval-LP but incurs increased computational cost, providing options for users with different resource requirements.The comparison attributes the performance increase to applying deep learning to graph-based LLM evaluation.Source: §7.1 · p. 10

8 CONCLUSION

GraphEval concludes that viewpoint-graphs provide a lightweight basis for idea evaluation through label propagation and GNN prediction, with GraphEval-GNN additionally addressing novelty through plagiarism detection. The appendix illustrates viewpoint extraction and documents prompts used for GraphEval and its baselines.

8 CONCLUSION

  • GraphEval uses viewpoint-graphs to address complex and subjective idea evaluation, with GraphEval-LP and GraphEval-GNN predicting evaluations from connected viewpoints.GraphEval-GNN additionally incorporates a plagiarism detection mechanism for novelty assessment.Source: §8 · p. 10
  • Experiments demonstrate significant improvements in accuracy and macro F1 over multiple baselines while maintaining low resource expenditure.The conclusion positions GraphEval as an integration of graph-based approaches with LLMs for idea evaluation.Source: §8 · p. 10

A A SIMPLE EXAMPLE OF VIEWPOINT EXTRACTION

B.1 PROMPTS USED IN GRAPHEVAL

B.2 PROMPTS USED IN BASELINES

  • Baseline prompts specify research-idea evaluation criteria, decision standards, and examples, while ToT decomposes evaluation into eight steps.Research Agent instead uses problem, method, experiment, and final-decision steps.Source: §B.2 · p. 16

C DETAILS OF THE DATASETS

The experiments use the ICLR Papers and AI Researcher datasets with dataset-specific label and split handling, and report the prompts and hyperparameters used for evaluation.

C DETAILS OF THE DATASETS

  • The experimental datasets and their sizes and label distributions are documented in Table 5, with Accept (Oral) and Accept (Spotlight) combined for the AI Researcher Dataset.The combined labels address scarcity in that dataset.Source: C DETAILS OF THE DATASETS · p. 17
  • The ICLR Papers training set increases Accept (Oral) and Accept (Spotlight) representation to help learning-based methods capture less-represented samples under long-tail conditions.This adjustment changes the training distribution rather than the evaluation task description.Source: C DETAILS OF THE DATASETS · p. 17
  • For the AI Researcher Dataset, prompt-based methods are evaluated across the entire dataset, whereas other methods use an 85%:15% train-test split with repeated experiments averaged to reduce bias.The limited data volume motivates the different evaluation procedures.Source: C DETAILS OF THE DATASETS · p. 17

D HYPERPARAMETER CONFIGURATION

  • The appendix supplies relation-extraction prompts and the hyperparameter settings used in the experiments.The prompt and configuration templates are referenced in Tables 6 and 7.Source: Table 6 · p. 18, Table 7 · p. 18
  • Baseline prompt templates define idea-evaluation criteria and decision standards for the four review decisions.These materials are provided in Tables 8 and 9.Source: Table 8 · p. 19, Table 9 · p. 19

E GENERALIZATION EXPERIMENT

GraphEval generalizes beyond its primary research-idea setting to long-form factuality evaluation and across time-separated data, with reported performance and temporal generalization ability supported by the experiments.

E.1 GENERALIZATION ON LONG FORM TEXT EVALUATION TASK

  • The long-form evaluation appendix documents the prompt template used to assess research ideas, including criteria, decision standards, few-shot examples, and label distributions.These prompt components are summarized in Table 10.Source: Table 10 · p. 20
  • GraphEval performs well on the FActScore long-form text evaluation task, demonstrating adaptability to a task involving human judgments of atomic-fact support.The dataset contains LLM-generated celebrity atomic facts assessed by human annotators, with topics split into training, validation, and test sets at 7:1:2.Source: §E.1 · pp. 19–20

E.2 GENERALIZATION ABILITY ACROSS DIFFERENT TIMES

  • Training and validation on ICLR papers published before 2022 and testing on 2023 papers validates GraphEval’s temporal generalization ability.The experiment compares GraphEval with classic baselines in Table 21.Source: §E.2 · p. 20

F.1 EFFECTS OF VARIOUS LIGHTWEIGHT GRAPH NEURAL NETWORK ARCHITECTURES

GraphEval-GNN generalizes across lightweight architectures, relation-extraction strategies, and a large-scale review dataset. It outperforms the tested alternatives while retaining strong performance under class imbalance.

F.1 EFFECTS OF VARIOUS LIGHTWEIGHT GRAPH NEURAL NETWORK ARCHITECTURES

  • GraphEval-GNN outperformed GraphEval-SGC and GraphEval-LightGCN because the alternatives sacrificed individualized node information for memory and speed.Source: §F.1 · p. 20

F.2 COMPARATIVE IMPACT OF ALTERNATIVE RELATION EXTRACTION METHODS

  • GraphEval-GNN outperformed the Hybrid relation-extraction method, whose prompted-LLM connections may reduce edge density while increasing hallucination risk and token costs.Source: §F.2 · p. 21

G SCALABILITY GENERALIZATION

  • 9.8% higher accuracy and 8% higher macro F1 than Fine-tuned BERT establish GraphEval-GNN’s strongest reported large-scale generalization result on ASAP-Review.GraphEval-GNN also outperformed Prompted LLM and remained best despite rare Accept (Oral) and Accept (Spotlight) labels below 4%.Source: G SCALABILITY GENERALIZATION · p. 21

H ACCURACY EVALUATION OF VIEWPOINTS

The generated viewpoints closely match their source ideas at both overall consistency and entity-level factuality. These evaluations support the accuracy and rationality of GraphEval’s viewpoint representations.

H ACCURACY EVALUATION OF VIEWPOINTS

  • 99.47% and 99.82% consistency rates show that generated viewpoints closely match the original ideas on ICLR Papers and AI Researcher, respectively.Consistency was assessed with LLaMa-3.1 (405b), where a positive judgment indicated that a viewpoint was consistent with its source idea.Source: H ACCURACY EVALUATION OF VIEWPOINTS · p. 21, Table 25 · p. 29
  • Entity-level Precision, Recall, and F1 Score all exceeded 0.9 on both datasets, further validating viewpoint accuracy and rationality.The metrics were computed after aggregating the constructed viewpoints and comparing them with the corresponding ideas.Source: H ACCURACY EVALUATION OF VIEWPOINTS · p. 21, Table 26 · p. 29

Related papers

Loading 2503.12600v2…