PaperlayerView exact revision on arXiv

Source-linked AI summary

Visual Analysis of LLM-based Entity Resolution from Scientific Papers

Siyu Wu, Yi Yang, Weize Wu, Ruiming Li, Yuyang Zhang, Ge Wang, Huobin Tan, Zipeng Liu, Lei Shi

arXiv:2609.06037v1cs.IR

TL;DR

Scientific entity resolution remains difficult because specialized literature contains diverse text, complex terminology, and limited high-quality labeled data, while evaluating LLM outputs is challenging. The paper introduces an interactive visual analytics pipeline that combines LLM-based batch resolution with labeled MOF data, error analysis, and RAG interpretation. The framework achieves 90.73% overall accuracy on 122 well-labeled MOF synthesis paragraphs, while its RAG interface still lacks comprehensive configuration-comparison support.

  • Problem

    Diverse scientific writing, chemical entity names, limited labeled datasets, and insufficiently actionable evaluation make LLM-based entity resolution difficult to deploy and refine.

  • Method

    The paper combines LLM-based batch entity resolution with a labeled MOF data pipeline, interactive visualizations, error analysis, and RAG configuration.

  • Results

    90.73% overall accuracy was achieved on 122 well-labeled MOF synthesis paragraphs.

  • Takeaways & Limitations

    The framework enables non-programming users to interactively inspect, interpret, and refine scientific-literature entity resolution.

  • Takeaways & Limitations

    The RAG implementation does not support simultaneous comparison of configurations or easy tracking of individual example contributions.

Abstract

from arXiv · show

This paper focuses on the visual analytics support for extracting domain-specific entity from extensive scientific literature, a task with inherent limitations using traditional named entity resolution methods. With the advent of large language models (LLMs) such as GPT-4, significant improvements over conventional machine learning approaches have been achieved due to LLM's capability on entity resolution integrate abilities such as understanding multiple types of text. This research introduces a new visual analysis pipeline that integrates these advanced LLMs with versatile visualization and interaction designs to support batch entity resolution. Specifically, we focus on a specific material science field of Metal-Organic Frameworks (MOFs) and a large data collection namely CSD-MOFs. Through collaboration with domain experts in material science, we obtain well-labeled synthesis paragraphs. We propose human-in-the-loop refinement over the entity resolution process using visual analytics techniques, which allows domain experts to interactively integrate insights into LLM intelligence, including error analysis and interpretation of the retrieval-augmented generation (RAG) algorithm. Our evaluation through the case study of example selection for RAG demonstrates that this human-machine collaborative approach improved single-document entity resolution accuracy by approximately 30%.

1. Introduction

Scientific entity resolution must handle diverse, structured relationships in literature, while traditional and emerging methods still face accuracy, consistency, and interpretability challenges. The paper proposes an interactive visual analytics framework that combines LLM-based resolution with human-guided analysis and reports 90.73% overall accuracy on 122 labeled MOF synthesis paragraphs.

  • Traditional rule-based and conventional machine-learning approaches struggle with diverse sentence structures and specialized MOF synthesis terminology.
  • LLM-based resolution offers advantages across diverse text formats and can adapt to new domains through prompt engineering and few-shot learning.
  • LLM applications still face inaccurate numerical extraction, inconsistent structured outputs, and limitations in complex reasoning.
  • The proposed visual analytics approach supports performance visualization, interactive error analysis, and interpretation of RAG-based entity resolution.
  • The framework combines an interactive LLM pipeline with visualizations that let users manipulate model parameters and inspect resolution outcomes.
  • 90.73% overall accuracy was achieved on 122 well-labeled MOF synthesis paragraphs after crossvalidation.

2. Related Work

Prior work has explored LLM-based structured extraction and visual analytics for interpreting machine-learning outputs. This paper extends those directions by combining interactive visualization with batch LLM-based entity resolution for MOF synthesis literature.

  • LLM-based Approaches for Structured Information Extraction: LLM research has addressed structured extraction through generative labeling, fine-tuning, and human-in-the-loop entity and relationship extraction.
  • LLM-based Approaches for Structured Information Extraction: Materials-science applications have used databases and LLMs for single-query knowledge retrieval.
  • LLM-based Approaches for Structured Information Extraction: This work develops an integrated visual analytics system combining LLMs with interactive visualization for batch entity resolution in MOF synthesis literature.
  • Visual Analytics for Interpreting the Results of ML Methods in NLP domain: Visual analytics systems support interpretation of machine-learning outputs through interactive graphs, embedding comparisons, and analysis of language-model results.
  • Visual Analytics for Interpreting the Results of ML Methods in NLP domain: The paper extends prior visual analytics approaches by focusing on LLM-based entity resolution from scientific literature.

3. Background and Problem Definition

Scientific literature contains diverse, structurally complex information that remains difficult to resolve reliably, even with LLMs. The paper defines a visual evaluation framework to help domain experts inspect errors, assess limitations, and refine extraction workflows.

  • Background: Scientific literature’s diversity and structural complexity make extracting and structuring domain knowledge difficult.
  • Background: Traditional entity-resolution methods require substantial domain expertise and manual effort and adapt inflexibly to new domains.
  • Background: LLMs process diverse text with limited additional training and can adapt through zero-shot or few-shot learning.
  • Problem Definition: Diverse writing styles, chemical entity names, and limited high-quality labeled datasets constrain real-world LLM-based entity-resolution workflows.
  • Problem Definition: Domain experts face substantial challenges when evaluating and refining LLM extraction performance.
  • Visual Analysis Requirements: Statistical metrics quantify overall accuracy but do not reveal actionable error patterns or model limitations.
  • Visual Analysis Requirements: The proposed framework enables direct result inspection, interactive error analysis, and graph-based assessment of algorithmic limitations.
  • Visual Analysis Requirements: Visual tools help practitioners validate and refine extracted data without extensive computational expertise.

4. System Overview

The system combines labeled-data construction, relevant-paragraph detection, configurable LLM entity resolution, and visualization of structured outputs. Its pipeline uses labeled MOF synthesis data, prompt and retrieval settings, and interactive result analysis.

  • System Overview: The platform provides batch resolution, few-shot LLM processing, integrated tools, and visual analysis for scientific-literature entity resolution.
  • System Overview: The workflow proceeds through interactive labeling, relevant-paragraph detection, LLM configuration and resolution, and result visualization.
  • Data Labeling and Entity Resolution Pipeline: Experts crossvalidate labels for MOF synthesis entities and produce 122 well-labeled synthesis paragraphs in JSON format.
  • Paper Upload and Relevant Paragraph Detection: An offline machine-learning model classifies uploaded paragraphs according to whether they describe MOF synthesis.
  • LLM Configuration and Entity Resolution: Labeled paragraphs and entities support LLM resolution, while the final stage compares resolved entities with the labeled database visually.
  • LLM Configuration: Prompt engineering, labeled examples, relevant paragraphs, model temperature, and few-shot sample size control the LLM output.
  • Result Visualization: The interface organizes structured outputs into accuracy matrices, heatmaps, network diagrams, and other visual analyses.

5. Visualization Design

The visualization system was designed with materials-science experts to support interactive analysis of LLM extraction results for MOF synthesis paragraphs. It combines performance comparison, detailed inspection, and controls for refining model outputs.

  • Visualization Design: Expert collaboration shaped an interface for understanding LLM extraction performance from MOF synthesis paragraphs across multiple perspectives.Iterative design sessions and expert feedback informed the system’s analytical workflow.
  • Visualization Design: Users can compare extraction performance across synthesis parameters to identify shortcomings for particular entity types.The system supports both overview analysis and in-depth investigation.
  • Visualization Design: The interface highlights extracted terms, displays BM25-ranked alternative examples, and provides threshold controls for selecting significant terms.Users can select or deselect examples to refine the model.
  • Visualization Design: The main view organizes overall metrics, detailed results, and extracted paragraphs from abstract to concrete.This top-to-bottom structure helps users locate areas of interest.

Overall Metrics View

The Overall Metrics View provides a quick assessment of the model’s extraction performance using four evaluation metrics.

  • Overall Metrics View: The view displays Precision, Recall, F1, and ACC to summarize the LLM’s general extraction performance.These metrics are presented in Fig.1(a).

Detail Result View

The Detail Result View compares extraction performance across synthesis parameters and metrics while supporting interactive filtering and alternative visual encodings. Additional tabs provide error and distribution analyses at varying levels of complexity.

  • Detail Result View: A heat map represents each synthesis parameter and metric with varied colors, while users can change its visual encoding.Available encodings include rectangular height, circular radius, plain text, and color with text.
  • Detail Result View: Users can toggle between evaluation metrics and TP, TN, FP, and FN statistics across parameters.The right-side panel supports switching between metrics and data statistics.
  • Detail Result View: Filtering or sorting extracted paragraphs automatically recalculates panel data and updates the corresponding charts.The extracted-paragraph panel includes shortcut controls for single selection and direct focus.
  • Detail Result View: Error Map View and Data Distribution View are available through separate tabs to provide interaction complexity suited to different users and reduce information overload.These advanced views are discretely accessible from the middle section of the page.

Error Map View

The Error Map View represents relationships among parameter-specific false positives and false negatives, while the distribution view compares existing and LLM-extracted paragraphs interactively. Together, these views support focused investigation of errors and data patterns.

  • Error Map View: The error graph represents parameter-specific false positives and false negatives as nodes, with borders encoding error counts and links encoding co-occurrence.Links connect error nodes when a synthesis paragraph contains multiple errors.
  • Error Map View: Users can mask low-frequency connections, highlight neighboring nodes, and reposition nodes to inspect surrounding error relationships.Setting “Mask line” to 3 hides lines with values less than or equal to 3.
  • Data Distribution View: The Data Distribution View uses a two-dimensional scatter plot and adjacent pie charts to display paragraph and parameter distributions.The scatter plot includes database paragraphs and LLM-extracted paragraphs.
  • Data Distribution View: One-hot encoding and PCA produce paragraph embeddings based on selected synthesis parameters, including names and quantities.Each paragraph is decomposed into synthesis parameters before vector representation.
  • Data Distribution View: Black, red, and blue points distinguish existing paragraphs, GPT-extracted paragraphs, and paragraphs present in both sources.Red and blue points use larger radii to emphasize new or matched data.
  • Data Distribution View: Tooltips, zooming, panning, axis controls, and point-size controls support interactive exploration of the scatter plot.Users can inspect detailed paragraph information by hovering over points.
  • Data Distribution View: Rectangular selection can pair LLM extraction points with corresponding ground-truth points and update parameter-distribution pie charts.Users can then choose a parameter for further inspection.

6. Case Study in Material Science

Three case studies evaluate LLM-based entity resolution on 122 labeled MOF synthesis paragraphs, covering overall extraction, error patterns, and RAG example selection. Visual analysis identifies solvent-related weaknesses and shows that manually selected, structurally similar examples can substantially improve extraction.

  • Evaluation setup: The evaluation uses three case studies and 122 well-labeled MOF synthesis paragraphs to assess the entity-resolution approach.The cases examine overall extraction, error analysis, and RAG example selection.
  • Entity resolution pipeline: 89.54% precision, 96.85% recall, 93.05% F1-score, and 90.73% accuracy were achieved without human intervention.Most parameters exceeded 90% accuracy, although solvent name, active temperature, and active time were weaker.
  • LLM error analysis: Error analysis identified lower performance for solvent name and solvent amount, with 14 papers showing concurrent errors in both parameters.The error map also connected solvent amount, solvent name, active temperature, and active time at link mask level 3.
  • LLM error analysis: Distribution analysis compared LLM-extracted and ground-truth solvent-name results for 54 selected points using paired pie charts.The analysis focused on a dense cluster in the embedding visualization.
  • Example selection in RAG: Carefully selecting five synthesis paragraphs with similar structures and conditions enabled extraction of all ground-truth entities in the RAG case.The selected examples included similar experimental details such as CdCl2.
  • Example selection in RAG: The study concludes that RAG effectiveness depends heavily on the quality and relevance of examples supplied to the model.Examples sharing structural patterns and experimental details with the target text improved entity identification and extraction.

7. Discussion and Limitations

The visual analytics system makes LLM extraction more inspectable by supporting hierarchical exploration, interactive error analysis, and transparent RAG example selection. Its main limitations are cumbersome comparison workflows, incomplete RAG configuration comparison, and testing on only 122 examples.

  • Discussion: Hierarchical views let users navigate from overall metrics to specific paragraphs without cognitive overload.This organization is one of three ways the system opens the extraction black box.
  • Discussion: Interactive error and distribution views reveal extraction-error patterns that manual inspection would make difficult to identify.These patterns support domain experts in developing targeted improvement strategies.
  • Discussion: RAG example-selection visualization makes example influence more transparent, interpretable, and controllable.The system exposes how different examples affect extraction results.
  • Limitations: The system makes comparative analysis cumbersome because users must return to task selection and re-select extraction tasks to view performance charts.The current RAG implementation also cannot simultaneously compare configurations or track individual-example contributions.
  • Limitations: Scalability remains uncertain because the system was tested on only 122 examples.Future work also calls for improved interaction and a fully labeled dataset.

8. Conclusion

The paper integrates LLMs with an interactive visual framework for batch entity resolution in scientific literature. Prompt engineering, structured outputs, visualization, and user interaction support more accurate and adaptable analysis.

  • The framework integrates large language models with visual entity resolution for scientific literature analysis.
  • The fully interactive system enables batch entity resolution without requiring programming expertise.
  • Prompt engineering and structured outputs improve LLM accuracy and reliability while minimizing typical automated entity-resolution errors.
  • Visualization provides detailed data representations and lets users adjust model parameters for specific research requirements.
  • The pipeline is transferable to similar entity-resolution tasks in other domains.
Loading 2609.06037v1…