PaperlayerView exact revision on arXiv

Source-linked AI summary

Ontology-Guided Multi-Agent Extraction of Evaluation Objects from Academic Review Texts: Evidence from Chinese Library and Information Science

Haolin Chen, Hongyi Dong, Yu Zhu, Yijia Hong, Leiqing Niu, Jiyuan Ye

arXiv:2608.29526v1cs.CL

TL;DR

Computational approaches to evaluative texts have focused largely on general review scenarios, while academic reviews involve abstract knowledge entities. This paper proposes an ontology-constrained staged multi-agent framework for Chinese academic reviews, and reports that it outperforms rule-based and zero-shot baselines across several evaluation measures.

  • Problem

    Computational methods have mainly addressed general review scenarios, while scholarly reviews involve abstract knowledge entities that require specialized object-type frameworks.

    Source: Mining Scholarly Review Texts · p. 2

  • Method

    The paper proposes an ontology-constrained, staged multi-agent framework for extracting evaluation objects from Chinese academic review texts.

    Source: CONCLUSION · p. 10

  • Results

    The full system outperformed rule-based and zero-shot baselines across Entity-level F1, Strict Typed F1, type accuracy, and boundary control; versus B2, it gained 18.50 percentage points in Entity-level F1 and 32.8 percentage points in Strict Typed F1.

    Source: Overall Performance · p. 8, CONCLUSION · p. 10

  • Takeaways & Limitations

    The framework supports structured utilization of evaluative knowledge in Chinese academic review texts.

    Source: CONCLUSION · p. 10

    All three agents use the same backend model, so the study does not establish advantages of heterogeneous multi-model collaboration.

    Source: Limitations · p. 10

Abstract

from arXiv · show

Academic reviews, scholarly commentaries, and book reviews serve as sources of evaluative statements about theories, methods, literature, institutions, and policies, providing valuable evidence for scholarly evaluation. Existing scientific entity extraction methods mainly target research articles and are less effective for evaluation objects, which are often abstract, context-dependent, and characterized by ambiguous type boundaries. This study proposes an ontology-guided multi-agent framework for evaluation object extraction. The framework combines candidate discovery, ontology-constrained classification, and domain review. Experimental results show that it achieves a Precision of 90.33%, Recall of 84.55%, Entity-level F1 of 87.34%, Strict Typed F1 of 79.78%, and Type Accuracy of 91.35%, substantially outperforming rule-based and zero-shot baselines. Ablation results indicate that the multi-agent workflow improves recall and stability, while ontology-based boundary constraints enhance fine-grained classification and reduce category confusion. The framework supports the structured utilization of evaluative scholarly texts and provides methodological support for evidence-based research evaluation and STI mining.

INTRODUCTION · RELATED WORK · Mining Scholarly Review Texts

Scholarly reviews contain evaluative knowledge, but much of it remains unstructured and difficult to extract because evaluation objects are abstract and have ambiguous type boundaries. The study addresses this with an ontology-constrained, multi-agent framework for extracting evaluation objects.

INTRODUCTION

  • As scholarly publishing expands, evaluative knowledge in conventional reviews, commentaries, and book reviews remains dispersed and not systematically structured.Prior work has examined expert recommendation, open review, and post-publication evaluation, but substantial evaluative knowledge remains in narrative texts.Source: INTRODUCTION · p. 1
  • Evaluation objects are difficult to extract reliably because they are abstract, context-dependent, and have ambiguous type boundaries.Surface lexical features and single-pass models are described as inadequate for this challenge.Source: INTRODUCTION · p. 1
  • Existing scholarly ontologies organize bibliographic records, scholarly resources, and research activities but lack dedicated representations for evaluation objects and their conceptual boundaries.The passage names FRBR, BIBO, VIVO, and CERIF as examples.Source: INTRODUCTION · pp. 1–2
  • The proposed evaluation ontology has object, dimension, and viewpoint layers, and organizes evaluation objects into four top-level categories with a hierarchical type system.The categories are actors, artifacts, abstract entities, and events; this study focuses on the object layer.Source: INTRODUCTION · p. 2
  • Separate agents handle candidate discovery, ontology-constrained classification, and domain review to improve extraction recall, type accuracy, and auditability.Source: INTRODUCTION · p. 2

Mining Scholarly Review Texts

  • Review articles, commentaries, and book reviews help academic communities organize and evaluate knowledge, with scholarly judgments central to their value.These texts also help portray fields, identify research directions, and organize knowledge.Source: INTRODUCTION · p. 1, Mining Scholarly Review Texts · p. 2
  • Citation analysis, co-word analysis, and topic modeling reveal macro-level patterns but offer limited support for identifying evaluation objects, relationships, and dimensions in review texts.The cited approaches include Hjørland (2013) and Leydesdorff & Nerghes (2017).Source: Mining Scholarly Review Texts · p. 2
  • Computational analysis of evaluative texts has focused mainly on product and online reviews, while academic reviews evaluate abstract entities such as theories, methods, disciplines, and debates.The passage argues that scholarly reviews require specialized object-type frameworks and extraction approaches.Source: Mining Scholarly Review Texts · p. 2

Entity Extraction from Scientific Texts · Ontology-Guided and Multi-Agent Extraction

Scientific-text extraction has advanced toward contextual and document-level understanding, but conceptual boundaries, domain shifts, and limited full-text reasoning remain challenges. Evaluative scholarly texts and their knowledge structures are underexplored, motivating ontology-guided multi-agent extraction for Chinese academic reviews.

Entity Extraction from Scientific Texts

  • Scientific extraction has progressed from word embeddings to pretrained and domain-specific language models, while research has expanded toward document-level understanding and structured knowledge discovery.The passage describes improvements in named-entity recognition and relation extraction alongside this progression.Source: Entity Extraction from Scientific Texts · p. 2
  • Long-form scientific extraction is constrained by domain shifts, confusion between fine-grained entities and broader concepts, and the need for full-text context beyond current document-level reasoning.The resulting bottlenecks include conceptual-boundary delineation, knowledge constraints, and modeling complex semantic relationships.Source: Entity Extraction from Scientific Texts · p. 2
  • Evaluative scholarly texts remain underexplored: prior work emphasizes communication, document characteristics, topics, citations, and concept normalization, with limited support for identifying evaluation objects and judgments.This leaves a substantial gap in fine-grained knowledge organization and entity extraction for evaluative academic texts.Source: Entity Extraction from Scientific Texts · pp. 2–3

Ontology-Guided and Multi-Agent Extraction

  • Ontology-based information extraction uses explicit conceptual specifications to constrain extraction and support accuracy, consistency, and interpretability, but scholarly ontologies offer limited evaluative-knowledge modeling.Existing applications support bibliographic description, citation relationships, research communities, and research-activity management, rather than stance, judgments, and support–challenge relations.Source: Ontology-Guided and Multi-Agent… · p. 3
  • Existing scholarly ontologies insufficiently represent evaluative statements and semantic dependencies, while single-model extraction risks boundary drift, type confusion, and factual instability.Multi-agent research indicates that role specialization, iterative discussion, and structured verification can improve stability and consistency in complex tasks.Source: Ontology-Guided and Multi-Agent… · p. 3
  • Systematic typologies of evaluation objects and ontology-guided multi-agent extraction remain largely unexplored in Chinese academic review texts, a gap this study addresses.The paper identifies limited attention to knowledge structures embedded in evaluative academic texts.Source: Ontology-Guided and Multi-Agent… · p. 3

METHODOLOGY · Academic Evaluation Ontology

The Academic Evaluation Ontology sets semantic boundaries for evaluative entities and organizes evaluation objects, dimensions, and viewpoints; this section focuses on the object layer as the anchor for extraction. Its hierarchy and explicit boundary rules support consistent classification of semantically adjacent targets.

Academic Evaluation Ontology

  • The ontology prioritizes what is evaluated, rather than how publications, authors, and institutions are connected, to provide stable semantic boundaries for evaluative-text extraction.It is intended for academic review articles, scholarly commentaries, and book reviews.Source: Academic Evaluation Ontology · p. 3
  • The ontology design draws on Guarino’s (1998) ontological distinctions, FRBR/LRM (IFLA Study Group on the Functional Requirements for Bibliographic Records, 1998), Ranganathan (1967), Kuhn (1962), Hjørland (2002, 2013), and OpenAlex (Priem et al., 2022).These sources inform the top-level structure, Artifact and Organization modeling, abstract knowledge objects, and compatibility with scholarly knowledge infrastructures.Source: Academic Evaluation Ontology · p. 3
  • The ontology has three layers—Evaluation Object, Evaluation Dimension, and Evaluation Viewpoint—and this study validates the object layer as the referential anchor for the other two.Objects identify what is evaluated, dimensions specify aspects of evaluation, and viewpoints capture judgments, including their polarity, intensity, and supporting evidence.Source: Academic Evaluation Ontology · p. 4
  • The object layer has four top-level classes—Agent, Artifact, Abstract Entity, and Event—and is refined into 4 L1 classes, 14 L2 subclasses, and 59 L3 leaf types.The hierarchy supports coarse-to-fine classification; Abstract Entity is identified as the most challenging category in the extraction task.Source: Academic Evaluation Ontology · p. 4
  • Explicit rules distinguish methodology, method, and technique; classify organizations by primary function; and separate theory, model, and framework by semantic role.The rules are applied in manual annotation and automatic extraction to reduce type drift and boundary ambiguity.Source: Academic Evaluation Ontology · pp. 4–5
  • Database versus Information System classification depends on the evaluation perspective, while Policy denotes formal documents and Policy Initiative denotes implementation actions.Databases are evaluated for content, coverage, and data quality; Information Systems for functionality, interface design, retrieval performance, and usability.Source: Academic Evaluation Ontology · p. 5
  • Annotation and system outputs use the most specific interpretable type, reserving “Other” for unresolved cases and retaining evidence and uncertainty sources for ontology refinement.Source: Academic Evaluation Ontology · p. 5

Multi-Agent Collaborative Knowledge Mining System Architecture

The system divides academic evaluation entity extraction into candidate discovery, ontology-constrained validation, and domain review, passing structured intermediate representations between stages.

Architecture

  • Three specialized agents separate high-recall discovery, constraint-based validation, and domain-consistency review to limit error propagation and improve extraction consistency.Each agent's structured output becomes the next module's input rather than requiring a single model to generate final results from scratch.Source: Multi-Agent Collaborative Knowledge… · p. 5

Agent 1: Candidate Discovery

  • Agent 1 broadly identifies potential evaluation targets, tolerates moderate over-generation, and excludes expressions lacking independent academic meaning or explicit identifiability.It produces candidates for later validation rather than performing final classification.Source: Multi-Agent Collaborative Knowledge… · p. 5

Agent 2: Ontology-Constrained Validation and Classification

  • Agent 2 discards candidates that are not identifiable ontology-mappable objects, then classifies valid entities using ontology rules and evaluative context.This converts high-recall candidates into structured annotations while reducing false entities from generic terms, abstract concepts, and evaluative expressions.Source: Multi-Agent Collaborative Knowledge… · p. 5

Agent 3: Domain Review

  • Agent 3 reviews classifications for type drift, boundary inconsistencies, and insufficient evidence, suggesting revisions and producing an internal confidence assessment.Confidence is only an internal reference; final performance is evaluated against the silver-standard corpus validated through independent human review.Source: Multi-Agent Collaborative Knowledge… · p. 5

EXPERIMENT DESIGN Research Questions

The experimental design is organized around four questions evaluating overall extraction performance, the multi-agent workflow, ontology-based type boundaries, and the framework under stricter typed evaluation.

Research Questions

Corpus Construction and Validation · Main Baselines and Ablation Settings

The study builds an ontology-constrained silver-standard corpus from 200 Chinese LIS review texts and compares a rule-based baseline, a zero-shot LLM baseline, ablations, and the full system. The corpus is independently checked, and its silver-standard status qualifies how evaluation results should be interpreted.

Corpus Construction and Validation

  • Ontology-guided review of candidates from 200 CSSCI-indexed Chinese LIS reviews produced a silver-standard corpus of 25,859 entities and 5,557 evaluative sentences.DeepSeek-chat generated high-recall candidates; DeepSeek V4-Flash reviewed existence, boundaries, and types, removed false positives, and supplemented missed entities. Entities were required to be contiguous spans with the most specific contextually appropriate type where possible.Source: Corpus Construction and Validation · p. 6
  • Independent review of 388 sampled entities found Cohen’s κ = 0.833 between researchers and κ = 0.909 and 0.910 against the silver standard.The sample was approximately 1.5% of all entities; reviewers noted some boundary inconsistencies, redundant extractions, and type misclassifications.Source: Corpus Construction and Validation · p. 6
  • Because the reference is a silver standard rather than a fully independent gold standard, evaluation provides comparative evidence under ontology-constrained corpus construction.Random-sample human validation and agreement statistics were used as a quality check to reduce potential circularity.Source: Corpus Construction and Validation · p. 7

Main Baselines and Ablation Settings

  • The main experiments compare B1, B2, A1, A2, A3, and Full on all 200 test documents with a shared label space; LLM settings use DeepSeek-chat and identical decoding.B1 is the rule-based lower bound, while B2 tests direct zero-shot prompting without ontology boundary constraints, a multi-agent workflow, or internal validation.Source: Main Baselines and Ablation Settings · p. 7
  • A1 removes the multi-agent workflow, A2 removes Ontology Constraints, and A3 removes both while retaining basic preprocessing, schema constraints, field validation, and minimal extraction instructions.These variants isolate the contributions of task decomposition and ontology-based boundary constraints.Source: Main Baselines and Ablation Settings · p. 7
  • B2 is an external zero-shot prompting baseline, whereas A3 is a minimized ablation within the same engineering workflow, so they are not intended for direct superiority comparisons..Source: Main Baselines and Ablation Settings · p. 7
  • Full combines few-shot prompting, a multi-agent workflow, and ontology-based boundary constraints.Source: Main Baselines and Ablation Settings · p. 7

Supplementary Domain Model Stability Analysis

The supplementary analysis introduces two domain-specific models to examine stability and performance differences from the proposed framework. Because they operate independently and cannot be integrated into its multi-agent workflow, they are excluded from the main ablation analysis.

Supplementary Domain Model Stability Analysis

  • B3 (LISGPT-14B) and B4 (HssaLLM-32B) were added as supplementary comparison models to examine stability and performance differences from the framework, but excluded from the main ablation analysis.Both operate as independent systems that cannot be directly integrated into the proposed multi-agent framework.Source: Supplementary Domain Model Stability… · p. 7
  • B3 uses LISGPT-14B with ontology-based boundary constraints but no agent workflow, testing direct-prompting stability with a medium-scale LIS-domain model.LISGPT-14B was developed for the Library and Information Science domain.Source: Supplementary Domain Model Stability… · p. 7
  • B4 uses HssaLLM-32B with ontology-based boundary constraints but no agent workflow, testing direct-prompting stability with a larger Humanities and Social Sciences model.HssaLLM-32B was developed for Humanities and Social Sciences.Source: Supplementary Domain Model Stability… · p. 7

Evaluation Metrics … Role of the Multi-Agent Workflow and Ontology Constraints

The study evaluates extraction, classification, and fine-grained distinction, finding that the full ontology-guided multi-agent system outperforms baselines while its two core mechanisms contribute to performance.

Evaluation Metrics

  • Strict Typed F1 requires correct entity boundaries and types, while Precision, Recall, and Entity-level F1 assess extraction performance.Type Accuracy measures classification correctness among matched entities; Invalid Rate and Intra-group Confusion reflect output validity and neighboring-category confusion.Source: Evaluation Metrics · p. 7

Overall Performance

  • RQ1 compares the full system with B1, a shallow rule-based approach, and B2, a general-purpose LLM prompted directly in one round, on the same test set.B1 provides a lower-bound reference under limited semantic modeling capability.Source: Overall Performance · p. 8
  • The full system beats B1 and B2, raising Entity-level F1 by 18.50 percentage points and Strict Typed F1 by 32.8 points over B2.Against B1, Entity-level F1 rises from 27.47% to 87.34%, with substantially higher precision and recall.Source: Table 1 · p. 8, Overall Performance · p. 8

Role of the Multi-Agent Workflow and Ontology Constraints

  • Without the multi-agent workflow, A1’s Entity-level F1 falls to 72.74% and Strict Typed F1 to 57.41%.The single-step prompt weakens recall and strict type matching by compressing candidate discovery, classification validation, and domain review.Source: Role of the Multi-Agent Workflow and… · p. 8
  • Without detailed ontology boundary constraints, A2 reaches 76.94% Entity-level F1 but only 56.60% Strict Typed F1 and 73.57% Type Accuracy.Agent specialization alone does not resolve fine-grained ambiguity; explicit type definitions, inclusion and exclusion criteria, and guidance on commonly confused categories remain necessary.Source: Role of the Multi-Agent Workflow and… · p. 8
  • Removing both mechanisms in A3 yields 74.84% Entity-level F1 and 53.51% Strict Typed F1, indicating complementary contributions.The B2–A3 comparison is diagnostic: B2 tests direct prompting, whereas A3 represents the proposed workflow with both core mechanisms removed.Source: Role of the Multi-Agent Workflow and… · p. 8

Error and Boundary Analysis

The analysis uses five predefined confusion groups because Macro-F1 and L1-group F1 are unavailable, and finds ontology boundary constraints especially valuable in boundary-sensitive classification.

Error and Boundary Analysis

  • RQ4 analyzes remaining errors using five predefined confusion groups because Macro-F1 and L1-group F1 are not reported in the current experiments.The groups target critical evaluative-extraction errors and cases that clarify the effects of ontology-based boundary constraints.Source: Error and Boundary Analysis · p. 8
  • The full system maintains low confusion across five predefined groups, with particularly large reductions in Methodology/Method/Technique versus A2 and A3.Table 2 presents the intra-group confusion results.Source: Error and Boundary Analysis · p. 8, Table 2 · p. 9
  • Ontology-based boundary constraints contribute most in conceptually boundary-sensitive classification scenarios.The especially large reductions in Methodology/Method/Technique confusion versus A2 and A3 support this conclusion.Source: Error and Boundary Analysis · pp. 8–9
  • Models often identify entity boundaries correctly but still confuse a methodology with a method, or vice versa.Type Accuracy alone therefore does not comprehensively evaluate system performance; Strict Typed F1 and Intra-group Confusion reflect both boundary recognition and fine-grained type discrimination.Source: Error and Boundary Analysis · p. 9

Additional Analysis of Domain Model Performance · DISCUSSION · Theoretical Contributions and Practical Implications

The supplementary analysis finds that domain-specific models have relatively low recall under direct prompting, while the discussion frames evaluation object extraction as knowledge organization enabled by ontology-guided boundaries and staged review. The framework is presented as a reproducible way to turn scholarly evaluative texts into structured evidence for research assessment and STI analysis.

Additional Analysis of Domain Model Performance

  • Both domain-specific models show relatively low recall under direct prompting, and remain insufficiently adapted to evaluative target extraction in this setting.The passage describes these models as supplementary references and notes their future potential.Source: Additional Analysis of Domain Model… · p. 9
  • Evaluative targets can span local contextual cues and ontology boundaries, so extraction may require staged candidate generation and validation rather than a single pass.The task is also underexplored, with limited high-quality task-specific training data.Source: Additional Analysis of Domain Model… · p. 9

Theoretical Contributions and Practical Implications

  • The study treats evaluation object extraction as a knowledge organization problem, defining objects as semantic anchors that connect evaluation dimensions and viewpoints.This ontology provides a conceptual basis for structuring dispersed evaluative statements as evidence for research assessment and STI analysis.Source: Theoretical Contributions and… · p. 9
  • Ontology-based boundaries make implicit disciplinary judgments executable by requiring category decisions to reflect functional attributes, epistemological status, and evaluative context.For categories such as theory/model/framework and methodology/method/technique, this explains Full’s advantages over A2 in Type Accuracy, Strict Typed F1, and Intra-group Confusion.Source: Theoretical Contributions and… · p. 9
  • The multi-agent workflow organizes the same backend model into stages: high-recall candidate generation, ontology-constrained filtering and classification, and domain review of structured records.Its contribution is the inspectable workflow, not the use of multiple different models.Source: Theoretical Contributions and… · p. 9
  • Explicit intermediate states, error isolation, independent verification, and structured review explain Full’s superiority over A1 in Recall, F1, and boundary control.The staged process lets candidate omissions, type confusions, and review disputes be identified and corrected at different stages.Source: Theoretical Contributions and… · pp. 9–10
  • The framework offers a reproducible workflow for converting academic reviews, scholarly commentaries, and book reviews into structured evaluative evidence.That evidence may support responsible research assessment, disciplinary mapping, and STI platforms that incorporate expert judgment embedded in scholarly discourse beyond citation-based indicators.Source: Theoretical Contributions and… · p. 10

Limitations … GENERATIVE AI USE

The study frames evaluation-object extraction from Chinese academic reviews as ontology-constrained knowledge structuring and reports that its full system outperforms two baselines. It identifies limits in model self-assessment and disciplinary validation, and proposes cross-disciplinary extension and richer evaluation knowledge graphs as future work.

Limitations

  • The study uses the same backend model for all agents, treats reviewer scores as internal diagnostics rather than expert evaluation, and validates empirically only on Chinese Library and Information Science reviews.Some weaker-performing models received higher Reviewer Confidence scores, indicating insufficient calibration; other humanities and social science disciplines require further testing and calibration of entity types, evaluative vocabularies, and boundaries.Source: Limitations · p. 10

Future Research

  • Future work may test ontology transfer to additional disciplines and extend object extraction to evaluation dimensions, viewpoints, and their semantic relationships.These extensions could support a more comprehensive academic evaluation knowledge graph; the study also identifies reducing inference costs as a future focus.Source: Future Research · p. 10, CONCLUSION · p. 10

CONCLUSION

  • Evaluation-object extraction involves abstract objects, context-dependent boundaries, and closely related types, addressed through candidate discovery, ontology-constrained classification, and domain review.The study frames extraction as knowledge structuring governed by domain-specific semantic constraints and develops an object-level ontology for evaluative contexts in Chinese Library and Information Science.Source: CONCLUSION · p. 10
  • On the complete 200-document test set, the full system outperformed rule-based and single-model zero-shot baselines in Entity-level F1, Strict Typed F1, type accuracy, and boundary control.Supplementary small-sample, multi-seed experiments found that direct prompting of domain-specific models still had insufficient recall and output variability; these were not the primary basis for comparison.Source: CONCLUSION · p. 10
  • Ablations indicate that multi-agent workflow improves extraction stability and error traceability, while ontology-based boundary constraints improve fine-grained type classification and discrimination of related concepts.The three-agent design centers on staged intermediate outputs, error isolation, independent validation, and structured review.Source: CONCLUSION · p. 10

GENERATIVE AI USE

  • The authors used generative AI for Python programming, language editing, text condensation, and formatting, then reviewed, verified, and revised its outputs.The authors assume full responsibility for the research design, data analysis, interpretation, and submission content.Source: GENERATIVE AI USE · p. 11

Related papers

Loading 2608.29526v1…