Source-linked AI summary
Ontology-Guided Multi-Agent Extraction of Evaluation Objects from Academic Review Texts: Evidence from Chinese Library and Information Science
Haolin Chen, Hongyi Dong, Yu Zhu, Yijia Hong, Leiqing Niu, Jiyuan Ye
TL;DR
Existing computational approaches provide limited support for extracting abstract, context-dependent evaluation objects from academic review texts. The study proposes an ontology-guided multi-agent framework combining candidate discovery, ontology-constrained classification, and domain review. On 200 test documents, the full system outperforms rule-based and zero-shot baselines, while ablations associate task decomposition with recall and stability and ontology constraints with fine-grained classification.
Problem
Existing extraction methods mainly target research articles or general reviews, while academic review texts contain abstract evaluation objects with ambiguous type boundaries.
Method
The framework combines staged candidate discovery, ontology-constrained validation and classification, and independent domain review.
Results
The full system outperforms rule-based and zero-shot baselines in Entity-level F1, Strict Typed F1, type accuracy, and boundary control on 200 test documents.
Takeaways & Limitations
The framework supports structured utilization of evaluative scholarly texts for evidence-based research evaluation and STI mining.
Takeaways & Limitations
The main experiments use the same backend model for all agents, so they do not establish advantages of heterogeneous multi-model collaboration.
Abstract
from arXiv · showhide
Academic reviews, scholarly commentaries, and book reviews serve as sources of evaluative statements about theories, methods, literature, institutions, and policies, providing valuable evidence for scholarly evaluation. Existing scientific entity extraction methods mainly target research articles and are less effective for evaluation objects, which are often abstract, context-dependent, and characterized by ambiguous type boundaries. This study proposes an ontology-guided multi-agent framework for evaluation object extraction. The framework combines candidate discovery, ontology-constrained classification, and domain review. Experimental results show that it achieves a Precision of 90.33%, Recall of 84.55%, Entity-level F1 of 87.34%, Strict Typed F1 of 79.78%, and Type Accuracy of 91.35%, substantially outperforming rule-based and zero-shot baselines. Ablation results indicate that the multi-agent workflow improves recall and stability, while ontology-based boundary constraints enhance fine-grained classification and reduce category confusion. The framework supports the structured utilization of evaluative scholarly texts and provides methodological support for evidence-based research evaluation and STI mining.
RELATED WORK
Academic review texts contain evaluative knowledge that is valuable for understanding scholarly judgments and research trajectories, but existing computational approaches provide limited support for extracting these evaluation objects. Their abstract and domain-specific nature requires specialized object types and extraction methods.
- Academic and book reviews organize research trajectories, formulate judgments, and advance scholarly agendas through evaluative knowledge.
- Citation analysis, co-word analysis, and topic modeling reveal macro-level patterns but provide limited support for identifying evaluation objects, relationships, and dimensions.
- Existing evaluative-text mining studies mainly address product and online reviews rather than abstract academic objects such as theories, methods, disciplines, and scholarly debates.
- Scientific literature extraction faces challenges in entity boundaries, type classification, and cross-document semantic understanding, especially under domain-specific vocabulary shifts.
- Evaluative scholarly texts remain underexplored, with research emphasizing communicative functions and document characteristics over internal evaluative knowledge structures.
Ontology-Guided and Multi-Agent Extraction
The study develops an Academic Evaluation Ontology to represent what is evaluated, how it is evaluated, and the judgments formed in evaluative scholarly texts. Explicit ontological boundaries organize heterogeneous objects and reduce confusion among semantically adjacent types.
- Theoretical Foundations: The ontology is intended to model evaluative knowledge beyond bibliographic relationships, including entities evaluated, evaluative dimensions, and resulting judgments.
- Ontology Structure: The Academic Evaluation Ontology models Evaluation Objects, Evaluation Dimensions, and Evaluation Viewpoints as the core semantic layers of evaluative texts.Objects identify targets, dimensions specify evaluative aspects, and viewpoints capture judgments with polarity, intensity, and evidence.
- Ontology Structure: The Evaluation Object layer contains Agent, Artifact, Abstract Entity, and Event, distinguishing stable entities from temporally bounded scholarly or institutional processes.
- Ontology Structure: The implementation refines the object hierarchy into 4 L1 classes, 14 L2 subclasses, and 59 L3 leaf types for progressively fine-grained classification.
- Boundary Rules and Annotation Principles: Boundary rules formalize distinctions among methodology, method, technique; theory, model, framework; organizations; databases, information systems; and policies, policy initiatives.
Multi-Agent Collaborative Knowledge Mining System Architecture
The proposed system decomposes evaluation object extraction into specialized agents following a candidate–validation–review paradigm. This separation targets candidate omission, type confusion, boundary inconsistencies, and insufficient evidence at distinct workflow stages.
- The framework assigns candidate discovery, ontology-constrained validation and classification, and domain review to three specialized agents.The architecture uses staged intermediate outputs rather than one agent handling the complete workflow.
- Agent 1: Candidate Discovery: Agent 1 prioritizes high recall by comprehensively identifying potential evaluation targets and allowing moderate over-generation for later processing.
- Agent 2: Ontology-Constrained Validation and Classification: Agent 2 validates Agent 1 candidates and assigns hierarchical ontology-constrained categories using candidate evidence rather than extracting directly from raw text.
- Agent 3: Domain Review: Agent 3 independently re-examines classified results for type drift, boundary inconsistencies, and insufficient evidence, then provides revision suggestions and internal confidence assessments.
Research Questions
The experimental design addresses overall effectiveness, the contribution of task decomposition, the effect of ontology boundaries, and the advantages and limitations of strict typed evaluation. Figure 1 presents the proposed framework used to investigate these questions.
- RQ1 compares the ontology-constrained multi-agent framework with rule-based and direct-prompt zero-shot baselines for overall extraction performance.
- RQ2 examines whether task decomposition improves extraction recall, output stability, and error traceability.
- RQ3 evaluates whether ontology-based boundary constraints improve fine-grained type classification and reduce confusion among neighboring concepts.
- RQ4 considers the complete framework’s advantages and limitations under stricter typed entity evaluation criteria.
- Figure 1 illustrates the Ontology-Guided Multi-Agent Entity Extraction System proposed to address the four research questions.
Corpus Construction and Validation
The study constructs a silver-standard corpus from 200 CSSCI-indexed Chinese LIS review texts and evaluates extraction settings designed to isolate workflow and ontology contributions.
- 200 CSSCI-indexed Chinese LIS review texts were randomly selected to construct the reference corpus.
- The DeepSeek-chat-based Full framework generated high-recall candidate entities, while DeepSeek V4-Flash performed ontology-guided corpus revision as an annotation reviewer.
- The resulting silver-standard corpus contains 25,859 entities and 5,557 evaluative sentences.
- Corpus Validation: Independent review of 388 randomly sampled entities yielded high inter-reviewer agreement, with Cohen’s κ = 0.833 and reviewer-to-standard κ values of 0.909 and 0.910.
- Experimental Settings: The main experiments compare B1, B2, A1, A2, A3, and Full on all 200 documents using the same label space and backbone model settings.
- Experimental Settings: A1 removes the multi-agent workflow, A2 removes ontology constraints, A3 removes both, and Full retains both mechanisms.
Supplementary Domain Model Stability Analysis
Supplementary experiments add two independent domain-specific language models to examine stability and performance differences under direct prompting.
- B3 and B4 are supplementary comparison models based on LISGPT-14B and HssaLLM-32B, respectively.
- B3 and B4 are excluded from the main ablation analysis because they are independent systems that cannot be integrated into the proposed multi-agent framework.
- B3 uses LISGPT-14B with ontology constraints but no agent workflow to assess direct-prompting stability in a medium-scale LIS model.
- B4 uses HssaLLM-32B with ontology constraints but no agent workflow to assess direct-prompting stability in a larger domain-specific model.
- The evaluation reports Precision, Recall, Entity-level F1, Type Accuracy, Strict Typed F1, Invalid Rate, and Intra-group Confusion across extraction and classification perspectives.
RESULTS
The Full system substantially outperforms rule-based and zero-shot baselines, while ablations show that multi-agent workflow and ontology constraints make complementary contributions.
- Baseline Comparisons: The Full system increases Entity-level F1 from 27.47% for B1 to 87.34%, showing that rule matching and conventional NER are insufficient for abstract evaluative targets.
- Baseline Comparisons: Compared with B2, the Full system improves Entity-level F1 by 18.50 percentage points and Strict Typed F1 by 32.8 percentage points.
- Ablation Analysis: A1 reduces Entity-level F1 to 72.74% and Strict Typed F1 to 57.41% after removing the multi-agent workflow.
- Ablation Analysis: A2 reaches 76.94% Entity-level F1, while Strict Typed F1 declines to 56.60% and Type Accuracy to 73.57% without detailed ontology boundary constraints.
- Ablation Analysis: Removing both mechanisms in A3 further lowers Entity-level F1 to 74.84% and Strict Typed F1 to 53.51%.
- Ablation Analysis: The ablation pattern indicates that the multi-agent workflow and ontology-based boundary constraints play complementary roles in system effectiveness.
- Interpretation: B2 and A3 should be treated as a diagnostic comparison rather than a direct superiority comparison because they represent different experimental purposes.
Error and Boundary Analysis
The analysis examines confusion among predefined boundary groups and supplementary domain-model performance. It emphasizes that ontology constraints help most in conceptually boundary-sensitive classifications, while direct prompting remains recall-limited.
- Boundary analysis uses five predefined confusion groups to identify where ontology-based constraints contribute most substantially.
- Ontology-based boundary constraints produce particularly large confusion reductions in the Methodology/Method/Technique group compared with A2 and A3.
- Strict Typed F1 and Intra-group Confusion assess extraction more comprehensively than Type Accuracy by jointly reflecting boundaries and fine-grained type distinctions.
- Each experimental setting was run three times, with final results reported as averages to assess output stability.
- Domain-specific models show relatively low recall under direct prompting, indicating that evaluative target extraction requires more than domain knowledge coverage alone.
DISCUSSION
The discussion frames evaluation-object extraction as knowledge organization and explains the framework’s gains through ontology constraints and staged multi-agent reasoning. It also identifies practical uses and important scope and validation limitations.
- Evaluation-object extraction is treated as a knowledge organization problem rather than merely a named entity recognition task.
- Ontology constraints convert disciplinary judgments into executable boundaries for distinguishing theory/model/framework, methodology/method/technique, and policy/policy initiative.
- The staged workflow assigns candidate generation, ontology-constrained classification, and structured domain review to three specialized agents.
- Explicit intermediate states, error isolation, independent verification, and structured review explain improvements over a single-prompt workflow in recall, F1, and boundary control.
- The framework can convert academic reviews and commentaries into structured evidence for research assessment, disciplinary mapping, and STI platforms beyond citation indicators.
- Empirical validation is limited to Chinese Library and Information Science, while Agent 3 scores remain internal diagnostics rather than substitutes for human or external expert evaluation.
Future Research
Future work extends the ontology across disciplines and expands extraction beyond objects to dimensions, viewpoints, and their relationships. The conclusion positions these directions alongside the framework’s structured-evidence contribution.
- Future research may transfer the object layer to additional disciplines to test ontology generalizability and scalability.
- Future studies may extract evaluation dimensions, viewpoints, and semantic relationships among objects, dimensions, and viewpoints.
- The framework addresses abstract objects, context-dependent boundaries, and closely related types through an ontology-constrained staged extraction process.
- On 200 documents, the full system outperforms rule-based and single-model zero-shot baselines across Entity-level F1, Strict Typed F1, type accuracy, and boundary control.
- The study contributes an object-level ontology and staged architecture, while future work targets lower inference costs, broader applicability, and richer evaluative relations.