Source-linked AI summary
When Does Bigger Help? A Controlled Study of LLM Scale for Ontology Learning
Hamed Babaei Giglou, Sören Auer, Jennifer D'Souza
TL;DR
The study addresses limited evidence about how LLM scale affects ontology-learning performance. It benchmarks 13 Qwen and GPT models in a controlled OntoLearner RAG framework, finding that scale effects vary by task, domain, architecture, and lineage rather than following model size alone. Non-taxonomic relation extraction remains especially difficult in abstract domains.
Problem
The effects of LLM scale on ontology-learning performance remain insufficiently characterized across tasks, domains, architectures, and model lineages.
Method
The study benchmarks 13 Qwen3.5, Qwen3.6, and proprietary GPT models with OntoLearner under controlled retrieval-augmented generation conditions.
Results
Model size does not produce proportional improvements: scaling mainly improves precision, dense 27B models can beat larger MoE models on term typing, and larger MoE models lead open-weight taxonomy discovery.
Takeaways & Limitations
Model size alone is insufficient for selecting LLMs for ontology learning; task, domain, architecture, and lineage must also be considered.
Takeaways & Limitations
Non-taxonomic relation extraction remains difficult, especially in the highly abstract Materials Data Science ontology, despite increasing model scale.
Abstract
from arXiv · showhide
The effect of Large Language Model (LLM) scale on ontology learning (OL) performance remains insufficiently characterized. We present a controlled evaluation of 13 models spanning dense and Mixture-of-Experts variants from the Qwen3.5 and Qwen3.6 lineages, together with proprietary GPT release variants, using the OntoLearner retrieval-augmented generation pipeline. All models are evaluated with the same embedding model, retrieval configuration, prompt templates, decoding settings, datasets, and metrics on term typing, taxonomy discovery, and non-taxonomic relationship extraction across four biomedical and materials science and engineering ontologies. Within the dense Qwen3.5 lineage, increasing parameter count primarily improves precision rather than recall, with the largest gains occurring between 9B and 27B parameters. However, the effect of scale is neither monotonic nor uniform across tasks and domains. Dense 27B models outperform substantially larger sparse models on term typing, whereas larger Mixture-of-Experts models achieve the strongest open-weight results on taxonomy discovery. Non-taxonomic relationship extraction remains difficult across model scales, particularly for the Materials Data Science ontology. Performance differences across matched Qwen variants and proprietary GPT releases further indicate that architecture and model lineage can outweigh nominal parameter count. These findings show that model size alone is an insufficient selection criterion for OL and provide empirical guidance for reproducible LLM-assisted ontology engineering.
1. Introduction
Ontology learning supports interoperable scientific knowledge, but the effects of LLM scale remain unresolved across tasks, domains, architectures, and model lineages. This study addresses that gap with a controlled benchmark of Qwen and GPT models using OntoLearner.
- Scientific automation produces heterogeneous data and terminology that require machine-actionable knowledge representations for reuse, integration, and reasoning.
- LLM-based ontology learning can automate term typing, taxonomy discovery, and non-taxonomic relation prediction with little or no task-specific training.
- Prior scaling research shows that parameter count alone incompletely predicts capability, with downstream performance sometimes plateauing, deteriorating, or changing direction as models grow.
- Ontology-learning tasks impose different semantic and structural demands, so scale may affect precision, recall, and relational reasoning differently across tasks and domains.
- Architecture, training composition, instruction tuning, and alignment may mediate or outweigh nominal parameter count, especially when comparing dense and MoE models.
- The research questions test within-family scaling, task and domain sensitivity, and architecture or lineage effects beyond nominal parameter count.
- The study evaluates 13 Qwen3.5, Qwen3.6, and proprietary GPT models with OntoLearner across term typing, taxonomy discovery, and non-taxonomic relation extraction.
2. Preliminaries
The paper evaluates three ontology-learning tasks: assigning ontology types to terms, discovering hierarchical relations, and extracting non-taxonomic semantic relations.
- Term Typing: Term typing assigns the most appropriate ontology type to a lexical term, mapping input L to output T.
- Taxonomy Discovery: Taxonomy discovery identifies hierarchical is-a relationships between ontology types associated with two lexical terms.
- Non-Taxonomic Relationship Extraction: Non-taxonomic relationship extraction identifies semantic relations other than is-a between ontology types associated with head and tail terms.
3. Related Work
Existing ontology-learning research compares models, prompting strategies, domains, and downstream applications, but rarely isolates model scale under controlled conditions. This study fills that gap with a fixed-protocol Qwen and GPT benchmark across scientific ontologies.
- Prior work spans classical methods, LLM-based ontology generation, prompting, fine-tuning, hybrid systems, and knowledge-graph applications.
- Table 1 compares representative LLM-based ontology-learning studies and indicates whether model size was explicitly varied or controlled.
- Existing studies compare LLM families or note size-related trends, but do not isolate scale within one family while fixing domain, prompting, and evaluation protocol.
- Cross-domain studies commonly vary domain without a scale sweep, leaving unclear whether within-family scaling reliably improves quality relative to computational cost.
- The paper combines LLMs4OL task formalization with OntoLearner to evaluate Qwen and GPT families consistently on biomedical and materials-science ontologies.
4. Methodology
The methodology uses OntoLearner as a controlled retrieval-augmented generation pipeline that separates context acquisition from LLM generation. A shared embedding model, retrieval setup, and prompts enable comparisons across models, tasks, and datasets.
- Pipeline: The pipeline contains semantic vector indexing, similarity-based retrieval, and task-specific LLM prompting and generation.
- Semantic Vector Indexing: A frozen Qwen3-Embedding-4B encoder maps ontology entities into a shared dense vector space before inference.
- Similarity-Based Retrieval: Each query is embedded with the same encoder, and single-pass dense retrieval selects the top-k nearest ontology entities by cosine similarity.
- Task-Specific LLM Generation: Task-specific prompts combine system rules, retrieved candidate entities, and the target input before zero-shot generation by the evaluated LLM.
- Task-Specific LLM Generation: The LLM selects ontology classes for term typing, tests direct subsumption for taxonomy discovery, and predicts candidate relations for non-taxonomic extraction.
- Models and Datasets: Four ontologies cover materials science and engineering and biomedical domains: MatWerk, MDS, OBI, and MFOEM.
- Controlled Evaluation: Keeping the embedding model, vector index, k=10, and prompt templates constant attributes observed performance variation to model scale and architecture.
5. Experimental Setup
The benchmark evaluates diverse open- and closed-source LLMs across parameter scales and architectures on four ontologies spanning biomedical and materials science and engineering domains.
- The benchmark includes dense and Mixture-of-Experts variants from Qwen3.5 and Qwen3.6, alongside frontier OpenAI GPT models.The models span multiple parameter scales and architectural designs.
- The four ontologies cover materials science and engineering and biomedical domains: MatWerk, MDS, OBI, and MFOEM.These domains provide heterogeneous conceptual structures and complexity levels for evaluating LLM-scale effects.
6. Results
Results show that scale improves ontology-learning performance unevenly: gains are task- and domain-dependent, architectures trade off across tasks, and release lineage can outweigh nominal parameter count.
- Open-Weight Model Scale: Small Qwen models below 9B parameters achieve high recall but poor precision, indicating weak rejection of negative RAG candidate classes.Term typing recall reaches 90.57%–100.00%, while precision is 9.06%–25.33%; precision falls to 1.36%–4.84% on harder tasks.
- Open-Weight Model Scale: +22.01 percentage points: Qwen3.5-27B reaches 58.14% F1 on MatWerk Term Typing versus 36.13% for Qwen3.5-9B.The gain is driven by precision increasing from 22.22% to 43.86% while recall remains 86.21%.
- Architecture and Lineage: Qwen3.5-122B-A10B achieves the strongest open-weight taxonomy-discovery F1 scores on MFOEM and MatWerk.It reaches 24.41% and 23.05%, respectively, versus 15.20% and 12.13% for the dense 27B model.
- Architecture and Lineage: Qwen3.6-27B scores 36.62% F1 on MatWerk Term Typing versus 58.14% for Qwen3.5-27B.The comparison indicates that architectural or alignment changes across releases can disrupt monotonic scaling behavior.
- Proprietary Frontier Models: GPT-5.5 outperforms GPT-5.6 successors across nearly all evaluated tasks and domains, while GPT-5.6-Luna can become highly over-conservative.On MFOEM Term Typing, Luna has 100% precision but 5.26% recall and 10% F1; on MDS relation extraction, it produces 0% precision, recall, and F1.
- Task Complexity: Taxonomy discovery plateaus below 9% F1 for models under 9B, with improvements appearing at 27B+ and frontier scale to 36%–42% F1.This task requires reasoning over hierarchical relationships between multiple concepts.
- Task Complexity and Domain: Non-taxonomic relationship extraction remains difficult: Qwen3.5-27B reaches only 12.24% F1 on MatWerk, while frontier models reach 16.77%–19.51%.The MDS domain shows strong scale resistance, with little improvement from 0.8B to 122B.
7. Conclusion
The study finds that LLM scale improves ontology-learning performance unevenly: gains depend on task complexity, domain characteristics, architecture, and alignment rather than parameter count alone.
- Larger models generally improve precision and decision-making ability, but scaling does not produce proportional gains across tasks and domains.
- Term typing benefits most from scaling, while taxonomy discovery requires larger capacity for relational reasoning.
- Non-taxonomic relation extraction remains difficult even for frontier-scale models, particularly in highly abstract domains such as Materials Data Science.
- Dense and Mixture-of-Experts architectures, along with model alignment strategies, affect performance beyond parameter count alone.
- Effective LLM-based ontology learning requires considering task structure, domain abstraction, and model architecture when selecting an approach.
Declaration on Generative AI
ChatGPT was used only for grammar, spelling, and readability checks; authors developed and verified the scientific content, design, analysis, and conclusions.
- Generative AI tools were used solely for grammar checking, spelling checks, and readability improvements.
- The authors reviewed and adapted all suggested changes for accuracy and appropriateness.
- The authors developed and verified the scientific content, research design, analysis, and conclusions without AI involvement.