Source-linked AI summary
LM-Lexicon: Improving Definition Modeling via Harmonizing Semantic Experts
Yang Liu, Jiaye Yang, Weikang Li, Jiahui Liang, Yang Li, Lingyong Yan
TL;DR
Definition modeling must generate precise definitions across rare senses and heterogeneous domains, but existing systems can produce incomplete, generic, or overly specific outputs. LM-LEXICON clusters data, trains semantic experts, and merges them into a sparse MoE with domain-level routing; it reports improvements over strong baselines and frontier models across five benchmarks.
Problem
Existing definition-modeling methods have limited rare-sense coverage and struggle with generic, overly specific, or domain-dependent definitions.
Method
LM-LEXICON clusters definition data into semantic domains, trains specialized experts, merges their weights into a sparse MoE, and uses semantic domain-level routing.
Results
Up to 10% improvement over strong baselines is reported across five benchmarks, with gains over frontier LLMs and supervised methods.
Takeaways & Limitations
The framework generates appropriate definitions across domains and genres while allowing specialized experts to be integrated for domain updates or collapsed into a single expert for efficient inference.
Takeaways & Limitations
The study focuses only on English definition modeling, leaving extension to other domains and similar semantic-intensive tasks for future work.
Abstract
from arXiv · showhide
We introduce LM-Lexicon, an innovative definition modeling approach that incorporates data clustering, semantic expert learning, and model merging using a sparse mixture-of-experts architecture. By decomposing the definition modeling task into specialized semantic domains, where small language models are trained as domain experts, LM-Lexicon achieves substantial improvements (+7% BLEU score compared with the prior state-of-the-art model) over existing methods on five widely used benchmarks. Empirically, we demonstrate that 1) the clustering strategy enables fine-grained expert specialization with nearly 10% improvement in definition quality; 2) the semantic-aware domain-level routing mechanism achieves higher expert efficacy (+1%) than conventional token-level routing; and 3) further performance gains can be obtained through test-time compute and semantic expert scaling. Our work advances definition modeling while providing insights into the development of efficient language models for semantic-intensive applications.
1 Introduction
Definition modeling automatically generates human-readable definitions from terms and their contexts, addressing the labor of continuously updating lexicons. LM-LEXICON targets limitations in semantic coverage and domain-sensitive meaning by specializing and routing semantic experts.
- Motivation: Definition modeling automatically generates definitions from a target term and its context, supporting lexicon construction.Traditional lexicon construction is labor-intensive, while language use requires updates for new terms, senses, meaning shifts, and domain knowledge.
- Challenges: Existing methods struggle with rare senses, generic or overly specific outputs, and terms whose meanings vary across domains.These limitations restrict semantic coverage and handling of semantic heterogeneity.
- Challenges: Dense architectures make precise domain-specific meaning representations difficult because polysemantic concepts share neurons through superposition.The lack of sparsification affects definition accuracy for words with distinct meanings across domains.
- Approach: LM-LEXICON trains parallel semantic experts on clustered data, merges their weights into a sparse MoE, and routes test samples with a semantic-aware router.The framework is described as Specialize-then-Synthesize and covers multiple domains and definition genres.
- Results: Up to 10% improvement over strong baselines is reported in automatic evaluation across five benchmarks.Human evaluation also reports strong performance across most criteria, particularly against frontier LLMs in semantic-intensive scenarios.
2 Related Work
Prior definition-modeling work progressed from word-embedding contexts to Transformer and causal language models, followed by LLM-based approaches. Related MoE research emphasizes efficient routing or specialized-model composition, providing context for LM-LEXICON's combination of both ideas.
- Upcycling to Mixture-of-Experts: MoE research has studied efficient token-level routing, while BTM and BTX specialize models across domains and compose their parameters or feed-forward networks.These lines of work motivate combining expert specialization with model merging.
- Definition Modeling: Early definition-modeling systems used pretrained word embeddings as global or local contexts to generate definitions.Later work adopted Transformer-based sequence-to-sequence and causal language models.
- Definition Modeling: Recent definition modeling includes LLM-based generation with models such as GPT-3.5 and GPT-4.
3 Methodology
LM-LEXICON follows a Specialize-then-Synthesize pipeline: cluster definition data by lexical semantics, train domain-specific experts, and merge them into a unified sparse MoE. A semantic domain-level router selects experts during training and inference while shared parameters are averaged.
- Overview: The framework has three stages: training-data partitioning, parallel expert training, and separate-expert merging.It begins from a pretrained seed model and targets improved multi-domain lexical-semantic performance.
- Dataset Construction: Definition-modeling data consists of context, term, and reference-definition triplets formatted into a prompt sequence.The prompt combines the context and target term for conditional definition generation.
- Clustering: Balanced k-means clusters prompt embeddings into semantically distinctive domains, with N = 4 selected using cluster cohesion and separation.The resulting partitions support expert learning for domains such as adjectives and proper nouns.
- Expert Training: Starting from a seed model, each expert is trained on its corresponding cluster, and loss masking propagates gradients only through predicted-definition tokens.The resulting language models are specialized in their respective domains.
- Model Merging: Expert merging places their feed-forward networks into MoE expert layers while averaging remaining parameters to mix capabilities across domains.This weight-level upcycling differs from averaging final output distributions or selecting one expert at test time.
- Routing: The semantic router sends representations toward domain centroids using cosine similarity and sparse top-k routing, with k = 2.The router and selected expert layers are lightly fine-tuned after merging; multihead-attention and other remaining weights are averaged across domains.
4 Experiments
Experiments evaluate LM-LEXICON across five benchmarks, comparing it with supervised, causal, and frontier language-model baselines. Results show strong benchmark performance, human-evaluation advantages, and gains from semantic clustering, domain-level routing, expert scaling, and test-time sampling.
- Main Results: LM-LEXICON nearly matches the paper’s reported 10% improvement in BLEU and ROUGE on 3D-EX over the prior state of the art.Table 2 compares LM-LEXICON-DENSE and LM-LEXICON-MOE with existing definition-modeling methods and frontier models.
- Main Results: LM-LEXICON achieves the highest reported Urban scores, 31.26% BLEU and 33.81% ROUGE, among the compared methods.The authors associate this result with modeling rare word senses and usages.
- Human Evaluation: LM-LEXICON-MOE consistently outperforms other evaluated models on most human-evaluation dimensions, especially accuracy, with a score of 4.6.The evaluation covers accuracy, clarity, conciseness, context appropriateness, and grammar and fluency.
- Ablation Study: Semantic embedding clustering outperforms lexical-based partitioning by about +7% BLEU and +1% ROUGE on 3D-EX.The authors report that semantic-targeted clusters may capture more precise senses and support more robust experts across domains.
- Ablation Study: Domain-level routing is more effective than the evaluated token-level and sequence-level routing policies, including top-2 token-level routing.The comparison treats semantic routing through specified domain clusters as especially beneficial for semantic-intensive tasks.
- Scaling and Test-time Compute: Expert scaling raises BLEU from 41.38% with N = 1 to 46.86% with N = 8, while best-of-N generation quality improves across five benchmarks.Oracle verification also boosts LM-LEXICON-DENSE by average ∆BLEU > 2%, whereas its benefit is more limited for LM-LEXICON-MOE.
5 Conclusion
LM-LEXICON combines domain experts with a sparse MoE model to generate appropriate definitions across domains and genres, outperforming frontier LLMs and strong supervised baselines.
- LM-LEXICON combines domain experts with a sparse MoE model for definition generation across domains and genres.
- LM-LEXICON significantly outperforms frontier LLMs and strong supervised baselines.
Limitations
The paper’s evidence is limited to English definition modeling, and training multiple expert language models requires substantial computation despite offline, asynchronous execution. It also identifies stronger generation verifiers as an unmet need for scaling test-time compute.
- The study focuses only on English definition modeling, limiting direct evidence for other domains and semantic-intensive tasks.
- Training N × M expert language models still requires an essential computation budget, although training can be offline and asynchronous.
- The authors encourage further exploration of parameter-efficient training methods based on LM-LEXICON.
- Stronger verifiers tailored to definition modeling and general language generation are currently unavailable or highly limited.
Ethics Statement
The study reports ethical safeguards covering data permissions, privacy, misuse prevention, responsible resource use, bias documentation, and fair compensation for annotators.
- The study used public data with appropriate permissions and measures for privacy protection and misuse prevention.
- The researchers documented potential biases and limitations and followed fair labor practices with appropriate annotator compensation.
- The computational resources were used responsibly.
A Additional Experiment Details
The appendix describes dataset preparation, semantic clustering, expert training, reward modeling, inference evaluation, implementation details, and resource accounting. These procedures include context-level data splitting, balanced semantic clusters, repeated evaluation runs, and reported training resources.
- Data Processing: Raw 3D-EX contains ten lexicon sources of term-context-definition triplets split at the word level into training, validation, and test sets.
- Data Processing: Entries with multiple example contexts are split into separate instances to expose the model to more contextual samples during training.
- Reward Modeling: The outcome reward model treats the golden definition as preferred and a model generation as alternate, framing reward modeling as binary classification with negative log-likelihood.
- Inference Setup: Each inference setting uses three runs with different random seeds, while coverage increases with samples through oracle verification against golden definitions.
C.1 Data Clustering Results
The clustering analysis evaluates cohesion within and separation between semantic domains, while examples illustrate representative cluster assignments and scaling in-context learning results on WordNet.
- Cluster cohesion and separation are reported in Tables 7 and 8 to assess the quality of the semantic partitioning.
- Four semantic clusters correspond to adjectives, scientific terms, proper nouns, and person names in the reported examples.
- The paper provides cherry-picked definition-modeling examples for each domain cluster.
- Figure 7 scales WordNet in-context learning results with k-shot demonstrations from 0 to 128 on a logarithmic scale.The comparison includes the Rerank-T5 prior state-of-the-art model.
D Human Evaluation Agreement
Human-evaluation agreement is assessed with Fleiss’s Kappa, which accounts for agreement among multiple raters while correcting for chance agreement. The resulting coefficients are reported for each criterion and model.
- Fleiss’s Kappa measures reliability of agreement among multiple raters while accounting for agreement occurring by chance.
- Po denotes observed agreement among the raters, whereas Pe denotes expected agreement by chance.
- Table 9 presents Fleiss’s Kappa coefficients for human-evaluation agreement across each criterion and model.
E Comparison of Different Definitions
The comparison examines generated definitions, implementation components, and evaluation procedures for LM-Lexicon-MoE. A representative case shows closer semantic alignment than several general-purpose models, while the code exposes the model’s expert-merging and routing design.
- Generated definitions: Closed-source models generate overly verbose definitions with redundant contextual information and multiple interpretations.
- Generated definitions: These models overemphasize derivative or secondary meanings, producing broad definitions that deviate from the core semantic meaning.
- Generated definitions: Models trained on general corpora struggle with uncommon word usages because of limited training examples, causing inaccurate definitions or contextual misinterpretation.
- Implementation: The merger combines expert models into a unified model by retaining expert layers and averaging other parameters.
- Implementation: The implementation provides semantic-expert merging code and a SemanticMoeLayer supporting token-level, sequence-level, and domain-level routing policies.The layer exposes expert count, active experts per token, and routing-policy parameters.
- Implementation: Domain-level routing assigns one selected expert responsibility for the entire sequence, unlike token-level routing that selects experts per token.
- Evaluation: Human evaluation rates five criteria independently on a 1–5 scale, with agreement summarized using Fleiss’s Kappa.