Source-linked AI summary
CMNIE: An Information Extraction Benchmark for Chinese Military News
Yan Yu, Mengna Zhu, Zhenyu Song, Hao Yang, Haiwen Chen, Mao Wang
TL;DR
Chinese military news requires joint extraction of events, arguments, entities, and relations, but existing resources provide limited support for this unified setting. CMNIE builds a benchmark with a unified domain schema and evaluates supervised and LLM-based extractors. Results show complementary strengths: explicit span modeling is strongest for exact arguments, while schema-conditioned generation leads relation extraction, and zero-shot systems struggle with exact boundaries.
Problem
Existing resources provide limited support for jointly modeling events, event arguments, entities, and relations in Chinese military news.
Method
CMNIE constructs a Chinese military news benchmark with unified annotations and evaluates supervised IE models, zero-shot LLMs, and fine-tuned LLM-based extraction methods.
Results
Explicit span modeling remains effective for exact arguments, while fine-tuned schema-conditioned generation achieves the strongest relation extraction results; zero-shot LLMs face exact boundary matching challenges.
Takeaways & Limitations
CMNIE supports research on schema adherence, span-sensitive extraction, and joint structured prediction in specialized Chinese news.
Takeaways & Limitations
The current release covers Chinese public military news from selected sources and periods, with some sparse event types and limited dictionary coverage of implicit or atypical events.
Abstract
from arXiv · showhide
Structured extraction from Chinese military news supports intelligence analysis, decision-making, and knowledge base construction. However, existing resources provide limited support for joint informa?tion extraction in this domain, especially when events, event arguments, entities, and relations must be modeled together. We present CMNIE, an information extraction benchmark for Chinese military news. Extend?ing military-domain resources beyond document-level event annotations, CMNIE jointly annotates event triggers, event arguments, named enti?ties, and entity relations under a unified domain schema. The dataset contains 13,000 instances collected from public Chinese military news, with manual annotations for 7 event types, 10 argument roles, 7 entity types, and 8 relation types. We evaluate supervised IE models, zero-shot large language models, and fine-tuned LLM-based extraction methods on a shared test set. Experimental results show that CMNIE remains chal?lenging, especially for relation extraction and exact matching of event?argument spans; zero-shot LLMs often identify relevant semantic units but fail to match gold span boundaries exactly. CMNIE provides a stan?dardized benchmark for studying schema adherence, exact span match?ing, and joint structured extraction in specialized Chinese news.
1 Introduction
CMNIE addresses the limited joint extraction support for Chinese military news by aligning events, arguments, entities, and relations in one benchmark. It evaluates supervised systems, zero-shot LLMs, and fine-tuned LLM-based methods under this unified schema.
- Military news IE must recover interdependent entities, relations, events, and event arguments, often across distant mentions and implicit relational expressions.
- CMNIE extends prior military-domain resources by jointly annotating events, arguments, named entities, and entity relations.
- Relaxed event scores are reported for zero-shot LLMs because generative outputs may identify the correct semantic unit while missing reference span boundaries.
- The benchmark evaluates supervised IE models, zero-shot LLMs, and fine-tuned LLM-based extraction methods.
2 Related Work
Related IE benchmarks differ in task scope and annotation units, while existing methods use varied span, joint-extraction, generation, and schema-conditioning strategies. CMNIE is positioned as a specialized resource that connects these benchmark and modeling developments to Chinese military information structures.
- Existing benchmarks cover multilingual entities, relations, and events; Chinese relation triples; broad Chinese event schemas; and document-level event arguments or relations.
- General-domain schemas and annotation units leave specialized Chinese military information structures only partially represented.
- IE methods range from contextualized span representations and globally informed joint extraction to unified structure generation.
- Recent LLM-based approaches encode specifications through annotation-guideline conditioning, IE supervision, schema-as-code representations, and zero-shot instruction following.
3 CMNIE Dataset
CMNIE is a jointly annotated Chinese military-news dataset built through multi-stage collection, schema design, expert-guided annotation, and quality control. It contains 13,000 instances with aligned event, argument, entity, and relation layers, exhibiting long-tailed and heterogeneous annotation patterns.
- 3.1 Data Construction: The corpus was collected from more than 40,000 public military-news articles, filtered to 13,000 candidate instances through dictionary matching and manual screening.Sources included Sina Military, Phoenix Military, and NetEase Military, covering articles published from August 2010 to February 2022.
- 3.2 Annotation Schema: CMNIE uses four annotation layers: seven event types, ten argument roles, seven entity types, and eight relation types.The schema includes event triggers and arguments alongside named entities and relations, with empty event annotations allowed for non-event instances.
- 3.3 Annotation Workflow and Quality Assessment: Annotation used two stages and multiple rounds, with 20 trained annotators and five domain experts manually reviewing spans, labels, and relation triples.Automatic event pre-labels served only as references, while experts handled guidelines, difficult cases, and disagreement review.
- 3.3 Annotation Workflow and Quality Assessment: Final audit scores were 95.3%, 95.1%, 99.4%, and 97.8% for event type accuracy, event argument recall, entity recall, and relation recall, respectively.Batches were accepted only when all four indicators reached at least 95%.
- 3.4 Dataset Statistics: 13,000 instances contain 6,997 event annotations, 23,087 event arguments, 97,508 entity mentions, and 40,252 relation mentions.The dataset is split at the instance level with event-label distributions kept as similar as possible across training, development, and test subsets.
- 3.4 Dataset Statistics: CMNIE has long-tailed distributions, with EQU and ORG dominating entities, Belong, Subordinate, and Equal dominating relations, and Manoeuvre the most frequent event type.Instances may contain dense entity or relation context without expressing every schema layer, producing empty relation and event annotations.
4 Benchmark Experiments
CMNIE evaluates supervised IE models, zero-shot LLMs, and fine-tuned LLM extraction methods across event extraction, NER, and relation extraction. Results show complementary model strengths, substantial exact–relaxed gaps for zero-shot event extraction, and persistent boundary-matching challenges.
- 4 Benchmark Experiments: The benchmark reports micro-averaged F1 for trigger, argument, named-entity, and relation extraction, including exact and relaxed event variants.Supervised and fine-tuned models share the training split, while all methods are evaluated on the same test split with a common label inventory.
- 4.2 Overall Results: OneIE achieves the highest AC score at 53.9, while GoLLIE with Qwen2.5-7B-Instruct leads TC and RE at 70.3 and 50.2.GoLLIE improves RE by 25.1 points over UIE, the strongest supervised RE baseline.
- 4.2 Overall Results: The three highest NER scores—71.9, 71.1, and 71.0—are closely clustered among leading systems.This indicates comparable NER performance among the strongest models.
- 4.2 Overall Results: Zero-shot LLMs trail the overall best results by 20.9 points for TC, 19.5 for AC, 24.9 for NER, and 26.0 for RE.Within the zero-shot group, leadership varies by task, with DeepSeek-V3.1-Chat leading TC/AC, GPT-5.1 leading RE, and Qwen2.5-72B-Instruct leading NER.
- 4.3 Exact and Relaxed Event Scores: Relaxation increases TC by 21.9 points for DeepSeek-V3.1-Chat and 21.3 for Llama-3.1-70B-Instruct.DeepSeek-V3.1-Chat reaches 71.3 relaxed TC versus 49.4 exact TC, showing the importance of boundary precision in zero-shot errors.
5 Discussion
CMNIE remains difficult because its data combine skewed event distributions, dense entities, implicit relations, and precise span requirements. Error analysis links event, entity, and relation mistakes, while the dataset is bounded by its Chinese public-news scope and sparse event types.
- 5 Discussion: CMNIE combines long-tailed event distributions, dense entities, frequent relations, empty-event instances, and implicit relation expressions.Models must distinguish background instances, learn sparse types, and recover precise boundaries.
- 5 Discussion: A single NER boundary error can invalidate an otherwise plausible relation prediction because RE requires two exact entity endpoints.Qualitative errors also include missed triggers, false event predictions, role confusions, and implicit relation expressions.
- 5.2 Data Availability, Limitations, and Ethical Considerations: CMNIE covers public Chinese military news from selected sources and periods, with some event types remaining sparse and the current release limited to Chinese.Dictionary retrieval favors explicit, frequent event expressions and offers limited coverage of implicit or atypical events outside the dictionary.
- 5.2 Data Availability, Limitations, and Ethical Considerations: The dataset is derived solely from publicly accessible news sources and has dual-use potential, requiring attention to copyright, legal, ethical, and harmful-deployment concerns.The release documents provenance and distributes permitted materials.
6 Conclusion
CMNIE unifies events, arguments, entities, and relations in a Chinese military news benchmark. Its results show complementary strengths between explicit span modeling and schema-conditioned generation, while exact–relaxed gaps identify boundary precision as a central zero-shot challenge.
- 6 Conclusion: CMNIE aligns events, event arguments, named entities, and entity relations under a unified annotation schema.The benchmark supports joint structured extraction in specialized Chinese news.
- 6 Conclusion: Explicit span modeling remains effective for exact arguments, whereas fine-tuned schema-conditioned generation achieves the strongest relation extraction results.These findings reveal complementary strengths across extraction approaches.
- 6 Conclusion: Exact–relaxed score gaps establish boundary precision as a central challenge for zero-shot extraction.CMNIE supports research on schema adherence, span-sensitive extraction, and joint structured prediction.