Source-linked AI summary
Retrieval-Augmented Multi-Prompt Ensemble for Minor-Grain Breeding Information Extraction
Hang Zhao, Jiahao Wang
TL;DR
Minor-grain breeding knowledge is recorded in unstructured, terminology-dense texts that are difficult to extract consistently. RAME combines retrieval, prompt diversity, and repeated voting; it ranks first on the Track-A leaderboard with a 0.499 Total Score.
Problem
Minor-grain breeding knowledge is recorded in unstructured, terminology-dense texts with variable forms and nested material names, making consistent extraction difficult.
Source: §1 · p. 1Method
RAME is a training-free framework combining hybrid retrieval-augmented few-shot selection, three prompt variants, and repeated LLM sampling with majority voting.
Source: §1 · pp. 1–2Results
0.499 Total Score; RAME ranks 1st and exceeds the official GPT-5.5-based Track-A baseline (0.448) by 11.4% relative improvement.
Source: §5.2 · p. 5, §6 · p. 6Takeaways & Limitations
Voting serves as a controllable test-time scaling knob for addressing single-pass LLM instability in structured extraction.
Source: §6 · p. 6RAME trades inference compute for accuracy through 3N decodes per document and is bounded by relation recall under exact-triple matching.
Source: §6 · p. 6
Abstract
from arXiv · showhide
This paper presents our system for CCL2026-Eval Task 5: Minor-Grain Breeding Information Extraction (MGBIE), which jointly extracts 12 entity types and 6 relation types from minor-grain breeding literature. We propose RAME (Retrieval-Augmented Multi-Prompt Ensemble), a training-free framework that elicits multiple LLM outputs under controlled diversity and aggregates them by majority voting to obtain high-confidence predictions. RAME combines (i) retrieval-augmented few-shot selection via a hybrid BM25-embedding retriever, (ii) a three-prompt ensemble (Strict, Relaxed, Balanced) spanning the precision to recall spectrum, and (iii) large-scale repeated sampling with majority voting to filter noisy predictions. Built on DeepSeek-V4-Flash, RAME achieves a Total Score of 0.499 (NER 0.730, RE 0.346) on the leaderboard, ranking 1st and surpassing the official Track-A baseline powered by GPT-5.5 (0.448), representing an 11.4% relative improvement. Code is available at https://github.com/king-wang123/CCL26-RAME.
1 Introduction
Minor-grain breeding information is difficult to extract consistently from terminology-dense texts with variable surface forms and nested material names. RAME addresses this challenge with training-free retrieval, prompt diversity, and repeated voting.
1 Introduction
- Minor-grain breeding texts contain dense domain terminology, variable surface forms, and nested material names that make consistent extraction difficult.MGBIE further involves scarce annotated data, fuzzy entity boundaries, and diverse relations that can span multiple sentences.Source: §1 · p. 1
- RAME combines hybrid retrieval-augmented few-shot selection, three complementary prompts, and majority voting over repeated LLM outputs.The prompts span stricter, more permissive, and balanced relation guidance; voting retains predictions recurring above a threshold.Source: §1 · pp. 1–2
- The authors present RAME as a training-free framework that ranks first on the leaderboard and analyze how voting threshold and sampling budget affect the precision–recall trade-off.The paper also releases its code for reproducibility.Source: §1 · p. 2
2 Related Work
Prior work frames information extraction as structured generation and applies prompting and self-consistency to improve extraction without task-specific supervised training. This paper situates MGBIE as a set-valued prediction problem for which stability remains a concern.
2 Related Work
- Information extraction has shifted toward structured generation, with UIE and InstructUIE casting NER and RE as text-to-structure generation (Lu et al., 2022; Wang et al., 2023a).Prompted annotation guidelines have enabled competitive extraction without per-task supervised training (Wang et al., 2025b; Sainz et al., 2024).Source: §2 · p. 2
- Self-consistency selects frequent sampled outputs, while prior structured-prediction work reports improved stability and performance from multi-round accumulative voting (Wang et al., 2023b; Wang et al., 2025a).Unlike voting over a scalar answer, MGBIE predicts sets of spans and triples.Source: §2 · p. 2
3 Task Definition
MGBIE evaluates extraction of typed breeding entities and semantic relations from text. Its coarse relation taxonomy, exact-match evaluation, and sparse relations define important task constraints.
3 Task Definition
- The task requires an entity set of character spans labeled with 12 breeding types and a relation set of head–tail triples labeled with 6 semantic relation types.Typical entity-type pairs are provided for relations, but valid pairs are not limited to them.Source: §3 · p. 2
- Relation extraction is difficult because the coarse-grained taxonomy permits valid head–tail pairs beyond the typical type combinations.The task also features scarce annotations, fuzzy entity boundaries, and relations that can span multiple sentences.Source: §3 · p. 2
- Evaluation uses exact matching: entity predictions require correct boundaries and types, while relation predictions require the full triple to match.Each subtask score combines F1, precision, and recall; the final score weights RE at 0.6 and NER at 0.4.Source: §3 · p. 2
- The provided 1,000 training documents form the retrieval pool, and the 1,000 test documents are divided into Phase-A and final Phase-B sets.Relations average 2.35 per document.Source: §3 · p. 2
4 Method
RAME retrieves relevant demonstrations, varies prompts and sampled examples across repeated LLM draws, votes on structured predictions, and aligns surviving entities to text spans.
4.1 Overview
- RAME retrieves demonstrations, runs three prompt variants across repeated draws with randomly sampled examples, votes over outputs, and aligns surviving predictions to character spans.Diversity comes from both prompt variants and randomly sampled demonstrations.Source: §4.1 · p. 3
4.2 Retrieval-Augmented Few-Shot Selection
- A hybrid retriever combines character n-gram BM25 for surface matches and embeddings for semantic similarity, then samples K demonstrations per draw from the top-M pool.Character n-grams help match out-of-vocabulary breeding identifiers.Source: §4.2 · pp. 3–4
4.3 Multi-Prompt Ensemble
- Strict guidance favors precision, Relaxed guidance favors recall, and Balanced guidance lists common but non-exhaustive patterns.The prompts share a schema and output format while differing in relation guidance.Source: §4.3 · p. 4
4.4 Majority Voting over Structured Outputs
- Voting counts occurrence-based entity and relation candidates at most once per draw and retains candidates appearing in at least θ of 3N draws.A small threshold favors recall, while a large threshold favors precision; the method uses a strict-majority threshold.Source: §4.4 · p. 4
4.5 Occurrence-Based Entity Alignment
- After voting, surviving entities are aligned to character spans using increasingly tolerant string matchers, and relations reuse their entities’ resolved spans.The matchers progress from exact substring matching to case-insensitive, whitespace-normalized, and punctuation-stripped matching.Source: §4.5 · p. 4
5 Experiments
RAME ranks first on the Track-A leaderboard, and the experiments examine its setup, component contributions, and voting behavior. The analysis shows that both the voting threshold and sampling budget shape performance.
5.1 Experimental Setup
- Leaderboard predictions use all 1,000 training documents for retrieval and the 600-document Phase-B test set; ablations use a 100-document development split and 900-document retrieval pool.The default configuration uses DeepSeek-V4-Flash, 60 draws per prompt, θ=91, and eight demonstrations from 30 retrieved candidates.Source: §5.1 · p. 4
5.2 Main Results
- 0.499 Total Score ranks RAME first, ahead of the GPT-5.5 Prompt-Engineering baseline at 0.448, an 11.4% relative improvement without parameter tuning.RAME scores 0.730 on NER and 0.346 on RE.Source: Table 1 · p. 5, §5.2 · p. 5
5.3 Ablation Study
- On the 100-document held-out split, merging all three prompts raises Total from the best single-prompt score of 0.427 to 0.438, a +0.011 gain; Strict leads single-prompt Total.Relaxed (P2) trades NER precision for slightly higher RE recall.Source: Table 2 · p. 5, §5.3 · p. 5
- Retrieval-augmented demonstrations add +0.042 Total, and repeated sampling with voting adds another +0.041, including +0.060 for RE.The paper attributes the retrieval gain to in-domain examples providing breeding-identifier surface conventions and says voting suppresses spurious, low-consensus relations.Source: §5.3 · p. 5, Table 2 · p. 5
5.4 Analysis
- Total peaks at θ=100 (0.529), while θ=91 scores 0.521; increasing sampling from N=1 to N=15 raises mean Total from 0.481 to 0.520, then gains saturate.From N=15–60, mean Total remains 0.519–0.523; increasing from 45 to 180 total draws adds 0.001, while variability contracts with more draws.Source: §5.4 · pp. 5–6, Figure 2 · p. 6
6 Conclusion
RAME combines hybrid retrieval, three complementary prompts, and repeated sampling with majority voting for MGBIE. It ranks first on Track-A, while its inference cost and relation recall under exact-triple matching remain limitations.
6 Conclusion
- RAME combines hybrid retrieval-augmented few-shot selection, a three-prompt precision–recall ensemble, repeated voting over occurrence-based outputs, and span alignment.It is training-free and uses 3N decodes per document.Source: §6 · p. 6
- RAME ranks first on the Track-A leaderboard with Total Score 0.499, an 11.4% relative improvement over the official GPT-5.5-based baseline, without parameter tuning.The paper describes voting as turning single-pass instability into a high-precision ensemble and a controllable test-time scaling knob.Source: §6 · p. 6
- RAME trades inference compute for accuracy through 3N decodes per document, and relation recall is bounded under exact-triple matching.The authors identify relation-centric and cross-sentence prompting, adaptive early stopping, and ensemble distillation as future directions.Source: §6 · p. 6
A Prompt Templates
The templates share an entity guide and output schema, while Strict, Relaxed, and Balanced prompts vary relation guidance and extraction rules. Each call supplies retrieved labeled examples before the target passage.
A Prompt Templates
- All three prompt variants share an entity-definition and output-schema skeleton, differing only in relation guidance and the final Rules line.The shared rules require verbatim entity substrings and numbered occurrences.Source: A Prompt Templates · p. 8
Shared Skeleton – Entity Guide + Output Schema
- The shared schema covers 12 entity types and six relation types, with entities represented as exact substrings and occurrence indices and relations linking labeled head and tail mentions.Entity types include crops, varieties, traits, genes, QTLs, markers, and breeding methods.Source: A Prompt Templates · p. 8
A Prompt Templates
- Strict enforces typed-pair constraints and prioritizes precision, whereas Relaxed permits any semantically fitting head–tail type combination and favors relation recall.Both require relations to use extracted entities as head and tail.Source: A Prompt Templates · p. 8
- Balanced presents common relation patterns as non-exhaustive, treats semantic definitions as authoritative, and prefers—but does not require—common patterns.Its rules say not to skip non-typical relations.Source: A Prompt Templates · p. 9
- Each call appends K retrieved demonstrations containing passage text and labeled JSON before the target passage and requests a JSON response.The template repeats the example format for K examples before introducing the real passage.Source: A Prompt Templates · p. 9
B Implementation Details
Implementation details cover the shared leaderboard and ablation configuration, model serving, development split, and output alignment and parsing.
B Implementation Details
- The same RAME configuration is used for leaderboard submissions and held-out-split ablations; Table 3 lists its hyperparameters.The supplied table passage identifies the table as RAME hyperparameters.Source: B Implementation Details · p. 9, Table 3 · p. 9
- DeepSeek-V4-Flash runs through sglang on eight NVIDIA H20 GPUs, with reasoning mode enabled and truncated empty draws retried using a doubled token budget.The service uses an OpenAI-compatible chat-completions API.Source: B Implementation Details · p. 9
- The fixed offline split assigns 900 of 1,000 training documents to retrieval and 100 to development, keeping experiment numbers comparable.The split indices are fixed across experiments.Source: B Implementation Details · p. 9
- Occurrence-based model references are aligned to spans through exact, case-insensitive, whitespace-normalized, then punctuation-stripped matching; unalignable relations are dropped.Malformed JSON draws contribute nothing to voting, and entities and relations are deduplicated before and after aggregation.Source: B Implementation Details · p. 10