Source-linked AI summary
A Guideline-Augmented Multi-Agent Framework for Schema-as-Code Biomedical Named Entity Recognition
Songtao Li, Yijia Zhang, Shidi Zhang, Jianyuan Yuan, Fengyu Zhang, Hongfei Lin
TL;DR
LLM-based BioNER leaves dataset-specific annotation semantics ambiguous and offers limited structural control over entity predictions. GAMA induces and verifies annotation guidelines, then plans and codes schema-constrained entities with dual-loop verification; across five datasets and multiple backbones, it consistently outperforms strong LLM-based baselines.
Problem
Retrieved demonstrations and external biomedical knowledge provide limited support for dataset-specific annotation semantics, while free-form generation can yield invalid formats, hallucinated or duplicated mentions, and boundary errors.
Source: §I · p. 1Method
GAMA induces and verifies dataset-specific rules, plans ranked mention-type hypotheses, codes them as schema-constrained entities, and verifies predictions with dual-loop refinement.
Source: §I · pp. 1–2, E. Verification and Dual-Loop Refinement · p. 4, C. Guideline-Guided Mention Planning · p. 4Results
GAMA consistently outperforms strong LLM-based baselines across five widely used BioNER datasets and multiple LLM backbones.
Source: §I · p. 2Takeaways & Limitations
GAMA produces more grounded and structurally valid biomedical entity predictions by combining verified guidelines, planning and coding, and dual-loop verification.
Source: §VI · p. 8
Abstract
from arXiv · showhide
Large language models (LLMs) have shown promising potential for biomedical named entity recognition (BioNER) through instruction following and in-context learning. However, existing LLM-based BioNER methods still face two key limitations. First, retrieved demonstrations and external biomedical knowledge provide limited support for dataset-specific annotation semantics, leaving entity boundaries, type scopes, and annotation conventions ambiguous. Second, free-form generation lacks sufficient structural control, often leading to invalid formats, hallucinated mentions, duplicated entities, and boundary errors. To address these limitations, we propose GAMA, a guideline-augmented multi-agent framework for schema-as-code BioNER. GAMA first induces candidate annotation rules from labeled training instances and verifies them against annotated data to construct reliable dataset-specific guideline memory. Guided by these verified rules, a planning component generates ranked span-type hypotheses with rationales, and a coding component converts them into schema-constrained entity objects. A verification module then checks span grounding, type validity, and structural compliance, and performs dual-loop refinement to correct invalid or low-confidence predictions. Experiments on five widely used BioNER datasets with multiple LLM backbones show that GAMA consistently outperforms strong LLM-based baselines. Ablation and parameter analyses further verify the effectiveness of the proposed components.
I. INTRODUCTION
BioNER methods based on supervised learning have progressed, but annotation requirements and dataset-specific semantics remain challenging for LLM-based extraction. GAMA combines explicit annotation guidelines with planning, schema-constrained coding, and verification; experiments report consistent gains over strong LLM-based baselines.
I. INTRODUCTION
- Supervised BioNER models have advanced, but large manually annotated corpora are costly to build because they require linguistic expertise and domain knowledge.BioNER supports downstream biomedical relation and event extraction.Source: §I · p. 1
- Retrieved examples and external biomedical knowledge provide limited support for dataset-specific boundaries, type scopes, and annotation conventions, while free-form generation can produce invalid, duplicated, or hallucinated entities.Boundary errors are also among the structural problems described.Source: §I · p. 1
- GAMA induces and verifies dataset-specific annotation rules, then uses guideline-guided planning, schema-constrained coding, and dual-loop verification to produce entity predictions.Planning ranks mention-type hypotheses with rationales; verification checks span grounding, type validity, and structural compliance.Source: §I · pp. 1–2
- GAMA consistently outperforms strong LLM-based baselines across five widely used BioNER datasets and multiple LLM backbones.Ablation studies further verify the effectiveness of its key components.Source: §I · p. 2
II. RELATED WORK
Prior BioNER work uses in-context examples, prompt content, external knowledge, and multi-agent collaboration to support extraction. GAMA's methods make dataset-specific annotation rules explicit by inducing candidate rules from training annotations and passing them to verification.
A. BioNER with In-Context Learning
- Prompt frameworks, structured prompts, UMLS knowledge, retrieved demonstrations, and chain-of-thought reasoning have been explored to improve in-context BioNER,, [9],,.These approaches emphasize task prompts, domain awareness, reasoning, and demonstration selection.Source: A. BioNER with In-Context Learning · p. 2
B. Multi-Agent Systems in Information Extraction
- Prior information-extraction systems assign specialized roles to agents for demonstration filtering, cross-task extraction, or generation evaluation,,.The passage characterizes role specialization, collaboration, and verification-oriented refinement as useful for structured extraction.Source: B. Multi-Agent Systems in Information… · p. 2
III. METHODS
- GAMA divides BioNER into rule induction and verification, ranked mention-type planning, schema-constrained entity coding, and prediction-level verification with iterative correction.The workflow is presented as a schema-constrained programming process with explicit guideline modeling.Source: §III · p. 2
A. Annotation Rule Induction
- Because datasets vary in boundary conventions, type granularity, and annotation scope, GAMA converts training annotations into candidate rules that the Rule Verification Agent validates and filters.Input–label pairs alone provide implicit rather than explicit annotation semantics.Source: A. Annotation Rule Induction · pp. 2–3
B. Rule Verification and Guideline Memory Construction
The Rule Verification Agent tests candidate annotation rules against training data and uses supported rules to build a dataset-specific guideline memory. Rules are ranked by support within entity types, with the top N retained for downstream modules.
B. Rule Verification and Guideline Memory Construction
- A candidate rule is supported when applying it helps recover at least one gold entity with the correct boundary and type.Rules are tested on training instances, and rules with no support are discarded.Source: B. Rule Verification and Guideline… · p. 3
- For each entity type, supported rules are ranked by their training-instance support, and the top N form the guideline memory.The rules are grouped by entity type before ranking.Source: B. Rule Verification and Guideline… · p. 3
- The resulting guideline memory provides structured annotation constraints for Planning, Coding, and Verification while filtering noisy or overly instance-specific rules without adding trainable parameters.Rule Verification filters candidate rules for consistency with dataset-specific standards.Source: B. Rule Verification and Guideline… · p. 3
C. Guideline-Guided Mention Planning
The Planning Agent combines the input sentence, verified guideline memory, and dataset schema to produce ranked mention-type hypotheses. Each hypothesis carries a confidence score and a rationale linked to relevant annotation rules.
C. Guideline-Guided Mention Planning
- The Planning Agent uses the sentence, verified guideline memory, and dataset schema to generate ranked mention-type hypotheses.The schema specifies valid entity types and dataset-specific output constraints.Source: C. Guideline-Guided Mention Planning · pp. 3–4
- Each hypothesis contains a candidate mention, predicted type, confidence score, and natural-language rationale grounded in the guideline memory.The confidence score ranges from 0 to 1.Source: C. Guideline-Guided Mention Planning · p. 4
- Confidence scores prioritize candidates during Coding and Verification, especially for overlapping spans or type conflicts, while rationales connect mentions to annotation rules.The rationales make type assignments explicit and provide interpretable evidence for downstream agents.Source: C. Guideline-Guided Mention Planning · p. 4
D. Schema-as-Code Entity Construction
The Coding Agent addresses the structural problems of free-form BioNER outputs by converting ranked mention-type hypotheses into schema-constrained entity objects. It preserves planning context for later verification and conflict handling.
D. Schema-as-Code Entity Construction
- Free-form text, tagging sequences, and loosely structured JSON can yield format inconsistencies, missing fields, or invalid entity types.These representations are difficult to integrate into a multi-stage verification workflow.Source: D. Schema-as-Code Entity Construction · p. 4
- The Coding Agent converts each mention-type hypothesis into an entity object with the mention in span and its predicted type in entity_type.The entity_type field is restricted to the dataset-specific valid type set.Source: D. Schema-as-Code Entity Construction · p. 4, Figure 2 · p. 4
- The Coding Agent preserves each hypothesis’s ranking score and rationale as auxiliary information for later verification and conflict handling.This standardizes the hypothesis while retaining its planning context.Source: D. Schema-as-Code Entity Construction · p. 4
E. Verification and Dual-Loop Refinement
GAMA verifies each generated entity object against the input text and dataset schema, then refines failures or lower-ranked hypotheses before producing the final prediction set.
E. Verification and Dual-Loop Refinement
- Each object is checked for span grounding, type validity, and structural compliance before inclusion in the final predictions.Grounding checks whether the mention appears in the input; type validity checks membership in the dataset’s type set; structural compliance checks the dataset schema.Source: E. Verification and Dual-Loop Refinement · p. 4
- Verification outcomes and diagnostic feedback indicate whether each object passes the checks and support refinement of its prediction.The final set consists of mention–type pairs whose objects pass verification.Source: E. Verification and Dual-Loop Refinement · p. 4
- The outer loop prioritizes higher-ranked Planning Agent hypotheses, while the inner loop uses diagnostic feedback to revise and re-check objects until they pass or reach the attempt limit.Only verified objects enter the final prediction set.Source: E. Verification and Dual-Loop Refinement · p. 4
- The refinement mechanism corrects object-level errors while preserving the Planning Agent’s hypothesis ranking.Its stated goal is text-grounded, schema-compliant entity predictions.Source: E. Verification and Dual-Loop Refinement · p. 4
A. Datasets
The evaluation uses five public BioNER datasets spanning several biomedical entity categories and compares GAMA with four representative LLM-based NER baselines.
A. Datasets
- The five datasets cover diseases, genes and proteins, chemicals, multiple biomedical entity types, and anatomical entities.They are NCBI Disease, BC2GM, BC4CHEMD, GENIA, and AnatEM, respectively.Source: A. Datasets · pp. 4–5
- Table II is captioned as a performance comparison across five BioNER datasets and different LLMs, with best results highlighted for each LLM–dataset combination.The supplied caption does not state the table’s cell values.Source: Table II · p. 5
B. Baselines
- The four comparison baselines are GPT-NER, CodeIE, CMAS, and EICL.They represent generation with self-verification, code-style structured prediction, cooperative multi-agent zero-shot NER, and entity-aware demonstration retrieval, respectively.Source: B. Baselines · p. 5
C. Base LLMs
GAMA is evaluated with several instruction-tuned LLM backbones without task-specific fine-tuning, using exact-match Micro-F1; it remains best across datasets as backbones scale from Llama3-8B to Llama3-70B.
C. Base LLMs
- The evaluated backbones span Llama3-8B and 70B, Qwen2.5-14B and 72B, GPT-3.5-turbo, and GPT-4o.The study includes instruction-tuned models from multiple model families, parameter scales, and closed- and open-source settings.Source: C. Base LLMs · p. 5
D. Implementation Details
- GAMA uses training-split labels only to induce and verify guideline memory, while baselines draw in-context examples from the same split; the reported results average three runs.Components communicate through structured intermediate outputs without updating model parameters; defaults include 10 retained rules per type, 3 ranked hypotheses, and up to 3 inner-loop iterations.Source: D. Implementation Details · p. 5
E. Evaluation Metrics
- All methods are evaluated with exact-match Micro-F1, requiring both the entity span boundary and type to match the gold annotation.Micro-F1 is reported as the primary metric.Source: E. Evaluation Metrics · p. 5
A. Main Results
- GAMA exceeds EICL by 2.94 points on BC2GM and 2.14 points on GENIA with Llama3-8B, and by 1.75 points on GENIA with Llama3-70B.GAMA remains best across datasets when scaling from Llama3-8B to Llama3-70B; the reported gains are especially relevant to boundary- and type-sensitive decisions.Source: A. Main Results · p. 6
B. Generalization across LLMs
GAMA ranks first across the reported datasets and LLM backbones. Ablations and parameter analyses examine its components and the effects of guideline-set size, hypothesis count, and refinement iterations.
B. Generalization across LLMs
- GAMA ranks first on every dataset and backbone; on NCBI, its score rises from 75.12 under Qwen2.5-14B to 78.91 under Qwen2.5-72B and 79.71 under GPT-4o.The gains are also reported against the strongest baseline, EICL, and are more pronounced on BC4CHEMD and AnatEM, which involve complex terminology and long-span entities.Source: B. Generalization across LLMs · p. 6
C. Ablation Study
- Removing any of five design choices reduces performance; removing guideline modeling costs 3.53–4.89 F1 points, while removing rule verification costs 2.79–3.96.Removing planning rationales yields the largest average degradation, and removing schema-as-code or the verification loop also reduces F1.Source: Table IV · p. 7, C. Ablation Study · p. 7
D. Effect of Top-N Rules
- Larger guideline sets can weaken focus on core constraints and add semantic noise, while Llama3-8B saturates earlier than Llama3-70B.The authors attribute these effects to longer contexts and low-frequency or weakly relevant rules; larger models handle larger rule collections more stably.Source: Table V · p. 7, D. Effect of Top-N Rules · p. 7
- Across three datasets, increasing hypothesis count and refinement iterations usually helps, but excessive values can add noise or redundant corrections; (k = 3, t = 3) is most effective.Gains are especially useful for complex or ambiguous entity structures, and the reported effects gradually saturate as both settings grow.Source: E. Impact of k and t · pp. 7–8, Table VI · p. 8
VI. CONCLUSIONS
GAMA combines dataset-specific annotation guidance with structured entity generation and verification for BioNER. Across five public datasets and multiple LLM backbones, it outperforms strong LLM-based baselines; the authors identify efficient guideline selection and broader biomedical information extraction as future work.
VI. CONCLUSIONS
- GAMA induces and verifies annotation guidelines, separates extraction into planning and coding, and validates entity objects through dual-loop refinement.The framework targets both missing dataset-specific annotation semantics and weak structural control in free-form generation.Source: §VI · p. 8
- Across five public BioNER datasets and multiple LLM backbones, GAMA consistently outperforms strong LLM-based baselines.Ablation and parameter analyses further support the effectiveness of its key components.Source: §VI · p. 8
- Future work will explore more efficient guideline selection and extend GAMA to broader biomedical information extraction tasks.Source: §VI · p. 8