PaperlayerView exact revision on arXiv

Source-linked AI summary

Calibrated Typed Decisions for Entity, Relation, and Event Extraction

Yang Wang, Hanlin Xiao, Yanan Ma, Chenghao Xiao, Chenghua Lin

arXiv:2610.17113v1cs.CLcs.AI

TL;DR

Schema-conditioned extraction still faces challenges transferring accurately across domains and annotation conventions. CALON combines span-based entity, relation, and event extraction with reward-augmented training, outperforming evaluated baselines on most of 40 benchmarks.

  • Problem

    Supporting user-specified schemas does not by itself ensure accurate extraction across new domains and annotation conventions.

    Source: §1 · p. 1

  • Method

    CALON is a unified encoder that predicts entities, relations, and events over spans and span pairs, combining likelihood supervision with relative rewards on sampled decisions.

    Source: §1 · p. 2

  • Results

    Across 40 benchmarks, CALON outperforms evaluated baselines on most, leads every reported relation extraction benchmark, and achieves the highest average F1 for event triggers and arguments with gold triggers.

    Source: §1 · p. 2, §6 · p. 8

  • Takeaways & Limitations

    The results support one extraction model transferring across datasets and handling span-pair tasks beyond conventional entity, relation, and event extraction.

    Source: §1 · p. 2, §6 · p. 8

    Training and evaluation are limited to English, transfer to other languages is untested, and candidate spans are capped at 32 tokens.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Information extraction models often perform well on the datasets they are trained on, but transferring to new domains and annotation guidelines remains challenging. Supporting user-specified types alone does not resolve this problem. We present CalOn, a unified 0.6B-parameter encoder for entity, relation, and event extraction across datasets and schemas. The model jointly encodes text and type definitions, then directly identifies entity spans, relations between spans, and event triggers and arguments. Its extraction decisions consider the requested types together, using their combined evidence to identify relevant candidates before assigning a type. This gives all three tasks a common probabilistic formulation while keeping every extracted mention anchored to the input text. We train on diverse sources, and supplement supervised learning with a reward that encourages correct extractions and penalises incorrect ones. Across 40 benchmarks, CalOn outperforms the evaluated baselines on most benchmarks. It leads all reported relation extraction benchmarks and achieves the highest average F1 for event triggers and arguments with gold triggers. The same model also performs strongly on aspect sentiment triplet extraction and pronoun resolution through its span-pair formulation. A correction for the probability bias introduced during training improves confidence estimates for entities, relations, and event arguments after calibration, with little change in F1.

1 Introduction

Schema flexibility alone does not ensure accurate extraction across domains and annotation conventions. CALON combines entity, relation, and event extraction with reward-based training, and outperforms evaluated baselines on most of 40 benchmarks.

1 Introduction

  • The remaining challenge is accurate extraction across domains and annotation conventions, despite models accepting user-specified schemas.Generative approaches can produce unsupported entities or fail to preserve mention boundaries, while reported performance of schema-flexible discriminative models varies across domains and tasks.Source: §1 · p. 1
  • For each candidate span or span pair, CALON samples type-score perturbations and compares each prediction’s reward with the average reward of the other samples.Correctly typed extractions receive positive rewards, false positives and incorrect types negative rewards, and no-extraction predictions zero reward.Source: §1 · p. 2
  • Across 40 benchmarks, CALON outperforms evaluated baselines on most, leads every reported relation extraction benchmark, and has the highest average F1 for event triggers and arguments with gold triggers.Its results also extend to span-pair tasks including aspect sentiment triplet extraction, pronoun resolution, coreference, and temporal and causal event relations.Source: §1 · p. 2
  • CALON is a unified encoder for user-specified schemas that predicts entities, relations, and events through spans and span pairs.The same formulation supports tasks beyond conventional information extraction.Source: §1 · p. 2
  • Its training objective combines likelihood supervision with relative rewards on sampled extraction decisions, without requiring an additional critic.Gaussian perturbations generate candidate predictions; reward comparisons favor correct extractions over false positives and incorrect type assignments.Source: §1 · p. 2

2 Related Work

Prior work supports schema-conditioned extraction through generation, instruction tuning, guidelines, and span or span-pair matching, while training data increasingly use LLM annotations and synthetic examples. Related work also discusses reward-based learning and probability calibration.

2 Related Work

  • Extraction systems support flexible schemas through structured generation, instruction tuning, annotation guidelines, or matching spans and span pairs to requested types.Training resources have expanded through LLM annotations and synthetic examples, while combining sources requires handling differences in annotation coverage.Source: §2 · p. 2
  • CALON adds rewards for sampled span and span-pair predictions, rewarding correct types and penalizing false positives and incorrect assignments.Prior work also includes null classes for rejecting non-entity spans and reinforcement learning through label-aware self-reflection.Source: §2 · p. 2
  • Calibration work covers post-training temperature scaling, training-time methods, threshold selection, and corrections for class-dependent sampling bias.CALON applies the correction principle to the lower loss weight assigned to null candidates in span and span-pair extraction.Source: §2 · p. 2
  • Figure 1 depicts schema-text encoding with type-candidate interaction and null-class scoring, alongside sampled-decision rewards added to supervised loss.The figure presents these as the model’s scoring and training components.Source: Figure 1 · p. 3

3 Methodology

CalOn jointly encodes text and queried schema types, then makes span and span-pair decisions for entities, relations, and events. Its training combines gold-label likelihood with sampled decision rewards, and decoding thresholds the probability of any queried type before assigning a type.

3.1 Encoder

  • The encoder represents spans and schema types as vectors, then lets queried types and overlapping candidates attend to one another.Span vectors use boundary-token states, the token mean, and a width embedding; type vectors use projected representations of type names and descriptions.Source: §3.1 · pp. 3–4

3.2 Typed Decisions with a Null Class

  • Queried types can be positive, negative, closed-world negative, or unknown; unknown types are excluded from training decisions, while event arguments are modeled as role-labeled relations from triggers to spans.At inference, a first pass predicts triggers and a second proposes argument spans before assigning their roles.Source: §3.2 · p. 4
  • Entities and ordered span pairs each choose among the null class and queried types, with reversed span pairs treated as distinct relation candidates.The formulation allows one candidate to have multiple gold types, such as coarse and fine entity types.Source: §3.2 · p. 4, §3.3 · p. 4

3.3 Training Objective

  • Training combines a weighted negative log-likelihood objective with rewards for sampled decisions that match gold labels and penalties for incorrect types.Null candidates receive reduced weight; sampled predictions are compared with the average reward of the other samples.Source: §3.3 · pp. 4–5

3.4 Decoding and Calibration

  • At inference, CalOn thresholds the total probability assigned to queried types, then labels selected candidates with their most probable type; a null-logit correction addresses bias from down-weighting null candidates.Entity temperature and threshold are fitted on training-source calibration splits; relation temperature is fixed at 1 and its threshold is fitted on REBEL with gold entities.Source: §3.4 · p. 5

4 Experimental Setting

The experiments train CalOn on diverse sources and evaluate it on 40 benchmarks spanning entity, relation, and event extraction. The setup uses validation data from training sources for model selection and calibration, with no target-dataset examples used for those purposes.

Training data

  • Training uses 3.1M instances from 84 sources, combining human-labeled data, LLM annotations, and synthetic examples across domains, genres, and schemas.The pipeline checks synthetic annotations, removes near duplicates, and filters texts that substantially overlap evaluation sentences.Source: §4 · p. 5

Evaluation

  • The evaluation covers 40 NER, RE, and EE benchmarks and reports exact-match micro F1, with equal benchmark weighting for average F1.Relation results include end-to-end and gold-entity F1; event results include trigger and argument F1 with gold and predicted triggers.Source: §4 · p. 5

Baselines

  • Baselines include GLiNER2-large, GLiFormer-large, InstructUIE, and GoLLIE-7B, with additional task-specific comparisons for NER and gold-entity relation extraction.Baselines use default decoding and the same scoring code; several receive type names only under their released interfaces.Source: §4 · p. 5

Implementation

  • Qwen3-0.6B is the backbone, and checkpoint, hyperparameters, temperature, and extraction thresholds are selected using validation data from training sources.No labeled examples from target datasets are used for training, model selection, or calibration.Source: §4 · p. 5

5 Results and Discussion

The section reports entity, relation, and event extraction results, then examines schema descriptions, model components, calibration, and inference efficiency.

5.1 Main Results

  • CALON has the highest average F1 on CrossNER, specialised entity extraction, and personal information extraction, while GLiFormer and InstructUIE lead on named exceptions.GLiFormer leads CrossNER Music and Science; InstructUIE leads MIT Restaurant, whose training data were included in its evaluated checkpoint.Source: §5.1 · p. 6

5.2 Schemas and Type Descriptions

  • Type descriptions add about 16 F1 points on CrossRE, while NER group averages change by only one to three points.Every evaluated entity type occurs by name in training, so the NER benchmarks test transfer across domains and annotation guidelines; CALON’s largest margin over the strongest baseline is on the least frequently annotated third of types.Source: §5.2 · pp. 6–7

5.1 Main Results

  • CALON leads every reported relation benchmark and has the highest average event trigger and gold-trigger argument F1.For relation extraction, average typing accuracy is 79% versus 40% when only annotated pairs are supplied; with all gold-entity pairs, average F1 is 49.8 versus 2.6. InstructUIE’s CASIE result raises its end-to-end event argument average above CALON’s.Source: §5.1 · p. 6, Table 2 · p. 7, Table 3 · p. 7

5.3 Ablation Study

  • Independent sigmoid scores reduce average NER F1 from 61.5 to 53.4, and thresholding an individual type also lowers NER F1.The results support a joint distribution over types and a null class, separating extraction decisions from type selection; description-conditioned decision reward mainly benefits RE, while treating unknown types as negatives lowers F1 across all three tasks.Source: §5.3 · pp. 7–8

5.4 Calibration

  • After null-class correction and refitting, expected calibration error falls from 0.25 to 0.10 for relations, 0.33 to 0.23 for entities, and 0.51 to 0.39 for event arguments, with little F1 change.Event predictions reuse entity and relation calibration parameters, and event arguments retain the highest average calibration error.Source: §5.4 · p. 8

5.5 Efficiency

  • CALON processes 120 sentences per second, 5 to 25 times baseline throughput, but uses more peak GPU memory than GLiNER2 and GLiFormer.Throughput is measured on MIT Movie with all 12 types, excluding model loading.Source: Table 4 · p. 8, §5.5 · p. 8

6 Conclusion

CALON is a unified encoder for entity, relation, and event extraction under user-specified schemas. Its benchmark results and calibration findings support a single model across these tasks.

6 Conclusion

  • The 0.6B-parameter CALON encoder handles entities, relations, and events through span and span-pair predictions under user-specified schemas.Its training combines likelihood supervision with relative rewards on sampled extractions, favouring correct predictions over false positives and incorrect type assignments.Source: §6 · p. 8
  • Across 40 benchmarks, CALON outperforms evaluated baselines on most benchmarks and leads every reported relation benchmark.It also has the highest average F1 for event triggers and arguments with gold triggers, and its span-pair formulation extends to tasks including aspect sentiment triplets and pronoun resolution.Source: §6 · p. 8
  • Probability correction and calibration improve confidence estimates across all three extraction tasks with little change in F1.Source: §6 · p. 8

Limitations

The paper identifies language coverage, candidate scope, training-data annotation, calibration transfer, and evaluation seeds as limitations.

Limitations

  • Training and evaluation are limited to English; transfer to other languages has not been tested.Source: Limitations · p. 9
  • Candidate spans are capped at 32 tokens, and relations are predicted only between spans in the same input, leaving longer mentions and cross-input relations outside scope.Source: Limitations · p. 9
  • Much training data is LLM-annotated; voting and judge checks reduce but cannot fully eliminate shared biases.Source: Limitations · p. 9
  • Calibration parameters fit to training-source validation data may remain imperfect after domain change, and results use a single seed.Repeating training and the full 40-benchmark evaluation would require substantial additional computation.Source: Limitations · p. 9

Ethical Considerations

The section sets privacy and review boundaries for extraction and describes the training-data preparation and evaluation procedures. It also notes that the optional Brier term is disabled in the reported model.

Ethical Considerations

  • Personal-information extraction can make sensitive details easier to aggregate, so applications should respect privacy and data-use conditions and review predictions before consequential decisions.The passage also cautions that LLM annotations may retain shared biases despite annotation checks.Source: Ethical Considerations · p. 9

A Method Details

  • The optional multi-class Brier term assigns equal target probability to gold classes, but β = 0 excludes it from the reported model’s training loss.The target vector assigns zero probability to the remaining classes.Source: A Method Details · p. 13

B.1 Data Preparation and Sampling

  • Training draws on human-labelled entity, slot, and event sources, alongside LLM-annotated web and Pile text and generated or reannotated data.The additional annotations include REBEL texts, CC-News articles, and synthetic combinations of genres, domains, and schemas.Source: §B.1 · p. 13
  • Generated annotations are checked through independent model votes and judge resolution; unresolved types are removed, near-duplicates and evaluation overlap are filtered, and sampling draws from 3.1M instances.The 5.12M-draw budget is distributed across sources, yielding approximately 35% entity, 41% relation, and 23% event examples.Source: §B.1 · p. 13

B.2 Evaluation Datasets and Scoring

  • Evaluation spans entity, relation, and event datasets, including converted span-pair tasks; benchmark scores use exact-match micro F1 with equal benchmark weight in averages.The conversions include ASTE, pronoun resolution, coreference, and temporal, causal, and subevent links; some relation and event evaluations use gold entities or triggers.Source: §B.2 · pp. 13–14
  • The evaluation also measures false positives on absent types and misses of gold mentions without an exact match.An absent-type false positive occurs when a query with no gold occurrence receives at least one prediction.Source: §B.2 · p. 14

B.3 Evaluation Adaptations

The evaluation adaptations standardize model inputs and candidate alignment while documenting differences in relation and argument scoring. The section also records prior training exposure, architecture limits, and calibration settings.

B.3 Evaluation Adaptations

  • Baseline inputs are adapted by model: GNER receives word-level tokens, GoLLIE uses type classes with optional descriptions, and GLiNER2 uses its structured event interface.For relation and argument comparisons, baseline predictions are aligned to gold entities or triggers using the stated span and text matching rules.Source: §B.3 · p. 14
  • GLiREL scores ordered pairs of gold entities and returns the highest-scoring relation type above threshold; the REBEL calibration split selected a no-relation label and threshold 0.05.On that split, retaining the no-relation label gave 18.1 F1 versus 17.2 without it.Source: §B.3 · pp. 14–15
  • Across 18 RE benchmarks, GLiREL’s mean typing accuracy is 40.0%, versus CALON’s 79.0%; with all ordered gold-entity pairs, average F1 is 2.6 versus 49.8.Typing accuracy tests positive pairs, while average F1 also requires rejecting unrelated pairs; GLiREL’s low F1 is associated with low precision.Source: §B.3 · p. 15
  • The evaluated InstructUIE checkpoint includes MIT Movie, MIT Restaurant, and CASIE exposure, while GoLLIE includes TACRED and ACE05; the comparison does not isolate exposure effects.The passage distinguishes shared source sentences from shared event types as different forms of prior exposure.Source: §B.3 · p. 15

B.4 Architecture and Training Settings

  • CALON uses spans of at most 32 tokens, texts of at most 1,024 tokens, combined text and schema of at most 3,072 tokens, and at most 24 entity spans for relation candidates.Its architecture includes two interaction blocks, four attention heads each, and bilinear scoring dimension 64.Source: §B.4 · p. 15
  • Training runs for 40,000 updates, while calibration fits entity temperature on training-source calibration data and selects extraction thresholds to maximize F1.The reported uncorrected settings are entity T = 1.219 and τ = 0.88, relation T = 1 and τ = 0.85; event triggers reuse entity parameters and arguments reuse relation parameters.Source: §B.4 · p. 15

C Additional Results

CALON leads the reported relation benchmarks and has the highest average event trigger and gold-trigger argument F1. Entity results are strongest on average across the reported NER groups, with dataset-level exceptions.

Entities

  • CALON has the highest average F1 on CrossNER, the other-domain group, and personal-information extraction, although GLiFormer leads on CrossNER Music and Science and InstructUIE on MIT Restaurant.InstructUIE scores 80.9 F1 on MIT Restaurant versus CALON’s 66.5, and both MIT datasets appear in its released training configuration.Source: C Additional Results · p. 15

Relations

  • 36.0 versus 10.6 F1: CALON’s end-to-end CrossRE average exceeds the strongest baseline, and it leads every reported relation benchmark in gold-entity and available end-to-end settings.Its lead on LitBank and MAVEN-ERE does not mean those tasks are solved; CALON’s scores on both remain low.Source: C Additional Results · pp. 15–16

Events

  • CALON has the highest average event trigger F1 and gold-trigger argument F1, but InstructUIE’s CASIE score raises its overall end-to-end argument average above CALON’s.Excluding CASIE, CALON has the highest average on all three event-extraction metrics; its gold-trigger argument F1 is 34.7 versus 28.4 for InstructUIE.Source: C Additional Results · p. 16
  • Dataset-level exceptions include stronger competing event scores on selected datasets and zero scores by some baselines on others, so task averages should be read alongside dataset results and training coverage.InstructUIE and GoLLIE each lead on particular M2E2 measures, while several baseline-dataset combinations score zero.Source: C Additional Results · p. 16

D Results without Type Descriptions

Without type descriptions, CALON retains several leading averages, but performance changes differ by task. The reported comparison evaluates models with names alone, and CALON’s training independently omits descriptions in some instances.

D Results without Type Descriptions

  • 53.7 and 57.2 F1: without descriptions, CALON leads the other-domain and personal-information NER averages, while GLiFormer leads CrossNER at 67.6 versus 66.2 F1.Models that support descriptions receive the same schema information as name-only models in this evaluation.Source: D Results without Type Descriptions · p. 16
  • 45.4 to 29.1: CALON’s gold-entity CrossRE average falls without descriptions, though it still leads every reported relation benchmark in both settings.The passage attributes useful information beyond relation type names to the relation definitions.Source: D Results without Type Descriptions · p. 16
  • CALON retains the highest average event trigger F1 without descriptions, but its gold-trigger argument F1 falls from 35.1 to 32.9, below InstructUIE’s 33.2.InstructUIE also retains the highest end-to-end argument average; the effects of descriptions vary across tasks and datasets.Source: D Results without Type Descriptions · p. 16

E Calibration per Benchmark

Calibration corrections reduce ECE for returned predictions with little change in F1, while analyses also examine type familiarity, inference cost, data scaling, and training-data coverage. The appendix documents model checkpoints, evaluation benchmarks, and the training sources and synthetic-data analysis.

E Calibration per Benchmark

  • Relation ECE decreases on 17 of 18 benchmarks and argument ECE on all seven, but temperature refitting does not improve every individual result.For event arguments, MLEE improves from 0.29 to 0.11, while ECE remains at least 0.30 on the other six datasets; the relation-candidate Brier score changes little.Source: E Calibration per Benchmark · p. 17

F Type Familiarity

  • CALON has the highest macro F1 in all three type-frequency groups, with ρ = 0.04 and 58.8 F1 for rare types versus GLiNER2’s 50.6.Frequencies come from CALON’s training data and do not measure baseline exposure; the analysis does not isolate training-data and architecture effects.Source: F Type Familiarity · p. 17

G Inference Cost

  • CALON processes 120 sentences per second, about five to eight times the throughput of GLiNER2, GLiFormer, and InstructUIE in the reported PyTorch comparison.Its weight memory is 1.2 GB versus 13.5 GB for the 7B models; GoLLIE is faster than CALON under a separate vLLM setup, so the PyTorch result does not establish a cross-framework speed advantage.Source: G Inference Cost · pp. 17–18

H Data Scaling

  • With descriptions, entity and relation averages improve by approximately five and eight points from 100K to 500K instances, while event-argument F1 improves about two points with the full data.Gains after 500K are smaller and non-monotonic; because training updates scale with dataset size and each run uses one seed, the experiment varies data and compute together and does not establish reliability of smaller differences.Source: H Data Scaling · pp. 18–19, Figure 3 · p. 19

K Training Data

  • Synthetic generation combines genres and domains with text and extraction schemas; checked absent types provide negative supervision, and pooled sources contain approximately 105K entity, 54K relation, and 85K event type names.Sentence-embedding projections place synthetic texts across broader regions than real-text sources and show overlap with some evaluation regions, but do not establish equivalent distributions or explain performance gains.Source: K Training Data · p. 19

E Calibration per Benchmark

  • Correction and temperature refitting reduce average ECE from 0.33 to 0.23 for entities, 0.25 to 0.09 for relations, and 0.51 to 0.39 for event arguments, with little change in F1.The reported ECE is computed over returned predictions, not all null and non-null candidates.Source: E Calibration per Benchmark · pp. 16–17, Table 9 · p. 21

I Models

  • Table 10 identifies the checkpoints used for the CALON backbone, baselines, data generation, annotation, and text-embedding analysis, along with model links and reported licenses.Source: Table 10 · p. 22

J Evaluation Datasets

  • Table 11 lists 40 evaluation benchmarks by task, domain, genre, type counts, and converted evaluation-input counts; inputs may be sentences, passages, or document chunks.The task conversions and scoring rules are described in Appendix B.2.Source: J Evaluation Datasets · p. 19, Table 11 · p. 22

Related papers

Loading 2610.17113v1…