PaperlayerView exact revision on arXiv

Source-linked AI summary

Xing USP: A Cross-Family Semantic Contract for Information Extraction and Intent-Slot Parsing

Zhongqiu Li, Boyang Wang, Ruiyu Fang, Shiquan Wang, Di Wu, Jiayi Liang, Feilong Zhang, Hongfeng Du, Xiaomeng Huang, Mengjiao Bao, Luwen Pu, Xuewei He, Yu Zhao, Chao Wang, Kaidong Yu, Bingkai Yang, Qingjun Zhu, Shaoyi Du, Shuangyong Song, Yongxiang Li

arXiv:2610.34343v1cs.CLcs.AI

TL;DR

Structured semantic parsing can recover a category while missing a value or its instance association. USP applies a shared contract and structural diagnostics across extraction and intent-slot tasks; in a six-model comparison, model profiles vary with the structural unit being evaluated.

  • Problem

    Category-based judgments leave value recovery and instance-level associations unexamined across structured semantic parsing tasks.

    Source: §1 · p. 1

  • Method

    USP provides a shared category–field–value representation that retains instance associations and evaluates outputs through five structural matching views.

    Source: §1 · p. 2, §3.7 · p. 5

  • Results

    The six-model comparison shows that model differences depend on the structural unit matched, making profiles change across diagnostic views.

    Source: §5 · p. 9, §6 · p. 9

  • Takeaways & Limitations

    The shared representation and explicit matching rules support analysis of structural agreement beyond a single aggregate score.

    Source: §6 · p. 9

    Inference budgets are not equalized, so the comparison characterizes performance under the evaluated execution conditions rather than a cost-matched protocol.

    Source: Limitations · p. 10

Abstract

from arXiv · show

Structured semantic parsing is typically judged by whether the correct category is assigned, which leaves value recovery and instance-level associations unexamined: a system may recognize an event yet bind an argument to the wrong instance, or identify an intent yet supply an incorrect slot value. We present USP, a cross-family semantic contract that represents categories, fields, values, and instance associations through a common interface for named-entity recognition, relation extraction, event extraction, and intent-slot parsing. The contract is paired with five diagnostic matching views---category agreement, field recognition, field-value pairing, instance-bound assignment, and complete-instance recovery---that ask distinct questions about the same output. Using this contract, we normalize 61 datasets into a unified corpus with shared schemas, prompts, and matching rules, and compare six models on a fixed test subset. Our fine-tuned Xing4.0-29B-Intent attains 57.70 task-balanced Field-Value Micro-F1 (equally weighted across the four task families) and ranks first in every family and on all five views; its margin over the strongest alternative widens from 2.93 points for category matching to 20.73 points for complete-instance matching. This widening gap shows that category agreement alone understates structural deficits. USP thus offers the community a shared representation, a normalized 61-dataset corpus, and a diagnostic evaluation protocol that make structural agreement measurable and comparable across extraction and intent-slot tasks.

1 Introduction

Structured parsing can identify the right category yet miss values or bind arguments to the wrong instance. USP provides a shared contract and diagnostic views for examining those structural differences across extraction and intent-slot tasks.

1 Introduction

  • Across six models, Xing4.0-29B-Intent ranks highest on every F1 view, with a wider margin for complete instances than for categories.The comparison uses a fixed test subset drawn from 61 public datasets.Source: §1 · p. 2
  • USP represents IE and intent-slot outputs with shared category, field, and value structure while retaining instance associations.The five matching views range from category agreement to complete-instance recovery.Source: §1 · p. 2
  • The normalized inventory spans 61 public datasets, and five matching views plus a separate parse-success measure examine structure across task families.The views cover categories, fields, values, binding, and complete instances.Source: §1 · p. 2
  • The six-model analysis distinguishes category agreement, value matching, complete-instance recovery, and parse success as different aspects of prediction quality.A correct category alone may leave an event argument bound to the wrong instance or an intent with an incorrect slot value.Source: §1 · pp. 1–2

2 Related Work

Prior work establishes shared representations for extraction and structured spoken-language tasks, while USP applies a common contract and diagnostic views across four task families. Related research also motivates schema descriptions and structural evaluation, but USP does not test description sensitivity.

2.1 Universal Information Extraction

  • Universal IE work shares schemas, structures, and extraction operations; USP studies instance associations and the same diagnostic views across four evaluated families.IEPile already groups event triggers and arguments by instance, so USP does not introduce instance association to IE.Source: §2.1 · p. 2

2.2 Schema Semantics and Robustness

  • Schema-guideline and robustness studies motivate distinguishing description coverage from evidence of benefit, while USP leaves controlled description sensitivity and schema perturbation untested.Schema retrieval is also outside USP’s contribution and evaluation scope.Source: §2.2 · pp. 2–3

2.3 Spoken Language Understanding and Structured Evaluation

  • SLU research includes compositional intent-slot structures and structural metrics; USP applies shared category, field, value, binding, and instance diagnostics across its four families.The paper identifies its contribution as the concrete shared contract and diagnostic application, not the general distinction between valid form and correct content.Source: §2.3 · p. 3

3 USP

USP uses one semantic contract for corpus normalization, schema-conditioned parsing, and evaluation across extraction and intent-slot tasks. Its sections define the representation and learning objective, describe corpus and schema choices, and specify structural diagnostics and corpus statistics.

3 USP

  • USP uses one semantic contract to govern corpus normalization, schema conditioning, structured output, and evaluation across task families.The task formulation maps an input and task-specific schema to an unordered collection of instantiated frames.Source: §3 · p. 3, §3.1 · p. 3

3.2 The USP Semantic Contract

  • The contract represents category labels with named argument groups and retains source-provided associations among values and relation, event, or call instances.The example figure presents schema-conditioned input and structured annotation under the contract.Source: §3.2 · pp. 3–4, Figure 1 · p. 4

3.3 Operational Requirements of the Contract

  • Parsing jointly selects categories, grounds schema-defined values, binds them to roles and instances, and serializes frames rather than exposing separate reasoning stages.Grounding can involve extraction, normalization, or contextual resolution.Source: §3.3 · p. 4

3.4 Corpus Normalization and Quality Control

  • The corpus contains 61 public datasets across NER, relation extraction, event extraction, and intent-slot sources, with preprocessing for deduplication, instruction assembly, and annotation repair.Evaluation uses a fixed test subset covering all four families; the inventory preserves partial coverage and missing values for audit.Source: §3.4 · p. 4

3.5 Schema Descriptions

  • Schema descriptions provide natural-language context for labels and optional argument criteria, but specificity varies by family and is near-constant for NER.Coverage does not establish an accuracy benefit, and controlled sensitivity remains future work.Source: §3.5 · p. 5

3.6 Unified Learning Objective

  • All task families share an autoregressive token-prediction objective conditioned on the serialized instruction and task-specific schema, without a separate loss for each operational requirement.The instruction combines task instructions, schema, and input.Source: §3.6 · p. 5

3.7 Structural Diagnostic Projections

  • The diagnostic projections separately assess categories, category–argument pairs, field values, instance binding, and complete instances; their scores are not necessarily nested or processing-stage pass rates.Parse success and strict Equality Accuracy are reported separately, and task-native metrics remain necessary for comparison with prior systems.Source: §3.7 · p. 5

3.8 Data Statistics

  • The final corpus retains about 2.75M instruction–output instances, combining the 61 public datasets and in-house XingChen annotations.Task-level splits are 0.82M NER, 0.91M relation extraction, 0.06M event extraction, and 0.97M intent-slot SLU instances.Source: §3.8 · p. 5

4 Experiments

The experiments audit contract conformance across the collection, compare six models on a fixed test subset, and examine performance under distinct structural matching views. Together, the results show that rankings and agreement vary with evaluation scope and matching unit.

4.1 Experimental Settings

  • The main comparison equally weights four task-family macros, treats failed requests as empty predictions, and reports views with distinct matching units.The evaluation uses a fixed test subset; parse success and strict Equality Accuracy are reported separately from the matching-view scores.Source: §4.1 · pp. 5–6

4.2 Contract Conformance and Repairability

  • Of 478,928 gold evaluation records across 57 dataset labels, 90.66% are valid, 6.11% invalid, and 3.23% ambiguous.Conformance varies sharply: 39 datasets are at least 95% valid, while ten are at or below 25%.Source: §4.2 · p. 6
  • Low conformance is concentrated in dataset-specific issues; audit preparation made MASSIVE, RAMS, and Wiki-ZSL appear entirely ambiguous, while five of seven intent-slot datasets are 85–100% valid.The audit classifies structural validity, not source-semantic preservation; it covers available evaluation splits and identifies potential repairs without applying them.Source: §4.2 · pp. 6–7

4.3.1 Cost-Efficient Screening on USP-Core-6K

  • USP-Core-6K uses outcome-blind, metadata-stratified sampling to screen additional models while retaining all four task families and dataset identities.Gemini-3.6-Flash leads the additional models in each family and overall; close rankings are descriptive, not significance results.Source: §4.3.1 · p. 7
  • Screening and main-comparison scores use different test subsets and cannot be compared directly; provider configurations couple model, decoding, and service effects.Core-6K preserves the ordering of the three corresponding full-benchmark runs in the calibration set and is used for initial screening.Source: §4.3.1 · p. 7

4.4 Structural Matching Analysis

  • Across five matching views, Xing4.0-29B-Intent leads, and category agreement understates differences in value and complete-structure recovery.Each view uses its own matching units, so the scores are distinct structural questions rather than sequential stages or a composite.Source: §4.4 · p. 7, Figure 2 · p. 8
  • Across 1,080 adjacent system–dataset–view comparisons, 190 profiles invert; four reference systems differ by 17.11–18.22 points between field and field-value views, versus 7.73 for the SFT row.The comparisons describe frozen rows without significance estimates.Source: §4.4 · pp. 7–8

4.3 Unified Schema-Driven Parsing Results

  • Xing4.0-29B-Intent scores highest in every task family and F1 matching view, while DeepSeek-V4-Pro has the highest Parse rate.Table 2 reports the main comparison on the fixed test subset.Source: §4.3 · p. 7, Table 2 · p. 8

5 Discussion

USP gives extraction and intent-slot outputs a shared language for examining categories, fields, values, and instance associations. The discussion argues that structural views add information beyond one score, while noting limits in cross-family comparison and evaluation scope.

5 Discussion

  • Figure 4 compares parser acceptance and Field-Value F1 across model–dataset pairs grouped by task, but parser acceptance includes empty outputs.Models with similar parser-acceptance rates can have substantially different extraction scores.Source: Figure 4 · p. 9
  • The shared contract pairs category labels with named fields and values while retaining instance associations, enabling the same diagnostic questions across extraction and intent-slot outputs.Its value in this study is representational and analytical; superiority to native task formulations would require a controlled comparison of both representations and evaluators.Source: §5 · pp. 8–9
  • A single score misses structural differences: model comparisons vary by matched unit, and the views are not successive processing-stage pass rates.Each view uses its own counting units; execution status is complementary because parser acceptance alone does not establish that a response was received.Source: §5 · p. 9
  • The shared abstraction does not erase annotation differences: NER links a type to a mention, while other families may bind multiple values within an instance.The conformance audit does not establish that conversion preserves source semantics; that requires a separate round-trip audit.Source: §5 · p. 9
  • Fixed-schema comparisons do not measure sensitivity to schema wording or input perturbations, and USP-specific representations and evaluators make published scores contextual rather than directly comparable.Robustness and external validity are separate dimensions illustrated by SGD-X and RUIE-Bench (Lee et al., 2022a; Zhu et al., 2025).Source: §5 · p. 9
  • MMLU-Pro results are a separate general-capability reference, not an explanation of unified-parsing results or a controlled base-versus-SFT comparison.They use one fixed scoring protocol and are reported in Appendix B.4.Source: §5 · p. 9

6 Conclusion

USP provides a shared semantic contract for information extraction and intent-slot parsing. Its diagnostic views distinguish structural agreement, and the six-model comparison shows that model profiles change with the unit being evaluated.

6 Conclusion

  • USP presents a shared semantic contract for information extraction and intent-slot parsing, with diagnostic views distinguishing category agreement, field-value recovery, and instance associations.The contract pairs a common representation with explicit matching rules.Source: §6 · p. 9
  • The six-model comparison shows that model profiles change with the structural unit being evaluated.The views distinguish category agreement, field-value recovery, and instance associations across task families.Source: §6 · p. 9
  • By making matching rules explicit, USP supports more informative analysis than a single aggregate score.The views examine outputs across task families.Source: §6 · p. 9

Limitations

The limitations concern conversion fidelity, fixed-schema evaluation, unequal inference budgets, and variation in annotation schemes across task families. Consequently, USP comparisons describe agreement with a shared contract rather than equivalence of task difficulty.

Limitations

  • The conformance audit tests whether converted records satisfy the contract, not whether conversion preserves source semantics; a round-trip audit against original annotations is needed.The proposed audit converts USP records back to task-native formats and compares them with the originals.Source: Limitations · pp. 9–10
  • Fixed schemas leave sensitivity to schema wording and description content unmeasured; controlled description-text perturbations remain future work.The comparisons do not measure these effects.Source: Limitations · p. 10
  • Inference budgets are not equalized, so the comparison characterizes performance under evaluated execution conditions rather than a cost-matched protocol.Source: Limitations · p. 10
  • Annotation conventions and description specificity vary across families, so cross-family USP comparisons indicate shared-contract agreement rather than equivalent task difficulty.NER associates a type with one mention, whereas relation, event, and intent-slot records can bind multiple values within an instance.Source: Limitations · p. 10

A.1 Data Collection and Cleaning

USP’s corpus covers four task families across multiple sources and includes a fixed test subset for model comparison. Its preprocessing, schema descriptions, instruction format, and released distribution define the resources and scope of the appendix materials.

A.1 Data Collection and Cleaning

  • USP collects open datasets across NER, relation extraction, event extraction, and intent-slot SLU, including bilingual, multilingual, and multi-turn dialogue sources.The construction inventory contains 61 datasets; the six-model comparison uses a fixed test subset, with failed inference identified separately.Source: §A.1 · p. 16
  • The fixed test subset excludes NCBI-Disease, Wiki-ZSL, and CMeIE for documented task or split fallback; other exclusions reflect partial model coverage or MASSIVE’s size.Source: §A.1 · p. 16
  • The cleaning specification lists overlap checks and heuristic filters, but the available records do not show that the complete sequence produced the reported build.Filters include text with over 80% non-alphabetic characters, short unlabeled text, and text with over 80% stopwords.Source: §A.1 · p. 16

A.1.1 Preprocessing Utilities

  • Preprocessing structurally deduplicates repeated annotations, assembles model-ready instructions while retaining metadata, and repairs verified naming inconsistencies.Ambiguous naming collisions are left unchanged for manual review; preprocessing utilities operate independently of runtime inference and scoring.Source: §A.1.1 · p. 16

A.2 Schema Description Assets

  • Across 94,636 prompts, all 1,098 category slots and 3,354 of 3,388 argument slots have descriptions; description coverage does not measure prediction effects.The 34 empty argument descriptions are in intent-slot data; 182 NER argument slots use only two distinct descriptions.Source: §A.2 · p. 16
  • The instruction target contains category labels and argument groups mapping schema-defined fields to values, with each group representing one instance.Unselected categories and unobserved fields are omitted; selected categories without extracted arguments use an empty argument list.Source: §A.2 · p. 16

A.3 Data Statistics

  • The released distribution reports 2,753,000 samples from 61 public datasets and in-house annotations, spanning 19 domains and 52 languages.Each retained instance has instruction and output fields formatted for instruction tuning.Source: §A.3 · p. 17

B.1 Experimental Settings

The appendix specifies the matching metrics, training setup, and aggregation rules for the unified-parsing comparison. Overall scores equally weight the four task-family macros.

B.1 Experimental Settings

  • The comparison reports structural matching views separately from accuracy measures and excludes task-native metrics from its main ranking.Task-native and USP scoring must be paired for direct comparison with task-specific systems.Source: §B.1 · p. 17

B.2 Training Configuration

  • The main-comparison checkpoint was fine-tuned for three epochs on a mixture of dataset instances and general-purpose instructions.The mixture samples 100K stratified instances per collected dataset, alongside a proportional share of general-purpose instructions.Source: §B.2 · p. 17

B.3 Unified Schema-Driven Parsing: Full Results

  • The appendix’s construction inventory covers 61 datasets, while its scope is distinct from the model-comparison subset.The full-results appendix reports per-dataset results for the fixed test subset.Source: §B.3 · p. 17

B.3.1 Metrics and Aggregate Summary

  • Overall scores macro-average dataset-level results within each task family, then weight the four family macros equally.Parse is likewise averaged across datasets within each family and then across families, rather than pooled across records.Source: §B.3.1 · p. 17
  • Table 2’s matching-view scores use the four-family aggregation, and its three API results project full-benchmark runs onto the fixed test subset.The full runs contain 94,636 items; failure handling follows the main experimental settings.Source: §B.3.1 · p. 17

B.3.2 USP-Core-6K Calibration and Scope

USP-Core-6K is assessed against full-benchmark results using the same four-family aggregation. It preserves the calibration models’ ordering, but is presented as a screening suite rather than a universally valid substitute.

B.3.2 USP-Core-6K Calibration and Scope

  • The 6,000-record Core-6K suite is nested within an 18,000-record confirmation suite, with selection based only on outcome-blind metadata.Selection uses a fixed seed and retains dataset identities from the broader evaluation pool.Source: §B.3.2 · p. 17
  • Core-6K comparisons use dedicated runs and the v2 evaluator; failed requests count as empty predictions in the 6,000-record denominator.Reported token usage reflects provider usage attached to retained predictions, not total billed usage across retries.Source: §B.3.2 · pp. 17–18
  • The four added measurements use the same frozen inputs and prompt, but model inference budgets are not controlled across models.Historical rows also differ in decoding and service settings.Source: §B.3.2 · p. 18
  • Core-6K preserves the three calibration models’ ordering, with absolute task-balanced error below half a point for each.The authors caution that close or anomalous cases require escalation to the confirmation suite or full benchmark, so Core-6K is reported separately from the main-test ranking.Source: §B.3.2 · p. 18, Table 5 · p. 18

B.3.3 Per-Dataset Results for the Main Comparison

The per-dataset results report Field-Value Micro-F1 across the fixed test subset and summarize observed dataset-level maxima. Failed inference contributes zero to aggregate scores, and maxima do not establish statistical significance.

B.3.3 Per-Dataset Results for the Main Comparison

  • Tables 6–9 report per-dataset Field-Value Micro-F1 across event extraction, NER, relation extraction, and intent-slot datasets.The table captions identify the corresponding task-family results in the fixed test subset.Source: §B.3.3 · p. 18, Table 6 · p. 19
  • CASIE and RAMS inference failures for Xing4.0-29B-Intent are marked with dashes and contribute zeros to aggregate scores.Other numeric zeros indicate scored predictions.Source: §B.3.3 · pp. 18–19
  • Table 10 counts datasets where each model has the highest score under the aggregate failure-handling policy; Xing4.0-29B-Intent leads in 31 cells, a descriptive count that does not establish statistical significance.Source: §B.3.3 · pp. 18–19

B.4 Supplementary MMLU-Pro Evaluation

The supplementary evaluation measures the same fine-tuned checkpoint on MMLU-Pro using a shared zero-shot protocol and deterministic answer extraction. Its scores are internal and not directly comparable with the public leaderboard.

B.4 Supplementary MMLU-Pro Evaluation

  • MMLU-Pro scores profile the same fine-tuned checkpoint used for unified parsing, under a protocol distinct from public leaderboard evaluation.All rows use the same frozen 12,032-question test set and a shared zero-shot instruction requesting reasoning followed by a boxed answer.Source: §B.4 · p. 19, Table 11 · p. 21
  • A deterministic scorer extracts answer letters by ordered rules, while empty or unparseable responses count as wrong.The denominator is 12,032, and the scorer does not use semantic-similarity or LLM-judge rescue.Source: §B.4 · p. 19, Table 11 · p. 21
  • Two independent implementations agreed on per-record answers, parse outcomes, applied rules, and correctness decisions on the shared test set.Confidence intervals use paired question-level bootstrap resampling with common indices across models.Source: §B.4 · p. 19
  • The reported internal zero-shot scores are not comparable to the public MMLU-Pro leaderboard, which uses a different prompt and evaluator.Responses that reach the generation limit before a recognized final answer count as wrong, making the reported values conservative for models with longer reasoning traces.Source: §B.4 · pp. 19–20, Table 11 · p. 21

Related papers

This paper cites

Loading 2610.34343v1…