PaperlayerView exact revision on arXiv

Source-linked AI summary

Structured Prediction as Translation between Augmented Natural Languages

Giovanni Paolini, Ben Athiwaratkun, Jason Krone, Jie Ma, Alessandro Achille, Rishita Anubhai, Cicero Nogueira dos Santos, Bing Xiang, Stefano Soatto

arXiv:2101.05779v3cs.LGcs.CL

TL;DR

Structured prediction often relies on task-specific discriminative classifiers and may not exploit pretrained knowledge about label semantics. TANL instead translates between augmented natural languages, achieving competitive or state-of-the-art results across tasks, including with one multi-task model and in low-resource settings. Its generative approach has quadratic token-generation complexity, motivating more efficient attention mechanisms for long sequences.

  • Problem

    Task-specific discriminative classifiers use class-number targets, limiting unified architectures and their ability to leverage pretrained knowledge about label semantics.

  • Method

    TANL frames diverse structured prediction problems as text-to-text translation between augmented natural languages, with outputs decoded into structured information.

  • Results

    TANL matches or outperforms task-specific models across the considered tasks, achieves new state-of-the-art results on several benchmarks, and supports comparable or better multi-task results.

  • Takeaways & Limitations

    A unified generative framework can handle diverse structured prediction tasks while providing benefits in multi-task and low-resource settings.

  • Takeaways & Limitations

    Token generation has O(L^2) time complexity, which can increase computation time for long input sequences.

Abstract

from arXiv · show

We propose a new framework, Translation between Augmented Natural Languages (TANL), to solve many structured prediction language tasks including joint entity and relation extraction, nested named entity recognition, relation classification, semantic role labeling, event extraction, coreference resolution, and dialogue state tracking. Instead of tackling the problem by training task-specific discriminative classifiers, we frame it as a translation task between augmented natural languages, from which the task-relevant information can be easily extracted. Our approach can match or outperform task-specific models on all tasks, and in particular, achieves new state-of-the-art results on joint entity and relation extraction (CoNLL04, ADE, NYT, and ACE2005 datasets), relation classification (FewRel and TACRED), and semantic role labeling (CoNLL-2005 and CoNLL-2012). We accomplish this while using the same architecture and hyperparameters for all tasks and even when training a single model to solve all tasks at the same time (multi-task learning). Finally, we show that our framework can also significantly improve the performance in a low-resource regime, thanks to better use of label semantics.

1 INTRODUCTION

TANL reframes structured prediction as translation between augmented natural languages, enabling one unified architecture across diverse tasks. It reports competitive or state-of-the-art results, including multi-task and low-resource gains.

  • Structured prediction covers NLP tasks whose outputs are structured objects, including graphs of entities and relations, semantic roles, and coreference links.
  • TANL addresses the limitations of task-specific discriminative classifiers by translating between augmented natural languages that encode and decode structured information.The framework is designed to leverage latent knowledge about label semantics in pretrained models.
  • The same architecture and hyperparameters are used across joint extraction, named entity recognition, relation classification, semantic role labeling, coreference, event extraction, and dialogue state tracking.Only the augmented natural language formats differ among tasks.
  • TANL achieves at least comparable results across all considered tasks and new state-of-the-art results on several joint extraction, relation classification, and semantic role labeling benchmarks.
  • A single multi-task model obtains comparable or better results than separately trained single-task models.
  • Improved transfer of label semantics significantly improves performance in the few-shot regime over previous approaches.

2 RELATED WORK

Prior NLP structured prediction methods largely extend classification with task-specific structured layers or classifiers. Related sequence-to-sequence work includes general text-to-text transfer and task-specific structured-output generation, while TANL emphasizes unified multi-task prediction.

  • Classical NLP structured prediction methods include Conditional Random Fields, Structured Perceptron, and Structured Support Vector Machines.
  • Recent state-of-the-art systems commonly place task-specific classifiers on pretrained language-model features, including BERT MRC for NER and SpERT for joint entity-relation extraction.
  • Sequence-to-sequence research has cast translation and summarization as natural-language text-to-text tasks and has generated structured outputs for selected prediction problems.
  • TANL is related to multi-task and transfer-learning work, particularly because it enables multi-task structured prediction and targets low-resource settings.

3 METHOD

TANL frames structured prediction as translation between task-specific augmented natural languages. The model decodes structured objects from generated text using replicated inputs, natural-language labels, and alignment procedures.

  • Format design: TANL frames structured prediction tasks as text-to-text translation between task-appropriate augmented natural languages.The input and output formats are designed to encode and decode structured information.
  • Format design: The output replicates the input sentence and adds bracketed patterns whose tags encode entity types and relations.Relations use X = Y, where X is the relation type and Y is the tail entity.
  • Format design: Natural-language entity and relation labels expose label semantics instead of abbreviations, leveraging pretrained knowledge about words such as person and location.Replicating input words also reduces ambiguity when an entity occurs multiple times.
  • Format design: Nested patterns represent hierarchical entities and allow multiple relations involving the same entity.The ADE example nests lithium within lithium toxicity and attaches two effect relations.
  • Decoding: Decoding removes special tokens and invalid fragments, then uses token-level Needleman-Wunsch dynamic-programming alignment to locate entities in the original input.Alignment can associate misspelled generated text such as Aciclovir with the corresponding input token sequence.
  • Decoding: For categorical tasks, output-sequence likelihood serves as a proxy for class scores and offers more robust evaluation when generation is imperfect in low-resource settings.This approach is used for tasks with a limited number of valid outputs.

4 STRUCTURED PREDICTION TASKS

The framework applies a common augmented-language approach across diverse structured prediction tasks. Task-specific formats mark entities, predicates, dialogue turns, triggers, or mentions so outputs can encode the required structures.

  • Task formats: Joint entity and relation extraction uses bracketed entity spans with type tags and relation assignments.The format preserves asymmetric head-to-tail relation order.
  • Task formats: Named entity recognition is handled as the entity-only special case of joint entity and relation extraction.
  • Task formats: Relation classification appends a head–tail relationship prompt and generates the relation type as its continuation.The prompt handles cases where the head entity does not precede the tail in the sentence.
  • Task formats: Semantic role labeling marks a predicate in the input and labels argument spans in the output according to their roles.Arguments may represent subjects, locations, times, or other predicate-related types.
  • Task formats: Event extraction detects typed triggers and extracts their typed arguments by processing one marked trigger at a time.The example identifies attacked as attack and injured as injury.
  • Task formats: Coreference resolution groups mentions by labeling later mentions with their first referring mention.The example links his to Barack Obama.
  • Task formats: Dialogue state tracking prefixes user and agent turns and generates slot names with predicted values inside belief delimiters.Unmentioned slots receive the value not given.

5 EXPERIMENTS

Experiments evaluate TANL in single-task, multi-dataset, and multi-task settings using a common pretrained T5-base architecture. TANL reaches state-of-the-art or competitive performance across tasks, transfers effectively to low-resource settings, and benefits from its design choices.

  • Single-task results: TANL achieves state-of-the-art performance on ADE, NYT, ACE2005, FewRel, TACRED, CoNLL-2005, and CoNLL-2012 in the single-task setup.It obtains a +6.2 absolute F1 improvement over the previous state of the art on NYT.
  • Single-task results: A single unchanged model architecture performs competitively across tasks without model modification, unlike typical task-specific discriminative models.Experiments start from a pretrained T5-base model with shared hyperparameters apart from dataset-specific settings.
  • Multi-task results: Multi-task training is on par with or better than single-task training, with sizable gains on small datasets.ADE relation F1 rises from 80.6 to 83.7, while CoNLL04 entity F1 rises from 89.4 to 90.6.
  • Multi-task results: The multi-task model has notably lower coreference scores when its 512-token limit requires splitting documents that use longer single-task sequences.The single-dataset coreference setup uses 1,536 input and 2,048 output tokens.
  • Low-resource settings: TANL shows data efficiency on FewRel, TACRED, and CoNLL04, including performance on scarce label types and low-resource training subsets.On CoNLL04, it outperforms SpERT using 0.8% to 6% of training data, while full-data performance is similar.
  • Low-resource settings: Pretraining on NYT and fine-tuning on limited CoNLL04 data yields up to +9 relation F1.This demonstrates transfer from a larger joint entity and relation extraction dataset to a low-resource setting.
  • Ablations: Ablations show that label semantics, the full augmented-language format, and optimal alignment each contribute to TANL effectiveness.Numeric labels and abridged output substantially or consistently hurt performance, especially in low-resource settings.

6 DISCUSSION AND CONCLUSION

TANL unifies structured prediction as text-to-text translation, handling diverse tasks within one generative framework and enabling seamless multi-task learning. Its main practical boundary is quadratic token-generation complexity, although more efficient attention mechanisms may reduce it.

  • 6 DISCUSSION AND CONCLUSION: TANL frames structured prediction as generative translation between augmented natural languages rather than task-specific discrimination.The augmented languages encode structured information in inputs and outputs, including structures for complex tasks.
  • 6 DISCUSSION AND CONCLUSION: The unified format supports diverse structured prediction tasks and makes multi-task learning seamless while achieving state-of-the-art performance on many tasks.The framework can output task-specific structures with little, if any, format error.
  • 6 DISCUSSION AND CONCLUSION: O(L^2) time complexity per generated token limits the current approach, especially for long sequences and complex tasks.The paper identifies Reformer and Linformer-style attention as possible future routes to O(L log L) or O(L) complexity.
  • 6 DISCUSSION AND CONCLUSION: Generative modeling is presented as promising but understudied for structured prediction, motivating further research in this direction.The authors connect their findings to a broader trend of solving traditionally discriminative NLP tasks generatively.

A EXPERIMENTAL SETUP, DATASETS, AND BASELINES

The experiments fine-tune a pre-trained T5-base model using a largely shared training configuration across structured prediction tasks. Evaluation reports average results, generally using micro-F1, with a separate larger setup for multi-task training.

  • A EXPERIMENTAL SETUP, DATASETS, AND BASELINES: All experiments fine-tune a pre-trained T5-base model with about 220 million parameters for downstream text-to-text structured prediction.T5-base is chosen for its text-to-text design and prior natural-language knowledge.
  • A EXPERIMENTAL SETUP, DATASETS, AND BASELINES: Most experiments share hyperparameters, including 8 V100 GPUs, batch size 8 per GPU, AdamW, learning-rate decay from 0.0005, and 256-token sequences.Some tasks use different maximum sequence lengths, as specified in the experimental setup.
  • A EXPERIMENTAL SETUP, DATASETS, AND BASELINES: The multi-task experiment trains for 50 epochs on 80 GPUs with batch size 3 per GPU and 512-token input/output sequences.

A.1 JOINT ENTITY-RELATION EXTRACTION

Joint entity-relation extraction is evaluated across CoNLL04, ADE, NYT, and ACE2005 using established dataset splits, cross-validation, and baselines. Low-resource studies compare TANL with SpERT and examine transfer from NYT to CoNLL04.

  • A.1 JOINT ENTITY-RELATION EXTRACTION: The evaluation covers CoNLL04, ADE, NYT, and ACE2005, spanning news, medical, distantly supervised, and multi-domain data.These datasets differ in entity and relation inventories, including nested entities in ADE.
  • A.1 JOINT ENTITY-RELATION EXTRACTION: CoNLL04 contains 922 training, 231 validation, and 288 test sentences, with four entity types and five relation types.Test results are averaged over 10 runs after 200 training epochs.
  • A.1 JOINT ENTITY-RELATION EXTRACTION: ADE uses 4,272 medical-report sentences with nested entities and evaluates macro-F1 through 10-fold cross-validation.It contains two entity types and one relation type; the multi-task experiment instead uses the first split once.
  • A.1 JOINT ENTITY-RELATION EXTRACTION: The comparison includes span-classification, question-answering, tagging, and encoder-decoder baselines such as SpERT, Multi-turn QA, ETL-Span, and WDec.
  • A.1 JOINT ENTITY-RELATION EXTRACTION: TANL outperforms SpERT with 0.8% (9 sentences) to 6% (72 sentences) of CoNLL04 training data, while performance is similar on the full training set.The low-resource advantage is especially pronounced for relation extraction.

A.2 NAMED ENTITY RECOGNITION

Named entity recognition experiments cover flat and nested datasets and compare TANL with established task-specific and sequence-to-sequence baselines. The setup also includes multi-dataset training across the four NER datasets.

  • A.2 NAMED ENTITY RECOGNITION: NER evaluation uses flat CoNLL03 and OntoNotes datasets alongside nested GENIA and ACE2005 datasets.
  • A.2 NAMED ENTITY RECOGNITION: CoNLL03 contains 14,041 training, 3,250 validation, and 3,453 test sentences across four entity types, with results averaged over 10 runs.The experiments train for 50 epochs.
  • A.2 NAMED ENTITY RECOGNITION: The experimental setup and results for single-dataset joint extraction and NER are summarized with standard deviations in Table 2.
  • A.2 NAMED ENTITY RECOGNITION: NER baselines include BERT-MRC, BERT-MRC + DSC, ClozeCNN, Seq2seq-BERT, and iterative second-best learning and decoding.These methods use machine reading comprehension, cloze pre-training, sequence-to-sequence generation, or iterative decoding.
  • A.2 NAMED ENTITY RECOGNITION: The multi-dataset NER experiment trains one model on all four datasets for 10 epochs and averages results over 5 runs.

A.3 RELATION CLASSIFICATION

TANL applies a unified augmented-language translation framework to relation classification on FewRel and TACRED. Its label semantics support few-shot and sparse-label settings, while alternative formats and likelihood-based prediction affect performance.

  • Datasets and formats: TANL experiments on FewRel and TACRED relation classification using augmented natural language formats.FewRel uses few-shot N-way K-shot evaluation with disjoint train, validation, and test relations.
  • Results: TANL outperforms BERTEM+MTB in certain FewRel settings, including 5-way 1-shot classification.The comparison includes BERT-pair and BERTEM+MTB baselines.
  • Format comparison: The chosen augmented format performs better than shorter or directly annotated alternatives, plausibly because its richer semantics and longer sequences aid training.The paper reports that the chosen format is more natural and closer to natural language.
  • Results: 77.8% recall is achieved for TACRED date of birth relations despite only 64 training labels.The result illustrates performance on a sparse relation type.
  • Label sparsity: TANL’s TACRED state-of-the-art result is attributed to shared semantics across labels, especially for sparse relation types.The dataset is highly imbalanced, with a few relation types accounting for approximately 60,000 of 68,000 training instances.
  • Prediction: Likelihood-based class scoring slightly improves relation-classification performance, particularly when few-shot label generation is imperfect.The method scores all possible outputs by sequence likelihood; the reported FewRel and TACRED numbers use this approach.

A.4 SEMANTIC ROLE LABELING

TANL evaluates semantic role labeling on CoNLL-2005 and CoNLL-2012 within its unified translation framework. The experiments use dataset-specific role formats and compare against established neural baselines across multi-dataset training settings.

  • Datasets: Semantic role labeling experiments use CoNLL-2005 and the English CoNLL-2012 OntoNotes subset.Because the tasks contain highly specific label types, the experiments use the datasets’ raw labels.
  • Label formats: CoNLL-2005 assigns semantic roles to verb predicates, including V, A0, A1, A2, A3, and adjunct labels.The passage defines examples such as A0 as acceptor and A1 as thing accepted for this dataset’s notation.
  • Label formats: CoNLL-2012 numbered-argument semantics depend on the verb and sense, so roles such as A0 and A1 lack consistent cross-verb generalizations.A0 commonly denotes an agent and A1 a patient or theme, but these interpretations are not universal.
  • Label formats: CoNLL-2012 also includes optional adjuncts and reference arguments, with reference labels such as R-A1 inheriting the referenced role.Adjuncts include temporal, locative, causal, modal, and other types.
  • Baselines: The baselines include Dependency and Span SRL, using Bi-LSTM and biaffine components, and BERT-SRL, which predicts spans from contextual and positional embeddings.The cited systems represent established neural alternatives for semantic role labeling.
  • Evaluation: The multi-dataset experiment trains one model on all datasets for 50 epochs and averages results over five runs.The supplied experiment description specifies the training duration and aggregation procedure.

B ABLATION STUDIES

CoNLL04 ablations test whether label semantics, natural output formatting, and alignment contribute to TANL. All three components help, with their importance varying by data availability and extraction target.

  • Ablation setup: The ablations compare TANL with numeric labels, abridged outputs, and a no-alignment variant.The study uses CoNLL04 joint entity and relation extraction to isolate these design choices.
  • Ablation setup: The experiments use all 1,153 training sentences or portions down to 0.8%, while evaluating on the full 288-sentence test set.Training epochs increase as the available training portion decreases.
  • Findings: All three components—label semantics, natural output format, and alignment—positively contribute to TANL effectiveness.The ablation study identifies each component as beneficial rather than reporting a single dominant component overall.
  • Findings: Label semantics become statistically relevant at 50% of the training data or less, whereas alignment has greater impact with larger training sets.With the full dataset, natural and numeric labels yield similar F1 scores.
  • Findings: For entity extraction, repeating the input sentence matters more than natural labels; for relation extraction, natural labels matter more.This contrast is reported for the left and right plots of the ablation results.

C ANALYSIS OF GENERATION ERRORS

The generation-error analysis examines how TANL’s output quality changes across CoNLL04 training-data sizes. Reconstruction errors dominate and alignment mitigates them, while low-resource training introduces additional format and entity errors.

  • Error types: The analysis categorizes reconstruction, format, entity, and label errors in generated augmented-language outputs.Reconstruction errors concern failure to replicate the input; entity errors concern relation tails lacking matching predicted entities.
  • Error patterns: Reconstruction errors are by far the most common, but the alignment step mitigates them.With the full CoNLL04 training set, the other error types occur very infrequently.
  • Low-resource behavior: When training uses limited data, format and entity errors occur and additional post-processing could benefit TANL.The paper leaves investigation of low-resource post-processing strategies for future work.
  • Scope: CoNLL04 is used for error analysis because it is the smallest benchmark considered, making its generation errors likely the most significant.The analysis therefore emphasizes a setting expected to expose generation weaknesses.
Loading 2101.05779v3…