PaperlayerView exact revision on arXiv

Source-linked AI summary

UnifiedSKG: Unifying and Multi-Tasking Structured Knowledge Grounding with Text-to-Text Language Models

Tianbao Xie, Chen Henry Wu, Peng Shi, Ruiqi Zhong, Torsten Scholak, Michihiro Yasunaga, Chien-Sheng Wu, Ming Zhong, Pengcheng Yin, Sida I. Wang, Victor Zhong, Bailin Wang, Chengzu Li, Connor Boyle, Ansong Ni, Ziyu Yao, Dragomir Radev, Caiming Xiong, Lingpeng Kong, Rui Zhang, Noah A. Smith, Luke Zettlemoyer, Tao Yu

arXiv:2201.05966v3cs.CL

TL;DR

Structured knowledge grounding has been studied through heterogeneous, task-specific approaches that limit systematic research. UnifiedSKG standardizes 21 tasks in a text-to-text framework, where T5 reaches state-of-the-art results on almost all tasks and multi-task prefix-tuning benefits most tasks.

  • Problem

    Heterogeneous SKG tasks have produced specialized, seemingly incompatible advances rather than systematic approaches spanning tasks and structured knowledge types.

  • Method

    UnifiedSKG standardizes 21 SKG tasks across six families and multiple domains by representing requests, structured knowledge, and outputs in a text-to-text format.

  • Results

    T5 achieves state-of-the-art results on almost all 21 tasks, while multi-task prefix-tuning benefits most tasks and largely improves overall performance.

  • Takeaways & Limitations

    UnifiedSKG provides a powerful, reproducible starting point for systematic SKG research across diverse tasks, knowledge sources, and learning settings.

  • Takeaways & Limitations

    UnifiedSKG assumes the correct structured knowledge type is provided, leaving resource search, action selection, and integration across multiple sources under-explored.

Abstract

from arXiv · show

Structured knowledge grounding (SKG) leverages structured knowledge to complete user requests, such as semantic parsing over databases and question answering over knowledge bases. Since the inputs and outputs of SKG tasks are heterogeneous, they have been studied separately by different communities, which limits systematic and compatible research on SKG. In this paper, we overcome this limitation by proposing the UnifiedSKG framework, which unifies 21 SKG tasks into a text-to-text format, aiming to promote systematic SKG research, instead of being exclusive to a single task, domain, or dataset. We use UnifiedSKG to benchmark T5 with different sizes and show that T5, with simple modifications when necessary, achieves state-of-the-art performance on almost all of the 21 tasks. We further demonstrate that multi-task prefix-tuning improves the performance on most tasks, largely improving the overall performance. UnifiedSKG also facilitates the investigation of zero-shot and few-shot learning, and we show that T0, GPT-3, and Codex struggle in zero-shot and few-shot learning for SKG. We also use UnifiedSKG to conduct a series of controlled experiments on structured knowledge encoding variants across SKG tasks. UnifiedSKG is easily extensible to more tasks, and it is open-sourced at https://github.com/hkunlp/unifiedskg.

1 Introduction

UnifiedSKG unifies 21 heterogeneous structured knowledge grounding tasks in a standardized text-to-text framework, enabling systematic benchmarking and extensible research. Experiments show strong T5 performance, gains from multi-task prefix-tuning, challenges for zero- and few-shot learning, and sensitivity to knowledge encoding.

  • Motivation: SKG tasks are heterogeneous because different structured knowledge types require specialized encoders, limiting compatibility across task-specific advances.The introduction identifies this heterogeneity as a main reason SKG progress has appeared unique and incompatible.
  • Framework: UnifiedSKG unifies 21 SKG tasks across six task families and multiple domains under standardized datasets, models, code, experiments, and metrics.It casts user requests, structured knowledge, and outputs into a text-to-text format.
  • Benchmarking: T5 finetuning with constrained decoding or reranking achieves state-of-the-art results on almost all 21 tasks, with performance generally increasing as model size grows.UnifiedSKG provides a powerful and reproducible starting point for SKG research.
  • Multi-task learning: Multi-task prefix-tuning benefits most tasks and substantially improves overall performance on both T5-base and T5-large, despite mixed results from simple multi-task learning.The framework enables knowledge sharing and cross-task generalization.
  • Zero- and few-shot learning: T0 struggles with zero-shot SKG, while GPT-3 and Codex struggle with few-shot SKG.UnifiedSKG serves as a challenging testbed for evaluating these capabilities with pretrained language models.
  • Analysis: T5 is sensitive to structured knowledge encoding variations, with sensitivity differing across tasks, and T5-3B can still generate invalid outputs despite fewer errors at larger sizes.UnifiedSKG supports controlled encoding experiments and comprehensive cross-task error analysis.

2 Related Work

Prior SKG work with pretrained language models has explored multiple ways to encode structured knowledge, while SKG outputs span heterogeneous formats and structured meaning representations remain challenging. UnifiedSKG builds on task-format unification by covering six task families in a broader text-to-text framework than prior QA-only unification.

  • SKG with PLMs: Prior SKG systems linearized structured knowledge and concatenated it with text, extending this approach with positional encoding, templates, planning, and attention-based methods.Examples include row/column embeddings, template-based linearization, planning, and modifications to transformer attention matrices.
  • SKG with PLMs: Hierarchical encoding represented table structure with tree-based or hierarchical transformers and encoded row, column, or knowledge-graph-triple structure.The cited approaches used tree-based transformers for tables, row and column representations, and hierarchical transformers for KG triples.
  • SKG with PLMs: SKG outputs include logic forms, SQL, dialogue states, natural language, answer sets, and Boolean values, making structured meaning representations difficult for PLMs trained primarily on natural language.The output formats are heterogeneous, and the passage identifies structured meaning representation as a specific challenge.
  • Task format unification: Task unification has been explored through question answering, textual entailment, and graph-to-sequence formats, while UnifiedSKG focuses on text-to-text unification for six SKG task families.Unlike UnifiedQA’s QA-only scope, UnifiedSKG addresses a broader range of SKG tasks and must linearize structured knowledge for text-to-text input.

3 The UNIFIEDSKG Framework

UNIFIEDSKG unifies 21 diverse structured-knowledge-grounding tasks across six families and multiple domains in a text-to-text framework. It linearizes heterogeneous inputs and outputs, while supporting individual-task training and broader multi-task, few-shot, and zero-shot learning studies.

  • UNIFIEDSKG selects 21 SKG tasks across six task families and multiple domains according to the guiding principle of diversity.
  • The task families include semantic parsing, question answering, data-to-text generation, fact verification, conversational tasks, and formal-language-to-text translation.
  • All tasks map a user request, structured knowledge input, and optional dialogue context to an output through input and output linearization.Linearization converts heterogeneous inputs and outputs into text sequences, enabling unified structured-knowledge representations.
  • Individual-task training minimizes token-averaged negative log-likelihood, with beam search used by default for decoding.
  • UNIFIEDSKG also supports multi-task, few-shot, and zero-shot learning studies with pretrained language models.

4 Experiments and Analysis

UnifiedSKG evaluates T5 across 21 structured knowledge grounding tasks, finding strong individual-task performance and benefits from multi-task prefix-tuning. Analyses show uneven transfer from non-SKG pretraining, difficulties in zero/few-shot learning, and limitations of automatic evaluation and generation quality.

  • Individual-task performance: Vanilla T5-3B achieves state-of-the-art performance on nearly all individual SKG tasks, often without extra unsupervised in-domain pretraining.Simple modifications such as constrained decoding or reranking are used when necessary.
  • Effect of pretraining on non-SKG tasks: T0-3B underperforms T5-3B on Spider but outperforms it on MWoZ and TabFact.T0-3B is further pretrained on many mostly non-SKG tasks.
  • Multi-task learning: Multi-task prefix-tuning outperforms single-task finetuning and prefix-tuning on most tasks while largely surpassing naive multi-task learning.The authors attribute mixed multi-task finetuning results to heterogeneous structured knowledge across domains.
  • Exploring task knowledge transfer: Tasks benefit from sequential transfer when they use the same structured-knowledge source, whereas parallel tasks with different knowledge and outputs show no positive transfer.The transfer experiments train on task A before finetuning on task B.
  • Zero-shot and few-shot learning: T0 performs poorly in zero-shot SKG, while Codex outperforms GPT-3 on structured-program generation tasks including Spider and MultiWoZ.PLMs struggle especially with induction tasks such as TabFact, where performance is close to random guessing; better prompting may improve GPT-3 and Codex.
  • Human evaluation and error analysis: Less than 5% of T5 outputs are generally ungrammatical, while missing information and contradiction are common errors and gains usually come from reducing contradiction.Automatic metrics do not always match human evaluation, and larger models are not consistently better.

5 Conclusions

UnifiedSKG unifies 21 structured knowledge grounding tasks to support systematic research and benchmarks T5 across them. It shows strong supervised performance, benefits from multi-task prefix-tuning, and reveals task-dependent encoding effectiveness and challenging zero-/few-shot learning.

  • UnifiedSKG unifies 21 structured knowledge grounding tasks into a common framework for systematic research.
  • Finetuning T5 on individual tasks achieves state-of-the-art results on almost all 21 tasks, while multi-task prefix-tuning benefits most tasks.
  • Structured knowledge encoding variations differ in effectiveness across tasks, and UnifiedSKG provides a challenging testbed for zero-shot and few-shot learning.

6 Limitations · A Contributions

UnifiedSKG provides an extensible, reproducible foundation for systematic SKG research, while leaving resource selection and multi-source integration under-explored. The authors also document distinct contributions to implementation, task unification, writing, experimentation, analysis, and review.

  • 6 Limitations: UNIFIEDSKG standardizes SKG research and makes applying models to diverse tasks and framing new tasks straightforward.It supports systematic study of structured knowledge encoding, multitask learning, zero-shot learning, and few-shot learning.
  • 6 Limitations: UNIFIEDSKG supplies each task’s correct structured knowledge, but searching, acting on, and integrating multiple resources remain under-explored prerequisites.The limitation concerns building a unified multi-purpose SKG system from user requests.
  • 6 Limitations: UNIFIEDSKG’s popular-task selection risks disproportionate coverage of languages, domains, and populations, motivating additions of diverse multilingual tasks.The authors also call for finer-grained SKG error analysis and systematic study of PLM behavior.
  • A Contributions: Code implementation and the experiment pipeline were led by Tianbao Xie and Chen Henry Wu, with PICARD code and Torsten Scholak’s advice accelerating implementation.This passage specifically assigns responsibility for the UNIFIEDSKG framework’s code base and experiment pipeline.
  • A Contributions: Tianbao Xie, Peng Shi, Michihiro Yasunaga, Chen Henry Wu, and Ming Zhong implemented 21 tasks in text-to-text form, adapted metrics, and verified performances.The passage identifies these activities as the task-unification contribution.
  • A Contributions: Chen Henry Wu and Tianbao Xie wrote most of the paper, while Michihiro Yasunaga, Peng Shi, and Chengzu Li added results and analysis for corresponding parts.Peng Shi also drafted related work on SKG with PLMs.
  • A Contributions: Chen Henry Wu, Tianbao Xie, and Chien-Sheng Wu conducted experiments, with collaborators advising on model size, structured-knowledge conversion, results interpretation, comparisons, and multitask transfer.Additional authors reviewed the paper and provided feedback over multiple rounds.

B Results with Full Metrics

This section reports development-set performance with full metrics, averaging three random-seed experiments on representative tasks from each family. It also evaluates T5 variants on a widely used pre-processed version of KVRET.

  • B Results with Full Metrics: Three experiments with different random seeds are conducted on a representative task from each family.The results are reported for the development set with full metrics.
  • B Results with Full Metrics: Results are reported as averages and standard variances in avrvar format.
  • B Results with Full Metrics: The KVRET experiments use another more widely used pre-processed version rather than the version in the main tables.This version is evaluated with T5-base, T5-large, and T5-3b, with results shown in Table 13.

C Input and Output Length Analysis … D.2 Metric Details

UnifiedSKG analyzes how structured-input length affects performance and specifies its implementation and evaluation metrics. Long inputs require truncation, while longer retained inputs improve accuracy on WikiTableQuestion.

  • C Input and Output Length Analysis: Large structured knowledge inputs can be arbitrarily long, requiring truncation to fit limited GPU capacity.Inputs and outputs are tokenized with Huggingface Transformers’ T5Tokenizer.
  • C Input and Output Length Analysis: WikiTableQuestion accuracy increases as the input truncation length becomes longer.This motivates studying more effective encoding of large structured inputs, including sparse attention.
  • D Experimental Setup: The experiments use T5 as the backbone language model, with DeepSpeed for T5-3B memory savings and larger batch sizes for WikiSQL, WikiTQ, and TabFact.Batch size is 32 by default and 128 for WikiSQL, WikiTQ, and TabFact; optimization uses Adafactor for T5-base and T5-large and AdamW for T5-3B.
  • D.1 Implementation Details: For most tasks, 1024 tokens is a practical input length, while 2048 tokens can hold all inputs for most tasks.The distribution excludes MTOP because its inputs concentrate on a relatively small field.
  • D.1 Implementation Details: All tasks use learning rate 5e-5 with linear decay, and experiments run on NVIDIA Tesla V100 and A100 GPUs.Development performance is evaluated every 500 steps, using the average development metric for checkpoint selection.
  • D.2 Metric Details: Evaluation uses task-specific metrics: exact match for most semantic parsing and QA, F1 for WebQSP, sacre-BLEU for data-to-text, accuracy for fact verification, and BLEC for high-fidelity NLG.Unless specified, results use T5-large and report development-set performance.

D.3 T0 Zero-shot Experimental Details … D.6 Hyperparameters

The appendix details zero-shot prompting for T0, GPT-3, and Codex, along with decoding settings, prompt construction, human evaluation, and model hyperparameters. It uses task-specific instructions and structured-knowledge formats while specifying evaluation and decoding procedures.

  • D.3 T0 Zero-shot Experimental Details: T0 3B inputs follow the most similar instructions from Sanh et al. (2021), with new instructions created when no suitable one exists.The resulting inputs are tested directly on T0 3B.
  • D.3 T0 Zero-shot Experimental Details: Task-specific T0 instructions cover database-to-SQL, table question answering, triples-to-text generation, and dialogue-state prediction.Examples include Spider, WikiTQ, DART, and MultiWoZ formats.
  • D.4.1 Hyperparameter Settings: Greedy decoding uses temperature 0 for GPT-3 and Codex on Spider, WikiTQ, MultiWoZ, and TabFact, while Codex temperature tuning on DART and SQL2Text shows no significant difference.GPT-3 is not tuned for those latter tasks to stay within budget.
  • D.4.1 Hyperparameter Settings: Maximum output length is 256 for Spider, WikiTQ, MultiWoZ, and SQL2Text, but 4 for TabFact, with “\n” used as the stop token.For GPT-3 and Codex, maximum length counts input and output tokens together.
  • D.4 GPT3 and Codex Details: GPT-3 and Codex use simple prompt words to concatenate the request, linearized structured knowledge, and context before generating completions.For WikiTQ, the prompt includes the linearized table, request, and an answer cue; alternative Spider knowledge formats produced similar results.
  • D.5 Human Evaluation: Human evaluation uses eight task-familiar authors, who assign each model output a 0–1 correctness score and classify errors such as missing information, contradiction, hallucination, or ungrammaticality.An output is correct only when it properly answers the request without grammar or wording mistakes.
  • D.5 Human Evaluation: The human-evaluation guidelines define task inputs and references for DART, FeTaQA, KVRET, Logic2Text, SQL2Text, and ToTTo.These include triples, tables, dialogues, logic expressions, SQL queries, and highlighted-table metadata.
  • D.6 Hyperparameters: Semantic parsing uses greedy search with beam size 1, while long linearized sequences use an input length of 1024 to hold the maximum input.The appendix states that the reasons for this setting are explained in Appendix C.

E Training Details

The section compares fine-tuning and prefix-tuning training procedures, noting that prefix-tuning generally requires more steps but can reach comparable results with continued training.

  • Training Details: Prefix-tuning generally needs more training steps than fine-tuning but can achieve comparable results with continued training.Prefix-tuning uses random initialization, following Li and Liang (2021).

F Task Unification … F.3 Output Format

The framework unifies heterogeneous structured knowledge inputs by linearizing tables, highlighted tables, relation-triples, knowledge graphs, and ontologies into sequences, then standardizes outputs according to their types.

  • F Task Unification: Structured knowledge inputs are converted into sequence representations to support a unified text-to-text framework.The input formats include tables, highlighted tables, relation-triples, knowledge graphs, and ontologies.
  • F.2 Linearization: A table is linearized as column information followed by rows, with special tokens marking table boundaries.The representation uses the form “col: c1, ..., cN row 1 : r1 row 2 : r2... rM ”, where N and M denote columns and rows.
  • F.1 Term Definition: A highlighted table combines the page title, section title, and each highlighted cell’s value with its column and row headers.A highlighted table contains metadata and highlighted cells whose contents entail the text description.
  • F.1 Term Definition: Relation-triples represent subject-predicate-object relationships, while knowledge graphs use entity-relation triples and retrieved subgraphs are serialized identically.Each relation-triple is formatted as “sub : rela : obj”, and multiple triples are joined by “ | ”.
  • F.1 Term Definition: An ontology lists each slot with all possible values in “slot : value1, ... valueslotn ” format, joining different slot-value lists with “ | ”.The ontology defines the possible slots, which are typically domain-dependent.
  • F.3 Output Format: Natural-language and formal-language outputs remain unchanged because they are already sequences.This preserves outputs that already match the target sequence format.
  • F.3 Output Format: Sets of answers are comma-space joined, Boolean values map True to “entailed” and False to “refuted”, and dialogue states serialize slot-value pairs sequentially.Dialogue states capture user goals as slot-value pairs, following the cited prior formulation.

G Input and Output Examples for Each Task … H.1.3 Case 3:

The paper illustrates UnifiedSKG by representing diverse structured-knowledge tasks as text-to-text mappings, then showing case-study differences in T5 predictions for Spider queries.

  • G.1 Spider: Spider maps a natural-language singer-count question to SQL over a concert database.The example asks, “How many singers do we have?” and outputs select count(*) from singer.
  • G.2 GRAILQA; G.3 CompWebQ: GRAILQA and CompWebQ represent knowledge-base questions with entity-relation context and logical-form outputs.Examples cover the Soviet Red Army and Liam Hemsworth, including relation paths, entity types, and compositional queries.
  • G.4 WebQsp; G.5 MTOP; G.6 WikiSQL: WebQSP, MTOP, and WikiSQL pair structured contexts with questions and formal outputs for entity retrieval, intent-slot parsing, and table querying.Examples include identifying Spain’s king, parsing a call to Nicholas and Natasha, and finding a player’s position from a table.
  • G.7 WikiTableQuestions; G.8 HybridQA; G.9 MultiModalQA; G.10 FeTaQA: WikiTableQuestions, HybridQA, MultiModalQA, and FeTaQA demonstrate table and text-grounded question answering across relational, textual, and filmography or awards data.The examples ask about the team preceding Crettyard, a marathoner’s placing, Ben Piazza’s film role, and Andy Karl’s Olivier Award.
  • G.11 DART; G.12 ToTTo; G.13 MultiWoZ2.1: DART, ToTTo, and MultiWoZ2.1 cover structured-to-text generation and task-oriented dialogue with database-like inputs and natural-language outputs.Examples include describing a school, verbalizing a governor record, and tracking hotel and train booking constraints.
  • G.14 KVRET; G.15 SParC; G.16 CoSQL; G.17 SQA Structured Input:; G.18 TabFact: KVRET, SParC, CoSQL, SQA Structured Input, and TabFact show grounded interaction, conversational SQL, structured question answering, and table reasoning.The examples include traffic-avoiding directions, airline-country lookup, car-model aggregation, submarine-captain questions, and a Turkish Cup statement.
  • G.19 FEVEROUS; G.20 SQL2Text; G.21 Logic2Text: FEVEROUS, SQL2Text, and Logic2Text include evidence-based verification, SQL-to-language conversion, and logical-form-to-text generation.Examples state that Alan Dale is a narrator, translate a professor-selection SQL query, and verbalize that three structures finish in 2006.

H.2 FeTaQA case … I Natural Language Template Examples

The case studies illustrate how T5 models translate structured inputs into natural-language answers across FeTaQA, KVRET, SQL2Text, DART, Logic2Text, and ToTTo, with model-specific successes and errors. The section also presents natural-language templates for describing Spider databases, tables, columns, primary keys, and foreign-key relations.

  • H.4 SQL2Text case: For SQL2Text, T5-large correctly paraphrases the SQL query as requesting company names and main industries for companies headquartered outside the United States.T5-base retains the USA wording, while T5-3B omits the company names.
  • H.5 DART case: All three T5 variants correctly verbalize the DART facts about Cotto’s English food, city-centre location, nearby landmark, price range, and high rating.The predictions differ mainly in sentence organization while preserving the listed attributes.
  • H.6 Logic2Text case: For Logic2Text, T5-3B correctly states that 3 games at Qualcomm Stadium started at 5:15 pm, while T5-base and T5-large are marked incorrect.The correct prediction preserves both the stadium filter and the time condition.
  • I.1 Spider Template: The Spider templates describe database contents, primary keys, table columns, and foreign-key relationships using placeholders for database, table, and column names.They include templates for overall database descriptions, table descriptions, and foreign-key descriptions.
Loading 2201.05966v3…