Source-linked AI summary
PICARD: Parsing Incrementally for Constrained Auto-Regressive Decoding from Language Models
Torsten Scholak, Nathan Schucher, Dzmitry Bahdanau
TL;DR
Fine-tuned language models often generate invalid SQL despite needing to satisfy formal constraints. PICARD constrains autoregressive decoding through incremental parsing, and it lifts fine-tuned T5 models to state-of-the-art results on Spider and CoSQL. The method remains compatible with otherwise unmodified models and existing vocabularies.
Problem
Pre-trained language models have unconstrained token outputs and may generate unusable SQL that fails lexical, grammatical, logical, or semantic correctness requirements.
Method
PICARD incrementally parses model output and rejects inadmissible tokens during decoding, without modifying the model’s pre-training, fine-tuning, decoder, or vocabulary.
Results
PICARD lifts a T5-3B model to state-of-the-art results on Spider and CoSQL using established exact-match and execution-accuracy metrics.
Takeaways & Limitations
PICARD provides simple constrained decoding that significantly improves otherwise unmodified fine-tuned T5 models on Spider and CoSQL.
Takeaways & Limitations
PICARD does not yet check whether compared expressions have compatible types or whether set-operation column types match.
Abstract
from arXiv · showhide
Large pre-trained language models for textual data have an unconstrained output space; at each decoding step, they can produce any of 10,000s of sub-word tokens. When fine-tuned to target constrained formal languages like SQL, these models often generate invalid code, rendering it unusable. We propose PICARD (code and trained models available at https://github.com/ElementAI/picard), a method for constraining auto-regressive decoders of language models through incremental parsing. PICARD helps to find valid output sequences by rejecting inadmissible tokens at each decoding step. On the challenging Spider and CoSQL text-to-SQL translation tasks, we show that PICARD transforms fine-tuned T5 models with passable performance into state-of-the-art solutions.
1 Introduction
PICARD constrains language-model decoding through incremental parsing while remaining compatible with existing autoregressive models and vocabularies. It rejects inadmissible tokens during generation and improves fine-tuned T5 performance on Spider and CoSQL.
- 1 Introduction: PICARD addresses the difficulty of making language models satisfy SQL’s lexical, grammatical, logical, and semantic constraints.Pre-trained models can generate invalid SQL that fails required correctness conditions.
- 1 Introduction: PICARD is an incremental parsing method for constrained decoding that works with existing autoregressive language-model decoders.It can be enabled optionally at inference time without changing pre-training or fine-tuning.
- 1 Introduction: PICARD rejects inadmissible token predictions at each decoding step, using constrained beam search rather than checking only completed sequences.The method was illustrated with top-k candidate checking during beam search.
- 1 Introduction: Unlike approaches requiring custom vocabularies or architectures, PICARD operates directly on the model’s readable SQL output.It also avoids filtering only finalized hypotheses with very large beam sizes.
- 1 Introduction: PICARD significantly improves fine-tuned T5 models and lifts T5-3B to state-of-the-art performance on Spider and CoSQL.On Spider, PICARD lets T5-Base outperform T5-Large without it, and T5-Large outperform T5-3B without it.
2 The PICARD Method
PICARD incrementally filters language-model tokens using lexical, grammatical, and guarded SQL parsing informed by database schemas. Its guards enforce scope and column-resolution constraints, while additional type-related checks remain future work.
- 2 The PICARD Method: PICARD integrates with greedy and beam search by warping model scores and uses the current hypothesis, token scores, and SQL schema information.Schema information includes table names, column names, and table-column membership.
- 2 The PICARD Method: At each generation step, PICARD checks the top-k tokens and assigns −∞ to predictions that fail its incremental parsing checks.The method offers off, lexing, parsing without guards, and parsing with guards modes.
- 2 The PICARD Method: Lexing mode detects malformed keywords and schema-invalid identifiers without requiring lexical items to appear in grammatical order.It converts partial detokenized output into SQL keywords, punctuation, operators, literals, and identifiers.
- 2 The PICARD Method: Parsing without guards checks grammatical structure by constructing an abstract syntax tree and rejecting invalid clause order or expression compositions.It also handles alias bindings, nested-query scope, and missing columns in qualified references.
- 2.3 Parsing with Guards: Guards require referenced tables or aliases to enter scope and ensure unqualified columns resolve to exactly one available table.These checks reject predictions such as selecting maker and model from car_makers when model_list is the only matching table.
- 2.3 Parsing with Guards: PICARD leaves further checks for same-type comparisons and compatible column types in UNION, EXCEPT, and INTERSECT queries to future work.These are identified as conceivable additional checks beyond the current guards.
3 Experiments
Experiments evaluate PICARD on Spider and CoSQL using fine-tuned T5 baselines, reporting substantial accuracy gains, fewer invalid SQL outputs, and modest decoding overhead. Ablations show that incremental guarded parsing benefits from larger beams and outperforms simpler checking modes.
- Experimental setup: Experiments train on Spider’s 7,000 examples and evaluate on Spider development/test sets and CoSQL development/test sets.CoSQL training combines Spider and CoSQL data for dialog state tracking.
- Metrics: Spider evaluation uses exact-set-match, execution, and test-suite execution accuracy, while CoSQL uses question-match and interaction-match accuracy.Interaction-match accuracy is joint accuracy across all questions in an interaction.
- Spider results: 12% of T5-3B Spider development predictions produce execution errors without PICARD, versus 2% unusable predictions with PICARD.The remaining PICARD failures occur when beam search finds no valid SQL prediction.
- Spider results: 75.5% development and 71.9% test exact-set-match accuracy are reported for T5-3B with PICARD on Spider.Execution accuracy is 79.3% on development and 75.1% on test; the results are described as on par or higher than the closest competitor.
- CoSQL results: PICARD significantly improves CoSQL performance, with fine-tuned T5-3B achieving state-of-the-art results.The reported CoSQL metrics are question-match and interaction-match accuracy on development and test sets.
- Efficiency: On Spider with beam size 4, decoding averages 2.5 seconds per sample without PICARD and 3.1 seconds with PICARD on an NVIDIA A100-SXM4-40GB GPU.The paper characterizes PICARD as fast despite this measured overhead.
- Beam-size analysis: PICARD increases performance for every tested T5 size, with gains strengthening from beam size 1 to 2 and saturating above beam size 4.The number of top-k tokens processed has modest to negligible performance impact, depending on model size.
- Ablations: Incremental lexing yields small beam-independent gains, whereas parsing with or without guards improves rapidly with beam size and guarded parsing leads clearly.Checking only when finalizing hypotheses remains effective but is substantially weaker than normal incremental operation, with the gap shrinking at large beam sizes.
4 Conclusion
PICARD provides simple constrained decoding for large pre-trained language models, improving fine-tuned T5 performance on Spider and CoSQL and lifting T5-3B to state-of-the-art results.
- PICARD is a new method for simple and effective constrained decoding with large pre-trained language models.
- Rejected hypotheses never enter the beam, rather than merely being removed after a complete beam is finalized.
- PICARD improves otherwise unmodified, fine-tuned T5 models on Spider and CoSQL text-to-SQL tasks.
- T5-3B with PICARD reaches state-of-the-art results on exact-match and execution accuracy metrics.