PaperlayerView exact revision on arXiv

Source-linked AI summary

COLLIE: Systematic Construction of Constrained Text Generation Tasks

Shunyu Yao, Howard Chen, Austin W. Hanjie, Runzhe Yang, Karthik Narasimhan

arXiv:2307.08689v1cs.CLcs.AIcs.LG

TL;DR

Existing constrained-generation benchmarks often use fixed constraint types that can be easy for state-of-the-art models. COLLIE specifies compositional constraints across text levels, builds instances from corpora, and evaluates models; GPT-4 performs best but averages only 50.9% constraint satisfaction.

  • Problem

    Existing constrained-generation benchmarks often focus on fixed constraint types that have proved easy for state-of-the-art models such as GPT-4.

    Source: §1 · p. 1

  • Method

    COLLIE uses a grammar to specify compositional constraints across generation levels and automates extracting corpus values, rendering instructions, and evaluating generations.

    Source: §1 · p. 1, §3 · p. 3

  • Results

    GPT-4 performs best among the evaluated models but achieves only 50.9% average constraint satisfaction, with success varying by constraint position.

    Source: §1 · p. 2

  • Takeaways & Limitations

    COLLIE-v1 provides fine-grained insights into model capabilities and shortcomings across 13 constraint types.

    Source: §6 · p. 9

    The representative constraint structures cannot capture all dimensions and nuances of model capabilities.

    Source: §6 · p. 9

Abstract

from arXiv · show

Text generation under constraints have seen increasing interests in natural language processing, especially with the rapidly improving capabilities of large language models. However, existing benchmarks for constrained generation usually focus on fixed constraint types (e.g.,generate a sentence containing certain words) that have proved to be easy for state-of-the-art models like GPT-4. We present COLLIE, a grammar-based framework that allows the specification of rich, compositional constraints with diverse generation levels (word, sentence, paragraph, passage) and modeling challenges (e.g.,language understanding, logical reasoning, counting, semantic planning). We also develop tools for automatic extraction of task instances given a constraint structure and a raw text corpus. Using COLLIE, we compile the COLLIE-v1 dataset with 2080 instances comprising 13 constraint structures. We perform systematic experiments across five state-of-the-art instruction-tuned language models and analyze their performances to reveal shortcomings. COLLIE is designed to be extensible and lightweight, and we hope the community finds it useful to develop more complex constraints and evaluations in the future.

1 Introduction

COLLIE addresses limits in constrained-generation benchmarks by enabling compositional constraints and automating their construction and evaluation. Its initial dataset spans 13 constraint types, and GPT-4 achieves only 50.9% average constraint satisfaction.

1 Introduction

  • Existing constrained-generation benchmarks focus on particular constraint types, require tailored data pipelines, or assess only specific LM capabilities.The paper also notes challenges in constructing datasets for scalable, comprehensive evaluation.Source: §1 · p. 1
  • COLLIE uses a grammar-based framework to specify compositional constraints across generation levels and modeling challenges, then automates value extraction, instruction rendering, and evaluation.The framework is designed to be modular and extensible.Source: §1 · pp. 1–2
  • 50.9% is GPT-4’s average constraint satisfaction rate on COLLIE-v1; success ranges from 100% for beginning a sentence with a specified word to 40%-60% for ending with one.The dataset contains 2,080 instances across 13 constraint types.Source: §1 · p. 2
  • The paper introduces COLLIE, releases COLLIE-v1, and evaluates five state-of-the-art LLMs to provide insights for model and benchmark development.The dataset uses Wikipedia, CC-News, and Project Gutenberg.Source: §1 · p. 2

2 Related Work

Prior work covers controllable generation, grammar-based tests, and scalable benchmarks, but the paper identifies limits in existing datasets’ constraint coverage, linguistic diversity, and construction effort. COLLIE combines compositional constraint design with values drawn from natural-language corpora.

2 Related Work

  • Controllable-generation work steers topics or attributes [17] [9] [20] or constrains lexical inclusion and exclusion [15] [13] [10] [25] [26] [27] [23], but existing datasets may not fully assess modern instruction-tuned LLMs.The authors attribute this difficulty to recent controllability gains from instruction tuning [30].Source: §2 · p. 2
  • Grammar-based benchmarks in question answering [37], instruction following [7] [31], and visual reasoning [18] systematically generate tests, but their synthetic data limits linguistic diversity and practical applicability.COLLIE instead extracts constraint values and examples from natural-language corpora.Source: §2 · p. 2
  • Scalable benchmarks such as HELM [24], BIG-Bench [33], MMLU [14], TaskBench500 [22], and Natural Instructions [36] require considerable human effort and may become obsolete as stronger models emerge.The paper proposes compositionality as a way to build automatic, scalable benchmarks with minimal human effort that can co-evolve with model capabilities.Source: §2 · p. 3

3 COLLIE: A Grammar-based Framework for Constrained Text Generation

COLLIE represents constraints with a grammar spanning text levels and logical compositions of count and position requirements. Its pipeline extracts target values from corpora, renders instructions, and evaluates generated text against the constraints.

3 COLLIE: A Grammar-based Framework for Constrained Text Generation

  • COLLIE’s grammar defines a full constraint specification as a generation level plus a logical composition of one or more base-constraints, which may apply to the whole output or a positioned part.A positioned text can be nested, such as the last sentence of the third paragraph.Source: §3 · p. 3
  • The grammar’s two base-constraint types are counts of strings or units and positional requirements that equate a text part to a specified string.Examples include limiting occurrences of “happy” and requiring the third word to be “happy.”Source: §3 · p. 3
  • The grammar can be extended with base-constraints such as part of speech or sentiment by implementing semantic checks, which the paper leaves to future work.A constraint structure is a family of specifications differing only in values; a constraint is one specification with concrete values.Source: §3 · p. 3
  • COLLIE is intended to support a wide range of constraints through logical compositions of base-constraints across different text levels.Table 1 lists the constraint structures used in COLLIE-v1 with simplified example values.Source: §3 · p. 3, Table 1 · p. 4
  • COLLIE’s structures combine generation levels and challenges: some require counting, positional control, or multi-level counting, while passage-level pass01 requires semantic planning for coherent constrained text.The most conceptually demanding structures combine counting and positional challenges across levels.Source: §3 · p. 4
  • The construction pipeline begins with manually specified constraint structures, then extracts values from corpora, renders natural-language instructions, and evaluates generations against the constraints.Researchers may specify a value range; extracted examples provide natural solutions, and evaluation derives a True/False constraint-satisfaction result.Source: §3 · pp. 4–5

4 Dataset Construction: COLLIE-v1

COLLIE-v1 contains 2,080 constraint instances across 13 structures, built by extracting target values from natural-language corpora. Its structures span word, sentence, paragraph, and passage generation, while validation includes automated filtering and length statistics across sources.

4 Dataset Construction: COLLIE-v1

  • COLLIE-v1 contains 2,080 constraint instances across 13 structures and 1,435 unique prompts, with future releases able to add constraints, metrics, and data sources.Source: §4 · p. 5

4.1 Constraint specification and extraction

  • The 13 structures span word, sentence, paragraph, and passage generation; five are single-level and eight are multi-level, with counting and position challenges.There are 3 word-level, 4 sentence-level, 5 paragraph-level, and 1 passage-level structures.Source: §4.1 · p. 5
  • Targets are sampled from corpora so each constraint-target pair has a natural solution and targets follow corpus-induced plausible distributions.Documents are chunked into strings at the needed level, then target values are extracted or swept over specified ranges.Source: §4.1 · p. 5

4.2 Data sources

  • Wikipedia, CC-News, and Project Gutenberg supply diverse subject matter, news dialogue and current events, and varied styles from different periods, respectively.Each source uses source-specific filtering and cleaning, including copyright filtering for CC-News and Markdown and footnote removal for Gutenberg.Source: §4.2 · pp. 5–6

4.3 Data validation and statistics

  • Validation samples 300 documents per source and up to 100 targets per structure and source; mean passage length is 11.8 paragraphs in Gutenberg versus 6.1 in Wikipedia.Automated filtering removes many strings to prioritize high recall, and paragraph and passage examples for para02, para03, and pass01 receive manual cleaning checks.Source: §4.3 · p. 6, Figure 2 · p. 6
  • Figure 3 presents weighted-average performance and per-constraint satisfaction for five models, while Figure 4 plots average pass rates across k using 20 samples per prompt.Source: Figure 3 · p. 7, Figure 4 · p. 7

5 Results & Analysis

The section evaluates five language models in zero-shot constrained generation and analyzes how performance varies across constraints. GPT-4 leads the comparison, but results show substantial difficulty with position, counting, and compositional constraints.

5.1 Main results

  • GPT-4 consistently outperformed the other models in zero-shot generation, achieving more than twice the constraint satisfaction rate of non-GPT models.The performance ordering was GPT-4, then GPT-3.5 and PaLM, followed closely by Vicuna-7B and Alpaca-7B.Source: §5.1 · p. 7
  • Simple existence constraints posed little challenge to most models, while longer generations with more counting or position constraints were only partially solved by GPT-4.For word03, para04, para05, and pass01, GPT-4 achieved satisfaction rates between 40% and 70%; other models had no satisfactory performance.Source: §5.1 · p. 8
  • GPT-4’s average pass@20 exceeded 63%, versus 32% for GPT-3.5, while arbitrary-position and mixed-count tasks remained challenging across models.The passage notes GPT-4’s 63% constraint satisfaction rate is far from perfect.Source: §5.1 · p. 8

5.2 Analysis

  • Model performance on a given constraint structure was highly consistent across data sources, suggesting constraint logic mattered more than target values or data distribution.The passage identifies this pattern across all models and points to Figure 8(g).Source: §5.2 · p. 8
  • Models generally handled first-position constraints well, but all struggled at arbitrary positions and only GPT-4 showed partial success with last-position constraints.The position effect was less sensitive to constraint level.Source: §5.2 · p. 8
  • Counting characters within a word was easier than counting characters within a sentence, and exact-equality tasks were harder than range or single-bound tasks.The comparisons include sent01 for exact equality, para03 for a range, and word01, sent03, and para04 for upper or lower bounds.Source: §5.2 · pp. 8–9
  • Adding a logical constraint made sent03 harder than word01, despite the added sentence-level requirement being expected to be manageable.The reported performance on sent03 uniformly trails performance on word01.Source: §5.2 · p. 9
  • GPT-4 improved by 20% after the second feedback round but plateaued at 66% after three additional rounds, with gains varying by task.Word03’s reported satisfaction rate increased from 62.1% to 10%; word02, sent01, and sent02 remained challenging.Source: §5.2 · p. 9

6 Conclusion

The conclusion presents COLLIE as a framework for constructing constrained-generation datasets and reports its use in evaluating models with COLLIE-v1. The accompanying appendices describe instruction rendering and feedback tools, while noting limits in automated processing and benchmark coverage.

6 Conclusion

  • COLLIE-v1 contains 1,132 constraints from 13 types and three data sources, and evaluation of five LLMs provides fine-grained insights into model capabilities and shortcomings.The authors frame COLLIE as a grammar-based framework that automates extraction of constraint values, instruction rendering, and performance assessment.Source: §6 · p. 9
  • Representative constraint structures cannot capture all dimensions and nuances of model capabilities, and automated filtering and processing can affect downstream constraint quality.The authors plan additional utilities such as grammar checkers and parsers to mitigate processing effects.Source: §6 · p. 9

A.1 Instruction rendering

  • The instruction renderer parses constraints as trees, renders prompts for each base constraint, and concatenates them for multi-constraints.Language models are offered as an option to improve fluency in edge cases without changing the original meaning.Source: §A.1 · p. 13

A.2 Feedback rendering

  • The feedback renderer can state both the target counts and the generated counts when a generation misses its constraint.Language models can optionally polish the rule-based feedback.Source: §A.2 · p. 13

B Dataset

The extraction pipeline loads and segments documents, filters and cleans text, constructs passages where needed, and extracts constraint targets. Source-specific sampling, filtering, and post-processing keep the resulting examples manageable and adapted to each corpus.

B.1 Extraction overview

  • Extraction loads documents, chunks and filters text, post-processes surviving strings, constructs passages when needed, and extracts constraint targets.For passage-level constraints, consecutive surviving paragraphs are joined; an excluded paragraph separates passages.Source: §B.1 · pp. 13–14
  • For each source, the pipeline samples 300 documents, up to 100 sequences per document, and up to 100 targets per constraint structure and source.The sampling is intended to prevent over-representation from very long documents.Source: §B.1 · p. 14

B.2 Text filters

  • Text filters are source-specific and remove URL-like strings, all-caps text, strings without qualifying sentences, copyright statements, and caption-like strings.The no-sentences heuristic keeps a string if at least one tokenized sentence contains a period and exceeds two characters.Source: §B.2 · p. 14

B.3 Text post-processing

  • Source-specific post-processing removes Markdown artifacts and extra whitespace, replaces single newlines with spaces, and removes bracketed text.The bracket-removal step is described as useful for removing references inside text.Source: §B.3 · pp. 14–15

B.4 Data sources

The dataset draws on three text corpora and a separate word list, with source-specific splitting, filtering, and post-processing. The authors report that the datasets contain no PII to their knowledge.

B.4 Data sources

  • The authors report that none of the datasets used contains PII, as far as they are aware.Figures 9 and 10 provide statistics on extracted constraints for individual and grouped data sources, respectively.Source: §B.4 · p. 15
  • Wiki uses the 20220301.en training split, paragraph splitting on two newlines, three filters, and Wiki-specific removal of text preceding a newline.For passage-level constraints, text containing the vertical-line character is also omitted; the source is licensed under CC BY-SA 3.0 and the GNU Free Documentation License.Source: §B.4 · p. 15
  • CC-News uses its Huggingface Datasets training split, splits documents on single newlines, applies four filters, and uses no post-processing.The passage describes the filters as copyright, URL, caption, and no-sentences filters.Source: §B.4 · p. 15
  • Gutenberg uses the processed Gutenberg, dammit dataset, splits on two newlines, applies two filters, and then runs four post-processors in sequence.The post-processors remove Markdown and brackets, replace single newlines with spaces, and collapse consecutive whitespace; documents are in the U.S. public domain.Source: §B.4 · p. 15
  • Word-level constraints draw from a newline-separated English word list without filtering or post-processing, and the authors say the extraction code will be released under an MIT license.Source: §B.4 · p. 15

C Additional Experimental Results

Two of 1,132 constraints were blocked by the PaLM-2 API for guardrailing reasons. The examples involve sentence- and passage-level prompts.

C Additional Experimental Results

  • 2 of 1,132 constraint prompts were blocked by the PaLM-2 API for guardrailing reasons.Source: C Additional Experimental Results · p. 15
  • One blocked prompt asked for a sentence containing three specified words in the CC-News c07 instance.Source: C Additional Experimental Results · p. 15
  • In ccnews_c14, the prompt requested a passage whose paragraphs each ended with one of four specified sentences, including a 6.6 percent yield claim and a ∼15 percent share slump in 2018.It also instructed the model to generate only the passage, without extras such as “Paragraph 1” or “Answer:”.Source: C Additional Experimental Results · p. 15

C.1 One-shot experiments

A preliminary one-shot experiment compared zero-shot prompting with prompts preceded by a fixed constraint and corpus example. Results were very similar for GPT-3.5 and GPT-4, suggesting that generation under constraints, rather than understanding them from examples, was the main difficulty.

C.1 One-shot experiments

  • One-shot results were very similar to zero-shot results for both GPT-3.5 and GPT-4.Table 2 reports 0- versus 1-shot results across all constraints.Source: §C.1 · p. 16, Table 2 · p. 16
  • The experiment prepended a fixed constraint and its corpus example as an input-output pair for each constraint structure.It used an internal dataset version that preceded COLLIE-v1.Source: §C.1 · pp. 15–16
  • The authors interpret the similar results as suggesting that task difficulty lies mainly in generation under constraints rather than understanding constraints from similar examples.Source: §C.1 · p. 16

C.2 Constraint satisfaction rates

Constraint satisfaction remains consistently high for particular constraint structures across data sources, with performance compared across models, constraints, and datasets. The section also reports dataset sizes and model-specific generation sample sizes.

C.2 Constraint satisfaction rates

  • Constraint satisfaction rates remain consistently high for a given constraint structure regardless of data source, across the models and datasets compared.Figure 8(f)–(g) summarize performance by constraint group and by constraint and dataset.Source: §C.2 · p. 16, Figure 8 · p. 16
  • GPT-4 and GPT-3.5 had 20 trials per prompt, PaLM had 30, and Vicuna-7B and Alpaca-7B had 10 each.Generation failures meant some model sample totals were not multiples of the number of prompts.Source: §C.2 · p. 16, Figure 9 · p. 17, Figure 10 · p. 17

C.3 Additional evaluations

Additional evaluations examine word validity and passage coherence alongside constraint satisfaction. GPT-4 and GPT-3.5 outperform other models on passage coherence, while word-level results show that satisfying constraints can coincide with fabricated words.

Word validity evaluation

  • Word validity is checked against an English vocabulary list supplemented with eight uncommon valid words because the list is incomplete.The listed additions include “supercalifragilisticexpialidocious” and “procrastinatrix.”Source: §C.3 · p. 17
  • GPT-4 performs best on long-word generation, but for the i-th-letter constraint all models fail while GPT-4 fabricates words and GPT-3.5 produces valid ones.For the last-character constraint, only GPT-4 satisfies the constraint, but it frequently generates made-up words; other models produce no valid satisfying words or strings.Source: §C.3 · p. 18

Passage coherence evaluation

  • Passage coherence is scored by GPT-4 on a 1–10 scale, using three samplings per generated text and averaging the scores.The figure caption notes that roughly one trial was taken per model and error bars show standard error across the dataset.Source: §C.3 · p. 18, Figure 12 · p. 18
  • GPT-4 and GPT-3.5 outperform the other models in passage coherence, with GPT-4 comparable to ground-truth passages.GPT-4 scores each generated passage in three independent runs; Figure 12 reports average scores and standard errors.Source: §C.3 · p. 18, Figure 12 · p. 18

Related papers

Loading 2307.08689v1…