Source-linked AI summary
JSONSchemaBench: A Rigorous Benchmark of Structured Outputs for Language Models
Saibo Geng, Hudson Cooper, Michał Moskal, Samuel Jenkins, Julian Berman, Nathan Ranchin, Robert West, Eric Horvitz, Harsha Nori
TL;DR
Structured-output generation needs systematic evaluation beyond compliance guarantees, especially across efficiency, schema-feature coverage, and task quality. This paper introduces JSONSchemaBench and evaluates six constrained-decoding frameworks with complementary real-world and official test-suite data, finding differentiated framework capabilities and improved downstream performance under constrained decoding.
Problem
Existing evaluations provide limited systematic evidence about constrained-decoding effectiveness across frameworks, realistic schemas, efficiency, coverage, and output quality.
Method
The paper introduces JSONSchemaBench with 10K real-world JSON schemas and evaluates six frameworks using efficiency, coverage, and quality metrics alongside the official JSON Schema Test Suite.
Results
Guidance consistently achieves the best downstream-task performance, with approximately a 3% improvement over the LM-only approach in every task.
Takeaways & Limitations
The benchmark and evaluation framework provide comparisons intended to inform structured-generation research and help identify effective constrained-decoding tools.
Takeaways & Limitations
Empirical coverage approximates true coverage using only finitely many experimental outputs, while the benchmark excludes GitHub-Trivial and GitHub-Ultra from experiments.
Abstract
from arXiv · showhide
Reliably generating structured outputs has become a critical capability for modern language model (LM) applications. Constrained decoding has emerged as the dominant technology across sectors for enforcing structured outputs during generation. Despite its growing adoption, little has been done with the systematic evaluation of the behaviors and performance of constrained decoding. Constrained decoding frameworks have standardized around JSON Schema as a structured data format, with most uses guaranteeing constraint compliance given a schema. However, there is poor understanding of the effectiveness of the methods in practice. We present an evaluation framework to assess constrained decoding approaches across three critical dimensions: efficiency in generating constraint-compliant outputs, coverage of diverse constraint types, and quality of the generated outputs. To facilitate this evaluation, we introduce JSONSchemaBench, a benchmark for constrained decoding comprising 10K real-world JSON schemas that encompass a wide range of constraints with varying complexity. We pair the benchmark with the existing official JSON Schema Test Suite and evaluate six state-of-the-art constrained decoding frameworks, including Guidance, Outlines, Llamacpp, XGrammar, OpenAI, and Gemini. Through extensive experiments, we gain insights into the capabilities and limitations of constrained decoding on structured generation with real-world JSON schemas. Our work provides actionable insights for improving constrained decoding frameworks and structured generation tasks, setting a new standard for evaluating constrained decoding and structured generation. We release JSONSchemaBench at https://github.com/guidance-ai/jsonschemabench
1 Introduction
The paper studies whether constrained decoding is efficient, broadly supports JSON Schema features, and preserves semantic quality, then introduces a multidimensional benchmark to evaluate these questions.
- Motivation: Structured-output applications require language models to produce machine-consumable results that obey predefined formats and constraints.Constrained decoding masks invalid tokens during generation so outputs conform to a specified structure.
- Motivation: JSON Schema has become a standard specification language for structured-output constraints across applications and commercial language-model APIs.It provides a domain-specific way to define constraints for JSON data.
- Research Questions: The evaluation addresses efficiency, coverage, and quality: generation speed, support for JSON Schema features, and effects on downstream task accuracy.The paper explicitly asks whether constrained decoding slows generation, supports the specification’s expanding features, and harms semantic quality.
- Benchmark: JSONSchemaBench contains 10K real-world JSON schemas organized across datasets with varying complexity and domains.The benchmark is paired with the official JSON Schema Test Suite for detailed feature-coverage analysis.
- Contributions: The study evaluates six constrained-decoding frameworks and proposes a fine-grained assessment of schema compliance and downstream-task effects.The frameworks include Guidance, Outlines, Llamacpp, XGrammar, OpenAI, and Gemini.
2 Background and Related Work
The background motivates systematic comparisons of constrained-decoding frameworks and introduces constrained decoding as token masking guided by formal output constraints.
- JSON Schema: JSON Schema describes JSON structure through composable constraints on properties, arrays, strings, and other data features.Its specification defines keywords that can be combined to enforce diverse constraints.
- Constrained Decoding: Constrained decoding masks tokens that violate predefined constraints at each generation step.The included algorithm updates the constraint state, computes a mask, filters model logits, and appends sampled tokens until EOS.
- Evaluation Gap: Prior evaluation studies did not compare multiple constrained-decoding frameworks using broad real-world benchmarks.Existing benchmarks focused on specific tasks or artificial formal-grammar settings with unclear real-world relevance.
3 The JSONSchemaBench
JSONSchemaBench is designed as a diverse, large, and multidimensional benchmark built from real-world and test-suite schemas spanning varied complexity.
- Design Goals: The benchmark goal is to cover common real-world constraint types with enough scale and fair multidimensional metrics for reliable evaluation.The design explicitly targets diversity, size, and comprehensive assessment.
- Data Collection: The schema collection combines GitHub data, the official JSON Schema Test Suite, JSON Schema Store, a function-calling dataset, and Kubernetes configurations.Invalid schemas are filtered and schemas are standardized to their declared JSON Schema versions.
- Data Organization: GitHub schemas are divided into trivial, small, medium, large, and ultra collections according to schema size.This organization supports finer-grained evaluation across complexity levels.
- Experimental Scope: GitHub-Trivial and GitHub-Ultra are retained in the benchmark but excluded from experiments because they were considered too easy or too hard.GitHub-Ultra is retained as an aspirational target for future advances.
4 Efficiency
The efficiency evaluation measures compilation and generation latency under controlled inference settings, while addressing coverage bias through a shared-instance comparison.
- Metrics: Efficiency is measured with Grammar Compilation Time, Time to First Token, and Time per Output Token.These metrics separate compilation overhead, initial latency, and steady-state token-generation cost.
- Setup: Experiments use Llama-3.1-8B-Instruct with specified inference backends on one NVIDIA A100-SXM4-80GB GPU and a 12-core AMD EPYC 7543 CPU.Batch size is one for all experiments.
- Fairness: Efficiency metrics are computed on the intersection of instances covered by all engines to reduce bias from differing schema coverage.Averaging only covered instances could favor engines that process simpler and shorter schemas.
- Compilation Results: Guidance and Llamacpp have minimal compilation time because they compute constraints dynamically, whereas Outlines has significantly higher compilation time.XGrammar’s compilation overhead is partly mitigated by concurrent execution with prompt prefilling.
- Generation Results: Guidance achieves higher output-token efficiency through guidance acceleration, while Outlines and Llamacpp have lower throughput than the LM-only approach.With the Transformers backend, Guidance has significantly better TPOT than XGrammar.
5 Coverage
The paper evaluates constrained-decoding coverage using complementary real-world schemas and official JSON Schema tests, distinguishing whether frameworks accept schemas, generate compliant outputs, and precisely implement schema semantics. Guidance generally leads coverage and reliability measures, but empirical coverage depends on the model and sampling setup, while test-suite performance does not directly map to real-world coverage.
- Coverage Definitions: The evaluation defines declared, empirical, and true coverage to separate schema processing, observed compliant generation, and semantic equivalence to the original schema.True coverage is difficult to measure because schemas admit infinitely many JSON instances; empirical coverage is therefore an approximation.
- Coverage Definitions: Compliance Rate = CEmpirical/CDeclared estimates how reliably a framework guarantees compliance for schemas it accepts.Declared coverage is an upper bound for both empirical and true coverage.
- JSONSchemaBench Results: Guidance has the highest empirical coverage on six of eight datasets, while Llamacpp leads on Washington Post and JSON Schema Store.Closed-source engines rank last on all but one dataset, and LM-only approaches drop on harder and domain-specific datasets.
- JSON Schema Test Suite Results: Guidance achieves full coverage on 13 test-suite categories and moderate coverage on 21, exceeding Llamacpp, XGrammar, and Outlines at these thresholds.Guidance has the single highest coverage in 19 categories, compared with 10 for XGrammar, one for Outlines, and none for Llamacpp.
- JSON Schema Test Suite Results: Guidance has the fewest total failures and especially minimizes under-constrained errors, whereas XGrammar minimizes compilation errors but has the most under-constrained failures.This indicates a trade-off between compilation robustness and permissiveness across frameworks.
- Caveats: Test-suite performance does not straightforwardly correspond to empirical coverage because feature prevalence differs across real-world schemas and under-constraining can delegate validation to the language model.Empirical coverage is also influenced by the language model and sampling methods used in the experiments.
6 Quality
The quality experiment examines whether constrained decoding affects reasoning-task accuracy, using three tasks with structured JSON outputs. Across frameworks, constrained decoding improves performance over unconstrained generation, with Guidance leading consistently.
- 6.1 Setup: The experiment evaluates constrained decoding’s effect on accuracy across the Last Letter, Shuffle Objects, and GSM8K reasoning tasks.All outputs use JSON objects containing “reasoning” and “answer” fields.
- 6.1 Setup: The study uses Llama-3.1-8B-Instruct with the original task setup and prompts from prior work.
- 6.2 Results: Constrained decoding achieves higher performance than unconstrained generation regardless of the evaluated framework.
- 6.2 Results: Guidance performs best across all tasks, improving approximately 3% over the LM-only approach on each task.The paper attributes this result potentially to Guidance’s token-healing implementation.
7 Conclusion
The conclusion presents JSONSchemaBench and a three-dimensional evaluation framework as resources for assessing constrained decoding under realistic conditions. The benchmark contains 10K real-world schemas drawn from diverse applications and complexity levels.
- 7 Conclusion: The paper proposes an evaluation framework covering efficiency, schema-feature coverage, and output quality.
- 7 Conclusion: JSONSchemaBench comprises 10K real-world JSON schemas organized into datasets spanning varied complexity and diversity.
- 7 Conclusion: The benchmark includes schemas from open-source repositories, analytics frameworks, container orchestration, publishing, function calling, and a schema store.
A.1 Data Processing
The data-processing pipeline validates, cleans, deduplicates, and extracts schemas before measuring feature and format distributions. It uses multiple checks to improve dataset validity and diversity.
- A.1 Data Processing: Schemas are validated against JSON Schema Draft2020-12 and additional Rust and JavaScript validators, with invalid schemas removed.
- A.1 Data Processing: Duplicate schemas are removed while ignoring key ordering when determining redundancy.
- A.1 Data Processing: The pipeline excludes empty schemas, unresolved external references, mismatched or missing draft versions, and unrelated fields.
- A.1 Data Processing: Regular-expression escaping issues are fixed, and schemas embedded below the JSON-file root are extracted.
- A.1 Data Processing: The benchmark counts feature keywords across 10K schemas and separately plots format-keyword usage, including date-time, email, and URI formats.
B Coverage Experiment Details
The coverage experiment estimates framework support through theoretical and empirical measures, using documented feature support and generated-output validation. Its setup fixes the model, decoding procedure, validator, and timeout conditions.
- B Coverage Experiment Details: The coverage prompt uses a simple instruction with two-shot examples, as shown in Figure 3.
- B Coverage Experiment Details: Greedy zero-temperature decoding produces one output per schema, making empirical coverage a Top 1 Empirical Coverage measure.
- B Coverage Experiment Details: Generated JSON is validated with Python’s jsonschema library under Draft2020-12 with string-format checks enabled.
- B Coverage Experiment Details: Theoretical coverage measures the proportion of schemas whose features are supported by the grammar engine and serves as an upper bound on true coverage.
- B Coverage Experiment Details: Theoretical support is summarized by feature in Figure 4 and by dataset in Table 12.
D JSON Schema Test Suite Experiment Details
The JSON Schema Test Suite experiment measures whether frameworks accept every valid instance and reject every invalid instance, while adapting the harness to each engine’s available interfaces. The analysis removes unsupported external-resource and format tests, leaving 43 of 45 categories.
- A framework passes a test case only when it permits every valid instance and prevents every invalid instance.
- The format category is removed because the JSON Schema standard ignores format by default, while optional format tests are reserved for future work.
- 43 of the original 45 test categories remain after dropping tests requiring remotely hosted schemas.
- The evaluation tokenizes each JSON instance and advances the framework’s constraints one token at a time to simulate generation.
- The harnesses use native token-mask interfaces where available and adapt underlying libraries when frameworks do not expose those interfaces publicly.
E Efficiency Experiment Details
Efficiency is measured under a controlled setup that accounts for model and vocabulary size, compilation, inference, and caching. Reported tables summarize median timing metrics for LlamaCpp and Hugging Face Transformers backends.
- The efficiency experiments use Llama-3.1-8B-Instruct with a 128K-token vocabulary to balance computational efficiency and model capability.
- Grammar compilation is not cached because every dataset schema is unique, whereas prefix caching is used during inference when applicable.
- Table 13 reports median GCT, TTFT, TPOT, TGT, and FF metrics for engines using LlamaCpp as the inference engine.
- Table 14 reports median efficiency metrics for engines using Hugging Face Transformers as the inference engine.
F Quality Experiment Details
The quality experiments standardize prompts and JSON schemas across tasks, validate generated JSON against Draft 2020-12 schemas, and measure structured-generation timing. Results indicate that engine strengths are not uniform across GSM8K instances.
- Shuffle Objects and GSM8K use the same prompt and JSON schema, while Last Letter receives a corrected prompt and JSON formatting.
- GSM8K responses use JSON fields for reasoning and the final answer.
- No single engine outperforms the others across all GSM8K instances because each engine has non-empty exclusive regions of correct instances.
- The generation pipeline separates grammar compilation from engine calling and records grammar compilation, time-to-first-token, and total generation time.
- Generated JSON is validated against the schema with the jsonschema library using Draft 2020-12 and enabled format checking.