PaperlayerView exact revision on arXiv

Source-linked AI summary

Where vs What: Decomposing Structural and Content Failures in LLM-Generated Structured Outputs

Yiwei Zhang, Chengke Wu, Li Wang, Jianqiang Li

arXiv:2608.25358v1cs.AI

TL;DR

Existing evaluations conflate structural placement errors with value errors in structured outputs. The paper introduces SCD to separate these dimensions, finds that structure degrades first across nested JSON and tables, and uses SCD-derived rewards in SA-RLVR to improve structural placement.

  • Problem

    Existing evaluations treat structured-generation failures monolithically, obscuring the distinction between value correctness and structural placement.

  • Method

    SCD independently measures structural fidelity and content accuracy across nested JSON and table tasks, while SA-RLVR converts SCD metrics into verifiable GRPO rewards.

  • Results

    Structural fidelity degrades earlier and faster than content accuracy as complexity grows; SA-RLVR lifts JSON Value Placement Accuracy from 0.26 to 0.63 versus 0.28 for matched SFT.

  • Takeaways & Limitations

    Structural addressing is an independent capability that can be evaluated and optimized directly, with improvements demonstrated across JSON and table topologies.

  • Takeaways & Limitations

    The controlled synthetic tasks limit ecological coverage, and SCD is most suitable where target values and intended structural positions can be programmatically verified.

Abstract

from arXiv · show

Structured outputs such as JSON and tables are central to modern LLM-based systems, yet generation failures are evaluated monolithically, conflating two distinct error modes: placement errors (correct values at wrong positions) and value errors (wrong values at intended positions). We introduce Structure-Content Decomposition (SCD), a framework that independently measures structural fidelity and content accuracy. Applying SCD to nested JSON and table tasks across six models (7B to frontier), we uncover a consistent phenomenon: structural fidelity degrades earlier and more sharply than content accuracy as complexity increases. At the highest complexity, even DeepSeek-V4-Flash (with reasoning) misplaces 35% of recalled values, while Qwen2.5-7B misplaces 74%. Controlled ablations suggest that this pattern is associated with reliance on semantic shortcuts rather than topological understanding of output structure. Based on these findings, we propose SA-RLVR, converting SCD metrics into verifiable rewards for reinforcement learning via GRPO. SA-RLVR successfully optimizes structural addressing across distinct topologies: it lifts JSON Value Placement Accuracy (VPA) from 26% to 63% while generalizing to held-out schemas; moreover, it consistently drives VPA improvements in the table domain, demonstrating that structure-aware rewards can directly enhance multi-domain structural positioning.

1 Introduction

Structured outputs require both correct values and correct structural placement, but existing evaluations conflate these failure modes. SCD separates them and reveals that structural fidelity degrades earlier and faster than content accuracy as complexity increases.

  • Structured outputs such as JSON and tables are critical to LLM workflows because both value correctness and structural placement matter.
  • Existing holistic metrics cannot distinguish incorrect values from correctly recalled values placed at wrong structural positions.This distinction matters because the two failure modes imply different underlying deficiencies and interventions.
  • SCD separates structural fidelity from content accuracy across a three-level evaluation framework.It is applied to nested JSON and table tasks across six models ranging from 7B-scale to full-scale systems.
  • Structural fidelity degrades earlier and faster than content accuracy across two topologies, two task paradigms, and six models.Controlled ablations associate this pattern with reliance on semantic shortcuts rather than robust topological addressing.
  • SA-RLVR raises JSON Value Placement Accuracy from 0.26 to 0.63, compared with 0.28 for a matched SFT baseline, while generalizing across schemas.The method converts SCD metrics into verifiable rewards for GRPO.

2 Related Work

Prior work addresses positional reasoning, structured-output generation, and verifiable-reward reinforcement learning, but commonly evaluates structured outputs with holistic or format-level criteria.

  • Prior studies document positional sensitivity in long-context retrieval and the need to reason jointly over table coordinates and content.
  • Constrained decoding methods guarantee format validity but cannot ensure correct value placement.In SCD terms, format validity corresponds to Level 1, whereas placement is evaluated at Levels 2–3.
  • Existing structured-output benchmarks typically report binary schema conformance, function-call exact match, or end-to-end execution outcomes.
  • Existing RLVR approaches use holistic rewards that conflate structural and value-level errors, while preference methods lack structure-specific gradients.

3 Structure-Content Decomposition

SCD decomposes structured-output quality into hierarchical format, structural-path, and value-placement checks, with diagnostic metrics separating recalled values from misplaced values across JSON and tables.

  • SCD mirrors generation through three levels: valid format, required structural paths, and correct values at those paths.
  • Format Validity checks whether the output is parseable and well-formed; failed outputs do not proceed to later levels.
  • Schema Compliance Rate compares required schema paths with actual output leaf paths, while Value Placement Accuracy checks planted values at their intended paths.
  • Value Presence measures whether planted values appear anywhere, enabling Structural Compliance Gap and Displacement Rate to diagnose misplaced recalled values.DR is defined as 1 − VPA/VP.
  • SCD uses tree leaf paths for nested JSON and row-column cells for tables, preserving the same hierarchy across distinct topologies.Tables test locating and modifying existing structure rather than generating structure from scratch.
  • The framework’s dual-domain design tests whether structure-first degradation generalizes beyond a single topology or task paradigm.

4 Experimental Setup

The experiments use deterministic, algorithmically generated JSON and table tasks with controlled complexity, enabling independent measurement of structural addressing and content accuracy across six models.

  • Controlled tasks use deterministic ground truth, uniquely identifiable planted values, and systematically varied complexity to isolate structural addressing from content accuracy.
  • JSON complexity varies recursive depth while holding binary-tree topology constant: S has depth=2, M depth=3, and L depth=4.The levels contain 12, 15, and 20 planted values respectively, with 70% targeting deep paths at M and L.
  • Table complexity increases through larger layouts, repeated or ambiguous structures, and greater target-cell distance from the table origin.
  • The table dataset contains 4,490 unique instances across 1,030 layout structures, synthesized from 53 real-world seed templates.
  • Six models span frontier closed-source systems, large MoE models, reasoning models, and Qwen models down to 7B scale.
  • As JSON complexity rises from S to L, VP remains high while VPA drops sharply; at L, DR reaches 35% for DeepSeek-V4-Flash and 74% for Qwen2.5-7B.

5 Results: Structure Degrades First

Across nested JSON and tables, structural placement degrades earlier than value preservation as complexity increases. Ablations associate this gap with recursive depth, positional distance, semantic cues, and field-name disambiguation.

  • 5 Results: Structure Degrades First: As JSON complexity rises from S to L, VP remains high while VPA drops sharply, producing a scissors pattern of correct values at wrong positions.At L-level, DeepSeek-V4-Flash misplaces 35% of recalled values, versus 74% for Qwen2.5-7B.
  • 5.1–5.2 Ablations: Increasing recursive depth raises Qwen2.5-7B displacement from near-zero to 28.5%, identifying nesting as a major structural-failure driver.Changing planted-value count at fixed JSON complexity produces no monotonic DR trend.
  • 5.3 Cross-Domain Consistency: At L-level table complexity, strong models preserve the JSON-like VP–VPA gap, while weaker models also lose value accuracy.DR ranges from 17–28% for strong models to 98% for Qwen2.5-7B.
  • 5.2 Table Ablations: In tables, increasing target-row distance raises DR from 35.8% to 44.6% while VP remains above 0.80, linking positional distance to placement failure.Unlike JSON, increasing table target count also raises DR, from 14.7% at n=5 to 23.6% at n=15.
  • 5.4 Why Does Structure Degrade First?: Replacing descriptive field names with opaque navigation and repeating names both increase displacement while leaving VP stable.DR rises by 7.5 percentage points with opaque cues and by 8.2 points with repeated names.
  • 5.4 Why Does Structure Degrade First?: A composite heuristic of semantic matching, positional approximation, and name disambiguation may preserve values while failing to bind them to recursive structural addresses.Recursive depth weakens these cues by multiplying structurally distinct positions with similar contexts.

6 SA-RLVR: Structure-Aware RL

SA-RLVR converts decomposed structural metrics into verifiable GRPO rewards and tests online exploration against imitation. It substantially improves JSON placement and generalization, while table transfer mainly improves formatting rather than coordinate placement.

  • 6.1 Method: SA-RLVR converts SCD metrics into deterministic, verifiable rewards and applies GRPO to improve structural placement beyond a matched imitation baseline.The reward emphasizes VPA and retains schema-level feedback while excluding VP, which ignores position.
  • 6.2 Results: Evaluation spans JSON-ID, two JSON OOD splits, Table-ID, and Table-OOD across in-distribution and out-of-distribution conditions.Figure 4 reports VPA growth beyond the SFT ceiling over 500 steps and outperformance on JSON-ID and both OOD splits.
  • 6.2 Results: On JSON-ID, VPA improves from 0.264 for Base to 0.629 with SA-RLVR, while VP rises from 0.310 to 0.869.JSON OOD gains are larger, including +247% on OOD-Eco.
  • 6.2 Results: SFT reaches only 0.281 VPA, whereas SA-RLVR substantially exceeds it through online reward-guided exploration.The matched SFT baseline isolates imitation from exploration using the same evaluation metrics for selection.
  • 6.2 Results: SA-RLVR improves table format validity from 26.5% to 85.5%, but coordinate-level VPA remains approximately 0.06–0.09.Mixed-domain training therefore transfers strongly to formatting but not substantially to grid-coordinate placement at 7B scale.
  • 6.3 Reward Ablation: All RL variants outperform SFT at VPA 0.58–0.63 versus 0.28, while the composite reward best balances placement and schema compliance.The composite reward preserves VPA while improving SCR over VPA-only.

7 Conclusion

SCD shows that structural addressing is an independent capability: as complexity grows, models preserve correct values but increasingly place them incorrectly, and SA-RLVR improves placement accuracy.

  • SCD reveals a consistent “structure degrades first” pattern across nested JSON and table tasks.
  • Displacement rates exceed 24% for frontier models and 70% for smaller models as complexity increases.
  • Controlled ablations associate these failures with semantic shortcuts rather than topological addressing.
  • SA-RLVR lifts Value Placement Accuracy from 0.26 to 0.63, while matched SFT reaches only 0.28.
  • The findings support evaluating and optimizing structural addressing directly rather than treating it as a byproduct of content generation.

Limitations

The study’s conclusions are bounded by synthetic, programmatically verifiable tasks, limited table templates, known target positions, and small-scale training with restricted transfer evidence.

  • Controlled synthetic tasks provide deterministic ground truth but may not capture noisy instructions, underspecified schemas, equivalent outputs, or downstream execution criteria.
  • Table conclusions require caution because the table domain is synthesized from a limited set of real-world templates.
  • SCD is most suitable when target values and intended structural positions are known and correctness can be programmatically verified.
  • SA-RLVR’s metric-derived rewards may encourage metric-specific behavior rather than fully general structural understanding.
  • Training evaluates SA-RLVR only on Qwen2.5-7B-Instruct with LoRA, while larger-scale behavior is inferred rather than directly trained.
  • The JSON-dominated training mixture shows limited cross-topology transfer, with Table-OOD VPA remaining near baseline.

D Detailed Experimental Results

The supplementary experiments report full SCD metrics, additional JSON model results, table-domain results, and ablations that extend the main diagnostic and training analyses.

  • The appendix reports full diagnostic metrics, including L-level JSON results, additional JSON pilots, and complete table-domain results.
  • The L-level JSON results include full SCD metrics and identify models using explicit thinking or reasoning mode.
  • Additional ablations cover both JSON and table settings discussed in the main text.
  • Table fill-row ablations show that increasing row distance and target count increases displacement while VP remains substantially higher than VPA for DeepSeek-V3.

F SA-RLVR Training Details

SA-RLVR trains Qwen2.5-7B-Instruct with LoRA using SCD-derived online verifiable rewards, emphasizing exact placement while retaining schema compliance.

  • SA-RLVR trains Qwen2.5-7B-Instruct with LoRA using SCD metrics as online verifiable rewards.
  • The reward emphasizes exact value placement while preserving a schema-compliance signal.
  • VP is excluded because emitting correct values at wrong locations can optimize it without achieving correct placement.
  • The training mixture contains approximately 3,400 prompts spanning synthetic JSON, real-schema JSON, and table prompts.
  • The matched SFT baseline uses 2,907 best-of-10 examples selected by the same SCD scoring pipeline and the same LoRA configuration.

G Qualitative Failure Cases

The qualitative cases illustrate placement failures in both nested JSON and tables, where recalled values appear at incorrect structural positions. These examples contrast intended paths or cells with displaced output values.

  • JSON sibling-path displacement: JSON sibling-path displacement places the value intended for the left metadata code under the right sibling.The target paths distinguish left.metadata.code = "ALPHA" from right.metadata.code = "BRAVO", while the output excerpt shows "BRAVO" under left.
  • JSON sibling-path displacement: The JSON example preserves the recalled value but assigns it to the wrong sibling path.The output excerpt contains the value "BRAVO" within the left object rather than its specified right-object location.
  • Table cell displacement: Table cell displacement within the correct row moves the target value into the Name cell instead of the Status cell.The target row places Ada under Name and Pending under Status; the target modification is Status -> "ALPHA", but the output row places "ALPHA" under Name.
Loading 2608.25358v1…