Source-linked AI summary
Parser States Already Know: Structure-Conditioned KV Persistence for Structured Generation
Linze Wu, Xinrui Chen
TL;DR
Structured generation needs reliable machine-executable outputs, but existing KV compression largely ignores parser-derived structural responsibility. PASK converts parser structure into layer-group-specific persistence decisions using calibrated task-risk floors and distortion-based allocation, achieving stronger compressed inference results and serving efficiency.
Problem
Existing KV compression uses model-side importance signals but leaves task-relevant parser structure and structured-generation risk largely unused.
Method
PASK uses parser transitions and layer groups to assign persistence actions, with offline task-error calibration setting protection floors and attention-output distortion allocating residual capacity.
Results
17.39 percentage points: PASK outperforms the strongest compressed baseline on average across eight BFCL subcategories at total KV budget 0.33 on Qwen3-4B.
Takeaways & Limitations
PASK improves the reliability–efficiency trade-off for compressed structured generation while enabling lightweight structure-conditioned online decisions.
Takeaways & Limitations
PASK requires one-time offline calibration, and its current implementation uses three layer groups and three discrete persistence actions.
Abstract
from arXiv · showhide
Structured generation underpins large language model (LLM) agents that produce JSON, SQL, and function calls, where a single wrong field can cause the downstream action to fail. Constrained decoding already tracks parser transitions to enforce formal validity, and these transitions expose how generated tokens participate in schema-critical decisions such as required fields, arguments, and structural boundaries under the active grammar. Existing KV compression largely leaves this task-relevant structural signal unused. We introduce PASK (Parser-Aware Structural KV Persistence), which turns parser-derived structure into layer-group-specific KV persistence decisions. PASK addresses the mismatch between model-side KV sensitivity and task-level structured risk by using task-error sensitivity to set minimum protection floors and attention-output distortion to allocate residual KV capacity. An offline calibration stage compiles these signals into a persistence policy, leaving only lightweight structure-conditioned lookup online. At a targe total KV budget of 0.33, PASK outperforms the strongest compressed baseline by 17.39 percentage points on average across eight BFCL non-live and Live subcategories on Qwen3-4B. In end-to-end serving, PASK achieves up to 2.2x higher throughput and 3.3x lower TPOT, while using 0.53x the peak GPU memory of Full KV.
1 Introduction
PASK uses parser transitions to protect structurally consequential generated history that model-side KV importance may underweight. It combines task-risk protection with distortion-based allocation through offline calibration and lightweight online lookup.
- Structured generation supports machine-executable outputs, where one incorrect field, argument, or function choice can invalidate the downstream action.
- Parser transitions expose generated tokens’ structural roles, including required keys, enum values, arguments, boundaries, and scaffold tokens, beyond token legality.
- Model-side KV sensitivity can under-protect history that later changes a legal function, argument, or field value and therefore the task outcome.
- PASK maps parser-derived structure and Transformer layer groups to DROP, RETAIN-LOW, or RETAIN-HIGH persistence actions.
- Offline calibration uses task-error sensitivity for protection floors and attention-output distortion for residual allocation, compiling decisions into a lightweight online lookup policy.
- 17.39 percentage points: PASK improves over the strongest compressed baseline on average across eight BFCL subcategories at total KV budget 0.33 on Qwen3-4B.
2 Related Work
Prior KV compression and structure-aware memory methods use model, context, or heuristic signals but do not encode generated tokens’ parser-derived structural responsibility. PASK uses parser transitions and layer groups to assign task-risk-calibrated persistence actions.
- KV Compression: KV compression methods reduce memory through quantization, eviction, selective retention, and cache management, using attention, distortion, recency, or token-importance signals.
- KV Compression: Existing KV policies capture model-side importance but do not directly represent whether a generated token is a required key, argument, enum value, or schema boundary.
- Structure-Aware Memory: Structure-aware memory methods incorporate temporal, semantic, program, or structural-role signals derived from attention patterns, program context, or execution heuristics.
- PASK: PASK’s overview combines parser transitions and layer groups in an offline-calibrated table assigning high-precision retention, low-precision retention, or release.
- Structure-Aware Memory: PASK instead uses constrained-decoding parser transitions to assign layer-specific persistence actions to generated tokens under task-risk calibration.
3 Method
PASK uses parser-derived structural roles and Transformer layer groups to control generated-KV persistence. Offline calibration sets risk-based protection floors and distortion-based upgrades, then compiles the policy for lightweight online lookup.
- Parser-Derived Persistence Buckets: PASK maps each accepted parser transition to a structural bucket and layer group, then selects RELEASE, RETAIN-LOW, or RETAIN-HIGH persistence.Structural tags encode token class, schema responsibility, grammar state, and proximity to critical parser regions; layer groups allow depth-dependent actions.
- Dual-Metric Offline Calibration: Task-error sensitivity determines each bucket’s minimum protection, while attention-output distortion ranks residual KV allocation above that floor.The two metrics separate task-level protection requirements from model-side numerical sensitivity.
- Dual-Metric Offline Calibration: PASK builds a nested candidate family through adjacent upgrades, evaluates candidates with complete development-set generation, and selects the lowest-cost policy within a reliability tolerance.The selected action, protection floor, and upgrade scores are stored in a frozen lookup table.
- Online Structure-Only Lookup: During decoding, PASK performs structure-only lookup with constant-size budget state, demoting above-floor actions or promoting eligible actions as budget changes.The online path uses parser-derived structure and frozen tables without online importance estimation.
- Prompt-Side Companion: A prompt-side companion applies the same calibrated floor-and-upgrade principle to prefill tokens grouped by input-derived semantic labels and layer groups.The composed prefill policy is selected on development data under a separate tolerance.
4 Experiments
Experiments evaluate PASK on BFCL function-calling tasks across Qwen3 model scales, measuring accuracy, memory, serving efficiency, ablations, and policy selection. PASK preserves reliability under compressed inference while reducing memory and latency, with benefits tied to both task-error protection and distortion-based allocation.
- Task accuracy: PASK reaches 88.00% overall accuracy on Qwen3-4B BFCL non-live, versus 91.83% for Full KV and 78.67% for TriAxialKV.On BFCL Live, PASK reaches 72.01% overall accuracy versus 75.22% for Full KV and 58.57% for the strongest reported non-PASK compressed baseline.
- Memory: PASK reduces peak process memory from about 40.0 GiB for Full KV to 21.0 GiB, a 47.5% reduction.SnapKV reaches about 24.5 GiB with a 38.8% reduction under the same comparison.
- Speed: 51.6% lower TPOT and 98.9% higher throughput occur for 8K-input/1K-output workloads on A100 with PASK.At 32K input and 1K output, TPOT reduction reaches 69.8% on A100, while the 32K-input/256-token A100 case trades throughput for lower per-token latency.
- Ablation study: Removing the task-error protection floor raises the required decode budget from 50.08% to 98.49% for the same development reliability gate.Removing distortion ranking while retaining the floor requires 60.67% decode budget, compared with 50.08% for the full V+D policy.
- Hyperparameter selection: Development selection chooses the lowest measured stage KV budget satisfying the inclusive reliability gate, using αD = 5% for decode and αP = 2% for prefill.Candidate policies are scanned on development data before the selected prefill and decode policies are combined for end-to-end evaluation.
5 Conclusion
PASK uses parser transitions to condition KV persistence in constrained structured generation, combining task-error protection floors with distortion-based capacity allocation. On BFCL, it improves accuracy and reduces memory while retaining lightweight online lookup.
- 5 Conclusion: PASK conditions generated KV-state persistence across Transformer depths on structural information exposed by parser transitions.Task-error sensitivity sets minimum protection floors, while attention-output distortion allocates additional KV capacity.
- 5 Conclusion: 88.00% accuracy is reached on Qwen3-4B non-live evaluation at an approximate total KV budget of 0.33.
- 5 Conclusion: 47.5% lower peak GPU memory is achieved relative to Full KV on Qwen3-4B non-live evaluation.
- 5 Conclusion: The decode-policy ablation supports separate roles for task-level protection and distortion-based residual allocation.
Limitations
PASK requires offline calibration before deployment and currently uses a simplified persistence-policy design. Finer layer partitioning and additional precision levels remain future extensions.
- Limitations: PASK incurs a one-time preprocessing cost because offline calibration constructs the structure-conditioned policy table.
- Limitations: The current implementation uses three layer groups and three discrete persistence actions, limiting policy granularity.Finer-grained layer partitioning or additional precision levels may provide further flexibility.