PaperlayerView exact revision on arXiv

Source-linked AI summary

Counterexamples to Local Reconstruction Gain as a Proxy for Final Fidelity in Residual Completion

Yasuto Hoshi, Daisuke Miyashita, Jun Deguchi

arXiv:2609.34063v1cs.LGcs.CL

TL;DR

The paper asks whether improving a layer’s attention-output reconstruction necessarily improves final-model fidelity. It tests residual completion with frozen backbones, finding counterexamples in direct-runtime evaluation and mixed results for a complementary layerwise diagnostic. The results show local reconstruction and final fidelity are related but not interchangeable.

  • Problem

    It is unclear whether improving same-input attention-output reconstruction through residual completion also improves the final model’s distributional fidelity to dense FULL.

    Source: §1 · p. 1

  • Method

    The study evaluates learned Top-K+ϕ and training-free RESA with frozen backbones, comparing completion interventions with all-abstain EXACT TOP-K and adding restoration and masking studies.

    Source: §1 · p. 2

  • Results

    Two fixed Qwen3-0.6B/Multi-LexSum interventions improve prespecified local reconstruction but worsen final KL fidelity on discovery and prompt-token-disjoint holdout requests.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Local reconstruction is useful evidence about estimator behavior, but by itself does not predict final-model fidelity; masking repairs do not guarantee an advantage over EXACT TOP-K.

    Source: §8 · p. 8, §6 · p. 8

    The central result depends on the prespecified primary local criterion, and final-fidelity gain measures KL to FULL rather than task utility; no unique downstream mechanism is isolated.

    Source: §7 · p. 8

Abstract

from arXiv · show

Residual completion augments query-aware sparse attention by estimating the contribution of tokens omitted from the exact sparse computation. We ask whether improving a layer's attention-output reconstruction on the same incoming Q/K/V and selected support necessarily improves the fidelity of the final model output. We study training-free RESA and learned Top-K+$\phi$ with frozen backbone language models. A prespecified single-layer screen yields two Qwen3-0.6B/Multi-LexSum interventions for which direct-runtime measurements show positive prespecified request-aggregate local reconstruction gain but worse final KL fidelity than the corresponding all-abstain Exact Top-K baseline on both discovery and prompt-token-disjoint holdout requests. Exact restoration at the same layer instead improves final fidelity, showing that the reversal is specific to approximate completion in these cases. In complementary multi-layer experiments, a task-independent local diagnostic often repairs the tested completion estimators, although the repaired models do not consistently outperform Exact Top-K. Together, these results show that better local reconstruction need not translate into better final-model fidelity.

1 INTRODUCTION

The paper tests whether residual completion that improves attention-output reconstruction at one layer also improves the final model’s fidelity. Direct-runtime counterexamples and complementary masking experiments show that these endpoints are related but not interchangeable.

1 INTRODUCTION

  • Residual completion estimates the aggregate contribution of tokens omitted from query-aware sparse attention, while selection determines which tokens are evaluated exactly.The question is whether better same-input attention-output reconstruction necessarily yields a final distribution closer to dense FULL, measured by KL divergence.Source: §1 · p. 1
  • Later residual blocks and attention layers can transform a local correction and change subsequent query-dependent token selections, so local and final fidelity measure different quantities.Prior work motivates examining downstream consequences of sparse-attention perturbations, and this paper directly tests that relationship for residual completion.Source: §1 · p. 1
  • Two fixed Qwen3-0.6B/Multi-LexSum interventions improve prespecified local reconstruction but worsen final KL fidelity on both discovery and prompt-token-disjoint holdout requests.The completion and abstention runs have identical incoming hidden states, Q/K/V, and selected token IDs at measured steps; the attention outputs are those actually returned by the model.Source: §1 · p. 2
  • At the same selected layer, exact dense restoration improves final fidelity where approximate completion reduces it, though the control does not identify the downstream source of divergence.The comparison indicates the reversal is specific to approximate completion in these cases, not simply to moving that layer toward dense attention.Source: §1 · p. 2
  • Task-independent negative-G masking improves tested ϕ/RESA models in 11 of 12 fidelity comparisons, but repaired models do not consistently outperform EXACT TOP-K.The same rule gives mixed results for PISA-0TH, suggesting the diagnostic’s outcomes depend on the estimator.Source: §1 · p. 2
  • Across the experiments, local reconstruction quality and final-model fidelity are related but not interchangeable; improved reconstruction alone does not ensure improved final fidelity.Layerwise diagnosis can repair an estimator without making the repaired model better than abstention.Source: §1 · p. 2

2 COMPLETION AND ITS EVALUATION

The evaluation compares completion with all-abstain EXACT TOP-K while retaining the full KV cache and a fixed selected support at the measured layer. It separates same-input local reconstruction from final-model fidelity to dense FULL.

2 COMPLETION AND ITS EVALUATION

  • EXACT TOP-K normalizes attention over the selected support with completion disabled; completion adds an estimate of the excluded contribution while the full prefix KV cache is retained.The exact branch includes four sink tokens, the most recent 64 prompt tokens, score-based Top-K tokens from the middle prompt region, and generated tokens.Source: §2 · p. 2
  • Local reconstruction gain compares post-WO completion and Exact Top-K attention outputs with the dense output, computed from the same incoming Q/K/V.The measured outputs are the dense attention output, the Exact Top-K output, and the output with residual completion enabled.Source: §2 · p. 2
  • Request-level local gain aggregates the median gain across predetermined measured steps, then averages across requests; alternative aggregates are reported separately.The primary local measure uses normalized squared reconstruction error, with a validity threshold for cases where the Exact Top-K residual is too small.Source: §2 · p. 2
  • Final-fidelity gain compares each model’s KL divergence to FULL against the corresponding all-abstain EXACT TOP-K path, with positive gain favoring completion.Models receive the same prompt and dense-model-generated continuation tokens; the primary final measure uses all evaluated decode predictions, while a separate measure restricts evaluation to locally sampled steps.Source: §2 · pp. 2–3

3 EXPERIMENTAL DESIGN

The experimental design combines a frozen-backbone long-context screen with fixed direct-runtime interventions, an exact-restoration control, and task-independent layer masking. Paired request resampling and prespecified multiplicity procedures are used for uncertainty.

3 EXPERIMENTAL DESIGN

  • The main setting uses a 65,536-token prompt and K = 16, with Llama and Qwen backbones, learned ϕ and training-free RESA estimators, and RULER and HELMET tasks.Backbone parameters are frozen; tested budgets K ∈ {8, 16, 32} form an aggressive fixed-read regime, and task selection uses the FULL–EXACT TOP-K performance gap.Source: §3 · p. 3
  • The primary direct-runtime comparison follows two screened Qwen3-0.6B/Multi-LexSum actions, ϕ L15 and RESA L23, fixed before follow-up and evaluated against estimator-specific EXACT TOP-K baselines.The holdout excludes duplicate prompts and prompts matching discovery prompts; screening and direct-runtime estimates are treated as distinct because RESA’s operational baseline differs.Source: §3 · p. 3
  • Direct-runtime measurements compare completion, FULL, and all-abstain EXACT TOP-K; at sampled steps, completion and abstention share bitwise-equal incoming states, Q/K/V, and selected token IDs.The study observes the attention outputs returned by the model rather than recomputing completion offline.Source: §3 · p. 3
  • The exact-restoration control replaces sparse attention with dense attention only at the selected layer, while other layers remain sparse; its block-state metric differs from same-input local reconstruction gain.Final-fidelity gain remains measured relative to EXACT TOP-K.Source: §3 · p. 4
  • Task-independent masking disables completion at layers with negative mean local gain on 30 unlabeled 64K calibration sequences drawn equally from FineWeb, arXiv Summarization, and BIGPATENT.The repaired hybrid is compared with unmodified completion and EXACT TOP-K, with same-count anti-ranked and depth-matched random masks as controls; the rule is also tested on adapted PISA.Source: §3 · p. 4
  • Uncertainty uses paired resampling of requests after within-request aggregation; the 36-action screen adjusts 72 directional statements, and direct-runtime follow-up adjusts four directions per split.The screen uses one-sided percentile bounds at α/72; direct-runtime uses 100,000 bootstrap draws and cutoff α/4, while other tables report paired 95% intervals.Source: §3 · p. 4

4 RESULTS

Two fixed Qwen3-0.6B/Multi-LexSum interventions improved same-input local reconstruction yet worsened final fidelity on discovery and prompt-token-disjoint holdout requests. Exact restoration improved final fidelity in those cases, while local-diagnostic repairs did not consistently beat Exact Top-K.

4.1 DIRECT-RUNTIME LOCAL IMPROVEMENT AND FINAL DEGRADATION

  • Both fixed interventions had positive prespecified local reconstruction gain but negative final-fidelity gain versus all-abstain EXACT TOP-K on discovery and holdout requests.The directions held under the stated sensitivity analysis for all four action–split comparisons.Source: §4.1 · p. 4, Table 1 · p. 5
  • On the ϕ holdout, row-mean G, energy-pooled G, and absolute local error reduction had intervals crossing zero, limiting the counterexample to the prespecified median-based criterion.The three nominal-only FWE cases did not show CI-negative final-fidelity gain on prompt-token-disjoint holdouts.Source: §4.1 · p. 4

4.2 EXACT RESTORATION AND APPROXIMATE COMPLETION HAVE OPPOSITE EFFECTS

  • At both selected-block and final endpoints, approximate completion was worse than EXACT TOP-K, whereas exact restoration at the same layer was better.The block metric compares against the global FULL trajectory, unlike the same-input local reconstruction metric; the control does not locate where the mismatch arises.Source: §4.2 · p. 5, Table 2 · p. 5, Figure 1 · p. 6

4.3 REPAIRING COMPLETION DOES NOT ESTABLISH AN ADVANTAGE OVER EXACT TOP-K

  • Negative-G masking improved final fidelity over unmodified completion in 11 of 12 tested ϕ/RESA conditions, but repaired hybrids had 3 positive, 4 unresolved, and 5 negative comparisons with EXACT TOP-K.The table reports paired-interval counts over three tasks.Source: §4.3 · p. 6, Table 3 · p. 6
  • The repaired hybrid remained 0.076 worse than the EXACT TOP-K baseline, so the repair did not surpass the all-abstain comparison.Under the same negative-G masking rule, PISA-0TH produced 2 positive, 5 unresolved, and 5 negative fidelity repairs.Source: §4.3 · p. 7
  • Broader observational results were condition-dependent: all 12 Llama conditions had positive local and final-fidelity gains, while eight Qwen conditions combined positive post-WO local gain with negative final fidelity.The focused Qwen3-8B/RESA study found 7 of 18 actions positive, 10 unresolved, and 1 negative in final fidelity; these were not additional replications of the core direct-runtime result.Source: §4.3 · p. 7

5 RELATED WORK

Prior work studies sparse attention, residual estimation, cache changes, and downstream effects. This paper isolates a completion intervention while holding the retained cache and exact-support rule fixed.

Query-aware selection and residual estimation

  • Query-aware methods such as QUEST and SparQ select query-dependent subsets, while residual estimators approximate contributions omitted by a sparse branch; this study isolates one completion intervention’s relation to final output.The related-work discussion contrasts these approaches with sampling-based MagicPIG and evaluates the realized local reconstruction error rather than only the resulting sparse-attention method.Source: §5 · p. 7

Cache compression, eviction, and reconstruction

  • Cache-compression methods alter retained or fetched KV content, whereas this study keeps the full prefix cache and exact selected support available and confines intervention to completion output.ReST-KV is identified as close because it uses layer-wise output-reconstruction discrepancy to guide KV eviction.Source: §5 · p. 7

Propagation and final-output sensitivity

  • Downstream-sensitivity work measures effects of sparse-output shifts or layerwise KV perturbations; this study connects those perspectives to residual completion with cache and exact-support rules fixed.The discussion also situates cache diagnostics and counterfactual sparse-attention evaluation in this area.Source: §5 · p. 7

6 DISCUSSION

The direct-runtime experiments establish local-to-final sign reversals for two fixed interventions, including prompt-token-disjoint holdouts, but do not estimate their frequency. Exact restoration and masking experiments further show that local diagnostics do not by themselves determine final-model ordering.

6 DISCUSSION

  • Two fixed interventions showed positive same-input local reconstruction but negative final KL fidelity on prompt-token-disjoint holdouts, without estimating how often such counterexamples occur.Alternative local aggregations were less stable, especially for the ϕ holdout.Source: §6 · p. 7
  • At the same selected layer, exact restoration improved final fidelity while approximate completion reduced it, but the responsible downstream transformation remains unresolved.The control rules out the simple explanation that moving the layer toward its dense-attention output necessarily harms the model.Source: §6 · p. 7
  • Negative-G masking improved ϕ/RESA fidelity in 11 of 12 tested conditions relative to unmodified completion, yet repairs did not consistently beat EXACT TOP-K and the rule did not transfer uniformly to PISA-0TH.The discussion frames local reconstruction as informative about estimator behavior, not a substitute for measuring model-level fidelity and utility.Source: §6 · p. 8

7 LIMITATIONS

The evidence is bounded by the evaluated requests, metrics, estimators, and systems; several comparisons do not establish broader coverage or task-utility effects.

Experimental coverage

  • The direct-runtime core tests two Qwen3-0.6B/Multi-LexSum interventions at 64K and K = 16; holdout requests are prompt-token-disjoint, not established source-document- or legal-case-disjoint.Sensitivity covers nearby aggressive budgets, and the 8B study is post-hoc; Llama/Qwen comparisons also vary in positional settings.Source: §7 · p. 8

Metrics and interpretation

  • The primary result is specific to within-request-median local gain: alternative local weightings are less uniform, and final-fidelity gain measures KL fidelity to FULL, not task utility.No targeted one-layer utility loss is statistically resolved; later support may change, and no unique downstream mechanism is isolated.Source: §7 · p. 8

Estimators and systems

  • The study evaluates frozen estimator configurations; end-to-end retrieval accuracy, latency, bandwidth, and serving-optimal budgets are outside its measured endpoints.PISA-0TH is an adapted estimator-family control with a different auxiliary representation.Source: §7 · p. 8

8 CONCLUSION

The experiments show that local reconstruction gain does not necessarily predict final-model fidelity. Complementary materials document the tested estimators, configurations, and measurement procedures.

8 CONCLUSION

  • For two fixed interventions, positive prespecified local gain coexists with worse final dense-model fidelity on discovery and holdout requests; exact restoration improves final fidelity, while masking does not guarantee an advantage over EXACT TOP-K.Local reconstruction remains useful evidence about an estimator, but does not by itself predict final output fidelity.Source: §8 · p. 8

REPRODUCIBILITY STATEMENT

  • Deterministic bootstrap streams and request-level outputs support reconstruction of the reported tables without additional model inference.The supplement reports screening, estimator settings, task-selection criteria, fixed masks, paired request identities, and uncertainty calculations.Source: REPRODUCIBILITY STATEMENT · p. 8

AI USE STATEMENT

  • Generative AI assisted with translation, literature search and summarization, manuscript editing, and analysis-code development; the authors are responsible for final content and validation.Source: AI USE STATEMENT · p. 8

A.1 ATTENTION, POSITIONAL SETTINGS, AND ESTIMATORS

  • Full KV caches are retained; exact support combines four initial tokens, 64 recent prompt tokens, middle-region Top-K tokens, and generated tokens, while Top-K+ϕ and RESA estimate omitted contributions differently.Top-K+ϕ uses positive-feature representations; RESA combines an end-of-prefill typical-query prior with the selected sparse branch during decode.Source: §A.1 · p. 11

A.2 FROZEN ϕ CHECKPOINT RECIPE

  • Frozen ϕ checkpoints use 200,000 training steps and 65,536-token prefill packs; teacher-only calibration targets normalized-attention entropy 0.9995 and head-specific coverage 0.95.Qwen uses YaRN factor 2 in this recipe, while Llama retains its native positional configuration.Source: §A.2 · p. 11

A.3 DIRECT OBSERVATIONS AND UNCERTAINTY

  • Direct-runtime comparisons use identical prompts and teacher tokens, checking that sparse completion and abstention share incoming states, Q/K/V, and selected token IDs at sampled positions.Dense reference attention uses FP32 softmax; instrumentation is separately checked against measurement-disabled logits on the first two requests per evaluation.Source: §A.3 · pp. 11–12
  • Normalized gain is undefined below the residual-energy floor; uncertainty resamples requests, and Table A1 contrasts alternative local weightings and absolute error reduction.Its sensitivity intervals are ordinary paired 95% intervals; the primary four-direction analysis uses median G and all-step F.Source: §A.3 · p. 12, Table A1 · p. 13

B DIRECT-RUNTIME SENSITIVITIES

The appendix details task selection, screening and sensitivity protocols, alongside controls and calibration-based masking analyses. It also distinguishes comparison baselines and the scope of the reported stability and repair tests.

B DIRECT-RUNTIME SENSITIVITIES

  • Discovery and holdout bounds pair positive local-gain lower bounds with negative final-fidelity upper bounds for both fixed interventions.Discovery ϕ L15: LG = 0.363, UF = −0.0090; Discovery RESA L23: LG = 0.085, UF = −0.0019; Holdout ϕ L15: LG = 0.214, UF = −0.0152; Holdout RESA L23: LG = 0.068, UF = −0.0017.Source: B DIRECT-RUNTIME SENSITIVITIES · p. 13

C.1 TASK SELECTION

  • The three stress tasks are selected from eligible tasks with positive headroom across four backbones and all 12 model–budget points; NIAH Single-1 is the no-headroom control.The eligibility screen uses 64K/K = 16 and K ∈ {8, 16, 32}.Source: §C.1 · p. 14, Table A2 · p. 14

C.2 ALL CANDIDATE LAYERS AND SCREENING ENDPOINTS

  • The 36-action screen crosses two Qwen backbones, two estimators, three candidate layers and three tasks; two actions survive the 72-direction multiplicity correction.Screening local gain uses dense-model replay, while final-fidelity gain uses the screening baseline.Source: §C.2 · p. 14, Table A3 · p. 15

C.3 PROMPT-TOKEN-DISJOINT HOLDOUTS UNDER THE SCREENING PROTOCOL

  • None of the three nominal-only FWE holdout cases has a CI-negative final-fidelity gain, despite CI-positive captured-state local reconstruction gain.The 50 holdout requests are prompt-token-disjoint; these negative holdout results are not evidence for the direct-runtime core.Source: §C.3 · p. 14, Table A4 · p. 16

C.4 NEARBY K VALUES

  • All six nearby-K intervals retain the stated local-reconstruction/final-fidelity ordering, with smaller RESA final-fidelity effects as K increases.The fixed-action sensitivity covers K ∈ {8, 16, 32} in a deliberately aggressive fixed-read regime, not substantially denser supports.Source: §C.4 · pp. 14–15, Table A5 · p. 16

D EXACT RESTORATION AND BASELINE-SPECIFIC EFFECT SIZES

  • The exact-restoration control compares sparse abstention, approximate completion and dense attention at one layer while other layers remain sparse; screening and direct-runtime baselines are kept stage-specific.For RESA L23, the reported final-fidelity gain shifts from approximately −0.004980 to −0.002874 because the completion KL is unchanged while the baseline KL differs.Source: D EXACT RESTORATION AND… · p. 17, §D.1 · p. 17, Table A6 · p. 17

E.1 CONDITION-LEVEL REPAIR AND COMPARISON WITH EXACT TOP-K

  • Negative-G masking disables the complete set of layers with negative generic mean post-WO gain; repair is compared both with unmodified completion and with a same-count anti-ranked control.The depth-matched control evaluates ten fixed masks; its observed ranges are not confidence intervals, and its count is not a calibrated p-value.Source: §E.1 · p. 18, §E.2 · p. 18, Table A7 · p. 18

E.3 GENERIC-CALIBRATION STABILITY

  • Calibration analyses report selected-set stability across balanced corpus prefixes, corpus-specific estimates and leave-one-corpus-out comparisons, with alternative summaries that do not redefine the frozen mean-G mask.The calibration table defines robust counts using layers negative by confidence interval across all three corpora and every leave-one-corpus-out analysis.Source: §E.3 · pp. 18–19, Table A10 · p. 20, Table A11 · p. 20

F.1 PISA-0TH MASKING

The section tests negative-G masking on PISA-0TH and reports unmodified all-layer completion outcomes for Llama and Qwen conditions.

F.1 PISA-0TH MASKING

  • Across 12 PISA-0TH conditions, negative-G masking yields positive fidelity repair in 2, unresolved repair in 5 and negative repair in 5; every utility interval is unresolved.The estimator-family transfer test adapts a zeroth-order PISA estimator with one summary per 64-token omitted-middle-region block.Source: §F.1 · p. 21

G.1 ALL-LAYER COMPLETION ENDPOINTS

  • All 12 Llama conditions have CI-positive local-reconstruction and final-fidelity gains, while eight Qwen conditions pair CI-positive post-WO local gain with CI-negative final-fidelity gain.These are observational all-layer completion comparisons on the same N = 50 mechanism request sets, not the single-layer direct-runtime experiment.Source: §G.1 · p. 22, Table A13

G.2 FOCUSED QWEN3-8B RESULT

The focused Qwen3-8B RESA experiment evaluates 18 layer–task–context actions selected from layers with positive generic-calibration local gain.

G.2 FOCUSED QWEN3-8B RESULT

  • Among 18 Qwen3-8B RESA actions, final-fidelity gain is CI-positive in 7, unresolved in 10 and CI-negative in 1.The sole CI-negative action is native-32K Multi-LexSum at L6, with G = 0.393 [0.349, 0.422], selected-block Gblock = 0.057 [0.043, 0.067], and F = −0.0024 [−0.0040, −0.0008].Source: §G.2 · p. 22
Loading 2609.34063v1…