Source-linked AI summary
Pooling Helps, Learned Weighting Hurts In-Context: Decomposing Group Attention
Michael Fore, James Mason Inder, Mrishika Nair, Praneetha Vaddamanu, Sharlina Keshava
TL;DR
Although univariate models remain strong baselines, the contribution of cross-variate attention pathways to multivariate and in-context forecasting is unclear. The paper edits attention weights at inference to separate pooling from learned weighting, finding that pooling helps broadly while learned weighting harms many in-context configurations and uniforming the first block recovers performance across those tested.
Problem
Despite many methods incorporating covariates into forecasting, univariate models remain strong baselines, and the paper examines which cross-variate attention pathway contributes in multivariate and in-context settings.
Source: §1 · p. 1Method
The paper edits Chronos-2’s attention matrix at inference to separate the effect of the V/O pooled summary from learned Q/K weighting across encoder blocks.
Source: §2 · p. 2Results
Pooling is positive on 18 of 20 network configurations; learned weighting is negative on 9 of 10 ICL conditions, 4 of which are worse than univariate inference, while uniforming block 0 recovers every tested ICL configuration.
Source: §1 · p. 2, §3.1 · p. 3Takeaways & Limitations
For in-context forecasting, the authors suggest selecting examples with meaningful patch-index alignment or removing learned Q/K weighting from group attention in block 0.
Source: §4 · p. 4The edits localize effects to a pathway and block but cannot establish what the block computes or whether phase alignment explains the MV–ICL performance difference.
Source: §4 · p. 4
Abstract
from arXiv · showhide
Group attention, introduced by the time series forecasting model Chronos-2, attends over the variates of a group at a fixed patch index and serves both multivariate (MV) and in-context learning (ICL) forecasting. Rather than evaluating this cross-variate attention design as a whole, we ask which part of the mechanism earns the benefit and probe its applicability to both MV and ICL regimes. By editing the attention matrix $\alpha$ at inference we separate the two pathways a head comprises: V/O, which projects a weighted summary of the group, and Q/K, which decides the weights. Uniform pooling (V/O without any Q/K weighting) is positive on 18 of our 20 sensor-network configurations, while the learned weighting (Q/K) splits by group type: its contribution is positive or negligible for MV, but materially degrades 8 of the 10 sensor-network ICL configurations, leaving 4 of them worse than univariate inference. By isolating the impact of different layers, we find that uniforming $\alpha$ in the first block alone improves every ICL configuration we test.
1 Introduction
The paper separates cross-variate attention into pooling and learned routing to test their contributions to multivariate and in-context forecasting. Uniform pooling helps both modes, but learned weighting is neutral-to-harmful for in-context groups; uniforming the first block recovers performance across the tested configurations.
1 Introduction
- The study asks which pathway in cross-variate attention earns its benefit by editing learned Q/K weights at inference to isolate pooling from routing.A head’s Q/K projection defines the weights in a weighted average, while V/O determines what gets averaged.Source: §1 · p. 1
- Chronos-2’s group attention supports both shared-window multivariate groups and in-context groups drawn from the same or similar sensors at other times.The mechanism attends over variates at a fixed patch index, and the paper tests it because ICL examples need not be time- or phase-aligned.Source: §1 · p. 1
- Uniform pooling helps both multivariate and in-context groups, while learned Q/K weighting helps only multivariate groups and leaves several in-context configurations worse than univariate inference.Uniforming α in the first block alone, without retraining, fully recovers performance on every tested in-context configuration.Source: §1 · pp. 1–2
- The usual explanations of cross-variate harm—irrelevant or noisy channels and optimization pathology—do not apply to the reported in-context result.The examples benefit through V/O, and the inference-time edits involve no training.Source: §1 · p. 2
2 Method
The method edits Chronos-2’s attention matrix at inference to distinguish the contribution of group pooling from learned weighting. It evaluates four settings across multivariate and in-context groups on sensor networks, using paired bootstrap intervals and a material-effect threshold.
2 Method
- Each head forms a V × V attention matrix over group variates at a patch index, and its weighted group summary is projected into each variate’s representation.The edits replace α after softmax in each of twelve encoder blocks while leaving V/O, layer normalization, and univariate time-domain attention untouched.Source: §2 · p. 2
- The four α settings are identity, uniform, row-shuffled, and learned; their successive comparisons isolate pooling, the effect of weighting, and whether weight identity matters.Uniform pooling measures V/O alone; shuffled versus learned weighting tests whether which variate receives larger weight matters.Source: §2 · p. 2
- The evaluation covers five sensor networks, with multivariate groups sharing a window and in-context groups using disjoint-time windows from one station across two context/horizon protocols.The protocols are 204/52 and 320/96, yielding 20 configurations, ten per mode.Source: §2 · p. 2
- Contributions are reported as percentage-point changes in MAE over the previous setting, with 95% paired-bootstrap intervals over windows and materiality defined beyond ±0.1 points.The threshold is calibrated using block 5, whose effects are immaterial in all 20 conditions.Source: §2 · p. 2
- Figure 2 shows contributions across five networks, multivariate and in-context groups, and two protocols, with paired-bootstrap intervals and filled markers for material effects.The material-effect threshold is an interval lying beyond ±0.1 MAE points.Source: §2 · p. 2, Figure 2 · p. 3
3 Results
Uniform pooling generally helps, while learned weighting separates by group type: it helps multivariate groups but often harms in-context groups. The harm compounds across blocks, and making block 0 uniform improves every tested in-context configuration.
3.1 Learned weighting helps multivariate groups and hurts in-context groups
- Learned weighting helps 8 of 10 multivariate conditions but is negative on 9 of 10 in-context conditions.Pooling is positive on 18 of 20 network configurations.Source: §3.1 · p. 3
- The released mechanism is worse than no cross-variate attention on 4 of 10 in-context configurations, versus none of the multivariate configurations.Pooling on those same in-context groups is strongly positive, pointing to identity-dependent weighting rather than uninformative groups.Source: §3.1 · p. 3
3.2 Harm is a property of the stack, and block 0 is where to break it
- Uniforming block 0 improves all ten in-context configurations, materially in eight by +0.4 to +20.5 points.On multivariate groups, the repair never materially helps and materially hurts on the tidal networks where learned weighting helped.Source: §3.2 · p. 3
- Cumulative ICL block prefixes reach −16.09, with material harm in 18 of 25 ICL cases versus 1 of 25 MV cases.Figure 3 compares independent single-block edits with cumulative leading-block prefixes.Source: §3.2 · p. 3, Figure 3 · p. 4
- Removing all group-attention Q/K weights improves mean ICL performance by +5.92, while removing block 0 Q/K alone improves it by +7.2.This indicates learned weighting is not uniformly harmful even within ICL.Source: §3.2 · p. 4
- Replacing uniform group attention’s learned V/O projection with a plain cross-variate mean significantly degrades ICL performance.The uniform setting retains the learned V/O projection.Source: §3.2 · p. 4
4 Discussion
For ICL, V/O’s identity-agnostic work is useful while Q/K weighting subtracts; benefiting from example series requires meaningful patch-index alignment or a block-0 architecture edit. The inference-time edits localize effects but do not explain why weighting helps tidal networks or becomes costly with depth.
4 Discussion
- For ICL, V/O contributes useful work while Q/K’s variate-specific weighting subtracts, so example-series benefits require meaningful patch-index alignment or removing block-0 Q/K.The proposed alignment is natural in the multivariate forecasting scenario.Source: §4 · p. 4
- The edits localize pathway and block effects but cannot explain the tidal-network exception or why depth makes learned weighting costly.The authors also leave phase-alignment controls in ICL and transfer to models trained with the modified structure for future work.Source: §4 · p. 4
- The findings may partly reflect Chronos-2’s training distribution, since its multivariate ability comes from synthetic structure imposed on univariate series.Whether other covariate-attention designs behave similarly is left open.Source: §4 · p. 4
A All configurations
Appendix A1 reports the configurations behind the aggregate counts and shows additional corpora and relatedness controls.
A All configurations
- Table A1 lists the configurations underlying §3.1’s aggregate counts, with sensor-network rows plotted in Figure 2 and LSTF corpora and relatedness controls shown numerically.The relatedness controls use variate sets chosen to be maximally correlated or anti-correlated with the target.Source: A All configurations · p. 6
B Controls
Appendix B tests whether the learned V/O projection can be replaced by a cross-variate mean.
B Controls
- Table A2 tests replacing the learned V/O projection with either a raw or norm-matched cross-variate mean.Source: B Controls · p. 6
C The restriction sweep
The appendix records the restriction sweep’s individual conditions and corresponding pooling term at ctx-204.
C The restriction sweep
- Table A3 lists conditions behind the restriction counts, while Table A4 gives the corresponding pooling term; both use ctx-204.Source: C The restriction sweep · p. 6
D The repair intervention
Uniforming attention in block 0 improves every in-context configuration on three metrics, with material costs confined to tidal multivariate groups.
D The repair intervention
- Table A1 reports 39 configurations as additive contributions in points of univariate-baseline MAE, with sensor-network rows plotted in Figure 2.The UV MAE baseline is in each dataset’s native units.Source: D The repair intervention · p. 6, Table A1 · p. 7
- Replacing the V/O projection with a raw or norm-matched mean costs far more than the reported effect, so uniform α is not mere averaging.Source: Table A2 · p. 7
- 18 of 25 ICL multi-block subsets are materially negative (worst −16.09), versus 1 of 25 MV subsets (worst −2.43).Source: Table A3 · p. 8
- Pooling grows with active-block count on 8 of 10 datasets, opposite the learned-weighting pattern.Source: Table A4 · p. 8
- Block-0 repair improves every ICL configuration on all three metrics; its only material costs occur on tidal multivariate groups.Source: Table A5 · p. 9
- Nearly all recoverable effect sits in blocks 0 and 1; no single later block recovers any, while block 5 shifts forecasts by at most 0.081 points.Source: Table A6 · p. 10