PaperlayerView exact revision on arXiv

Source-linked AI summary

Refinement Symmetry in Multimodal Transformers

Yuhao Du, Shunian Chen

arXiv:2609.32669v1cs.AIcs.CV

TL;DR

Standard attention assigns each key equal prior weight, so token multiplicity can affect aggregation when a signal’s representation changes. The paper formalizes refinement symmetry and uses mass-proportional weighting with a coupling-based stability analysis. At twofold merging, measure weighting improves all-partition-correct accuracy over global count weighting on both evaluated benchmarks.

  • Problem

    Standard attention assigns each key one unit of prior weight, although tokenization changes how many keys represent a signal.

    Source: §1 · p. 1

  • Method

    The paper formalizes refinement symmetry, characterizes mass-proportional weighting, and uses a physical coupling to bound attention changes under retokenization.

    Source: §1 · p. 1

  • Results

    At twofold merging, inherited group mass improves all-partition-correct accuracy over global count weighting on MVBench and WorldSense.

    Source: §1 · p. 1

  • Takeaways & Limitations

    Refinement symmetry separates represented information from token multiplicity and connects mass-proportional aggregation to partition robustness.

    Source: §6 · p. 9

    Deployment evaluation covers two benchmarks with frozen encoders and four partition rules, and robustness gains depend on compression budget.

    Source: LIMITATIONS · p. 9

Abstract

from arXiv · show

Attention weights depend on token counts, which change with the representation of a signal. We study refinement symmetry: splitting a representation while preserving content, position, visible context, and total mass should preserve its contribution. Building on proportional and quadrature attention, we show that split invariance forces the local mass factor to be linear for any fixed positive attention kernel, provided that factor is nondecreasing. For changed representations, a physical coupling bounds attention error by separating feature change from weight reallocation. In Qwen2.5-Omni-7B, duplicating half the visual tokens threefold changes 255 of 3,586 MVBench answers under standard attention; measure weighting preserves every answer under matched visibility. Under natural frame resampling, it reduces distributional drift. At twofold merging of a frozen video encoding, a five-seed evaluation shows an all-partition-correct accuracy gain of 1.04 percentage points over global count weighting (average group mass) and 0.93 points over standard attention. The advantage over global count also holds on WorldSense but depends on the compression budget. The result is a representation principle with a measured benefit in robustness across partitions.

1 INTRODUCTION

Standard attention gives each key equal prior weight, so changing token multiplicity can change aggregation even when represented content is unchanged. The paper formalizes refinement symmetry and reports that measure weighting preserves predictions under matched duplication while improving correctness across tested partitions.

1 INTRODUCTION

  • Duplicating half the visual representations changes MVBench answers under standard attention, while measure weighting preserves every answer with matched positions and visibility.The figure reports changes to 150 and 255 of 3,586 answers under standard attention, and none under measure weighting.Source: §1 · p. 1, Figure 1 · p. 2
  • Refinement symmetry requires aggregation to remain unchanged when represented mass is split while content, position, and visibility are preserved.The paper characterizes this requirement and uses a physical coupling to bound attention error when features and scores change.Source: §1 · p. 1
  • For a fixed positive attention kernel, invariance to every equal split selects mass-proportional weights among monotone local mass rules.A physical mass coupling separates value perturbation from attention reallocation in the stability bound.Source: §1 · p. 1
  • At twofold merging, inherited group mass improves all-partition-correct accuracy over global count weighting on MVBench and WorldSense.Figure 1 reports improvement under all four tested partitions on both benchmarks.Source: §1 · p. 1, Figure 1 · p. 2

2 REFINEMENT SYMMETRY AND STABILITY

The section defines exact refinement as splitting represented signal mass without changing content, positional features, or visible context. It characterizes mass-proportional weighting, bounds changes under retokenization through a physical coupling, and relates sampling density to aggregation bias.

2.1 REPRESENTED MASS AND EXACT REFINEMENT

  • An exact duplicate split divides a cell’s positive represented mass among children while copying its content, positional features, and visible context.Standard attention can shift a two-token output from 1/2 to 2/3 when only the value-one token is duplicated; splitting its mass preserves 1/2.Source: §2.1 · p. 2

2.2 CHARACTERIZATION OF THE LOCAL WEIGHTING RULE

  • Among normalized rules w(m)K(s) with fixed positive kernel and nondecreasing local mass rule, invariance to every equal split holds if and only if w(m)=cm for c>0.The sufficiency is additive mass weighting; necessity follows from equal-split invariance and monotonicity.Source: §2.2 · p. 3

2.3 NATURAL RETOKENIZATION AS A COUPLED PERTURBATION

  • A mass coupling matches portions of signal represented by two partitions, separating changes in token values from changes in attention allocation.The resulting bound has no explicit token-count factor; for exact copies, the feature and paired log-kernel discrepancies are zero.Source: §2.3 · p. 3

2.4 SAMPLING DENSITY AND OPERATOR CONSISTENCY

  • With continuous scores and values, a positive continuous kernel, and vanishing cell diameters on a compact domain, mass quadrature converges to the normalized continuum integral.Under nonuniform sampling density, standard attention’s bias is Covπ(v, ρ)/Eπ[ρ], so finer sampling need not remove allocation error.Source: §2.4 · p. 3

3 MEASURE WEIGHTING IN MULTIMODAL TRANSFORMERS

The section calibrates represented mass across modalities, accounts for positional support geometry, and extends weighting to visibility and overlapping coverage. It distinguishes mass allocation from the relative modality priors and feature supports used in multimodal attention.

3.1 REFERENCE CALIBRATION ACROSS MODALITIES

  • Effective token mass is the product of a modality-specific scale and cell measure, with split invariance fixing linear dependence on cell measure but not relative modality priors.Changing one modality’s scale changes its odds against other co-visible types.Source: §3.1 · p. 4

3.2 POSITION AND SUPPORT GEOMETRY

  • The primary deployment comparison holds physical point positions fixed, while support averaging separately averages rotary logits rather than their nonlinear softmax.For general supports, the change in the averaged rotary matrix is bounded by min{2, ωmaxW1(p, p′)}; equal mass alone does not control this change.Source: §3.2 · p. 4

3.3 VISIBILITY AND OVERLAPPING COVERAGE

  • For a fixed query, visible token mass is the measure of its cell intersected with the visible region, preserving splits even when a causal boundary cuts a cell.For overlapping coverage, nonnegative functions assign mass across tokens while integrating a mixture of token kernels that retains feature disagreements.Source: §3.3 · p. 4

4 EXPERIMENTS

The experiments test whether weighting tokens by represented mass preserves attention under duplication and resampling, and whether it improves robustness across merged partitions. Results span controlled probes, video benchmarks, operator-error tests, and a temporal rate-transfer task.

4.1 EVALUATION DESIGN

  • The evaluation combines released-model probes, trained deployment benchmarks, controlled operator and temporal studies, and paired inference with clustered intervals and multiplicity controls.Deployment uses five trained seeds per benchmark; trained comparisons use prespecified contrasts, while additional-budget comparisons are descriptive unless stated otherwise.Source: §4.1 · p. 4, §4.4 · p. 6

4.2 CONTENT-PRESERVING DUPLICATION

  • Standard attention changes 150 answers at split factor two and 255 at factor three, while measure weighting preserves every answer under matched visibility.Full-modality duplication multiplies standard cross-modal attention odds by the duplication factor; measure-weighted logit deviations remain at the measured arithmetic floor.Source: §4.2 · p. 5, Table 1 · p. 5

4.3 NATURAL FRAME RESAMPLING

  • The natural-resampling evaluation compares KL divergence from the native 16-frame option distribution across five nonreference frame budgets.The paired accuracy difference and answer-flip differences are unresolved in the reported evaluation.Source: §4.4 · p. 6

4.4 ADAPTIVE MERGING AT DEPLOYMENT

  • At twofold merging, all-partition-correct accuracy rises by 1.04 pp over global count weighting and 0.93 pp over standard attention on MVBench.On WorldSense, robust accuracy rises by 0.74 pp against global count weighting, while contrasts against standard attention do not resolve; the advantage depends on compression budget.Source: Table 2 · p. 6, §4.4 · pp. 6–7, Table 3 · p. 7

4.5 OPERATOR ERROR UNDER ADAPTIVE PARTITIONS

  • At N = 32, measure weighting reduces mean absolute error from 0.1524 to 0.0076 for activity-driven partitions and from 0.1005 to 0.0020 for variation-driven partitions.Uniform partitions give the same output under both rules, and all 6,144 field–partition–budget comparisons satisfy the coupled bound.Source: §4.5 · p. 7, Figure 3 · p. 8

4.6 RATE TRANSFER AND POSITIONAL ABLATIONS

  • Across ten paired seeds, measure-minus-standard accuracy gains increase with rate ratio in 17 of 20 adjacent steps, while the equal-rate gain is +0.09 pp.In a separate thirty-seed panel, the ratio-32 gain is 16.8 pp; the study also reports a cross-modal log-odds slope of 1.007 ± 0.085 against log q.Source: §4.6 · pp. 8–9, Figure 4 · p. 8

5 RELATED WORK

Prior work treats attention as weighted integration and studies stability across sequence lengths; this paper positions refinement analysis alongside these approaches and token-budget methods.

5 RELATED WORK

  • Token Merging’s multiplicity correction is measure weighting for equal-mass base patches, while continuum, neural-operator, and spherical attention use function-space or quadrature formulations.This paper characterizes finite split invariance and bounds perturbations beyond exact copies.Source: §5 · p. 9
  • Prior measure-based regularity analyses give token-count-independent bounds, while this paper’s coupling bound separates value change from reallocation and retains feature distortion after mass correction.The network-level composition result specifies additional assumptions.Source: §5 · p. 9
  • Variable-byte patches and fixed-length image packing illustrate variable token budgets, while vision–language compression selects tokens by attention, similarity, clustering, or video structure.These choices determine which features remain; refinement analysis asks how inherited mass enters aggregation and what error remains when features or support change.Source: §5 · p. 9

6 CONCLUSION

The conclusion frames refinement symmetry as separating represented information from token multiplicity. It connects mass-proportional aggregation and perturbation bounds to prediction preservation and partition robustness.

6 CONCLUSION

  • Refinement symmetry separates represented information from token multiplicity, characterizes mass-proportional aggregation through finite invariance, and bounds attention error when content or support changes.With matched content, position, and visibility, measure weighting preserves predictions under duplication; at twofold merging, it improves all-partition-correct accuracy over global count on both benchmarks.Source: §6 · p. 9

LIMITATIONS

The stated scope and deployment constraints limit where exact invariance and reported robustness claims apply.

LIMITATIONS

  • Exact invariance applies to copied content, position, and visibility; other retokenizations require perturbation analysis.Deployment evidence covers two benchmarks with frozen encoders and four partition rules; adaptive encoding requires measuring feature changes.Source: LIMITATIONS · p. 9
  • Robustness gains depend on compression budget, while applying the depth bound to a pretrained decoder requires measuring blockwise constants.Further pipeline savings require reducing encoder and acquisition costs; modality scales remain task-dependent, and native-reference calibration preserves a checkpoint’s prior at its native partition.Source: LIMITATIONS · p. 9

REPRODUCIBILITY STATEMENT

The appendix records reproducibility materials and develops a refinement criterion, sampling-density analysis, and split-invariance results. Its claims are stated for specified representations, assumptions, and local weighting rules.

REPRODUCIBILITY STATEMENT

  • The source archive includes figures, tables, and LaTeX files, while the supplement provides numerical results, figure data, rendering code, deterministic checks, and source hashes.The appendices also provide proofs, implementation conventions, protocols, and statistical procedures.Source: REPRODUCIBILITY STATEMENT · p. 10

A.1 REFINEMENT CRITERION

  • For partition families closed under common refinement, a reduction is refinement-invariant exactly when it factors through the represented measure and field; the layer criterion applies to cell-constant outputs.For ordered sequences, centroid features, or changing causal horizons, those structures must also enter the representation and refinement relation.Source: §A.1 · p. 12
  • Covariance alone does not select a density: any finite measure absolutely continuous with respect to the reference measure yields an invariant reduction, so consistency with a specified operator selects the intended density.The uniqueness claim for Proposition 1 is limited to its stated local class.Source: §A.1 · p. 12

A.2 QUADRATURE CONSISTENCY AND SAMPLING-DENSITY BIAS

  • Under the stated continuity and compactness assumptions, representative-point sampling converging to density ρ yields a limit biased from the intended functional by Covπ(v, ρ)/Eπ[ρ].The figure compares both limits on increasingly fine nonuniform partitions, and the difference between the limits produces nonvanishing error.Source: §A.2 · p. 13, Figure 5 · p. 13
  • For smooth interval integrands and regular partitions, midpoint quadrature has O(N^-2) error; the standard estimator also converges at second order under the stated smooth-grid conditions.Those rate conditions are stronger than the assumptions needed for existence of the limit.Source: §A.2 · p. 13

A.3 SPLITTING, UNIQUENESS, AND MASKS

  • Mass-proportional weighting preserves contributions under arbitrary mass splits, and invariance to every equal split uniquely forces linear weights among monotone local mass rules.Network-level duplication additionally requires identical incoming representations, positional treatment, and visible total mass; causal child visibility can violate equivalence to a parent token.Source: §A.3 · pp. 13–14

A.4 LEARNING A MEASURE EXPONENT

This section characterizes when a shared measure exponent is identifiable from attention probabilities and how its gradient depends on score alignment. It also bounds what these results establish: positive discrepancy resolves one probability ambiguity, not general identification through values or downstream losses.

A.4 LEARNING A MEASURE EXPONENT

  • For one row, the exponent gradient is a covariance between score gradients and log-measure offsets; contributions across rows, heads, or layers can cancel.The covariance uses the uniform empirical distribution over keys.Source: §A.4 · p. 14
  • Log-measure variance separates variation among physical keys from their common offset relative to auxiliary keys, and neither component alone determines loss sensitivity.Sensitivity also depends on the gradient direction.Source: §A.4 · p. 14
  • A shared bias vector can trade off with the measure exponent without changing attention probabilities across rows exactly when D_id = 0.The proposition applies to a finite family of rows or tokenizations with fixed score arrays.Source: §A.4 · p. 14
  • Positive D_id removes this particular ambiguity in attention probabilities, but does not establish identification through arbitrary values, downstream losses, or a jointly trainable score function.A single configuration with constant b within each type has D_id = 0; several configurations need not, because one shared bias vector must explain them all.Source: §A.4 · p. 15
  • With a uniform physical offset, probabilities depend on the exponent and offset only through u = αℓ; a unique finite profile minimizer implies α*ℓ = u*.The gradient can still be small through saturation or alignment, and baseline attention share p0 is not generally the physical-token fraction.Source: §A.4 · p. 15

B.1 A SHARP CHANGE-OF-WEIGHT BOUND

The appendix bounds attention changes by separating feature perturbations from reallocation of attention mass, then extends the analysis to geometry, network propagation, modality scales, visibility, and implementation choices.

B.1 A SHARP CHANGE-OF-WEIGHT BOUND

  • TV(P, Q) ≤ tanh(∆/4) bounds the change in attention weights from the logarithmic range ∆ of their likelihood-ratio tilt.The coupling lifts both attention distributions to a shared support, where value diameter controls the effect of reallocation; the bound is sharp.Source: §B.1 · pp. 15–16

B.2 SUPPORT GEOMETRY AND ROTARY SCORES

  • For rotary scores, support displacement bounds changes in averaged rotary matrices, while equal mass alone provides no such geometric control.For intersecting cells, Lipschitz value and log-kernel fields yield bounds in terms of cell diameters; the analysis also covers asymmetric supports.Source: §B.2 · pp. 16–17

B.3 ALIGNMENT MAPS AND NETWORK PROPAGATION

  • Layerwise alignment errors propagate through subsequent block Lipschitz constants, and a logit error below half the reference top-two margin preserves the argmax.The bound covers complete blocks, but its depth product can be large even without a token-count factor.Source: §B.3 · pp. 17–18

B.4 MODALITY SCALES AND REFERENCE CALIBRATION

  • Local weighting invariant to every binary mass split must be linear in mass, while attention probabilities are unchanged by type rescaling only when co-visible types share a factor.Factors must be constant on each connected component of the co-visibility graph.Source: §B.4 · p. 18

B.5 PHYSICAL VISIBILITY AND OVERLAPPING REPRESENTATIONS

  • Integrating visible mass over physical regions preserves aggregation across subdivisions, but changed visibility boundaries or support-sensitive scores require additional perturbation control.Small boundary displacement alone need not imply a small output change without control of affected kernel mass and normalizers.Source: §B.5 · p. 19

C.1 MEASURES, REFERENCE SCALES, AND MASKS

  • The attention implementation adds log reference mass after score transformation, so temperature or softcapping applied to the combined quantity can alter split behavior.Masked keys receive zero weight, and each query must retain at least one visible key.Source: §C.1 · p. 20

C.2 SUPPORT-INTEGRATED POSITION

  • Support integration gives sinc factors for uniform intervals and Gaussian attenuation for variance-matched supports, but integrating logits does not commute with softmax.Thus these factors alone do not establish exact geometric refinement covariance of an attention layer.Source: §C.2 · p. 20, Figure 6 · p. 21

C.3 PRIMITIVE CHECKS AND THEIR SCOPE

  • Measure-aware budgets preserve admission in the constructed routing case, but discrete all-or-nothing child decisions can diverge from a parent decision at capacity boundaries.The primitive checks are deterministic constructed examples, and mask or budget choices are not resolved by log-measure bias alone.Source: §C.3 · pp. 20, 22

C.4 FLOATING-POINT AND TRAINING REPRODUCIBILITY

The paper distinguishes algebraic attention invariance from floating-point behavior and treats seeds or paired questions—not repeated rates—as the independent units for inference.

C.4 FLOATING-POINT AND TRAINING REPRODUCIBILITY

  • Float32 controls confirm that algebraic split invariance can still show rounding differences when attention length or kernel dispatch changes.A zero-mask control produced bitwise-identical float32 logits but bfloat16 deviations on all eight clips, up to approximately 1.1 × 10^-2.Source: §C.4 · p. 22

D.1 ESTIMANDS AND UNITS

  • For trained-model comparisons, paired observations use the same initialization and data seed, so repeated rates do not add independent training runs.When averaging multiple rates, the analysis first averages within seed.Source: §D.1 · p. 22, §C.4 · p. 22
  • Benchmark accuracy compares paired correctness indicators, while video-clustered deployment contrasts address dependence among questions from one source video.An answer change between two incorrect options does not count in the exact McNemar test; a nonsignificant test does not establish equivalence.Source: §D.1 · p. 22
  • Question-level paired intervals describe variation across evaluated questions, not training variation or guaranteed coverage under within-video dependence.WorldSense intervals use a video-cluster bootstrap; other benchmark intervals are conditional on the evaluated question set and its sampling assumptions.Source: §D.1 · pp. 22–23

D.2 MULTIPLICITY AND EXPLORATORY RESULTS

The experiments assess how measure weighting behaves across controlled rate imbalance, signal partitions, positional transfer, token merging, and video-rate changes. The findings include substantial gains at high rate imbalance, reduced operator error under adaptive partitions, and reduced rate-dependent attention allocation.

D.3 INDEPENDENT FACTORIAL REPLICATION

  • At rate ratio 32, the independent factorial replication found a +16.8 pp gain with 95% CI [+13.5, +20.2], while the equal-rate gain remained unresolved.Nineteen of twenty adjacent mean-gain steps increased with rate ratio; the exact seedwise sign-flip test gave p = 9.3 × 10^-10.Source: §D.3 · p. 23
  • Against learned per-modality bias in the thirty-seed panel, there were nineteen of twenty ascents (exact sign-flip p = 1.9 × 10−9), with equal-rate gain +1.15 pp (95% CI [+0.28, +2.03]; nominal p = 0.012).The coarseness coefficient was +1.44 ± 0.56 pp (SE; nominal p = 0.016), indicating an association with absolute resolution; rate imbalance alone does not account for it.Source: §D.3 · pp. 23–24

E.2 OUTCOMES AND COUPLED VERIFICATION

  • The coupling bound held for all 6,144 comparisons, but was worst-case: the largest observed error-to-bound ratio was approximately 0.10.Tightness was attained by a two-atom construction, not by these signals.Source: §E.2 · pp. 24–25
  • At 32 cells, adaptive partitions had mean absolute errors of 0.1524 versus 0.0076 for activity-driven partitions and 0.1005 versus 0.0020 for variation-driven partitions, standard versus measure weighting.Under uniform partitions, both methods had mean absolute error 0.0021; the comparison uses a fixed discrete fine-grid reference, not exact continuous integration.Source: §E.1 · p. 24, §E.2 · p. 25

F.1 SIGNAL GENERATION AND OPTIMIZATION

  • The template estimator reached approximately 99.1% accuracy across eight transfer rates, providing an achievable reference rather than a Bayes bound.The task uses a four-layer, 1.77M-parameter model; non-temporal shortcut probes scored 17.7–19.7% against 20% chance.Source: §F.1 · p. 25

F.2 CROSS-RATE TRANSFER

  • The matched eight-seed full-model versus shared-clock transfer gain was 2.94 pp, with 95% CI [0.61, 5.26], but did not meet the conservative multiplicity threshold.The figure encodes positional family by marker shape and compares seed-mean held-out accuracy across models.Source: Figure 7 · p. 26, §F.2 · p. 26, Table 5 · p. 26

F.4 TOKEN MERGING AND SUPPORT EXTENT

  • At merge factors 2, 4, and 8, models without extent attenuation lost 11.1, 19.1, and 21.0 pp; retaining the original extent reduced accuracy by 0.7, 11.2, and 13.3 pp.The results favor tracking represented extent on this task but do not identify a uniquely superior support distribution.Source: Figure 8 · p. 27, §F.4 · p. 27

G.1 RATE-DEPENDENT ATTENTION ALLOCATION

  • In Qwen2.5-Omni-7B, measure weighting reduced the mean log-attention-odds slope against video rate from 0.4646 to 0.0536, an approximately 88% reduction.The positive residual shows that the correction does not make resampled video representations identical.Source: §G.1 · p. 28, Figure 9 · p. 28

G.2 DUPLICATION WITH CONTROLLED VISIBILITY

Under matched visibility, measure weighting preserves answers when representations are duplicated, whereas standard attention changes some MVBench answers. Additional probes compare local allocation, resampling, calibration, model scale, tokenization, fine-tuning, and masked-token loss weighting.

G.2 DUPLICATION WITH CONTROLLED VISIBILITY

  • At split factor three, standard attention changed 255 of 3,586 MVBench answers, while measure weighting changed none under matched visibility.At factor two, standard attention changed 150 answers; measure weighting again changed none.Source: §G.2 · p. 28

G.3 UNEQUAL SAMPLING AND COUNT CONTROLS

  • A single per-video count cannot distinguish unequal within-video allocations; with two equal-span groups, a count rule supplied those groups can reproduce their masses.The result separates measure weighting from a per-modality count rule, not every geometrically informed estimator of density.Source: §G.3 · p. 29

G.4 NATURAL FRAME-BUDGET CHANGES

  • Natural frame-budget changes reduced mean reference-answer KL divergence by 0.0168 nats, while accuracy and answer-consistency differences remained unresolved.The measure and reference-calibrated count biases were equal across the 3,240 question–budget evaluations.Source: §G.4 · p. 29

G.5 WORLDSENSE CALIBRATION; G.6 GENERALITY ACROSS MODEL SCALE

  • On the 3B checkpoint, measure weighting reduced frame-budget KL by 0.0154 nats per question; paired accuracy and any-flip differences were unresolved.Under WorldSense rate changes, calibrated count and measure weighting gave identical answers.Source: §G.5 · p. 29, §G.6 · p. 30, Table 7 · p. 30

H.1 CONTENT-ADAPTIVE VIDEO TOKENIZATION

  • Across three content-adaptive tokenizations, consistency differences were unresolved, and the best observed local-count window was selected on the same data.That same-data selection makes the observed best consistency optimistically biased.Source: §H.1 · pp. 30–31, Table 9 · p. 31

H.2 LORA FINE-TUNING ON MVBENCH

  • In the 2,786-question LoRA evaluation, the q = 8 difference was +0.72 pp, with 95% CI [−0.16, +1.59] and nominal exact McNemar p = 0.125.With one trained seed per arm, intervals characterize question variability only.Source: §H.2 · p. 31, Table 10 · p. 31

H.3 MODALITY WEIGHTS IN A MASKED-TOKEN OBJECTIVE

  • For the per-token objective, setting w = 1/q produced higher held-out loss than w = 1 at q = 4, 8, and 16.Measure normalization enforces the stipulated refinement behavior of a fixed loss field but does not determine the best learned task tradeoff.Source: §H.3 · p. 32, Table 11 · p. 32

H.4 CROSS-RATE DEPLOYMENT AFTER TRAINING

At the primary twofold-merging budget, measure weighting improves all-partition-correct accuracy over both global count and standard attention on MVBench, while WorldSense supports the advantage over global count. The section also tests cross-rate deployment, temporal positional quantization, overlapping resolutions, pretrained support, and the deployment evaluation’s partitions and inference protocol.

H.4 CROSS-RATE DEPLOYMENT AFTER TRAINING

  • Across three seeds, the cross-rate accuracy-difference slope is unresolved, and natural frame changes do not satisfy the fixed-representation split assumptions.The deployed baseline already spans 37.27–38.73% mean accuracy across rates, with its minimum at the training rate.Source: §H.4 · p. 33

H.5 TEMPORAL POSITIONAL QUANTIZATION

  • At off-lattice declared rates, 14 of 2,160 evaluations change answers; none change at the four on-lattice rates.The tested frame rates are 1, 2, 3, 5, 6, 10, 12, and 24 fps; the Figure 11 intervals describe perturbation distributions, not confidence in a mean.Source: §H.5 · p. 33, Figure 11 · p. 33

H.6 OVERLAPPING SPATIAL RESOLUTIONS

  • Neither overlapping-resolution size set resolves an accuracy difference from measure weighting, and the views are not refinements of one partition.The calibration sets the relative image/text prior, so these results concern that implementation choice rather than the exact partition theorem.Source: §H.6 · pp. 33–34, Table 12 · p. 34

H.7 PRETRAINED SUPPORT PANEL

  • Without task training, the pretrained support panel finds +0.72 pp all-partition-correct accuracy over standard attention and +0.44 pp over calibrated count at twofold reduction.Point support matches or exceeds empirical characteristic-function geometry for every arm and budget.Source: §H.7 · p. 34

I DEPLOYMENT EVALUATION

  • The deployment comparison holds dense video encoding fixed while grouping temporal tokens, then evaluates standard, global-count, and measure-weighted attention on the same grouped features and physical positions.The intervention changes post-encoder representations while holding raw-video encoding fixed.Source: I DEPLOYMENT EVALUATION · p. 34

I.1 ACQUISITION, PARTITIONS, AND TRAINING

  • The deployment panel uses source-disjoint MVBench splits and five shared checkpoints trained only at the native partition, then evaluates unseen groupings.The primary budget and four contrasts were specified before data collection, but the study has no external preregistration; inference combines conditional source-cluster and five-seed tests.Source: §I.1 · pp. 34–35, §I.2 · p. 35

I.2 PRIMARY ENDPOINTS AND INFERENCE

  • +1.04 pp against global count and +0.93 pp against standard attention in all-partition-correct accuracy at twofold reduction on MVBench; mean pairwise JSD falls by 0.0063 nats against global count.The JSD difference from standard attention is unresolved; decoder latency falls to 0.073 s at eightfold reduction, while encoder cost is unchanged.Source: §I.2 · p. 35, Table 13 · p. 36

I.3 A SECOND BENCHMARK: WORLDSENSE

  • +0.74 pp robust accuracy and 0.0049 nats lower mean pairwise JSD against global count at the primary budget on WorldSense.Against standard attention, the robust-accuracy gain is +0.21 pp with an interval spanning zero; the advantage over global count is supported on both benchmarks at the primary budget.Source: §I.3 · p. 36

I.4 REPRESENTATION AND ALLOCATION DIAGNOSTICS

The diagnostics separate changes to grouped input embeddings from attention redistribution, and the partition decomposition shows where measure weighting differs from global count. The measured embedding and attention quantities are descriptive diagnostics, not a validated decoder bound.

I.4 REPRESENTATION AND ALLOCATION DIAGNOSTICS

  • Embedding deviation increases with reduction, while the attention-distribution comparison satisfies the stated total-variation bound for every recorded layer–head comparison.The embedding deviation is identical across attention rules because the input groups are shared; the attention diagnostic is largest under standard attention in every budget–partition cell.Source: §I.4 · pp. 36–37, Table 14 · p. 37
  • The combined embedding and attention diagnostic is not a validated decoder bound; decoder-layer or network bounds require additional layer-specific quantities.The diagnostic panel uses one adapter seed, and explaining budget dependence requires further analysis of decoder layerwise errors.Source: §I.4 · p. 37

I.5 PER-PARTITION DECOMPOSITION

  • At K = 8, measure and count coincide exactly on uniform groups, so their differences arise on adaptive partitions; the keyframe partition is the hardest and measure weighting lifts it over both comparators.On keyframe, accuracy rises from 63.03% (standard) and 63.78% (count) to 64.07% (measure); its measure-versus-count gain turns negative at fourfold and eightfold reduction.Source: §I.5 · pp. 37–38, Table 15 · p. 38

Related papers

Loading 2609.32669v1…