PaperlayerView exact revision on arXiv

Source-linked AI summary

Where Position Matters: Profiling-Guided Layer-Wise RoPE Allocation in Transformers

Zhiwei Liu, Boao Kong, Bing Liu

arXiv:2610.39223v1cs.LGcs.AIstat.ML

TL;DR

Layer-wise RoPE allocation is motivated by limited calibration between positional dependence and task utility, coarse coverage rules, and repeated profiling costs across model scales. The paper profiles rotary-phase demand, assigns coverage under a fixed budget, and transfers profiles within model families; its reported experiments show matched-budget validation-loss improvements, with resource analysis identifying favorable trade-offs for physical MLA.

  • Problem

    Existing positional profiles have limited calibration to task-level utility, while uniform or predefined coverage rules offer limited layer-specific flexibility and profiling separately at each scale incurs repeated costs.

    Source: §1 · p. 2

  • Method

    The paper uses phase-off and coverage-benefit profiles to assign high and low rotary coverage under a fixed budget, and maps small-model profiles by normalized depth for larger-model allocation.

    Source: §1 · p. 2, §5.2 · p. 8

  • Results

    Matched-budget experiments show validation-loss improvements, and resource analysis identifies favorable quality–storage–compute trade-offs for physical MLA.

    Source: §7 · p. 11

  • Takeaways & Limitations

    Within-family profiles can guide target-scale architecture allocation before training, while physical MLA offers favorable quality–storage–compute trade-offs in the reported analysis.

    Source: §7 · p. 11

    The fixed high/low design is coarse, and profile transfer relies on within-family structural correspondence.

    Source: §8 · p. 11

Abstract

from arXiv · show

Rotary Position Embedding (RoPE) typically uses uniform coverage across Transformer layers despite heterogeneous positional requirements. Phase-off interventions reveal non-uniform, model-specific rotary-phase dependence, while coverage interventions relate it to coverage-upgrade benefits. We propose profiling-guided layer-wise RoPE allocation, assigning high/low coverage under a fixed budget using early coverage-benefit profiles. The framework supports fixed-width query/key representations and physical rotary-dimension allocation in Multi-head Latent Attention (MLA). Normalized-depth mapping transfers profiles within model families, enabling small-model measurements to guide larger-model allocations before pretraining. A brief analysis relates selection and transfer to score calibration and inter-layer interactions. From-scratch experiments demonstrate validation-loss improvements over available budget-matched baselines, while analytical resource accounting identifies favorable quality--storage--compute trade-offs for physical MLA. These results connect positional demand to transferable, efficient architecture design.

1 Introduction

RoPE coverage can vary across layers, but practical layer-wise allocation requires measures tied to task utility, explicit budgeted rules, and affordable profiling across model scales. The paper connects phase-off demand to coverage benefits, allocates rotary coverage under a fixed budget, and transfers profiles within model families.

1.1 Related Work

  • Existing approaches motivate layer-wise coverage, while the paper identifies gaps in calibrating positional dependence to task utility, allocation flexibility, and profiling costs across scales.Prior work includes partial RoPE, hybrid local/global attention, and frequency or head-level analyses; the paper frames its approach as connecting layer-wise phase dependence to coverage benefits and budgeted allocation.Source: §1 · pp. 1–2, §1.1 · p. 3

1 Introduction

  • Phase-off profiling measures each layer’s dependence on rotary phase, and selected-layer coverage interventions test whether those scores predict coverage-upgrade benefits.Across four pretrained models and 136 layers, the profiles are strongly non-uniform and model-specific, with strong ranking agreement between phase-off scores and measured upgrade benefits.Source: §1 · p. 2
  • The framework assigns higher coverage to layers with larger early coverage-benefit scores and lower coverage to others, subject to a fixed rotary-pair budget.It provides fixed-carrier allocation with complete query/key widths and physical rotary-dimension allocation for MLA.Source: §1 · p. 2
  • Normalized-depth mapping transfers small-model allocation profiles to larger models, where the target budget determines layer coverage before training.From-scratch experiments examine transferred allocations under different attention parameterizations.Source: §1 · p. 2

2 Preliminaries and Problem Formulation

The paper formulates rotary coverage as a layer-wise allocation under an aggregate rotary-pair budget. It describes RoPE and MLA representations and distinguishes fixed-carrier from physical allocation.

2 Preliminaries and Problem Formulation

  • RoPE rotates selected query/key dimension pairs by position-dependent phases and applies the identity to unselected pairs, while all dimensions participate in attention matching.The background describes attention over token representations and includes a causal mask.Source: §2 · p. 3
  • MLA uses a shared key/value latent and a decoupled rotary branch, with cache storage depending on the latent width and rotary width.The supplied passages identify the shared latent and positional key as relevant cache components.Source: §2 · p. 3
  • The allocation chooses active rotary-pair counts across layers under a total budget, using nested subsets from a shared frequency bank.Fixed-carrier allocation preserves complete query/key widths; physical MLA allocation changes rotary dimensions while keeping content-branch and latent widths fixed.Source: §2 · pp. 3–4

3 Layer-Wise Positional Demand and Coverage-Upgrade Benefits

The section measures layer-wise positional demand with phase-off interventions and tests whether demand rankings correspond to benefits from increasing rotary coverage. The measured profiles vary across depth and model, and show strong ranking agreement with upgrade benefits.

3.1 Experimental Setup and Positional-Demand Measurement

  • Profiling evaluates four frozen pretrained models on the same 500 text samples, using paired loss differences with matched inputs, masks, and evaluation positions.Phase-off sets a layer’s rotary transformation to the identity while preserving its pre-rotation query/key representations and value branch.Source: §3.1 · p. 4

3.2 Empirical Distribution of Layer-Wise Positional Demand

  • Phase-off demand is non-uniform across layers: positive demand is concentrated in the first depth quarter for three models, while Qwen3-30B has elevated demand at both network ends.In LLaMA-2-7B, layer 2 accounts for 40.73% of total positive demand; Table 1 summarizes positive-demand shares in the first and last depth quarters.Source: §3.2 · p. 4, Table 1 · p. 5, Figure 1 · p. 5

3.3 Relating Positional Demand to Coverage-Upgrade Benefits

  • Within each model, phase-off scores strongly rank layers by their benefits from upgrading coverage from 50% to 100%.The reported within-model Spearman correlations are 0.943, 1.000, 0.943, and 1.000 in the evaluated model order; the other upgrade intervals range from 0.943 to 1.000.Source: §3.3 · p. 5
  • Coverage benefits can arise at different intervals even within one model: LLaMA-2-7B layer 1 recovers most positional utility by half coverage, while layer 2 improves mainly from 50% to 75%.Calibration tests 25%→75%, 50%→100%, and 25%→100% upgrades using 100 samples.Source: §3.3 · pp. 5–6

4 Motivating Analysis

The motivating analysis relates score-based layer selection and cross-scale transfer to calibration accuracy and inter-layer interactions. Under stated conditions, calibrated profiles guide budgeted allocation, while profile discrepancy contributes to target allocation error.

4 Motivating Analysis

  • Under exact calibration and additive effects, the analysis states that the high/low allocation is optimal.The formal statement and proof are reported in Appendix A.2.Source: §4 · p. 6
  • Theorem 1 relates score-based selection quality to calibration accuracy and joint-configuration effects; sufficiently separated scores identify beneficial budget-preserving layer exchanges.The stated exchange condition is α(si − sj) > 2ε + 2δ.Source: §4 · p. 6
  • Theorem 2 connects normalized-depth profile agreement to target-layer selection, with profile discrepancy contributing an additional allocation-error term of 2Ktαtη.The formal statement and proof are reported in Appendix A.3.Source: §4 · p. 6

5 Profiling-Guided Layer-Wise RoPE Allocation

The allocation converts layer-wise positional demand into a fixed-budget architecture: the budget sets how many layers receive high coverage, profiles set their locations, and small same-family profiles can guide target allocations before training.

5 Profiling-Guided Layer-Wise RoPE Allocation

  • The budget determines how many layers receive high coverage, while positional profiles determine their locations; normalized-depth transfer lets small-model profiles guide larger target allocations before training.One early profile can support multiple target depths and rotary budgets.Source: §5 · p. 7, §5.3 · p. 8

5.1 Demand-Guided Budget Allocation

  • Early coverage-benefit scores prioritize layers for additional rotary pairs over a low-coverage base, subject to the prescribed total rotary-pair budget.Scores are measured after 10% of reference-model training; the selected high-coverage count is constrained to meet the budget.Source: §5.1 · pp. 7–8

5.2 Architectural Instantiations

  • Fixed-carrier allocation rotates selected frequency pairs and uses identity elsewhere while keeping query/key widths intact; physical MLA allocation changes rotary dimensions and projection structure.Within an experiment, coverage levels use nested subsets from a shared frequency bank.Source: §5.2 · p. 8

5.3 Within-Family Profile Transfer

  • Normalized-depth mapping transfers source scores to target layers, after which the target budget determines the allocation and the target model is trained from scratch.Piecewise-linear interpolation with constant endpoint extension is used for score mapping.Source: §5.3 · p. 8

5.4 Storage and Computational Complexity

  • Profile reuse amortizes configuration acquisition, while physical MLA’s rotary-related parameters, storage, and matrix multiplications scale linearly with total rotary budget.Fixed-carrier configurations preserve principal matrix dimensions and storage; their added computation comes from early reference training and profiling.Source: §5.4 · pp. 8–9

6 Experiments

Experiments find demand-guided placement improves on available matched-budget controls across Dense and MLA settings, with physical MLA showing storage and compute savings in selected transferred configurations.

6.1 Experimental Setup

  • Experiments use Dense and MLA models at 130M, 350M, and 1B; larger models reuse within-family 130M profiles, while targets are trained from scratch.Profiles are collected after 10% of reference training, and FLOPs include profiling costs.Source: §6.1 · p. 9

6.2 Dense Quality and Cost

  • Demand-guided Dense allocation outperforms Full, Uniform50, and Random50 at all three scales, with query/key widths and storage unchanged.The gains therefore reflect rotary-coverage placement across depth rather than model-capacity changes.Source: §6.2 · p. 9, Table 2 · p. 10
  • 1.0147× and 1.0019× Full are the end-to-end FLOPs ratios for fixed-carrier Dense at 350M and 1B, respectively; model storage is unchanged.These costs include source acquisition, while matrix-multiplication dimensions remain fixed.Source: §6.2 · p. 9, Table 2 · p. 10

6.3 MLA Quality and Physical Resources

  • Demand-guided MLA beats uniform and random half-budget controls at 130M and 350M, isolating layer placement at equal aggregate rotary width.At 350M, the transferred configuration slightly improves on Full while using fewer resources.Source: §6.3 · p. 10, Table 3 · p. 10
  • 0.965× storage and 0.9476× end-to-end FLOPs accompany a slight Full improvement for transferred 350M MLA; at 1B, it nearly matches Full at 0.964× storage and 0.9396× FLOPs.At 1B, budget-matched controls are unavailable; the 130M configuration approaches Full but has a 1.0305× end-to-end FLOPs ratio due to same-scale profile acquisition.Source: §6.3 · p. 10, Table 3 · p. 10

6.4 Layer Placement and Rotary Budget

  • Demand-guided placement beats reversed, last-ten-layer, and random alternatives at 130M MLA, and transferred 350M MLA beats random sets; Profile50 also beats Profile75 despite fewer rotary pairs.The Profile50/Profile75 comparison shows validation loss does not improve monotonically with rotary-pair count.Source: §6.4 · p. 10

6.5 Cross-Scale Transfer and Replication

  • 0.011348 validation loss is the average Transfer50 improvement over Uniform50 across two paired 350M MLA groups; 130M profiles also guide 350M and 1B allocations without target-scale probing.A separate target-direct comparison also improves 1B Dense against its paired baseline.Source: §6.5 · p. 10

7 Conclusion

The paper presents profiling-guided layer-wise RoPE allocation and within-family profile transfer as a route to budgeted architecture design. Experiments report matched-budget validation-loss improvements and resource trade-offs for physical MLA.

7 Conclusion

  • The paper presents profiling-guided layer-wise RoPE allocation, using early profiles for within-family cross-scale transfer and reporting matched-budget validation-loss improvements and physical MLA resource trade-offs.The conclusion frames these findings as budgeted architecture design.Source: §7 · p. 11

8 Limitations

The fixed high/low allocation simplifies architecture construction, but its coarse granularity leaves room for finer resource allocation.

8 Limitations

  • The fixed high/low design simplifies architecture construction, but its coarse granularity leaves room for finer resource allocation.Transfer also relies on within-family structural correspondence, motivating exploration across broader architectures and longer contexts.Source: §8 · p. 11

A Analysis of Profiling-Guided Rotary Allocation

The analysis decomposes allocation loss into single-layer gains and joint-intervention effects, then examines how calibrated scores support budgeted layer selection and normalized-depth profile transfer.

A.1 Setup and Reference-Consistent Loss Decomposition

  • The frozen-checkpoint objective compares allocation losses with the all-high reference, separating each layer’s restoration benefit from residual joint-intervention effects.The residual measures the difference between the joint loss change and the sum of single-layer changes at the same all-high reference.Source: §A.1 · p. 15

A.2 Calibrated Profiles and Budgeted Layer Selection

  • When scores calibrate to all-high reference gains and interaction errors are bounded, selecting the top K scores has zero loss gap in the exact-calibration case.A budget-preserving exchange from a lower- to a higher-scored layer reduces loss when α(si − sj) > 2ε + 2δ.Source: §A.2 · pp. 16–17

A.3 Normalized-Depth Profile Transfer

  • Normalized-depth interpolation transfers source profiles to target layers, with the target allocation’s loss gap bounded by calibration, source-measurement, depth-mapping, and interaction errors.The bound is 0 ≤ Ft(Ŝs→t) − Ft(S⋆t) ≤ 2Kt[εt + αt(εs + ηdepth)] + 2δt.Source: §A.3 · pp. 18–19

B.1 Early Reference Training and Layer-Wise Probing

The appendix accounts for the training and probing needed to acquire profiles, their cross-scale reuse, experimental settings, and the storage implications of physical MLA allocation.

B.1 Early Reference Training and Layer-Wise Probing

  • Early reference-training and layer-wise probing costs are computed from training-token fractions, intervention passes, and cumulative probing inputs.Phase-off measurement uses one reference pass plus one intervention per layer; low/high benefit measurement uses one reference pass plus two interventions per layer.Source: §B.1 · p. 19

B.2 Same-Scale and Cross-Scale Configuration Acquisition

  • Cross-scale configuration-acquisition cost depends jointly on reference-training tokens, per-token arithmetic, and the depth and computation of profiling.The combined cost ratio is expressed through early reference training and layer-wise probing.Source: §B.2 · p. 20

B.3 End-to-End Cost and Profile Reuse

  • End-to-end cost combines target training with profile acquisition, and reusing one source profile across targets amortizes the acquisition cost.For physical allocation, target structure changes the training term; cross-scale transfer and reuse reduce the profile-acquisition term.Source: §B.3 · p. 20

C.1 Models, Training, and Profiling Settings

  • Profiles for final architecture allocation were measured at 10% of reference training using matched-marginal scores and shared evaluation inputs within each comparison.The profiling analysis covered four pretrained models and used 500 shared text samples for phase-off profiling and 100 samples per configuration for coverage calibration.Source: §C.1 · p. 21

C.2 Resource Evaluation Settings and Normalization

  • Resource ratios are normalized to full rotation at the same model specification, while the reported configuration-cost accounting uses specified probing workloads and acquisition assumptions.The run summary leaves event-wise global sequence counts unspecified, so the main-table workload is an accounting assumption.Source: §C.2 · pp. 21–22

C.3 Parameter and Storage Counts for the Experimental Models

  • At 350M MLA, half-budget allocation reduces storage from approximately 723.95 MiB to 698.30 MiB, or 0.965×, while equal total rotary budgets preserve aggregate rotary-projection and positional-key-cache sizes.The half-budget example also reduces parameters from approximately 348.1M to 337.8M.Source: §C.3 · p. 23

C.4 Training and Configuration FLOPs for the Experimental Models

The results compare demand-guided layer placement and transferred profiles against alternative allocations, then assess quality and resource trade-offs across rotary budgets.

C.4 Training and Configuration FLOPs for the Experimental Models

  • For 350M MLA, the target-training ratio is 0.932642× and the end-to-end ratio including early reference training and profiling is 0.947582×; the 1B MLA end-to-end ratio is 0.939618×.Dense target-matrix arithmetic remains unchanged, while relative configuration-acquisition cost decreases with target scale.Source: §C.4 · p. 24
  • Using a 130M source profile instead of target-direct profiling reduces early-reference and probing costs for larger targets in both Dense and MLA.For 1B MLA, the early-reference ratio is 0.0199× and the probing ratio is 0.1101× of direct acquisition.Source: §C.4 · p. 24

C.5 Frozen Layer Sets and Layer-Selection Ablations

  • In 130M MLA, demand-guided allocation improves loss over the random-set mean by 0.030679, reverse ranking by 0.037628, and the final-ten-layer configuration by 0.087689.At 350M MLA, cross-scale allocation improves over the two-random-set mean by 0.011339.Source: §C.5 · p. 25

C.6 Rotary Budget and Quality–Resource Trade-offs

  • In 130M MLA, Profile50 reaches loss 3.381707, outperforming Profile75 at 3.390307 by 0.008600 while approaching full-rotation loss of 3.377954.The evaluated quality–budget response is non-monotone.Source: §C.6 · p. 25
  • For the three-quarter MLA budget, target-training FLOPs are 0.965097× and weight-plus-4K-cache storage is approximately 271.83 MiB, or 0.981×; end-to-end compute is 1.0654×.The end-to-end ratio includes same-scale early reference training and the specified probing workload.Source: §C.6 · p. 25

C.7 Cross-Scale and Paired Training

Cross-scale and paired-training experiments report lower validation loss for transferred and target-direct allocations than for Uniform50. Additional experiments quantify the loss and cache effects of narrower rotary width and show sharply different coverage responses across two LLaMA-2-7B layers.

C.9 Representative Coverage Responses

  • At half coverage, LLaMA-2-7B layer 1 recovers approximately 97.0% of full-rotation utility, while layer 2 recovers 3.1%; layer 2 reaches 99.3% at three-quarter coverage.The plotted utility is normalized by each layer’s phase-off loss on the same 100 calibration samples.Source: Figure 3 · p. 27, §C.9 · p. 28

C.7 Cross-Scale and Paired Training

  • Transfer50 reduces 350M MLA validation loss by 0.011348 on average relative to Uniform50 across paired training runs.The reported difference is computed within each matched training group before averaging.Source: §C.7 · p. 25, Table 14 · p. 27
  • At 1B Dense, target-direct allocation improves loss by 0.013850 over same-group uniform coverage after 10% of reference training.Its high-coverage set shares eleven layers with the transferred set, with Jaccard score 11/17.Source: §C.7 · pp. 25–26, Table 15 · p. 27

C.8 Additional Experiments with a Narrower KV Latent

  • Uniformly reducing rotary width increases loss by 0.014658 while decreasing the full-layer logical cache from 12,544 to 11,648 elements per token.The comparison uses a 1B MLA model with KV latent width 384.Source: §C.8 · pp. 27–28
Loading 2610.39223v1…