PaperlayerView exact revision on arXiv

Source-linked AI summary

Dynamic Positional Attention Modulation for Parameter-Efficient Fine-Tuning of Large Language Models

Dayan Pan, Jingyuan Wang, Xie Yu

arXiv:2610.07848v1cs.CLcs.AI

TL;DR

Existing PEFT methods often adapt uniformly despite heterogeneity across attention dimensions, heads, layers, and input tokens. DyPAM modulates query and key positional attention with input-conditioned, dimension-wise modulation and head- and layer-wise structure. Experiments report effectiveness and robustness compared with existing PEFT methods across backbones and tasks.

  • Problem

    Many PEFT methods use uniform, static adaptations despite attention heterogeneity across dimensions, heads, layers, and input tokens.

    Source: §1 · pp. 1–2

  • Method

    DyPAM adapts positional attention by modulating query and key representations with input-conditioned, dimension-wise modulation and head- and layer-wise structural modulation.

    Source: §1 · p. 2, §2.3 · p. 3

  • Results

    Experiments across multiple backbones and downstream tasks demonstrate DyPAM’s effectiveness and robustness compared with existing PEFT methods, with strong scalability across model sizes.

    Source: §1 · p. 2, §4.3 · p. 8

  • Takeaways & Limitations

    By reshaping how positional information influences attention, DyPAM adjusts how dependencies are formed across tokens, consistent with gains on multi-step and relational reasoning tasks.

    Source: §6 · p. 9

Abstract

from arXiv · show

Parameter-efficient fine-tuning (PEFT) has become a standard approach for adapting large language models to downstream tasks. However, most existing PEFT methods rely on uniform and static adaptations, without accounting for the structured heterogeneity of attention across dimensions, heads, layers, and input tokens. In practice, attention representations exhibit non-uniform behavior, and positional encoding mechanisms such as rotary positional embeddings (RoPE) induce dimension-dependent positional structure, making uniform adaptation suboptimal. In this work, we propose DyPAM (Dynamic Positional Attention Modulation), a PEFT method that adapts how positional information contributes to attention by operating directly on the query and key representations. DyPAM combines input-conditioned, dimension-wise modulation with head-wise and layer-wise structural modulation, performing fine-grained adaptation of positional attention aligned with the RoPE-induced structure without modifying the pretrained backbone. Extensive experiments on mathematical and commonsense reasoning benchmarks across multiple backbone models demonstrate that DyPAM consistently outperforms existing strong PEFT baselines.

1 Introduction

The introduction motivates structure-aware adaptation by describing heterogeneous attention and the limits of largely uniform PEFT methods. It presents DyPAM as a fine-grained positional-attention approach combining input-conditioned modulation with head- and layer-wise structure.

1 Introduction

  • Existing PEFT methods often adapt uniformly across layers, heads, and dimensions, while pretrained attention varies across those components and input tokens.The introduction argues that adaptation should account for these differences rather than treat model components uniformly.Source: §1 · pp. 1–2
  • Structure-aware adaptation should distinguish dimensions, account for head- and layer-wise differences, and remain conditioned on input tokens and context.The stated aim is to adapt dependency modeling while keeping the original attention architecture unchanged.Source: §1 · p. 2
  • DyPAM combines input-conditioned, dimension-wise modulation with head-wise and layer-wise structural modulation to adapt positional attention.The method models heterogeneity at token, dimension, head, and layer levels, aligning its modulation with RoPE-induced dimension pairing.Source: §1 · p. 2
  • DyPAM is presented as a parameter-efficient method that explicitly modulates positional attention in a fine-grained, structured manner.The introduction identifies this as the method’s central contribution.Source: §1 · p. 2

2 Preliminaries

The preliminaries review the attention, positional-encoding, and PEFT concepts used to describe DyPAM. They establish that attention needs positional information, RoPE induces dimension-dependent positional behavior, and common PEFT updates are often uniform.

2 Preliminaries

  • The preliminaries review attention, RoPE, and PEFT as background for describing the proposed method.This section introduces the concepts used in the following method description.Source: §2 · p. 2

2.1 Attention in Large Language Models

  • Self-attention projects token states into per-head query, key, and value representations and computes outputs per head; because attention itself lacks order, position is added to queries and keys.The passage identifies self-attention as the mechanism for token interactions and explains why positional information must be incorporated.Source: §2.1 · pp. 2–3

2.2 Rotary Position Embedding

  • RoPE applies position-dependent rotations to paired query and key dimensions, whose differing rotations produce non-uniform positional behavior across dimensions.Figure 2 depicts dimension responses to relative positional distances and a heatmap of their non-uniform sensitivity.Source: §2.2 · p. 3, Figure 2 · p. 3

2.3 Parameter-Efficient Fine-Tuning

  • PEFT keeps the pretrained backbone fixed while applying lightweight transformations, but common additive updates are static and uniform; DyPAM instead multiplicatively modulates attention representations conditioned on input and structure.In DyPAM, the modulation is applied to query and key representations and indexed by layers, heads, tokens, and feature dimensions.Source: §2.3 · p. 3

3 Method

DyPAM adapts positional information in attention through input-conditioned dimension-wise modulation combined with head-wise and layer-wise structural modulation. It operates on query and key representations and aligns the adaptation with RoPE-induced structure.

3.1 Framework Overview

  • DyPAM combines token-dependent dimension-wise modulation with persistent head- and layer-specific biases to adapt positional attention.The input-conditioned component captures token-dependent variation, while structural biases preserve preferences across inputs.Source: §3.1 · p. 3, §3.5 · p. 6

3.2 Query–Key Representations and Modulation Features

  • Low-rank projections of hidden states produce token- and head-specific modulation features that connect contextual inputs to query and key adaptation.The features are derived from the same hidden states used to form query and key representations.Source: §3.2 · p. 4

3.3 Input-Conditioned Dimension-Wise Modulation

  • DyPAM assigns one input-conditioned modulation value to each RoPE-aligned dimension pair, reducing parameter overhead while preserving pairwise positional structure.For grouped-query attention, key-side modulation is shared across heads using the same key representation, while query-side modulation remains head-specific.Source: §3.3 · pp. 4–5

3.4 Head-Wise and Layer-Wise Structural Modulation

  • Head-wise and layer-wise biases augment input-conditioned scores with structural preferences that persist across token positions.Query and key representations use separate bias vectors.Source: §3.4 · p. 5

3.5 Applying Modulation in Attention

  • Bounded modulation factors are applied to paired query and key dimensions before RoPE, aligning adaptation with positional structure.The paired coordinates share a factor, and the normalization centers modulation around the original scale to limit deviations from pretrained representations.Source: §3.5 · pp. 5–6

3.6 Training Details

  • DyPAM is trained end-to-end with the standard language-modeling cross-entropy loss.The model comprises the pretrained backbone together with DyPAM parameters.Source: §3.6 · p. 6

4 Experiments

DyPAM is evaluated across reasoning tasks, model families, and PEFT baselines, with analyses of overall performance, scalability, component contributions, sensitivity, and learned modulation. It consistently outperforms baselines at comparable parameter counts, and the analyses show structured, restrained modulation patterns.

4.1 Experimental Setup

  • Evaluation covers mathematical and commonsense reasoning benchmarks, three RoPE-based model families, and diverse PEFT baselines, with comparable trainable-parameter budgets for most methods.The research questions address overall performance, model size, components, hyperparameter sensitivity, and learned modulation patterns.Source: §4.1 · p. 6, §4 · p. 6

4.2 Overall Performance (RQ1)

  • Across mathematical and commonsense reasoning benchmarks, DyPAM consistently outperforms PEFT baselines under comparable parameter counts across evaluated backbones.The tables compare mathematical reasoning across three backbones and commonsense reasoning across three backbones.Source: §4.2 · p. 6, Table 1 · p. 7, Table 3 · p. 8
  • DyPAM’s query/key modulation is input-conditioned, unlike RoSA’s static RoPE-aware selection, and the paper attributes its consistent margin over RoSA to this finer-grained design.The comparison also reports more balanced improvements across tasks and models.Source: §4.2 · p. 7

4.3 Scalability Analysis (RQ2)

  • DyPAM remains stronger than the best PEFT baselines across Qwen3 scales from 0.6B to 8B, and its performance gap widens as model size increases.Table 2 reports macro-averaged accuracy on mathematical reasoning benchmarks.Source: Table 2 · p. 7, §4.3 · p. 8

4.4 Ablation and Sensitivity Analysis (RQ3, 4)

  • Removing any core component degrades performance, while weak or overly strong modulation underperforms proper modulation; the ablation and sensitivity analyses examine these effects.Figure 4 covers component removals and variation in modulation strength α.Source: §4.4 · p. 8, Figure 4 · p. 9

4.5 Analysis of Modulation Patterns (RQ5)

  • Learned query bias varies across layers and attention dimensions, while query/key modulation scales stay centered near 1 and vary by layer and training data.The reported visualizations characterize these patterns as structured and stable, preserving pretrained attention structure while allowing flexibility.Source: §4.5 · p. 8, Figure 5 · p. 9, Figure 6 · p. 9

5 Related Work

Prior PEFT work reduces trainable parameters through low-rank updates, scaling, structured reparameterization, and spectral methods. DyPAM is positioned among approaches to structured attention by applying fine-grained, input-conditioned modulation rather than static or coarse-grained adaptation.

5.1 Parameter-Efficient Fine-Tuning

  • PEFT spans low-rank additive updates, activation-modulating gating, geometrically constrained update spaces, and frequency-based adaptation; RoSA uses static RoPE-aware selection emphasizing low-frequency attention components.Examples include LoRA [16] and AdaLoRA [58] for low-rank updates, IA3 and LNTuning [60] for gating, OFT [39] and Bone [21] for geometric constraints, and FourierFT [10] for spectral adaptation; RoSA is [36].Source: §5.1 · p. 9

5.2 Structured Modeling in Transformer

  • DyPAM differs from static or coarse-grained approaches by generating input-conditioned modulation over query and key dimension pairs within attention.The discussion contrasts it with RoSA [36] and situates it amid work on structured attention and positional encodings, including ComRoPE [55].Source: §5.1 · p. 9, §5.2 · p. 9

6 Conclusion

DyPAM modulates positional attention through fine-grained, structured query/key adaptation aligned with RoPE. Across mathematical and commonsense reasoning benchmarks, it consistently outperforms strong PEFT baselines; runtime optimization and extension to diverse architectural mechanisms remain future work.

6 Conclusion

  • DyPAM operates directly on query and key representations, jointly modeling input-conditioned dimension-wise modulation with head-wise and layer-wise structure aligned to RoPE.The conclusion links this approach to heterogeneous attention across dimensions, heads, layers, and input tokens.Source: §6 · p. 9
  • Across mathematical and commonsense reasoning benchmarks, DyPAM consistently outperforms strong PEFT baselines.The conclusion relates its gains to tasks involving multi-step and relational reasoning.Source: §6 · p. 9
  • Future work will optimize runtime implementation and extend DyPAM across diverse architectural mechanisms.These are stated as directions for further work.Source: §6 · p. 9

A.1 Effect of Positional Encoding Schemes

The appendix compares activation patterns across positional encoding schemes and examines token-dependent variation. RoPE-based models show more dimension-structured and token-dependent patterns than the alternatives described.

A.1 Effect of Positional Encoding Schemes

  • RoPE-based LLaMA-3.2-3B has structured, dimension-dependent query activations, whereas ALiBi-based BLOOM-560M and learned-embedding OPT-350M are more homogeneous.Figure 7 compares query representations across attention dimensions under different positional encoding schemes.Source: §A.1 · p. 11, Figure 7 · p. 11

A.2 Token-Dependent Activation Patterns

  • Different tokens induce systematically different activation patterns in the RoPE-based model, while token-dependent structure is less pronounced under alternative positional encodings.Figure 8 compares token-type activation patterns across layers and dimensions for LLaMA-3.2-3B, BLOOM-560M, and OPT-350M.Source: §A.2 · p. 11, Figure 8 · p. 11

B Experimental and Implementation Details

The section documents the experimental settings, dataset statistics, and software environment used in the study. It also describes seed-based reproducibility checks and checkpoint selection.

B Experimental and Implementation Details

  • Experiments use seed 2333 as the primary run, with seeds 1000, 2000, 3000, and 4000 for significance and reproducibility checks.The checkpoint with the lowest loss on a random 300-sample validation set is used for testing.Source: B Experimental and Implementation… · p. 11
  • Default training uses AdamW at 1 × 10−3 with cosine scheduling, batch size 2, sequence length 2048, and modulation settings d_e=64, r=128, and α=0.3.Training uses BF16 mixed precision and DeepSpeed ZeRO Stage 1.Source: B Experimental and Implementation… · p. 11
  • Table 4 summarizes the training-dataset statistics, while Table 5 summarizes mathematical-reasoning test-dataset statistics.The supplied passages identify the tables’ scope but do not provide their cell values.Source: Table 4 · p. 12, Table 5 · p. 12

B.1 Software and Environment

  • The software environment includes PyTorch 2.7.0, Transformers 4.51.0, CUDA 12.8, and an NVIDIA GeForce RTX 4090 GPU.Baseline trainable-parameter counts are matched to a comparable scale except for extremely low-parameter methods.Source: §B.1 · p. 12

C Dataset Details

The experiments use Math10K and Commonsense15K for instruction tuning and evaluate on mathematical and commonsense reasoning benchmarks. Dataset statistics are summarized in tables.

C Dataset Details

  • Math10K supplies math word problems with step-by-step solutions, while Commonsense15K supplies commonsense questions in a normalized instruction format.Both datasets are from the LLM-Adapters benchmark suite [17].Source: C Dataset Details · p. 12
  • Mathematical evaluation covers MultiArith, GSM8K, AddSub, AQuA, SingleEq, SVAMP, and MAWPS; commonsense evaluation covers BoolQ, PIQA, SIQA, ARC, OBQA, HellaSwag, and WinoGrande.The passage describes these as established reasoning benchmarks and gives each task’s broad type.Source: C Dataset Details · p. 12
  • Table 6 summarizes statistics for the commonsense reasoning test datasets.The passage also identifies Table 5 as the mathematical test-dataset statistics table.Source: Table 6 · p. 12, C Dataset Details · p. 12

D Evaluation Protocol and Metrics

The evaluation protocol generates model outputs with greedy decoding using a unified instruction template. Task-specific extraction rules determine the predicted answers used for accuracy computation.

D Evaluation Protocol and Metrics

  • Outputs are generated with greedy decoding, with sampling disabled and max_new_tokens set to 256.Generation uses the Hugging Face Transformers generate() API.Source: D Evaluation Protocol and Metrics · p. 12
  • The unified prompt template frames each task as an instruction and asks for a response that appropriately completes the request.The template includes an instruction placeholder.Source: D Evaluation Protocol and Metrics · p. 12
  • Mathematical answers are extracted as numbers within absolute tolerance 10−3 or A–E choices for AQuA, while commonsense answers use exact-match labels.The passage states that the extraction and accuracy scripts are provided in the codebase.Source: D Evaluation Protocol and Metrics · p. 12

E Inference Overhead

DyPAM adds limited inference overhead relative to the base model and unmerged LoRA, with speed varying by batch size. Its input-conditioned design prevents merging into the backbone weights.

E Inference Overhead

  • 1.03% FLOPs overhead for DyPAM is below unmerged LoRA’s 1.13% on LLaMA-3.2-3B.The passage attributes the difference to avoiding low-rank updates to all linear layers and the small cost of per-token modulation.Source: E Inference Overhead · p. 12
  • DyPAM is about 10% faster than unmerged LoRA at batch size 1 and comparable at batch size 32.The passage reports negligible memory overhead.Source: E Inference Overhead · p. 12
  • DyPAM cannot be merged into backbone weights because its design is input-conditioned, though KV-cache reuse and kernel fusion could further reduce overhead.Source: E Inference Overhead · p. 12

Related papers

Loading 2610.07848v1…