RDP LoRA: Geometry-Driven Identification for Parameter-Efficient Adaptation in Large Language Models

Yusuf Çelebi; Yağız Asker; Özay Ezerceli; Mahmoud ElHussieni; Selva Taş; Reyhan Bayraktar; Fatma Betül Terzioğlu

Source-linked AI summary

RDP LoRA: Geometry-Driven Identification for Parameter-Efficient Adaptation in Large Language Models

Yusuf Çelebi, Yağız Asker, Özay Ezerceli, Mahmoud ElHussieni, Selva Taş, Reyhan Bayraktar, Fatma Betül Terzioğlu

arXiv:2604.19321v1Apr 21, 2026cs.LGcs.AIcs.CLcs.CV

TL;DR

Layer selection for LoRA remains difficult because internal representations have distinct layer-specific roles that uniform adaptation overlooks. The paper models hidden states as high-dimensional trajectories and uses training-free RDP pivots to select layers, achieving 81.67% on MMLU-Math with 13 layers versus 79.32% for full adaptation and 75.56% for random selection.

Problem
LoRA typically applies adaptation uniformly despite distinct geometric and functional roles across layers, leaving layer selection structurally underdetermined.
Method
The method applies RDP to hidden-state trajectories to identify structural pivots and uses them as a direct, training-free signal for sparse LoRA layer selection.
Results
81.67% MMLU-Math accuracy is achieved with 13 RDP-selected layers, versus 79.32% for full 36-layer adaptation and 75.56% for random 13-layer selection.
Takeaways & Limitations
Layer identity, rather than layer count or interval width alone, is central to efficient LoRA adaptation, and representation geometry supplies a training-free selection signal.
Takeaways & Limitations
The study evaluates a single benchmark and does not exhaustively explore model families, scales, or capacity settings, limiting generalizability.

Abstract

from arXiv · show

Fine-tuning Large Language Models (LLMs) remains structurally uncertain despite parameter-efficient methods such as Low-Rank Adaptation (LoRA), as the layer-specific roles of internal representations are poorly understood, leading to heuristic decisions about where adaptation should be applied. We model the evolution of hidden states as a high-dimensional geometric trajectory and propose using the Ramer-Douglas-Peucker (RDP) algorithm, a parameter-free and training-free polygon simplification method that preserves global structural transitions while eliminating locally redundant changes, to identify critical breakpoints along the representation path. Crucially, we use these geometric pivots not merely for analysis, but as a direct decision signal for determining which layers should be adapted during parameter-efficient fine-tuning. By integrating this geometry-aware layer selection strategy into LoRA fine-tuning of Qwen3-8B-Base, we achieve superior performance on MMLU-Math using only 13 RDP-selected layers (81.67%), significantly outperforming both full 36-layer adaptation (79.32%) and random 13-layer selection (75.56%), as well as the baseline Qwen3-8B-Base model (74.25%). These results demonstrate that leveraging the intrinsic geometry of representation trajectories provides a robust, interpretable, and training-free signal for optimizing layer selection during model adaptation.

1. Introduction

The paper addresses structurally uninformed layer selection in LoRA by treating hidden states as geometric trajectories and using RDP pivots to select adaptation layers. On Qwen3-8B-Base, 13 geometry-selected layers outperform full and random sparse adaptation on MMLU-Math.

LoRA reduces fine-tuning cost but typically applies adaptation uniformly across layers with distinct geometric and functional roles.
RDP provides a training-free, geometry-based method that identifies structural pivot points along high-dimensional hidden-state trajectories.It preserves global structural transitions while suppressing locally redundant variations.
Geometric pivots are used directly to prioritize layers associated with significant representational shifts for adaptation.
81.67% accuracy on MMLU-Math is achieved with 13 selected layers, compared with 79.32% for full 36-layer LoRA adaptation.

2. Related Work

The paper positions geometry-driven layer selection as a principled alternative to uniform, heuristic, or randomized PEFT strategies. RDP simplifies representation trajectories by retaining salient structural turns without training.

Existing LoRA and QLoRA methods apply updates uniformly, while sparse and fusion strategies often rely on heuristics, randomization, or module integration.
The method treats layer-wise representations as continuous paths where high-curvature turns indicate salient semantic shifts.
RDP removes local redundancies while preserving the global computational skeleton, yielding a dimension-neutral signal for selecting semantically dense layers without training.

3. Methodology

The methodology models layer-wise hidden representations as a discrete high-dimensional curve and applies RDP to extract structurally important layers. A target-driven threshold controls simplification, while the algorithm preserves salient deviations and discards low-variance segments.

Hidden representations V = {v1, v2, . . . , vL} form a trajectory across L transformer layers in a D-dimensional embedding space.
The method treats layer selection as structural simplification rather than a task-specific or gradient-based optimization problem.
RDP identifies pivot layers by preserving geometrically salient turns and discarding low-variance trajectory segments.
A target-driven RDP variant determines the simplification threshold instead of relying on a manually fixed ϵ.
Increasing ϵ produces progressively coarser trajectory approximations.
RDP takes an ordered point list and distance threshold ϵ, recursively splitting at the maximum-deviation point when dmax > ϵ; otherwise it returns the endpoints.
RDP suppresses noise-like micro-oscillations while maintaining dominant geometric structure in a 2D signal.
Because orthogonal distance is defined in any Euclidean space, RDP applies unchanged to 3D and high-dimensional embedding trajectories in a training-free manner.

3.3. Geometric Interpretation of Embedding Sequences

Embedding spaces organize concepts geometrically, with coherent regions and directional shifts reflecting semantic structure. RDP uses this geometry to filter semantic trajectories and retain their principal transition skeleton.

Embedding geometry places semantically related concepts near one another, making spatial organization a manifestation of learned conceptual relationships.
A 3D projection shows internally coherent clusters for mathematics, animals, and music as distinct semantic islands.
The embedding location of “apple” can favor the technology cluster over food, reflecting contextual polysemy in the pretraining corpus.
Coherent trajectory portions imply semantically coherent regions, while sharp directional changes signal transitions between conceptual areas.
RDP’s dimension-independent distance criterion provides a training-free way to estimate information density in 768-dimensional or higher embedding trajectories.
With ϵ = 1.12, RDP preserves pivots such as algorithm, integral, elephant, and lonely while removing micro-scale oscillations.

3.4. Layer-wise Trajectory Extraction and Attention-Weighted Projection

The method represents layer-wise Transformer transformations as a high-dimensional hidden-state trajectory, using attention-weighted projections to obtain one semantic vector per layer. This trajectory supports geometric identification of structurally significant regions for subsequent analysis.

Layer-wise representation construction: Each Transformer layer maps representations forward, so the layer-wise output matrix is reduced to a single vector for geometric analysis.The resulting vector z_l summarizes the semantic state at layer l.
Layer-wise representation construction: Attention-weighted projection produces layer-level vectors instead of mean pooling or last-token representations.Mean pooling may dilute signals from low-information tokens, while the final token alone may miss which contextual elements receive emphasis.
Trajectory formation: The vectors z_1 through z_L form a discrete hidden-state trajectory across the model’s high-dimensional representation space.This trajectory links geometric changes to the model’s internal layer hierarchy.
Reasoning-relevant band identification: The hybrid signal S(l) combines global geometric deviation and local velocity to identify a dynamically active layer interval.After smoothing, adaptive Otsu thresholding selects layers satisfying S(l) > τ as the semantic core.
Reasoning-relevant band identification: The trajectory is averaged across dataset samples, while PCA projection is used only to visualize its broad geometry.The visualization does not preserve full geometric fidelity, and the central segment is highlighted as the highest-curvature region.

3.6. Multi-Scale RDP Analysis: Extracting the Structural Backbone

The method applies RDP across hidden-state trajectories and target resolutions to identify structural pivot layers while balancing global and local representational changes.

Dataset-Level Trajectory Analysis: RDP is applied to hidden-state trajectories collected across a dataset to characterize the model’s global layer-wise topology.Using multiple inputs reduces stochastic variation from individual instances.
Distributional Consensus: Figure 7 reports how frequently layers are selected as pivots at target resolution t = 6.The distribution visualizes layer-selection frequency across the multi-scale analysis.
Multi-Scale Resolution: Multi-scale RDP addresses the inability of a fixed distance threshold to capture both global structure and fine local variation in high-dimensional trajectories.The method operates over target resolutions rather than a single threshold.
Multi-Scale Resolution: Target resolution t specifies the desired number of points retained after RDP simplification, starting at t = 3 and increasing until the full trajectory is recovered.The minimum meaningful target preserves the trajectory endpoints and at least one interior pivot.
Dynamic Thresholding: For each target resolution, the method searches for the minimal ϵ that retains at most t points, with larger ϵ producing coarser simplifications.The search exploits the monotonic decrease in retained points as ϵ increases.
Multi-Scale Voting: The multi-scale voting scheme emphasizes pivots appearing at coarse resolutions while retaining lower-weight contributions from finer-scale pivots.This favors globally critical layers without suppressing structurally consistent local transitions.

3.7. Geometric Importance Ranking and Adaptive Adaptation Priority

The method combines RDP-based structural importance with velocity-aware semantic change to rank layers and prioritize adaptation where hidden-state trajectories are most informative.

Layer Importance Ranking: The multi-scale RDP outputs feed a Layer Importance Ranking that identifies information bottlenecks during the model’s forward pass.The ranking converts structural analysis into an adaptation prior.
Structural Importance Index: The Structural Importance Index combines normalized RDP voting and normalized layer-wise semantic velocity through a tunable parameter β.β controls the trade-off between major directional pivots and regions of informational acceleration.
Adaptive Adaptation Priority: Higher-ranked layers receive priority for parameter-efficient fine-tuning, while less critical layers receive reduced or no adaptation.This replaces uniform layer treatment with geometry-informed adaptation allocation.

4. Experiments

Experiments evaluate geometry-based layer selection on Qwen3-8B-Base and additional models under shared training protocols, with results favoring sparse geometry-selected adaptation but limited by single-run and single-benchmark evaluation.

Core Experiments: 81.67% accuracy on MMLU-Math is achieved by adapting 13 geometry-selected layers, exceeding the unadapted and full-LoRA baselines.The unadapted model reaches 74.25%, while uniform Full LoRA reaches 79.32%.
Core Experiments: 75.56% accuracy from random 13-layer selection trails geometry-selected sparse LoRA, while all-band reasoning adaptation reaches 78.10%.These comparisons test whether performance depends on layer identity rather than sparsity or band constraints alone.
Core Experiments: Inverse selection reaches 78.48%, and geometry-weighted strategies reach 78.20%–79.23%, below uniform allocation on selected layers.The reported comparisons indicate that selecting layers mattered more than capacity tuning in these configurations.
Evaluation Caveat: All reported results come from a single run, leaving variance across multiple seeds for future work.This limits assessment of run-to-run variability.
Scale and Architecture: Across Qwen3-4B, Qwen3-14B, DeepSeek-LLM-7B, and Gemma-7B, geometry-selected sparse LoRA consistently outperforms random baselines.On larger or distinct architectures, it matches or exceeds Full LoRA with significantly fewer parameters, while gains on Qwen3-4B are modest.

5. Discussion

The discussion argues that LoRA performance and efficiency depend primarily on selecting structurally important layers rather than simply adapting more layers. Geometry-based selection provides a training-free signal, but the reported evidence is limited in architectural and task coverage.

Layer Selection as a Structural Decision: Layer selection directly determines LoRA performance and efficiency, with representational geometry identifying critical layers without training.The reported gains are attributed to structural positions rather than parameter count.
Layer Identity over Band or Sparsity: Adaptation is more sensitive to layer identity than to layer count or interval width, making selection a qualitative discrimination problem.Adapting all reasoning-band layers or random subsets underperforms RDP-selected pivots.
Selection versus Capacity Tuning: Uniform capacity on geometrically selected layers shows relatively stable performance, placing layer selection ahead of capacity tuning.Asymmetric capacity allocation may still be useful in some circumstances.
Geometry vs. Random Selection: Geometry-selected layers outperform random designs with the same number of adapted layers, indicating that RDP captures a relevant structural signal.
Generalization: Geometry-based selection can match or exceed Full LoRA with substantially fewer parameters, although Full LoRA performs marginally better on some architectures.
Limitations: The study does not exhaustively explore model families, scales, capacity settings, or benchmarks, limiting generalizability across architectures and tasks.The authors frame the findings as a limited exploration of the architectural design space.

6. Conclusion

The conclusion presents a geometry-driven LoRA framework that selects layers from hidden-state trajectories using RDP, avoiding gradients and task-specific training signals. It reports improved sparse adaptation against random selection and often against full adaptation, while proposing dynamic selection as future work.

6. Conclusion: The framework models hidden-state sequences as high-dimensional trajectories and uses RDP to identify structurally important layers for sparse LoRA adaptation.
6. Conclusion: Geometry-selected sparse LoRA improves over random layer selection and often surpasses full adaptation or approaches it with substantially fewer trainable parameters.
6. Conclusion: The authors suggest that adaptation performance depends more on structural layer locations than on the number of adapted layers.
Future Work: Future work could extend static layer selection to input-dependent inference or online training based on batch-level representation geometry.

A. Appendix: Additional Experimental Results

The appendix compares layer-adaptation strategies across four additional language models and reports MMLU-Math accuracy under five settings. Geometry-driven selection remains competitive with Full LoRA and outperforms random baselines, especially in larger models.

Additional Models: The appendix evaluates layer-adaptation strategies across Qwen3-4B-Base, Qwen3-14B-Base, Gemma-7B, and DeepSeek-LLM-7B-Base.
Comparative Results: Table 3 reports MMLU-Math accuracy for each model under five distinct adaptation settings.
Comparative Results: Geometry-driven selection consistently outperforms random baselines and remains competitive with Full LoRA, particularly in larger base models.
Comparative Results: Geometry-Weighted Sparse LoRA performs especially well on Qwen3-14B and DeepSeek-LLM-7B, suggesting geometric capacity allocation becomes more effective as model scale grows.