PaperlayerView exact revision on arXiv

Source-linked AI summary

YOLO-PEFT: Parameter-Efficient Fine-Tuning on YOLO Family

Xu Lin, WenJie Nie, Jinlong Peng, Weifu Fu, YueXiao Ma, Xiawu Zheng, Yong Liu

arXiv:2608.07051v1cs.CV

TL;DR

PEFT methods from language models do not safely transfer to heterogeneous real-time detector graphs, making adapter placement costly to validate. YOLO-PEFT plans placements under explicit structural and semantic constraints, and planner-selected LoRA outperforms Full-SFT on YOLO11s and YOLO12s while refusing unsafe RT-DETR-L configurations within calibrated coverage.

  • Problem

    PEFT lacks safe, systematic adapter-placement evidence for heterogeneous detector graphs, where validating module combinations can require costly trial and error.

  • Method

    YOLO-PEFT formulates detector PEFT as auditable constraint planning that filters targets, allocates ranks under a budget, and lowers accepted plans into a unified runtime.

  • Results

    +7.1 and +6.5 mAP50:95 over Full-SFT on YOLO11s and YOLO12s, while RT-DETR-L returns Refuse under the evaluated coverage.

  • Takeaways & Limitations

    Within the evaluated coverage, YOLO-PEFT replaces manual target-module trial and error with explicit planning and verified deployment paths.

  • Takeaways & Limitations

    Refusal safety is not validated over unseen detector architectures because no detector family was held out.

Abstract

from arXiv · show

Generic parameter-efficient fine-tuning (PEFT) methods transferred from language models can fail silently on real-time detectors, whose heterogeneous operators and detection-specific components impose placement constraints absent from regular Transformer stacks. We propose YOLO-PEFT, a structure-aware framework that formulates adapter placement as an auditable constraint-planning problem. Given a detector graph, a PEFT request, and a resource budget, YOLO-PEFT assigns operator and semantic roles, evaluates explicit operator-validity, detector-semantic, graph-interface, and deployment predicates, records a reason code for each excluded module, and either emits a budgeted target-module plan or returns Refuse before training. Under the official VOC07+12 trainval-to-VOC07 test protocol, planner-selected RS-LoRA reaches 0.7138 and 0.7307 mAP50-95 on YOLO11s and YOLO12s, respectively, compared with 0.6428 and 0.6662 for Full-SFT. On RT-DETR-L, all seven evaluated LoRA-family configurations cross the predefined catastrophic threshold, supporting a calibrated Refuse-to-Full-SFT decision within the evaluated coverage. A controlled YOLO11 audit further shows that LoRA reduces peak training memory by 43.9 percent, although training takes 1.72 times longer. Within the evaluated detector families, placement policies, and calibration coverage, YOLO-PEFT replaces manual target-module trial and error with explicit, inspectable planning while preserving verified train-save-merge-export paths; refusal on unseen detector architectures remains an open validation problem. Project Page: github.com/Tencent/YOLO-Master

1 Introduction

YOLO-PEFT recasts parameter-efficient adaptation for heterogeneous real-time detectors as auditable, multi-constraint planning rather than module-name selection. It filters unsafe placements, allocates budgeted adapters, estimates risk within calibrated coverage, supports Refuse-to-Full-SFT fallback, and preserves train–save–merge–export deployment paths.

  • Motivation and gap: Generic PEFT interfaces can miss operator contracts, detection semantics, graph interfaces, and placement risk, leaving architecture–adapter failures to costly trial and error.Detector-specific methods demonstrate selective adaptation but remain tied to particular architectures or policies.
  • Framework: YOLO-PEFT formulates adapter placement on a heterogeneous computation graph and evaluates operator-validity, detection-semantic, graph-interface, deployment, and budget constraints before training.Its parser assigns operator and semantic roles, while a bounded 10-dimensional architecture fingerprint supports auditable risk estimation within evaluated calibration coverage.
  • Safety and deployment: Refuse makes fallback to Full-SFT a first-class outcome when no reliable plan exists, while the runtime supports adapter-only persistence, reloading, merging, and ONNX/TensorRT export.The fallback grouped-convolution LoRA backend preserves convolution outputs up to numerical tolerance after wrapper removal.
  • Constraint-resolved planning: The ordered planner applies mandatory operator and semantic filters, detector-specific safeguards for DFL paths, attention, text fusion, and MoE routing, then assigns ranks within the resource budget.Unsafe targets are removed before budget-aware rank assignment over surviving modules.
  • Evidence: +7.1 and +6.5 mAP50:95 are achieved by planner-selected LoRA over Full-SFT on YOLO11s and YOLO12s, whereas RT-DETR-L returns Refuse within the stated evaluation coverage.Evidence spans a diagnostic sweep of 14 PEFT variants and five detector architectures, with no universal PEFT ordering across evaluated families.

2 Related Work

Prior PEFT work develops compact adaptation methods and vision-specific modules, while detector studies explore placement and task-specific tuning. YOLO-PEFT addresses the remaining gap by validating detector-wide plans across heterogeneous operators, interfaces, semantics, and export behavior.

  • General and vision PEFT: PEFT freezes most pretrained weights and learns compact task-specific parameters; LoRA uses low-rank factors, while IA3, AdaLoRA, DoRA, and HRA refine adaptation.Unified libraries target modules by name or type, and AutoPEFT searches adapter configurations in pretrained language models.
  • General and vision PEFT: Vision-side PEFT spans residual adapters, visual prompt tuning, feature scaling, AdaptFormer, convolutional bypasses, Conv-Adapter, and Pro-Tuning across CNNs and Transformers.Dense-prediction work includes LoRand for Swin detection and segmentation, while E3VA highlights that few trainable parameters need not minimize training memory or time.
  • Motivation and gap: Existing literature largely assumes a chosen backbone and head, whereas YOLO-PEFT validates detector-wide target plans against operator constraints, detection semantics, graph interfaces, and export behavior.This distinction frames YOLO-PEFT as a structure-aware validation and lowering framework for existing PEFT methods.
  • Detector PEFT: Detector-focused studies adapt MobileNetV2–SSD-FPNLite, Swin-based oriented detectors, Deformable-DETR, DiffusionDet, Grounding DINO, and DETR through gates, low-rank modules, or hyper-adapters.These works examine ranks, placement, incremental adaptation, or staged tuning, but do not establish a common detector-wide feasibility check.
  • Motivation and gap: Real-time detectors combine CNN, Transformer, text–image fusion, sparse expert routing, attention-centric blocks, and loss-coupled heads, creating heterogeneous operator and interface constraints.Examples include RT-DETR, YOLO-World, YOLOv12, and YOLO-Master; this heterogeneity motivates a common train–save–merge–export contract rather than another adapter parameterization.

3 Methodology

YOLO-PEFT treats adapter placement as an auditable constraint-planning problem over detector graphs, returning a budgeted placement or Refuse. It combines operator, semantic, graph-interface, budget, reliability, and deployment checks while recording explicit rejection reasons and preserving exportable models.

  • Constraint-planning formulation: The planner maps a detector graph, PEFT request, and adapter budget to either a placement π or Refuse.Refusal is valid when constraints have no feasible intersection or calibrated reliability predicts catastrophic degradation.
  • Auditable module representation: Each candidate module receives operator, semantic, graph-interface, and deployment metadata, with Boolean constraints producing reason codes.The representation distinguishes computational contracts, detector roles, tensor-flow interfaces, backend support, mergeability, and exporter support.
  • Role-aware filtering: GraphParser rejects unsafe targets before user ranges and budgets are applied, including depthwise, normalization/activation, unknown, DFL, router, and geometry-sensitive regression modules.Grouped convolutions use group-local rank allocation, while explicit user targets cannot re-enable mandatory exclusions.
  • Architecture-conditioned policies: Architecture-conditioned policies support calibrated detector families while leaving unmatched modules and unrecognised Transformer detectors unsupported.Known structures receive role profiles; custom detectors fall back to topology scanning, and unseen detector families are outside validation coverage.
  • Training and deployment contracts: The implementation preserves unchanged base-checkpoint loading, adapter-only checkpoint storage, ordinary-module restoration after merging, and export-compatible ONNX/TensorRT graphs.These four invariants ensure that a numerically promising placement remains trainable and deployable.

4 Experiments

Experiments show that YOLO-PEFT’s structure-aware constraints improve detector accuracy and stability, while its calibrated refusal and runtime contract prevent unsafe or non-deployable adapter use. The evaluated trade-off is lower peak memory but longer training, with refusal remaining limited to calibrated architectures and coverage.

  • Evaluation Protocol: The official VOC protocol uses 16,551 trainval images from VOC2007 and VOC2012, evaluated on 4,952 official VOC2007 test images.VOC2007 trainval contains 5,011 images and VOC2012 trainval 11,540 images.
  • Refusal and Stability: 7/7 (100%) of evaluated RT-DETR-L adapter configurations are unsafe under the ∆< −0.05 threshold, so the calibrated planner refuses rather than falsely accepting the least-bad adapter.The rule is calibrated on seven observed collapses and is not validated for unseen architectures; LOVO reaches 86.7% accuracy with F1 = 0.850.
  • Effectiveness: 0.7453 mAP50−95 for YOLO12s HRA exceeds 0.6662 for Full-SFT by +7.9 points, while planner constraints raise YOLO12s from 0.6900 to 0.7307 mAP50−95.Operator validity filtering raises accuracy to 0.7094, and detection-head semantic exclusions provide the largest stability leap.
  • Deployment Economics: 43.9% less peak training memory accompanies LoRA’s 1.72× longer training time: 16.03 versus 28.57 GB and 203.8 versus 118.2 s.Relative to Full-SFT, the adapter removes 12.54 GB but adds 85.6 s on the controlled YOLO11 configuration.
  • Rank Sensitivity: 0.7307 mAP at rank 16 improves over 0.7288 at rank 8, while rank 32 reaches 0.7363 with diminishing returns and substantially higher overhead.The study identifies r = 16 as a practical Pareto point for real-time detectors.
  • Runtime Contract: 0.7138 mAP is identical for naive substitution and the Contract layer on YOLO11s, but only the Contract layer preserves checkpoint, merge, and ONNX/TensorRT deployment invariants.After merging, the adapter branch is removed and the exported graph retains base-model operator cost.

5 Conclusion

YOLO-PEFT concludes that structure-aware, constraint-checked planning can produce mergeable, export-compatible adapter plans for evaluated detector families, with measured accuracy and memory benefits. Its recommendations and refusal decisions remain bounded by implemented constraints, calibration coverage, and the evaluated evidence.

  • Conclusion: YOLO-PEFT resolves adapter placements under operator, semantic, and budget constraints, then lowers accepted plans into mergeable and export-compatible runtimes.The framework parses detector graphs from supported, evaluated families before planning.
  • Conclusion: 43.9% lower LoRA peak VRAM accompanies 1.72× longer training in the controlled YOLO11 audit.Peak VRAM falls from 28.57 to 16.03 GB, while training time rises from 118.2 to 203.8 s.
  • Conclusion: 0.7454 mAP50:95 for HRA versus 0.6891 for Full-SFT in the seed-0 YOLO-Master-EsMoE-S stress test supports compatibility with a conv-only MoE host subset.BOFT and OFT terminate without metrics, and the result does not establish an expert-aware planner.
  • Conclusion: Current evidence is limited to VOC and evaluated families, leaving COCO, domain-shift, region-only fine-tuning, and unseen architectures for future validation.Future work also targets VLM detectors with cross-modal attention and vocabulary heads, plus mixture-of-transformers and mixture-of-experts detectors.
  • Conclusion: Planner-selected adapters should be used only after operator and semantic checks pass, while refusal before training is supported only for evaluated or calibrated architecture–method regimes.The conclusion characterizes this as a decision rule within implemented constraints and calibration coverage, not a universal adapter recommendation.

A Deployment and Export Audit

The deployment audit verified both unmerged and merged detector graphs through ONNX and TensorRT export checks. ONNX Runtime CPU execution stayed below the reported PyTorch error threshold, while TensorRT testing covered FP32 and FP16 dynamic-profile engines.

  • ONNX/TensorRT verification protocol: Both unmerged and merged graphs passed ONNX checker validation at opset 18 with static [1, 3, 640, 640] and dynamic batch, height, and width axes.
  • ONNX/TensorRT verification protocol: 1.24 × 10−3 maximum absolute error was the upper bound for ONNX Runtime CPU execution relative to PyTorch.
  • ONNX/TensorRT verification protocol: TensorRT engines were built and executed in FP32 and FP16 with dynamic optimization profiles, verifying numerical agreement, memory, throughput, and latency.

B Merge Equivalence of the Fallback Manual Conv2d Backend · B.1 Dense convolution as im2col matrix multiplication · B.2 Grouped convolution and per-group rank allocation

The fallback manual Conv2d backend preserves the frozen host convolution while implementing an additive LoRA branch through unfolded input patches. Dense and grouped convolutions are handled with im2col-compatible computation, unchanged bias merging, and group-preserving rank allocation.

  • B Merge Equivalence of the Fallback Manual Conv2d Backend: The backend keeps the host Conv2d frozen and computes an additive LoRA branch on unfolded input patches.This establishes the fallback backend’s merge-equivalence strategy.
  • B.1 Dense convolution as im2col matrix multiplication: For dense convolution, unfold maps x ∈ R^N×C_in×H×W to x̃ ∈ R^N×C_in k_h k_w×L.Here, N is batch size and L = H′W′.
  • B.1 Dense convolution as im2col matrix multiplication: The unfold operator uses the host convolution’s kernel size, stride, padding, and dilation.Matching these parameters aligns the unfolded patches with the host convolution.
  • B.1 Dense convolution as im2col matrix multiplication: The dense convolution over unfolded patches is computed using the im2col convention as x̃ᵀWᵀ.This produces an L × C_out output before reshaping.
  • B.1 Dense convolution as im2col matrix multiplication: The host convolution’s bias term is preserved unchanged during merging.Bias preservation maintains the host layer’s additive bias behavior.
  • B.2 Grouped convolution and per-group rank allocation: Grouped convolution uses per-group factors and allocates balanced rank r_g = r/G, requiring r to be divisible by G.The allocation is defined per group.
  • B.2 Grouped convolution and per-group rank allocation: Concatenating per-group updates yields a block-diagonal dense im2col update, preserving the host convolution’s group structure.Disjoint input-channel blocks prevent channel mixing across groups.
  • B.2 Grouped convolution and per-group rank allocation: The dense case is recovered by setting G = 1 and r_1 = r.This connects the grouped-convolution formulation to the dense-convolution case.

B.3 Proof of Proposition 1

The proof shows that grouped LoRA updates are equivalent to a convolutional weight increment with the same shape as the original weights. After adding the increment, the wrapped layer can be replaced by a plain Conv2d, up to floating-point operation ordering.

  • Grouped LoRA equivalence: For each group g, the LoRA branch is linear in unfolded patches with effective matrix B_gA_g^⊤.The matrix is reshaped to (C_out/G, C_in/G, k_h, k_w).
  • Grouped LoRA equivalence: Concatenating per-group updates produces a convolutional weight increment ΔW with the same shape as W_0.This establishes that the grouped updates can be represented in the convolutional weight space.
  • Mergeability: After applying W_0 ← W_0 + sΔW, the wrapper can be replaced by a plain Conv2d up to floating-point operation ordering.The replacement follows from absorbing the LoRA update into the convolutional weights.

B.4 Numerical tolerance and implementation notes … F.1 Full detector-family coverage

YOLO-PEFT specifies backend routing, merge and export contracts, runtime metadata checks, diagnostic evaluation, and detector-family coverage. The implementation supports verified deployment paths while exposing limitations in current compatibility checks and cross-family ranking stability.

  • B.4 Numerical tolerance and implementation notes: Merged and unmerged Conv2d LoRA paths are equivalent within standard floating-point tolerance only in deployment/evaluation mode, after wrapper removal and with adapter dropout disabled.The paths differ in operation ordering, while merge correctness is verified after wrapper removal.
  • C.1 Backend selection rules: The PEFT backend handles LoRA-family variants, while the narrower manual fallback covers plain Conv2d LoRA when PEFT is unavailable, bypassed, or explicitly requested.Quantized paths always use PEFT because quantization integration is backend-specific.
  • C.2 Quantized and wrapper-required paths: Wrapper-required training states cannot be exported unless merged into a plain YOLO module or processed through a PEFT-managed export path.Otherwise, the contract layer refuses export.
  • C.3 Export policy: Export requires unchanged base-checkpoint loading, adapter-only checkpoints, ordinary YOLO structure after merge, and modules compatible with ONNX / TensorRT tooling.PEFT-managed adapters become export-compatible only after a successful PEFT merge or equivalent unwrapped-module verification.
  • D Runtime Metadata Schema: Runtime metadata records backend, variant, freeze-BN and head-inclusion settings, target modules, and backend-specific fields, enabling loader selection and configuration consistency checks.The PEFT path writes runtime_metadata.json, while fallback writes fallback_meta.json and a fallback weight file; stronger deploy-time manifest checks remain extensions.
  • E.1 Numerical companion to Fig. 2: Table S2 separates the Full-SFT anchor from PEFT ∆mAP values and marks catastrophic cells, while the diagnostic matrix is a seed-0 sweep rather than a multi-seed aggregate.The matrix uses ∆<−0.05 for catastrophic entries and clips collapsed RT-DETR-l failures at ∆= −0.600 for visualization.
  • E.2 Pairwise rank agreement: Within-CNN variant rankings show the only strongly positive pairwise agreement, whereas attention-heavy, text-fusion, and Transformer-decoder detectors weaken or reverse those rankings.The appendix provides the numerical companion to Fig. 2 and pairwise rank-agreement diagnostics rather than duplicating the main-text matrix.
  • F.1 Full detector-family coverage: Eleven backbones underlie the aggregate architecture-coverage discussion, with closely related CNN detectors listed separately but sharing the reported LoRA+decay acceptance band.The coverage tables enumerate family status without adding per-backbone measurements, while non-CNN rows retain their explicit main-text status.

G Detailed Ablations · G.1 A1. Naive substitution vs. contract-managed substitution

The ablation compares naive module substitution with a full Contract layer to test whether contract management is required for deployment compatibility. It covers manual and PEFT-managed substitution backends and evaluates save, reload, merge, and export without manual graph surgery.

  • G.1 A1. Naive substitution vs. contract-managed substitution: YOLO-PEFT uses module substitution in both evaluated backends.The manual Conv2d backend is used for B2, while PEFT-managed wrappers are used for B1.
  • G.1 A1. Naive substitution vs. contract-managed substitution: B2 uses a manual Conv2d backend for module substitution.
  • G.1 A1. Naive substitution vs. contract-managed substitution: B1 uses PEFT-managed wrappers for module substitution.
  • G.1 A1. Naive substitution vs. contract-managed substitution: The ablation tests whether surrounding contract management is required for deployment compatibility.
  • G.1 A1. Naive substitution vs. contract-managed substitution: Table S5 compares a naive substitution baseline with the full Contract layer.
  • G.1 A1. Naive substitution vs. contract-managed substitution: ONNX/TRT/Ckpt records whether trained models can be saved, reloaded, merged, and exported without manual graph surgery.

G.2 A2. Planner-component ablation … G.6 A7. LM-inherited ranking vs. structure-aware placement

The ablations show that planner constraints, rank, PEFT variant, and training-side interactions materially affect detector outcomes, while short-horizon selector diagnostics do not establish a universally superior rule. Structure-aware placement is evaluated against language-model-inherited ranking, with collapse safeguards remaining within-scope rather than held-out-architecture evidence.

  • G.2 A2. Planner-component ablation: Planner ablations isolate operator validity, detection-head semantic filtering, and budget pruning as separate contributors on YOLO12s.The experiment uses LoRA r=16, α=32, RS-LoRA, and the core 600-epoch-cap/640-resolution protocol.
  • G.2 A2. Planner-component ablation: The budget-matched selector diagnostic finds no universally superior selector at its short training horizon.Four rules share a rank-4, 31,104-parameter budget; gradient top-k leads at seed 0, while backbone-only leads at seeds 28 and 42, with means within 0.0034 and standard deviations of 0.0093–0.0267.
  • G.3 A3. Rank sensitivity: Rank sensitivity shows monotone gains with rank while identifying r=16 as a practical total-parameter trade-off.The comparison uses YOLO12s on VOC under the core 600-epoch-cap/640-resolution protocol.
  • G.4 A4. Variant sweep under fixed placement: Variant rankings shift across YOLO11s and YOLO12s when planner placement is fixed, supporting architecture-conditioned variant selection.Only the local PEFT update rule varies in this ablation.
  • G.5 A5. RS-LoRA and DoRA interaction: Only the DoRA-without-RS-LoRA corner crosses the catastrophic threshold in the RS-LoRA × DoRA interaction ablation.The two-factor design tests whether collapse stems from the variant label alone or from interacting training-side priors.
  • G.6 A7. LM-inherited ranking vs. structure-aware placement: The structure-aware plan selects PEFT variants and targets from the detector graph instead of inheriting a language-model ranking.It uses the same backbone and training budget as the LM-inherited comparison.
  • G.6 A7. LM-inherited ranking vs. structure-aware placement: The collapse count comes from a separate 300-epoch/320-resolution diagnostic matrix and is not an independent held-out-architecture result.It motivates a within-scope safeguard for the evaluated comparison.

H Reproducibility Notes

The reported core results use the official Ultralytics VOC mapping and a controlled YOLO11 efficiency audit. LoRA reduces peak VRAM relative to Full-SFT but increases training time.

  • Protocol and data mapping: The core protocol trains on VOC2007 and VOC2012 train/validation images and evaluates on VOC2007 test, with no random or few-shot split.The historical W&B field val2007 aliases VOC2007 test rather than the VOC2007 validation directory; core runs resize images to 640.
  • Efficiency accounting: 43.9% lower peak VRAM accompanies LoRA versus Full-SFT in the controlled YOLO11 audit, at 16.03 versus 28.57 GB.The efficiency claims exclude external storage, distribution, and externally sourced memory estimates.
  • Efficiency accounting: 1.72× longer training accompanies LoRA versus Full-SFT in the controlled YOLO11 audit, at 203.8 versus 118.2 s.This comparison is limited to the controlled YOLO11 audit.
Loading 2608.07051v1…