PaperlayerView exact revision on arXiv

Source-linked AI summary

LocalProp: Neuro-Localized Memory-Efficient Backpropagation

Diana-Nicoleta Grigore, Iuliana Georgescu, Radu Tudor Ionescu

arXiv:2609.32517v1cs.AIcs.CVcs.LGmath.OC

TL;DR

End-to-end backpropagation across pre-training and fine-tuning is memory-intensive and biologically implausible, while localized methods have rarely covered the full vision-model pipeline. LocalProp applies local updates through I-JEPA pre-training, supervised fine-tuning, pruning, and recovery. It compares favorably with existing local optimization approaches while reducing peak training memory, with gradient span offering a practical accuracy-memory trade-off.

  • Problem

    End-to-end backpropagation is used across training stages, but its global pass is memory-intensive and biologically implausible, and prior localized methods rarely address the full pipeline.

    Source: §1 · p. 1

  • Method

    LocalProp applies localized gradient updates during I-JEPA pre-training and supervised fine-tuning, then uses structured pruning and local recovery under the same locality constraint.

    Source: §1 · p. 2, §3 · p. 5

  • Results

    LocalProp compares favorably with existing local optimization approaches while substantially reducing peak training memory; on CIFAR-10, a four-block window reaches 86.22% accuracy with 58% less peak memory than full fine-tuning.

    Source: §4.3 · p. 7, §5 · p. 9

  • Takeaways & Limitations

    Gradient-propagation span is a tunable coordination budget: larger windows can recover accuracy while requiring more activation memory.

    Source: §4.3 · p. 7, §5 · p. 9

    The study uses CIFAR-10, CIFAR-100, and ImageNet-1K; resulting models may inherit training-data biases, so sensitive deployments require further assessment.

    Source: ETHICS STATEMENT · p. 10

Abstract

from arXiv · show

The current deep learning training paradigm employs end-to-end backpropagation, regardless of the training stage, i.e. pre-training or fine-tuning. However, backpropagating through the entire model is neither biologically plausible nor memory efficient, since learning inside the brain is highly localized. Therefore, we propose LocalProp, a training procedure that locally updates the weights of a model. Our neuro-localized weight updates follow the "pre-training then fine-tuning" paradigm, where the pre-training is based on I-JEPA. After locally updating the weights, a pruning operation is performed, followed by a short final fine-tuning phase. Pruning helps by sending the learning signal from higher blocks to lower blocks. We perform experiments on several datasets, including large-scale benchmarks such as ImageNet, and empirically show that LocalProp reaches good performance at a fraction of GPU peak memory. By varying the number of jointly optimized blocks, we identify gradient-propagation span as a practical control over the accuracy-memory trade-off.

1 INTRODUCTION

End-to-end backpropagation is memory-intensive and biologically implausible, while prior localized methods have not addressed the full vision-training pipeline. LocalProp applies localized optimization across pre-training, fine-tuning, and pruning, and treats gradient span as a tunable design choice.

1 INTRODUCTION

  • End-to-end backpropagation couples layers under one objective and requires storing or recomputing activations across network depth, despite its success.The paper also motivates localization by the mismatch between global error signals and biological learning.Source: §1 · p. 1
  • On CIFAR-10, LocalProp improves feature separability in early ViT-S blocks but plateaus with depth, while full backpropagation continues to improve.Figure 1 reports linear-probe top-1 accuracy at each transformer block.Source: Figure 1 · p. 2
  • 86.22% top-1 accuracy at 1.46 GiB lets four-block LocalProp use 58% less memory than full backpropagation, which reaches 87.29% at 3.48 GiB.The accuracy difference is 1.07 percentage points.Source: Figure 2 · p. 2, §1 · p. 2
  • Earlier backpropagation-free methods were tested on small architectures and datasets and did not address the full pipeline of pre-training, fine-tuning, and pruning.The cited examples include Forward-Forward and NoProp.Source: §1 · pp. 1–2
  • LocalProp applies localized optimization across self-supervised pre-training, supervised fine-tuning, structured pruning, and recovery.The framework isolates transformer blocks or convolutional parameter groups and detaches gradients at module boundaries.Source: §1 · p. 2, Figure 3 · p. 3
  • Pruned models outperform same-size from-scratch counterparts, supporting that pruning preserves useful structure rather than merely selecting a smaller architecture.The method scores attention heads or convolutional channels and uses downstream performance to guide pruning.Source: §1 · pp. 2–3

2 RELATED WORK

Prior work explores localized optimization, multi-stage vision training, and pruning largely as separate concerns. LocalProp asks whether restricted gradient propagation can span the full model life-cycle, including pruning and recovery.

Localized optimization

  • Localized learning ranges from local goodness or denoising objectives to restricted-gradient methods that optimize modules separately.The cited approaches include Forward-Forward, NoProp, Greedy InfoMax, local-error methods, Decoupled Greedy Learning, and SEDONA.Source: §2 · pp. 3–4

Multi-stage vision pipelines

  • Vision pipelines combine self-supervised pre-training, supervised adaptation, and sometimes pruning or distillation, but localized methods rarely address both pre-training and adaptation.I-JEPA predicts target-patch latent representations from a visible context region.Source: §2 · p. 4

Pruning

  • LocalProp uses gradient-free ablation scores from downstream classification to assess structural importance, then recovers pruned models with block-local supervised updates.Pruning masks remain enforced during recovery.Source: §2 · p. 4

3 METHOD

LocalProp trains and adapts models with block-local objectives, then prunes components using downstream classification loss and performs local recovery. During pre-training, gradients are limited to one active block and its predictor at a time.

3 METHOD

  • The method has four stages: local I-JEPA pre-training, local supervised fine-tuning, structured pruning, and a final local fine-tuning cleanup.Recovery retains the accumulated pruning mask.Source: §3 · pp. 4–5
  • At each pre-training update, a round-robin schedule selects one block; its predictor matches target-encoder representations at masked patches using a Smooth L1 objective.The target encoder supplies detached representations, and image labels are not used during pre-training.Source: §3 · pp. 4–5
  • During supervised fine-tuning, one block at a time is optimized with its own classifier and cross-entropy objective, while final predictions use the last block’s classifier.The shared patch embedding is also optimized during the update burst for the first block.Source: §3 · p. 5
  • Pruning scores components by the final-classification loss after temporarily suppressing them, then retains the highest-scoring components within the prescribed budget.Candidates are processed from deepest to shallowest, and scoring requires a full-encoder forward pass but no gradients.Source: §3 · p. 5
  • The method’s phase summary reports that local pre-training improves later local fine-tuning, while recovery restores much of the performance lost during pruning.The table covers top-1 accuracy on CIFAR-10, CIFAR-100, and ImageNet-1K for ViT-S and ResNet-50.Source: Table 1 · p. 6

4 EXPERIMENTS

Experiments evaluate LocalProp on CIFAR-10, CIFAR-100, and ImageNet-1K using transformer and convolutional backbones. Recovered pruned models, local updates, and adjustable gradient spans show favorable accuracy-memory trade-offs, with outcomes varying by architecture and pruning strategy.

4.1 DATASETS

  • Experiments span CIFAR-10/100 at 32×32 and ImageNet-1K at 224×224, using ViT-S/16 and grouped ResNet-50 with local training setups.The ImageNet-1K ViT-S/16 pretraining uses one transformer block per step; ResNet experiments use detached group boundaries.Source: §4.1 · p. 6, §4.2 · p. 6

4.3 RESULTS

  • ViT-S pruning reduces active parameters 21.8%, dropping ImageNet-1K accuracy 37.12% to 24.96%, then recovery raises it to 34.81%; ResNet-50 drops 89.79% to 88.23% on CIFAR-10.Recovery benefits vary across ResNet-50 datasets, and the usefulness of recovery depends on how redundancy is distributed across architectures and datasets.Source: §4.3 · p. 7
  • 75.40% vs. 66.85% on CIFAR-10 and 34.81% vs. 21.48% on ImageNet-1K: recovered pruned ViT-S beats same-size scratch models.The advantage remains when the same-size CIFAR-10 architecture is locally I-JEPA-pretrained: 75.40% vs. 70.71%.Source: §4.3 · p. 7
  • A four-block window reaches 86.22% vs. 87.29% with full fine-tuning and uses 58% less peak memory; most gains arrive before 12 blocks.The number of jointly optimized blocks controls the accuracy-memory trade-off.Source: §4.3 · p. 7
  • Detached representation-level cosine matching performs best with little difference between neighboring- and final-block targets; logit-level KL is less effective, and cross-block signals add little.The interface-objective ablation is for ViT-S on CIFAR-10.Source: Table 4 · p. 8, §4.3 · pp. 8–9
  • Local objectives yield useful, similarly scaled gradients and early-block separability, but independent block optimization limits depth gains; LocalProp maps are more object-aligned than scratch.The clearer late-layer response is largely preserved after pruning.Source: §4.3 · p. 8, Figure 4 · p. 9
  • Among ViT-S pruning choices, late-block head pruning outperforms global head pruning at near-equal active-parameter budgets; preserving depth beats dropping blocks, and uniform pruning degrades most.Table 5 compares uniform pruning, global head pruning, block dropping, and late-block head pruning on CIFAR-10 and CIFAR-100.Source: §4.3 · p. 8, Table 5 · p. 8
  • LocalProp's advantage over alternatives widens on ImageNet; local-update memory savings also widen with input resolution, while total training time depends on schedule and accuracy target.Tables 2 and 6 report peak allocated memory and block-wise versus end-to-end fine-tuning costs for ViT-S.Source: Table 3 · p. 7, Table 2 · p. 7, §4.3 · p. 8

5 CONCLUSION

The conclusion presents LocalProp as a localized-gradient framework spanning pre-training, fine-tuning, and pruning with recovery. Across the evaluated datasets and architectures, it compares favorably with local optimization approaches while reducing peak training memory.

5 CONCLUSION

  • LocalProp applies localized updates across self-supervised pre-training, supervised fine-tuning, and structured pruning with recovery.Source: §5 · p. 9
  • Across CIFAR-10, CIFAR-100, ImageNet-1K, transformers, and convolutional networks, LocalProp compares favorably with local optimization approaches and reduces peak training memory.Source: §5 · p. 9
  • Local objectives provide useful, comparably scaled signals but miss coordination across depth; gradient propagation span is a tunable coordination budget.Source: §5 · p. 9
  • The work evaluates memory-efficient vision training on CIFAR-10, CIFAR-100, and ImageNet-1K; sensitive deployments require additional fairness, privacy, and misuse assessment.Source: ETHICS STATEMENT · p. 10

A.1 ADDITIONAL METHOD DETAILS

The ResNet-50 adaptation uses group-local predictive objectives that account for changing feature-map resolutions, then applies detached local fine-tuning and structured pruning with local recovery.

Local I-JEPA for ResNet-50

  • ResNet-50 is divided into 14 local groups, each followed by a 1×1 projection and a predictor of three residual depthwise-separable convolutional blocks.The first group contains the stem and first bottleneck; later bottlenecks through Stage 3 form individual groups, and the final group contains Stage 4.Source: §A.1 · p. 13
  • The convolutional objective averages target-mask features, then uses ℓ2 normalization and cosine distance to match masked-region semantics despite spatial downsampling.Unlike the ViT objective, it does not require element-wise correspondence and retains finer token-wise Smooth L1 supervision for ViT.Source: §A.1 · p. 13

Local supervised fine-tuning for ResNet-50

  • All 14 ResNet groups update on every mini-batch from detached inputs, using local classifiers whose summed losses update only their corresponding groups.The local classifiers are discarded after adaptation, and a linear probe evaluates frozen final-group representations.Source: §A.1 · pp. 13–14

Channel pruning and local recovery for ResNet-50

  • The pruning-order comparison removes 24.29% of parameters under matched checkpoints, budgets, scoring, and recovery, with the main configuration pruning deepest-to-shallowest.The table caption specifies the matched conditions and removal rate but does not provide the accuracy values.Source: Table 8 · p. 14

Additional analysis

  • Local gradients across depth have a mean norm ratio of 1.04±0.02 relative to gradients received by the same block under full backpropagation.Figure 5(a) compares gradients induced by block-local objectives with those under full backpropagation.Source: Figure 5 · p. 14

Channel pruning and local recovery for ResNet-50

  • Channel importance is the increase in final classification loss after temporary ablation; highest-scoring channels are retained, and masks keep removed parameters inactive during local recovery.The masks cover convolutional filters and associated batch-normalization parameters, and are enforced on both parameters and gradients.Source: §A.1 · p. 14

A.2 REPRESENTATION ANALYSIS

The representation analysis finds positive, though imperfect, alignment between local and full-backpropagation gradients, and examines pruning-order sensitivity and feature structure.

Pruning and recovery settings

  • The pruning and recovery configuration enforces structured masks throughout local recovery for both ViT-S attention projections and ResNet-50 convolutional and normalization parameters.Table 10 summarizes key pruning and recovery hyperparameters.Source: Table 10 · p. 15

Gradient analysis

  • Local-loss gradients remain positively aligned with full-backpropagation gradients across blocks and datasets, but alignment is imperfect, particularly on CIFAR-100.Their mean norm ratio is 1.04 ± 0.02, so the reported difference primarily concerns direction rather than scale.Source: §A.2 · pp. 14–15

Pruning score heatmap

  • Figure 5(b) shows attention-head importance scores, defined by final classifier loss after temporarily suppressing a head; its symmetric-log scale is linear near zero and logarithmic above ΔL = 0.005.Source: §A.2 · p. 15

Pruning-order ablation

  • Reversing the ResNet-50 pruning traversal produces only small differences in recovered performance under identical starting checkpoints, budgets, masks, and recovery.The compared orders are deepest-to-shallowest and shallowest-to-deepest.Source: §A.2 · p. 15

Additional feature visualizations

  • Across three additional ImageNet examples, feature-magnitude visualizations at different blocks use the same protocol and show spatial structure preserved across different object categories.Source: §A.2 · p. 15

A.3 REPRODUCIBILITY DETAILS

The appendix identifies the final configurations and evaluation measures, reports three-seed CIFAR-10 behavior, and describes the experimental environment and code availability.

Final configurations

  • According to the cited passage, Tables 7, 9, and 10 summarize the final configurations that were used to obtain the results reported in the main paper.Source: §A.3 · p. 15

Seed sensitivity

  • Across three CIFAR-10 seeds, configuration ordering remains stable, recovery restores most pruning loss, and recovered models outperform same-mask models trained from scratch.Table 11 reports top-1 accuracy and sample standard deviation.Source: §A.3 · p. 15, Table 11 · p. 16

Hardware, software, and randomness

  • Experiments ran on NVIDIA RTX 3090 and RTX 5090 GPUs with an Intel Core i9 under Ubuntu 20.04.6, using Python 3.8 and PyTorch 2.4.1.Python, NumPy, PyTorch, and CUDA devices were seeded before model initialization and data-loader construction; each main-table entry is one complete run.Source: §A.3 · pp. 15–16

Evaluation metrics

  • Top-1 classification accuracy is the primary downstream metric; efficiency is measured by active parameter count, peak allocated GPU memory, and GPU-hours.The main paper reports these efficiency measures.Source: §A.3 · p. 16

Code and model availability

  • The supplementary code archive contains implementations and configuration files for local pre-training, fine-tuning, pruning, and recovery on both architectures.The source code and trained models are stated to be released upon publication under CC BY-NC 4.0.Source: §A.3 · p. 16

Additional feature visualizations

  • Feature-magnitude visualizations compare the input and four training or pruning conditions across blocks 0, 5, and 11 for three ImageNet-1K validation examples.Feature magnitudes are jointly normalized across methods, with dashed contours marking the same object region.Source: Figure 6 · p. 17

Related papers

Loading 2609.32517v1…