PaperlayerView exact revision on arXiv

Source-linked AI summary

Progressive Permutation Alignment with Temperature-Controlled Spectral Sharpening

Yuta Horioka, Naoya Chiba, Yuki Uranishi

arXiv:2610.29396v1cs.LGcs.CV

TL;DR

Permutation symmetry can make functionally equivalent INR models distant in weight space, complicating interpolation. The paper aligns models progressively using temperature-controlled spectral annealing and a subsequent layer-wise refinement. It reports reliable alignment across architectures and reduced interpolation loss barriers for independently trained NeRFs.

  • Problem

    Permutation symmetry makes functionally equivalent models appear different in weight space, while simple interpolation can produce high loss barriers [8].

    Source: §1 · p. 1

  • Method

    The method aligns layers from input to output using Hungarian anchors and temperature-controlled spectral annealing, then performs sequential layer-wise weight-matching refinement.

    Source: §1 · p. 3, §5 · p. 14

  • Results

    The method demonstrates reliable alignment for independently trained NeRFs and reduced loss barriers in practical weight-interpolation experiments.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The reported NeRF results support using neural fields as weight-space representations with smoother interpolation.

    Source: §1 · p. 3

    The formulation omits bias terms, though the authors state it can be extended to include them.

    Source: §3.1 · p. 6

Abstract

from arXiv · show

In implicit neural representations (INRs), 3D signals are encoded directly in network weights, making stable weight-space operations essential. However, permutation symmetry causes independently trained yet functionally equivalent models to appear distant in weight space, often inducing large loss barriers under linear interpolation. Existing neuron alignment methods either commit early to discrete matches and become trapped in inconsistent local optima or rely on computationally expensive gradient optimization over continuous relaxations, both of which are unreliable in challenging cases such as NeRF. We adopt progressive alignment, a layer-wise strategy that follows inter-layer dependencies from input to output. At each layer, a correlation matrix is estimated and aligned with a discrete anchor obtained by the Hungarian algorithm, after which temperature-controlled spectral annealing gradually sharpens the correlation structure toward a permutation matrix. This update is deterministic and lightweight, avoiding the inner-loop gradient optimization required by continuous relaxation methods. A final refinement stage further optimizes the network layer-by-layer via weight matching, starting from the aligned solution. Experiments on MLPs, CNNs, and Neural Radiance Fields (NeRFs) show that the proposed method reliably recovers neuron correspondences and achieves stable alignment across architectures. In practical experiments aligning independently trained NeRF models, our method reduces the loss barriers under weight interpolation compared with existing approaches.

1 Introduction

INRs store signals in network weights, but permutation symmetry can make functionally equivalent models appear far apart and create loss barriers during interpolation. The paper proposes progressive, temperature-controlled alignment to address difficult cases such as independently trained NeRFs.

1 Introduction

  • Permutation symmetry allows hidden units to be reordered without changing a network’s function, while simple averaging or interpolation can produce high loss barriers.INRs encode continuous signals directly in weights, making weight-space operations relevant to their use as representations.Source: §1 · pp. 1–2
  • Progressive alignment follows inter-layer dependencies from input to output, aligning each layer’s correlation matrix to a Hungarian anchor before spectral annealing sharpens it toward a permutation matrix.The approach is presented as bridging continuous relaxation and discrete matching without iterative optimization.Source: Figure 1 · p. 2, §1 · p. 3
  • The method demonstrates reliable alignment for independently trained NeRFs and reduces loss barriers in practical weight-interpolation experiments.The introduction identifies noisy, ambiguous correlations as a challenge and contrasts gradual spectral annealing with early discrete decisions that can propagate errors across layers.Source: §1 · p. 3
  • The paper frames permutation alignment as a sequential problem and introduces temperature-controlled spectral projection to avoid premature commitment to local optima.The proposed two-stage strategy uses the alignment as an initialization for sequential layer-wise weight matching.Source: §1 · p. 3
  • The proposed approach is applied to NeRFs, with the paper reporting reliable alignment and reduced loss barriers for independently trained models.The paper presents this as a challenging practical setting for weight-space alignment.Source: §1 · p. 3

2 Related Work

Related work connects permutation symmetry to mode connectivity, reviews discrete and continuous alignment approaches, and describes both progressive strategies and INR-specific weight representations.

2.1 Permutation Symmetries and Mode Connectivity of Neural Networks

  • Permutation symmetry yields many equivalent network parameterizations and is connected to loss-landscape geometry and mode connectivity; suitable alignment has been conjectured to place independently trained models in one connected basin.This line of work extends to larger architectures, multi-model settings, and training-time alignment.Source: §2.1 · pp. 3–4

2.2 Permutation Alignment of Neural Networks

  • Discrete weight matching solves a layer-wise linear assignment problem, but incorrect matches can propagate and results may vary with update order, initialization, or random seed.Continuous alternatives include doubly stochastic relaxations and OT4P’s gradient optimization over orthogonal matrices [4,21,26,12].Source: §2.2 · p. 4
  • Progressive layer-wise approaches such as FedMA, ZipIt!, ProDistill, and PLeaS follow directional dependencies, which prior work associates with more stable alignment and merging.Spectral signals have also been used to resolve permutation ambiguity in transformer head matching.Source: §2.2 · p. 4

2.3 Weight Representations of INRs

  • INR weight representations support downstream learning, but permutation sensitivity can cause high loss barriers or representation collapse.Symmetry-aware architectures address the issue, yet stable post-hoc alignment of independently trained INRs remains challenging.Source: §2.3 · p. 5

3 Proposed Method

The method aligns networks progressively, using layer correlations and temperature-controlled spectral sharpening to move from continuous alignments toward discrete neuron correspondences. Its objective is to reorder hidden neurons to reduce weight-space distance while respecting dependencies between neighboring layers.

3.1 Problem Formulation

  • The objective is to find permutations that reorder one network’s hidden neurons to align it functionally with another while minimizing weight-space distance.The global loss sums squared differences between aligned layer weights; input and output permutations are fixed to identity, and the joint discrete problem is NP-hard.Source: §3.1 · pp. 5–6

3.2 Progressive Alignment Based on Correlation Maximization

  • With neighboring permutations fixed, each layer update maximizes a correlation matrix that captures neuron-wise consistency with adjacent layers.Progressing from input to output decomposes the combinatorial problem into iterative layer-wise subproblems.Source: §3.2 · p. 6

3 Proposed Method

  • Progressive alignment combines Hungarian anchors with temperature-controlled spectral annealing, sharpening layer correlations toward discrete permutations without iterative gradient optimization.The schedule preserves continuous structure early to explore global consistency, then emphasizes high-confidence modes as temperature decreases.Source: §3.3 · pp. 7–8

3.3 Temperature-Controlled Spectral Annealing

  • At each layer, the method solves a linear assignment with the Hungarian algorithm for an anchor permutation, applies temperature-controlled SVD spectral annealing, and interpolates between the annealed correlation matrix and identity according to current temperature.Source: §3.3 · pp. 7–8

4 Experiment

The experiments assess alignment accuracy on MLPs and CNNs, then test permutation recovery and interpolation between NeRF models. The proposed method recovers known permutations and substantially suppresses NeRF interpolation loss barriers.

4.1 Comparison Methods

  • The comparison includes naive interpolation, weight matching, progressive variants, OT4P, and the proposed method with or without final weight-matching refinement.The complete method uses spectral projection as an initialization followed by iterative layer-wise weight matching.Source: §4.1 · p. 9

4.2 Evaluation of Alignment Accuracy on MLPs and CNNs

  • On MLP5, VGG11, and ResNet18, weight matching and the proposed method achieve 100% precision and L1 distance ≈0, unlike OT4P and Progressive OT4P.OT4P takes about 40 minutes per VGG11 run and misses exact recovery on the CNNs; Progressive OT4P precision varies with network and temperature.Source: §4.2 · pp. 10–11, Table 1 · p. 10

4.3 Alignment and LMC Evaluation on NeRF

  • NeRF tests cover eight NeRF Synthetic scenes, using both random-permutation recovery and interpolation between independently initialized models.For interpolation, the study evaluates rendered views along parameter-space interpolation, with ten sampled test images per scene.Source: Evaluation on Random Permutation… · p. 11, Alignment of NeRF Models Trained from… · pp. 11–12
  • 100% precision and L1 distance ≈0 show that both the proposed method and weight matching restore the original NeRF weights, while Progressive OT4P reaches at most 6.3% precision.OT4P reaches about 99% at τ = 0.5 and 0.7 but falls to 14.55% precision at τ = 0.3.Source: Evaluation on Random Permutation… · p. 11, Table 2 · p. 12
  • At λ = 0.5, the proposed method and its sequential-refinement variant reach comparable average accuracy (14.22 vs. 14.40 dB PSNR) and substantially suppress NeRF interpolation loss barriers.Slight blurring remains at the midpoint, while overall scene structure and object contours are preserved; refinement mainly reduces sensitivity to the temperature schedule.Source: Table 3 · p. 12, Alignment of NeRF Models Trained from… · pp. 13–14

5 Conclusion

The conclusion presents progressive alignment with temperature-controlled spectral sharpening as a method for addressing permutation symmetry in Neural Fields and INRs. Experiments report accurate permutation recovery on MLPs and CNNs, reliable recovery and reduced loss barriers on NeRF, with independently trained-model evaluation limited to MLP-based NeRFs on NeRF Synthetic.

5 Conclusion

  • The method combines a Hungarian anchor with temperature-controlled spectral annealing, followed by sequential layer-wise refinement to improve consistency across layers.The conclusion describes the procedure as deterministic and efficient, avoiding costly iterative optimization used by gradient-based relaxations.Source: §5 · p. 14
  • Experiments demonstrate accurate recovery of known permutations on MLPs and CNNs, plus reliable recovery and reduced interpolation loss barriers on NeRF.The reported independently trained-model evaluation is limited to MLP-based NeRFs on NeRF Synthetic; other INRs and real scenes are future work.Source: §5 · p. 14

Supplemental Materials of Progressive

The supplied passage gives the supplemental-materials section title.

Supplemental Materials of Progressive

A Additional Baseline Comparisons: Activation Matching and STE

The supplemental comparison adds Activation Matching and STE as baselines at the interpolation midpoint. The proposed method outperforms STE across the reported metrics, while Activation Matching relies on data-dependent inputs.

A Additional Baseline Comparisons: Activation Matching and STE

B Ablation Study

The ablation study evaluates the contributions of the proposed method’s components and its sensitivity to temperature scheduling on the NeRF Synthetic dataset.

B Ablation Study

  • The ablation study evaluates the contribution of each component of the proposed method and sensitivity to the temperature schedule τ.The experiments use the NeRF Synthetic dataset.Source: B Ablation Study · pp. 18–19

B.1 Importance of Temperature-Controlled Spectral Projection

The ablation compares variants of the proposed method to assess spectral sharpening and sequential refinement. The reported results indicate that spectral projection and final refinement contribute to interpolation quality.

B.1 Importance of Temperature-Controlled Spectral Projection

  • The ablation includes Naive, Weight Matching, and Progressive Weight Matching baselines, alongside variants that omit spectral projection, refinement, or both.The complete method combines spectrally sharpened continuous alignment with final sequential layer-wise refinement.Source: §B.1 · p. 19
  • The complete method consistently improves performance over the variant without spectral projection or refinement.The ablation reports average NeRF Synthetic metrics at λ = 0.5 and summarizes interpolation trajectories.Source: §B.1 · p. 19, Table S2 · p. 19, Figure S1 · p. 20
  • Spectral projection is critical: using the raw correlation matrix is insufficient to navigate the complex loss landscape and suppress noise.The spectral sharpening emphasizes dominant permutation structures during progressive updates.Source: §B.1 · p. 19
  • Final sequential layer-wise refinement consistently improves PSNR after spectral sharpening, supporting the continuous solution as an initialization for discrete local optimization.The trajectory ablation also reports that refinement further stabilizes alignment.Source: §B.1 · p. 20, Figure S1 · p. 20

B.2 Effect of Temperature Scheduling for τ

The three-step temperature schedule provides the strongest and most stable alignment quality among the tested schedules. The supplementary results also report alignment computation times across architectures, including a large speed advantage over OT4P.

B.2 Effect of Temperature Scheduling for τ

  • The three-step temperature schedule is the clear optimal choice, outperforming the two-step and five-step schedules across evaluated metrics.The tested schedules are {1.0, 0.0}, {1.0, 0.5, 0.0}, and {1.0, 0.75, 0.5, 0.25, 0.0}; the three-step schedule is reported as most stable with and without refinement.Source: §B.2 · pp. 20–21, Table S3 · p. 21, Figure S2 · p. 21

C Additional Quantitative and Qualitative Results

  • The supplementary section reports computational efficiency across architectures, alongside further quantitative and qualitative interpolation evaluations.The evaluation includes execution time and extended interpolation comparisons.Source: C Additional Quantitative and… · p. 21

C.1 Computational Efficiency

  • 48 seconds for VGG11 alignment versus about 40 minutes for OT4P—roughly 50 times longer; NeRF refinement adds approximately 3 seconds.The method avoids the inner-loop gradient optimization used by continuous relaxation methods such as OT4P.Source: §C.1 · p. 22, Table S4 · p. 22, Table S5 · p. 22

C.2 Extended Qualitative Comparisons

Across the NeRF Synthetic scenes, the proposed method preserves interpolation quality better, especially near the midpoint, and produces more stable visual transitions than the compared methods. The supplementary material also derives a trace-maximization form for the layer-wise alignment objective.

C.2 Extended Qualitative Comparisons

  • The proposed method preserves higher average SSIM than the baselines, particularly near the middle of the interpolation path.Across all eight scenes, its SSIM drop around the midpoint is consistently reduced.Source: §C.2 · p. 23, Figure S3 · p. 23, Figure S4 · p. 25
  • The proposed method maintains lower RMSE and higher PSNR along the interpolation path, with the largest PSNR differences near the midpoint.The reported curves cover all eight NeRF Synthetic scenes.Source: §C.2 · p. 23, Figure S5 · p. 26, Figure S6 · p. 26
  • For the Lego scene, the proposed method preserves structure and appearance more consistently than the baselines, and sequential refinement further stabilizes the transition.The compared renderings use interpolation coefficients λ = 0.0, 0.25, 0.50, 0.75, and 1.0.Source: §C.2 · p. 24, Figure S7 · p. 27
  • In Chair and Ficus, baseline renderings often collapse or become view-inconsistent near λ = 0.5, while the proposed method reduces these failures and yields smoother transitions.Additional comparisons show renderings for Drums, Hotdog, Mic, Materials, and Ship.Source: §C.2 · p. 24, Figure S8 · p. 28, Figure S9 · p. 29

D Derivation of the Trace-Maximization Form

  • The derivation uses Frobenius-norm invariance and trace cyclicity to rewrite the layer-dependent alignment objective as maximizing Tr(P_l^T C_l).The layer permutation P_l is optimized with adjacent permutations held fixed.Source: D Derivation of the Trace-Maximization… · p. 35

E An example where Progressive Weight Matching fails

In a three-layer linear example, alternating Weight Matching settles on incorrect correspondences, while progressive spectral sharpening recovers the correct ones.

E An example where Progressive Weight Matching fails

Related papers

Loading 2610.29396v1…