Source-linked AI summary
HeuFouFT: Task-Guided Metaheuristic Coordinate Search for Fourier Fine-Tuning
Ruiheng Wang, Yubo Hou, Yakun Zhu, Tianle Shen, Tao Wan, Zengchang Qin
TL;DR
Fixed uniform and Gaussian band-pass Fourier sampling does not adapt selected coordinates to downstream tasks. HeuFouFT uses block-level task feedback and metaheuristic search to select coordinates; on GPT-2-Medium and E2E, its variants outperform fixed sampling and LoRA, with PSO also beating LoCA on four metrics.
Problem
Uniform and Gaussian band-pass FourierFT use fixed, task-agnostic coordinate rules, although coordinate choice matters under a limited spectral budget.
Source: §1 · p. 1Method
HeuFouFT combines block-level task feedback with surrogate-assisted metaheuristic search to select a discrete, layer-shared Fourier coordinate set before final fine-tuning.
Source: §1 · p. 1Results
Takeaways & Limitations
Task-guided search allocates limited spectral capacity effectively and, after coordinate selection, uses 15–18% of Full FT FLOPs.
Source: §5 · p. 4
Abstract
from arXiv · showhide
We introduce Heuristic-Guided Fourier Fine-Tuning (HeuFouFT), a task-guided framework for selecting trainable frequency coordinates in Fourier fine-tuning. Existing uniform and Gaussian band-pass schemes allocate a limited spectral budget through fixed, task-agnostic rules. HeuFouFT instead searches for coordinates using downstream performance. A coarse intensity map from lightweight block-level probes initializes three metaheuristic optimizers: Genetic Algorithm with Simulated Annealing (GA-SA), Particle Swarm Optimization (PSO), and Cuckoo Search (CS). During search, a Random Forest filters each population so that only the top 30% of candidates proceed to proxy fine-tuning. On E2E with GPT-2-Medium, all three variants outperform random-uniform FourierFT, Gaussian band-pass FourierFT, and LoRA across five metrics. PSO further outperforms LoCA, the best-performing baseline, on four metrics while using 37.6% fewer trainable spectral coefficients. Once coordinates are selected, HeuFouFT requires only 15--18% FLOPs of Full FT. These results show that task-guided search allocates limited spectral capacity more effectively than fixed sampling. Our code is publicly available.
1. INTRODUCTION
HeuFouFT replaces fixed, task-agnostic Fourier coordinate sampling with downstream-performance-guided search. On GPT-2-Medium and E2E, its three variants outperform fixed sampling and LoRA, with PSO also outperforming LoCA on four metrics.
1. INTRODUCTION
- PEFT keeps pretrained weights frozen while learning a small parameter set; LoRA uses low-rank factors, while spectral methods learn transform-domain coefficients.Related approaches include spectral-complexity-based F-Adapter and Fourier/cosine low-rank adaptation for vision in FouRA.Source: §1 · p. 1
- FourierFT's uniform and Gaussian band-pass rules are task-agnostic, so their selected coordinates cannot adapt to downstream tasks.LoCA instead jointly learns coefficient values and locations in a cosine representation.Source: §1 · p. 1
- Figure 1 shows localized LoRA-update spectral energy that uniform or Gaussian band-pass samples cover poorly, motivating task-feedback-guided coordinate selection.The spectrum motivates this choice but is not an oracle for FourierFT.Source: §1 · p. 1, Figure 1 · p. 1
- HeuFouFT uses a block-level task prior, surrogate assistance, and GA-SA, PSO, or CS to refine complete Fourier coordinate sets.The framework selects coordinates based on downstream performance rather than a fixed sampling rule.Source: §1 · p. 1
- All three variants outperform fixed sampling and LoRA across five metrics using 37.6% fewer spectral coefficients than FourierFT and LoCA configurations; PSO also beats LoCA on four metrics.These results are on GPT-2-Medium and E2E.Source: §1 · p. 2
2. FREQUENCY-DOMAIN PARAMETERIZATION AND MOTIVATION
FourierFT represents weight updates with a sparse set of selected Fourier components and trainable coefficients. With a fixed coefficient budget, coordinate choice determines the available adaptation directions, motivating task feedback rather than spectral magnitude alone.
2. FREQUENCY-DOMAIN PARAMETERIZATION AND MOTIVATION
- FourierFT places trainable coefficients at selected coordinates in a sparse spectrum, then applies an inverse two-dimensional DFT and real projection to form the update.Each adapted matrix learns n real coefficients while sharing the selected coordinate set and update scale.Source: §2 · p. 2
- For fixed budget n, selected coordinates determine the update subspace, while learned coefficients adjust component amplitudes.Figure 1 shows localized LoRA-update energy peaks that fixed uniform and annular sampling may miss.Source: §2 · p. 2
- Spectral magnitude alone cannot rank regions by adaptation value, and the preferred Gaussian band-pass setting varies across metrics, so task feedback is needed.Block-level probes supply feedback for selecting where to place the parameter budget.Source: §2 · p. 2
3. FREQUENCY-DOMAIN SAMPLING APPROACHES
The section contrasts fixed Fourier sampling with a task-informed pipeline that scores coarse frequency blocks and searches for complete coordinate sets. GA-SA, PSO, and CS provide distinct search strategies under a fixed sampling budget.
3. FREQUENCY-DOMAIN SAMPLING APPROACHES
- Uniform sampling selects n coordinates with equal probability, while Gaussian band-pass sampling favors an annulus set by frequency bias fc and bandwidth W; fc = 0 gives a low-pass region.Source: §3 · p. 2
3.1. Heuristic Information Foundation
- A 10 × 10 grid of block probes uses one-epoch lightweight fine-tuning on 10% of training data; normalized, weighted NLG metrics form an intensity map that correlates with LoRA-update energy (r = 0.85).Block scores reflect adaptation value rather than spectral magnitude, but cannot identify important coordinates within a block; search refines selection below block granularity.Source: §3.1 · p. 2
- A Random Forest predicts population fitness, sending only the top 30% of candidates to proxy fine-tuning and validation; its held-out R2 is 0.86.Search fitness uses validation HI after one-epoch fine-tuning on 10% of training data, while reported results use the held-out test set.Source: §3.1 · p. 2, §3.2 · p. 2
3.2. Metaheuristic Sampling Algorithms
- GA-SA evolves complete coordinate sets through crossover and mutation with simulated-annealing acceptance, while PSO updates sampling sets using particle velocities guided by personal and swarm bests.SA can accept lower-fitness candidates probabilistically; PSO's inertia decreases from 0.9 to 0.4 over 200 iterations.Source: §3.2 · pp. 2–3
- CS represents each coordinate set as a binary mask and perturbs it with heavy-tailed Lévy flights for long-range exploration, subject to budget and coordinate uniqueness.Its step size decays from 0.5 to 0.1 over the first 100 iterations.Source: §3.2 · p. 3
4. EXPERIMENTAL STUDIES
On GPT-2-Medium and E2E, HeuFouFT’s coordinate-search variants outperform fixed-sampling baselines and LoRA with fewer spectral coefficients, while using substantially less post-selection compute than Full FT. The experiments compare Gaussian sampling, optimizer behavior, and search and fine-tuning efficiency.
4. EXPERIMENTAL STUDIES
- The evaluation uses GPT-2-Medium on E2E and compares Full FT, LoRA, LoCA, and uniform or Gaussian band-pass FourierFT; HeuFouFT uses 29,952 parameters versus 48,000 for FourierFT and LoCA.HeuFouFT’s selected coordinates are fixed across three seeds, so reported standard deviations reflect fine-tuning stochasticity rather than search variance.Source: §4 · p. 3
4.2. Overall Performance
- PSO leads on BLEU, NIST, METEOR, and CIDEr, while CS leads on ROUGE-L; all three variants outperform fixed sampling and LoRA with 37.6% fewer coefficients than FourierFT and LoCA configurations.Table 2 reports GPT-2-Medium performance on E2E; its error terms are means and standard deviations over three seeds.Source: Table 2 · p. 3, §4.2 · p. 3, §4 · p. 3
4.1. Frequency Bias and Block Sampling
- Across six tested Gaussian frequency biases, the best setting differs by metric, and shifting the annulus does not optimize individual coordinates using task feedback.The tests use fc values from 0 to 500 in steps of 100, with W = 20.Source: §4.1 · p. 3
4.2. Overall Performance
- All three search variants share one intensity map, and adding simulated annealing to Vanilla GA improves all five metrics.The authors interpret the consistent gain as supporting simulated annealing’s role in escaping local optima of greedy evolution.Source: §4.2 · pp. 3–4
- PSO clusters around a compact middle-frequency region, CS explores more broadly through Lévy flights, and GA-SA retains a more balanced middle-to-high-frequency distribution.The paper reports that these distinct search profiles achieve similar downstream performance.Source: §4.2 · p. 4
4.3. Sampling Behavior and Efficiency
- PSO places 84.5% of coordinates in the middle band and has the highest energy concentration and sampling quality; CS places 79.6% in the high band and has the highest coverage efficiency.Figure 3 presents frequency ratios, energy concentration, coverage efficiency, and sampling quality; these profiles align with PSO leading four metrics and CS leading ROUGE-L.Source: §4.3 · p. 4, Figure 3 · p. 4
- After coordinate selection, HeuFouFT uses 15–18% of Full FT FLOPs and about 9% of its peak memory, compared with 25% and 12% for FourierFT and LoCA.The authors report a separate search cost and note that coordinate selection is paid once per task and can be amortized when the selected set is reused.Source: §4.3 · p. 4
5. CONCLUSION
HeuFouFT selects Fourier coordinates through block-level probes, Random Forest filtering, and metaheuristic search. On GPT-2-Medium and E2E, its variants outperform fixed sampling and LoRA with fewer coefficients, while PSO outperforms LoCA on four metrics.
5. CONCLUSION
- HeuFouFT combines block-level probes, Random Forest filtering, and metaheuristic search to select Fourier coordinates.The three searches reach similar performance despite selecting distinct frequency distributions, which the authors attribute to task-guided selection rather than the optimizer.Source: §5 · p. 4
- On GPT-2-Medium and E2E, all three variants outperform fixed sampling and LoRA across five metrics with 37.6% fewer coefficients than FourierFT and LoCA configurations; PSO further outperforms LoCA on four metrics.Source: §5 · p. 4
- After coordinates are selected, HeuFouFT requires only 15–18% of Full FT FLOPs.Source: §5 · p. 4
- The authors conclude that task-guided coordinate search can allocate limited spectral adaptation capacity effectively.Source: §5 · p. 4
A. FREQUENCY DOMAIN INFORMATION METRICS
The section defines frequency-domain measures for energy uniformity, low-frequency concentration, weighted coverage, and retained update energy.
A.1. Spectral Entropy
- Spectral entropy is the Shannon entropy of spectral density; higher values indicate more uniform frequency-domain energy and lower information density.Its range is [0, log(MN)].Source: §A.1 · p. 6
A.2. Spectral Information Density
- Spectral Information Density is spectral entropy divided by maximum possible entropy, with higher values indicating more uniform energy and lower information density.Its range is [0, 1], and maximum entropy corresponds to a completely uniform distribution.Source: §A.2 · p. 6
A.3. Low-Frequency Energy Ratio
- Low-Frequency Energy Ratio measures the proportion of total energy in a centered low-frequency region; higher values indicate a greater low-frequency share.The low-frequency region is usually circular or square, centered on the spectrum, with radius R.Source: §A.3 · p. 6
A.4. Coverage Efficiency
- Coverage Efficiency normalizes the weighted count of sampled points in high-value regions by their weighted area, using Heuristic Intensity × Energy Density as weight.The high-value regions satisfy HI(u, v) · ED(u, v) ≥ τ.Source: §A.4 · p. 6
A.5. Energy Concentration
- Energy Concentration measures how much key weight-update energy selected frequency points retain; it is the sampled energy divided by total frequency-domain energy.The measure ranges from (0, 1], with M selected points compared against the M highest-energy components.Source: §A.5 · pp. 6–7
B.1. BLEU
This section describes five text-generation metrics and the Random Forest surrogate’s role in screening frequency candidates, along with its feature and hyperparameter setup.
B.1. BLEU
- BLEU measures candidate-reference n-gram overlap with a length penalty, but is semantics-agnostic and can overvalue short texts.Its clipped n-gram precisions use uniform weights by default for 1–4-grams.Source: §B.1 · p. 7
B.2. NIST
- NIST uses information-weighted n-grams and a weighted arithmetic mean, emphasizing rare informative n-grams while underweighting low-frequency terms.Its scores range from 0 to 10.Source: §B.2 · p. 7
B.3. METEOR
- METEOR aligns candidate and reference texts using stem and synonym matches, penalizes word-order mismatches, and combines precision and recall harmonically.Its semantic and word-order sensitivity comes with higher computational complexity.Source: §B.3 · p. 7
B.4. ROUGE-L
- ROUGE-L measures similarity through the longest common subsequence, capturing paraphrasing and semantic coherence while ignoring word-order details.Its β parameter balances recall and precision.Source: §B.4 · p. 7
B.5. CIDEr
- CIDEr measures candidate-description coherence using TF-IDF-weighted n-gram similarity and geometric averaging across references.It rewards semantic diversity while ignoring syntax errors.Source: §B.5 · p. 8
C.1. Motivation and Design Rationale
- The Random Forest surrogate predicts candidate fine-tuning performance from spectral characteristics to screen low-potential candidates before metaheuristic optimization.Its stated purpose is to reduce redundant computation while preserving high-performance sampling points.Source: §C.1 · p. 8
C.2. Feature Selection for Spectral Characterization
- The surrogate uses seven spectral features computed from the DFT-transformed weight-update signal to represent energy distribution, information density, and frequency structure.The features include mandatory indicators and domain-specific supplementary features.Source: §C.2 · p. 8
C.3. Hyperparameter Configuration
- Random Forest hyperparameters were tuned to balance prediction accuracy and computational efficiency under the GPT-2 Medium and E2E setup.The optimal settings are referenced in Table 3, but are not specified in this passage.Source: §C.3 · p. 8
C.4. Training Algorithm Flow
The subsection identifies the Random Forest training procedure as a data-driven pipeline within HeuFouFT’s heuristic sampling framework.
C.5. Training Algorithm Flow
- Random Forest surrogate training follows a data-driven pipeline.The pipeline is integrated with HeuFouFT’s heuristic sampling framework and detailed in Algorithm 1.Source: §C.4 · p. 8
- The surrogate-training pipeline is integrated with HeuFouFT’s heuristic sampling framework.Algorithm 1 is cited as the detailed description of the pipeline.Source: §C.4 · p. 8
- Algorithm 1 provides the detailed account of the Random Forest surrogate-training pipeline.Source: §C.4 · p. 8
C.5. Computational Cost Evaluation
The evaluation reports lower computational cost from Random Forest filtering, while noting that data collection dominates its overhead but is incurred once.
C.5. Computational Cost Evaluation
- 68% lower overall computational cost follows Random Forest filtering of 70% of low-potential candidates.The evaluation uses a single NVIDIA A100 80GB GPU at FP16 precision.Source: §C.5 · p. 8
- Data collection accounts for 81% of RF-related cost, but is a one-time overhead across multiple metaheuristic iterations.RF inference takes less than 0.5 GPU-hours.Source: §C.5 · p. 8
- The reported cost is described as aligning with typical cloud-GPU fine-tuning expenses and as scalable to larger models such as Llama 3-7B.Source: §C.5 · p. 8
D. COMPUTATIONAL COST ANALYSIS OF META-HEURISTIC SAMPLING ALGORITHMS
Costs cover intensity-map probes, surrogate labeling, and metaheuristic searches; end-to-end compute and memory measurements include final fine-tuning.
D. COMPUTATIONAL COST ANALYSIS OF META-HEURISTIC SAMPLING ALGORITHMS
- 5,000 sampling schemes take 6.94 hours and 2.05 × 10^19 FLOPs to label, while map construction takes 1.25 hours and 8.2 × 10^17 FLOPs.Each scheme uses 30 seconds of lightweight fine-tuning; the map uses 100 nonoverlapping blocks, each taking 45 seconds.Source: D. COMPUTATIONAL COST ANALYSIS OF… · p. 8
- GA-SA search takes 175 minutes (2.92 hours) and 6.3 × 10^16 FLOPs across 175 iterations.Each iteration takes 60 seconds.Source: D. COMPUTATIONAL COST ANALYSIS OF… · p. 8
- CS search takes 75.5 minutes (1.26 hours) and 2.87 × 10^16 FLOPs, while PSO takes 180 minutes (3 hours) and 6.72 × 10^16 FLOPs.CS uses 151 iterations and PSO uses 240 iterations.Source: D. COMPUTATIONAL COST ANALYSIS OF… · p. 9
- End-to-end FLOPs include map construction, surrogate labeling and training, search, and final fine-tuning; measurements use one A100 80GB GPU with FP16.Wall-clock time excludes data loading, and each step is repeated three times for an average.Source: D. COMPUTATIONAL COST ANALYSIS OF… · pp. 8–9