Source-linked AI summary
Transfer Learning in Physics-Informed Neural Networks: Full Fine-Tuning, Lightweight Fine-Tuning, and Low-Rank Adaptation
Yizheng Wang, Jinshuai Bai, Mohammad Sadegh Eshaghi, Cosmin Anitescu, Xiaoying Zhuang, Timon Rabczuk, Yinghua Liu
TL;DR
PINNs generally require retraining when boundary conditions, materials, or geometries change. This paper evaluates parameter-based transfer learning—full finetuning, lightweight finetuning, and LoRA—in strong- and energy-form PINNs, finding that full finetuning and LoRA improve convergence speed and accuracy in most cases.
Problem
PINNs are typically limited to specific problems and require retraining when boundary conditions, material distributions, or geometries change.
Method
The study systematically evaluates full finetuning, lightweight finetuning, and LoRA for transfer across varying boundary conditions, materials, and geometries in strong- and energy-form PINNs.
Results
Full finetuning and LoRA significantly improve convergence speed and accuracy in most cases, while lightweight finetuning performs poorly in most scenarios.
Takeaways & Limitations
Transfer learning can improve PINN training efficiency and performance across varying boundary conditions, materials, and geometries.
Takeaways & Limitations
The study does not automatically determine LoRA rank, and all layers share the same rank.
Abstract
from arXiv · showhide
AI for PDEs has garnered significant attention, particularly Physics-Informed Neural Networks (PINNs). However, PINNs are typically limited to solving specific problems, and any changes in problem conditions necessitate retraining. Therefore, we explore the generalization capability of transfer learning in the strong and energy form of PINNs across different boundary conditions, materials, and geometries. The transfer learning methods we employ include full finetuning, lightweight finetuning, and Low-Rank Adaptation (LoRA). The results demonstrate that full finetuning and LoRA can significantly improve convergence speed while providing a slight enhancement in accuracy.
1. Introduction
PINNs typically require retraining when boundary conditions, material distributions, or geometries change. This paper systematically evaluates full finetuning, lightweight finetuning, and LoRA for transfer across these varying conditions.
- PINNs are limited to specific problems, so changes in boundary conditions, material distributions, or geometries necessitate retraining.
- Transfer learning fine-tunes a pretrained model for a related task, often requiring fewer iterations and less data.
- The study compares full finetuning, lightweight finetuning, and LoRA for PINNs in strong and energy forms.
- Experiments cover transfer across boundary conditions, material distributions, and geometries.
- Full finetuning and LoRA significantly improve convergence speed while slightly enhancing accuracy across most scenarios.
2. Preparatory knowledge
The paper introduces strong-form and energy-form PINNs as two mathematical representations of PDEs. Strong form uses differential equations and residual minimization, whereas energy form uses variational principles, energy integration, and admissible functions.
- Strong-form PINNs represent PDEs from a differential-element perspective, while energy-form PINNs use variational principles from an overall-energy perspective.
- Strong form: The strong form begins with domain and boundary operators acting on the field of interest over the domain and boundary.
- Strong form: Strong-form PINNs approximate the field variable with a neural network and optimize a loss based on PDE and boundary-condition residuals.
- Energy form: Energy-form PINNs use a variation as the weight function, yielding a Galerkin formulation based on domain and boundary energy.
- Energy form: The energy method constructs an admissible function that satisfies essential Dirichlet boundary conditions in advance.
- Energy form: Using the minimum potential energy principle, the method optimizes the standard neural-network parameters by minimizing the potential energy.
3. Method: Transfer learning in PINNs
The paper systematically evaluates parameter-based transfer learning in strong and energy PINNs, using pretrained source-task parameters to accelerate convergence on related target problems. It compares full fine-tuning, lightweight fine-tuning, and LoRA, which progressively restrict the trainable parameter set.
- The study evaluates transfer learning in both strong and energy forms of PINNs.
- Parameter-based transfer learning initializes target-task models with parameters pretrained on a source problem to achieve faster convergence.
- Full fine-tuning: Full fine-tuning initializes the new model with old-task parameters and optimizes all parameters on the new dataset.
- Lightweight fine-tuning: Lightweight fine-tuning freezes most pretrained parameters and trains a smaller subset, commonly the later layers in PINNs.Freezing early layers reduces trainable parameters and computational costs.
- LoRA: LoRA keeps pretrained weights W fixed and trains low-rank matrices A and B, producing target-task weights through W ∗= W + AB.The low-rank product has rank r, typically much smaller than the layer dimensions.
- LoRA: LoRA reduces training parameters by representing an update as a low-rank matrix product, with full fine-tuning as the limiting case r = min(d, m).
4. Results
The experiments evaluate transfer learning for PINNs across changing boundary conditions, material distributions, and geometries using strong- and energy-form formulations. Full fine-tuning and LoRA generally improve convergence speed and provide slight accuracy gains, while lightweight fine-tuning can be insufficient when problem changes are large.
- Experimental scope: The study systematically tests three transfer-learning schemes across boundary conditions, materials, and geometries in PINNs.The experiments use the strong form for changing boundary conditions and the energy form for material and geometry generalization.
- Boundary conditions: PINNs accurately simulate the Taylor-Green vortex across the tested frequencies and times.The comparison includes stream function and vorticity predictions against exact solutions.
- Boundary conditions: As frequency increases, PINN accuracy decreases because fully connected networks exhibit spectral bias toward lower frequencies.The frequency parameter controls the solution-space frequency, making higher-frequency cases harder to fit.
- Boundary conditions: Lightweight fine-tuning performs worst for large problem changes because training only the final layer provides too few learnable parameters.The reported boundary-condition experiments compare lightweight fine-tuning with full fine-tuning, LoRA, and no transfer learning.
- Efficiency: Full fine-tuning generally performs best, but its larger number of learnable parameters makes each epoch take longer; LoRA can also incur greater computation despite fewer trainable parameters.Automatic differentiation reduces lightweight fine-tuning’s efficiency gains, while LoRA’s internal matrix computations can exceed those of full fine-tuning.
- Materials and geometries: For material and geometry generalization, transfer learning improves convergence speed, while accuracy advantages are modest and task-dependent.In material tests, LoRA rank 4 yields the lowest errors; in geometry tests, LoRA achieves the highest accuracy, but the improvement is not significant.
5. Discussion
The discussion relates LoRA rank to source–target similarity and reports that scenario fusion can improve accuracy without increasing total iterations.
- The Rank in LoRA: Higher source–target discrepancy requires a higher optimal LoRA rank, increasing from 12 for π→2π to 56 for π→3π.The experiments use rank r = 4 in most cases, but the optimal rank varies with domain similarity.
- The Rank in LoRA: Domain similarity can be estimated from geometry, material, boundary conditions, and PDE type using feature vectors and cosine similarity.The source and target domains are represented by feature vectors V_s and V_t.
- The Rank in LoRA: Future work should automate LoRA rank selection and allow different hidden layers to use adaptive ranks.The current work uses a predetermined rank and shares the same rank across all layers.
- Feature Fusion in Transfer Learning: 100,000 source-domain iterations followed by 100,000 target-domain iterations yields higher accuracy than 200,000 iterations without transfer learning.Both training strategies use the same total number of iterations.
6. Conclusion
The paper evaluates transfer learning in strong- and energy-form PINNs across changing boundary conditions, materials, and geometries. It reports that LoRA and full finetuning generally improve convergence speed and accuracy, while lightweight finetuning performs poorly in most scenarios.
- Conclusion: Transfer learning is evaluated in strong- and energy-form PINNs across different boundary conditions, materials, and geometries.The study examines generalization across these three types of changing problem conditions.
- Conclusion: LoRA and full finetuning significantly improve convergence speed and accuracy in most cases, whereas lightweight finetuning performs poorly in most scenarios.This is the paper’s overall conclusion across the evaluated cases.
- Future Work: The paper identifies transfer learning for operator learning and PINO as future research directions beyond PINNs.It also proposes adaptive LoRA rank selection for different hidden layers.
- Conclusion: Scenario fusion can improve PINN accuracy while keeping the total iteration count unchanged.The paper describes sequential source- and target-domain training as a future direction for exploring useful scenario combinations.
Appendix A. Taylor Green Vortex boundary and initial conditions setup
The appendix specifies the analytical-solution-based initial and boundary conditions used for the Taylor Green Vortex on the simulation domain [0, 1]^2.
- Initial Conditions: The Taylor Green Vortex initial conditions are applied from the analytical solution in Eq. (26).The appendix introduces the initial conditions before giving their explicit form.
- Boundary Conditions: The simulation domain is [0, 1]^2, and the boundary conditions are obtained for that domain.The domain specification precedes the boundary-condition formulation.
Appendix B. Computation graph of PINNs
The appendix explains why lightweight finetuning provides limited efficiency gains in PINNs: differentiating network outputs creates additional computation graphs whose cost grows with PDE derivative order.
- Computation Graphs: Lightweight finetuning does not significantly improve efficiency over full finetuning because PINNs differentiate with respect to inputs and create new computation graphs.The additional graphs arise while constructing the differential operator.
- Computation Graphs: If the network graph has C_T steps, first- and second-derivative graphs require 2C_T and 4C_T steps, respectively.The appendix uses T = NN(x; θ) to illustrate this growth.
- Computation Graphs: Full finetuning has a maximum graph size of 2MC, while last-layer lightweight finetuning has (2M − 1)C + 1 steps.Here M denotes the highest PDE derivative order in the appendix’s formulation.
- Efficiency Implication: Lightweight finetuning’s backpropagation efficiency gain decreases as the highest derivative order M increases.The number of graphs traversed becomes less different from full finetuning for higher-order differential operators.
Appendix C. Computation of LoRA
LoRA reduces the number of trainable parameters but adds computation to gradient calculation, resulting in longer computation time than full finetuning.
- Gradient computation: The gradient computation cost for LoRA remains comparable to that for the full parameter matrix because W* and W have the same shape.
- Gradient computation: LoRA introduces an additional matrix multiplication during gradient calculation.The added multiplication increases the internal computation cost despite reducing the number of trainable parameters.
- Computational cost: LoRA takes longer to compute than full finetuning, as shown in Table 1.