POLARIS: Projection-Orthogonal Least Squares for Robust and Adaptive Inversion in Diffusion Models

1. Introduction

• DDIM inversion supplies the latent trajectory for diffusion editing and restoration, making its reliability foundational to those applications.The related-work discussion also describes faithful inverted latents as supporting a broad family of downstream tasks.Source: §1 · p. 2, §2 · p. 2
• Consecutive-step noise-prediction discrepancies accumulate during inversion, and CFG amplifies the error, contributing to artifacts and departure from the source image.Prior corrective methods address downstream effects and can add complexity, reduce efficiency, and weaken background preservation.Source: §1 · p. 2
• POLARIS reframes inversion from downstream error compensation to addressing the noise-prediction discrepancy between consecutive steps.The paper characterizes this discrepancy as the root cause of trajectory drift amplified by fixed CFG.Source: §1 · p. 2
• Across editing and restoration tasks, POLARIS substantially improves reconstruction fidelity and editing quality across multiple models with negligible overhead.The introduction reports experiments on Pick-a-Pic and COCO 2017 and describes the module as plug-and-play.Source: §1 · p. 2

2. Related Work

• Earlier inversion approaches optimize conditioning or latent codes, while downstream editing and restoration methods use attention manipulation, feature injection, or measurement-consistency updates.Examples include Null-Text Inversion and Negative Prompt Inversion for conditioning, and Prompt to Prompt, Plug-and-Play, DDRM, DDNM, and DPS for downstream tasks.Source: §2 · pp. 2–3
• POLARIS treats inversion as direct error minimization, using a dynamic per-step guidance scale rather than optimizing embeddings or latents to offset drift.Its closed-form update is derived from an orthogonality condition between conditional and unconditional noise-change vectors.Source: §1 · p. 2, §2 · p. 3

3. Preliminary

• DDIM defines a deterministic denoising process, mapping a sample at one timestep to a cleaner sample using a noise prediction and schedule.The preliminary section presents inversion as reversing this process to map a data sample back to noise.Source: §3 · p. 3
• In inversion, the ideal noise prediction cannot be computed directly because it depends on the unknown state, so the current-state prediction is used as an approximation.This approximation makes inversion computationally tractable.Source: §3 · p. 3
• Because of this approximation, DDIM inversion is nearly, but not perfectly, invertible.Source: §3 · p. 3

Part 1: Inversion

• The algorithm takes a source image and prompt as input and returns an inverted latent together with a recorded scale list.Source: Part 1: Inversion · p. 3
• After encoding the image, each inversion step computes conditional and unconditional noise-change vectors and records the step’s guidance scale.The procedure initializes the scale at the first step and computes the later scales from the changes between adjacent steps.Source: Part 1: Inversion · p. 3
• At each step, the guided noise prediction combines unconditional and conditional predictions, with the guidance scale controlling their contributions.Standard DDIM sampling and inversion typically use a constant scale across timesteps.Source: Part 1: Inversion · pp. 3–4
• Figure 2 contrasts accumulated error in existing CFG-based DDIM inversion with POLARIS’s closer-to-ideal latent path and projection-based error-minimization objective.During generation, the recorded ωt sequence is replayed for reconstruction.Source: Figure 2 · p. 4

4.1. The Exact Solution and Its Instability

• The exact approach minimizes ∥a + b∆ω∥² to choose a guidance-scale change recursively from an initial scale, with a measuring current-step changes and b the prior guidance direction.The scale for each step is built from the previous step’s value and the optimal change.Source: §4.1 · p. 4
• The exact scale computation is ill-posed: small prediction noise can make ∆ω error diverge as the historical guidance direction ∥b∥ approaches zero.Its dynamic scale also fluctuates extremely and produces no significant reconstruction improvement over fixed scale.Source: §4.1 · p. 5

4.2. A Robust Approximate Solution

• The squared error-term ratio exceeded 1 across most steps and averaged over 20; ∥a∥ averaged 3.2662 versus 0.7215 for |∆ω|·∥b∥.Figure 3 compares the average magnitudes of the current-step and history-dependent terms.Source: §4.2 · p. 5, Figure 3 · p. 5
• POLARIS neglects the history-dependent term and selects each timestep’s guidance scale by projecting the origin onto the line of current-step prediction changes.This minimizes the approximate error using information available at that step, avoiding the exact solution’s cascading error.Source: §4.2 · pp. 5–6
• Under ∥∆εφ −∆εc∥>0, the approximate ω(t) computation is well-posed and its error is bounded at the same order as input prediction noise.Source: §4.2 · p. 6
• POLARIS’s additional computational cost comes from calculating ω(t) using known variables and is described as negligible relative to models with many parameters.Source: §4.2 · p. 6

5.1. Diffusion Inversion and Reconstruction

• On COCO2017 and Pick-a-Pic, POLARIS significantly improves reconstruction over the baseline, while qualitative results show decreasing error as inversion steps increase.The baseline’s cumulative error grows with more steps; POLARIS’s decreasing error trend mitigates error accumulation.Source: §5.1 · pp. 6–7

5.2. Image Editing

• With SAGE, POLARIS improves Back-PSNR by near 1.0 dB and reduces Back-MSE by nearly 20%; with P2P (Mask), it reduces Back-LPIPS by approximately 40% and boosts Back-PSNR to nearly 30 dB.Table 2 compares editing on reconstruction, aesthetic, and preference metrics; bold* marks POLARIS-enhanced methods and underlining marks mask-based methods.Source: §5.2 · p. 7, Table 2 · p. 7
• POLARIS improves background fidelity, but the Aesthetic Score, a proxy for semantic correctness, does not improve to the same degree.The method targets approximation error rather than semantic accuracy and is described as complementary to semantic-alignment methods.Source: §5.2 · p. 7

5.3. Extension to Other Inversion Tasks

• POLARIS consistently improves reconstruction fidelity across restoration tasks and baselines, generally reducing MSE and increasing PSNR and SSIM across most evaluated tasks.The gains across DPS, DDRM, and DDNM indicate applicability across architectures and degradation types.Source: §5.3 · p. 7, Table 3 · p. 8
• Figure 6 reports a sensitivity analysis of the initial guidance scale ω0 and characterizes POLARIS as robust.Source: Figure 6 · p. 8

5.4. Ablation Study

• The ablation compares POLARIS’s dynamically computed scales after varied initial values with uniformly random scales from U(0, 2); Figure 7 identifies the precise error-minimizing trajectory as crucial.The initial guidance scale ω0 is tested at nine values from 0.0 to 10.0, with later scales computed dynamically.Source: Figure 7 · p. 8, §5.4 · p. 8

6. Conclusion

• POLARIS addresses approximate noise error in DDIM inversion by treating guidance scale as a dynamic per-step variable and minimizing error at its source.Source: §6 · p. 8
• The plug-and-play approach improves reconstruction fidelity and background preservation across editing tasks with virtually no computational overhead.Source: §6 · p. 9
• Future work targets complex non-rigid edits and possible extensions to video and 3D inversion.The conclusion describes complex non-rigid edits as a known challenge for existing models.Source: §6 · p. 9

7.1. Theorem 1: Inherent Flaws of the Exact Solution

• The exact-solution limit analysis finds divergence both when a*·u is nonzero and in the orthogonal case.For the orthogonal case, the analysis proceeds to a second application of L’Hôpital’s rule before establishing divergence.Source: §7.1 · p. 11
• As ∥b*∥→0, the exact solution is ill-conditioned and its error magnitude grows proportionally to ∥b*∥^-1 when δb→0.The limit analysis reports divergence under this condition.Source: §7.1 · p. 11

7.2. Theorem 2: Inherent Advantages of the Approximate Solution

• Under well-separated true signals and sufficiently small noise, the approximate solution’s computational error is O(∥∆δ∥).The bound makes output error first-order Lipschitz in the noisy input.Source: §7.2 · p. 12

8.1. Unified POLARIS Update Rule

• For a probability-flow ODE inverse step, POLARIS optimizes ωt to minimize the inversion error ∥τ(t)∥2.The unified rule approximates the error as a weighted combination of changes in unconditional and conditional predictions.Source: §8 · p. 12, §8.1 · p. 13
• The update can minimize error in score space or velocity space, with flow-based optimization described as offering a straighter, more numerically stable trajectory.In score space, the metric penalizes errors in the final generation steps as σt approaches zero.Source: §8.1 · p. 13

8.2. Invariance of Fixed-Scale Baseline

• For fixed ω, the generative step is identical across noise, score, and flow parameterizations.The baseline trajectory is mathematically independent of the parameterization space.Source: §8.2 · p. 13

8.3. Experimental Validation

• Across COCO 2017 and Pick-a-Pic tests, the score-based variant achieves the highest fidelity among the evaluated parameterizations.The evaluation used 100 samples per dataset, Stable Diffusion v1.5 outputs, 50 DDIM steps, and replayed ωt.Source: §8.3 · p. 13, Table 4 · p. 13

9.1. Computational Efficiency

• POLARIS boosts the DDIM baseline PSNR by +9.39 dB with only ∼3% overhead and is reported to offer the best fidelity-latency trade-off among the compared approaches.The efficiency benchmark used 50 inference steps on one NVIDIA RTX 4090.Source: Table 5 · p. 14, §9.1 · p. 14
• Across 10–100 inference steps, POLARIS adds an average 3.3% runtime over the fixed-scale baseline.The runtime comparison is reported in Table 6.Source: Table 6 · p. 14, §9.1 · p. 14

9.2. Image Reconstruction

• With guidance scales fixed to ω ≡1 for inversion and sampling, standard DDIM underperforms POLARIS across all metrics and tested step counts.The reconstruction comparison covers 10–100 inference steps at intervals of 10.Source: Table 7 · p. 14, §9.2 · p. 14

9.3. Ablations on Sampling Schedulers

• Across 10–100 steps on COCO2017, POLARIS has the lowest MSE and LPIPS and highest PSNR and SSIM among the four compared guidance schedules.The alternatives are a forward schedule, reverse schedule, cosine decay, and fixed-scale baseline.Source: §9.3 · p. 14, Table 8 · p. 15

9.4. Scaling POLARIS to Larger Diffusion Models

• POLARIS also scales to Stable Diffusion XL, with the paper reporting even better results on this larger backbone.The SDXL comparison spans Pick-a-Pic and COCO2017 across inference steps.Source: Table 9 · p. 15, §9.4 · p. 15

10.1. Baselines

• Restoration baselines include DDRM’s measurement-error correction, DDNM’s range/null-space decomposition, and DPS’s likelihood-gradient guidance.The passages describe these as approaches to general inverse problems.Source: §10.1 · p. 16
• Editing baselines use cross-attention injection in P2P and self-attention consistency in SAGE; P2P-Mask instead combines inversion choice with a spatially generated edit mask.P2P-Mask compares fixed-scale DDIM inversion with POLARIS and modulates guidance between edit and preserve regions.Source: §10.1 · p. 16
• The baselines include per-step null-text optimization, negative-prompt and direct-inversion corrections, and EDICT’s coupled-state approach.The descriptions contrast embedding optimization, prompt-space correction, recorded error-map injection, and a dual-latent design.Source: §10.1 · p. 17

10.2. Datasets

• Evaluation uses MS-COCO 2017 and Pick-a-Pic to assess editing across diverse scenes and prompt-aligned image preferences.COCO provides five human-written captions per image; Pick-a-Pic pairs prompts with generated candidates selected by human annotators.Source: §10.2 · p. 17

10.3. Metrics

• MSE and PSNR measure pixel-level reconstruction fidelity, with lower MSE and higher PSNR indicating better reconstruction.Calculations use 8-bit pixel values, and PSNR is expressed in decibels.Source: §10.3 · pp. 17–18
• SSIM and LPIPS assess structural or perceptual similarity, while AES and preference rate capture aesthetic appeal and participant judgments.Higher SSIM indicates stronger structural similarity; lower LPIPS indicates greater perceptual similarity.Source: §10.3 · p. 18

Image editing

• Figs. 8–9 show text-guided editing that preserves background details during complex semantic modifications.Source: §11 · p. 18, Figure 8 · p. 19, Figure 9 · p. 20

Image restoration

• Figs. 10–13 demonstrate POLARIS effectiveness across deblurring, super-resolution, inpainting, and colorization.Source: §11 · pp. 18–19, Figure 10 · p. 21, Figure 11 · p. 21

• The “Degraded” columns show decoded initial latents rather than pixel-space degradations, revealing the reverse-diffusion starting point and latent-space challenges.Source: §11 · p. 19