PaperlayerView exact revision on arXiv

Source-linked AI summary

TAG: Tangential Amplifying Guidance for Hallucination-Resistant Sampling

Hyunmin Cho, Donghoon Ahn, Susung Hong, Jee Eun Kim, Seungryong Kim, Kyong Hwan Jin

arXiv:2510.04533v2cs.CV

TL;DR

Diffusion-model hallucinations include outputs that contradict the data distribution or conditioning, while existing guidance can overlook local geometry. TAG amplifies tangential score-update components to steer sampling along the data manifold; the paper reports fewer hallucinations and improved fidelity, while noting limits to its local first-order analysis.

  • Problem

    Diffusion-model hallucinations can result from trajectories crossing low-density valleys between data modes, while common residual-scaling guidance does not account for local directional structure.

    Source: §1 · p. 1

  • Method

    TAG reweights sampling updates along normal and tangential directions, amplifying the tangential component as a plug-and-play method without additional denoising evaluations or retraining.

    Source: §1 · p. 2, §4 · p. 3

  • Results

    TAG steers trajectories toward higher-density regions of the data manifold, producing samples with fewer hallucinations and improved fidelity.

    Source: §7 · p. 9

  • Takeaways & Limitations

    The results motivate viewing diffusion guidance as geometric shaping of sampling trajectories and tangential components as semantic refinement signals.

    Source: §7 · p. 9

    The analysis establishes only local first-order likelihood gain and does not fully characterize reverse trajectories; its radial–tangential approximation may miss complex manifold geometry.

    Source: §7 · p. 9

Abstract

from arXiv · show

Diffusion models achieve state-of-the-art image generation but often produce semantic inconsistencies, or hallucinations. Existing inference-time guidance methods rely on external signals or architectural modifications, adding computational overhead. We propose $\mathbf{T}$angential $\mathbf{A}$mplifying $\mathbf{G}$uidance $\mathbf{(TAG)}$, a training-free, architecture-agnostic, plug-and-play guidance method that operates purely on trajectory signals. TAG uses an intermediate sample as a projection basis and amplifies the tangential components of the estimated score to correct the sampling trajectory. A first-order Taylor analysis shows that this steers the state toward higher-probability regions of the data manifold, reducing inconsistencies and improving fidelity while adding negligible overhead to existing samplers. Code is available at our Project Page (https://hyeon-cho.github.io/TAG/).

1. Introduction

Diffusion-model hallucinations can arise when sampling trajectories cross low-density valleys between data modes, producing attribute mismatches and structural inconsistencies. TAG instead separates update directions and amplifies the tangential component to steer sampling toward the data manifold.

1. Introduction

  • Hallucinations include samples that violate the data distribution or conditioning, and mode interpolation through low-density valleys can produce attribute mismatches and structural inconsistencies.Examples include mixed-up objects and anatomically implausible structures.Source: §1 · p. 1
  • Existing residual-scaling guidance is geometry-unaware, applying scalar magnification without accounting for local directional structure and potentially distorting the denoising trajectory.The cited examples include classifier-free guidance and its variants.Source: §1 · p. 1
  • TAG decomposes updates into radial and tangential directions, preserving the radial component while amplifying the tangential component associated with structural information.The paper motivates this score-level view through Tweedie’s identity and illustrates the tangential component’s semantic content in Figure 2.Source: §1 · p. 1, Figure 1 · p. 1, Figure 2 · p. 2
  • The paper reports that tangential amplification reduces out-of-distribution samples and introduces TAG as a plug-and-play method for standard conditioned or unconditioned diffusion backbones without extra denoising evaluations or retraining.Its contributions also include a proof that amplification steers trajectories toward the in-distribution manifold and an architecture-agnostic, computationally lightweight algorithm.Source: §1 · p. 2

2. Preliminaries

The preliminaries describe score-based diffusion as reversing a gradual noising process using a learned score and numerical samplers. They also position TAG among guidance approaches that seek to shape update directions rather than merely scale them.

2. Preliminaries

  • Score-based generative models learn a time-indexed score approximating the log-density gradient of noise-perturbed data and use it to reverse a gradual noising process.The framework connects discrete diffusion models with stochastic differential equations.Source: §2 · p. 2
  • Sampling uses a reverse-time SDE or probability-flow ODE, discretized with numerical methods such as predictor-corrector schemes or modern deterministic samplers.The forward process transforms complex data into a simple prior; the displayed forward SDE is dx = f(x, t)dt + g(t)dW.Source: §2 · p. 2
  • Prior inference-time guidance modifies the sampling update without retraining, but geometry-agnostic scaling can reduce diversity or perturb solver dynamics.Other approaches use projections to damp undesired components, while some methods depend on specific guidance algebra or task assumptions.Source: §2 · pp. 2–3
  • On a 2D branching distribution, the figure reports that TAG achieves the highest similarity to ground truth without additional NFEs, suppressing off-manifold outliers while retaining detail.The comparison includes no guidance, naive truncation, CFG, TAG, and ground truth.Source: Figure 3 · p. 3

3. Motivation and Intuition

The motivation treats high-probability regions of the corrupted distribution as noisy data manifolds near thin spherical shells. TAG preserves radial movement while assigning additional gain to the tangential direction along those shells.

3. Motivation and Intuition

  • Tweedie’s formula links the posterior mean under Gaussian corruption to the noisy observation through the score, which points toward increasing marginal density.The paper describes this score-driven adjustment as nudging the state toward higher-probability regions.Source: §3 · p. 3
  • Because Gaussian corruption concentrates probability near a thin shell, the paper models the high-probability region as a noisy data manifold and aims to move along it rather than change radius.The diffusion noise schedule already dictates radial movement.Source: §3 · p. 3
  • The update is split into a radial component normal to the shell and a within-shell tangential component, with the latter amplified by TAG.Figure 4 depicts the decomposition relative to successive noisy manifolds and the local tangent space.Source: §3 · p. 3, Figure 4 · p. 3
  • TAG preserves the radial first-order term while allocating extra gain to the within-shell component, steering the step toward higher-density regions along the manifold.The paper formalizes this as a constrained maximum-likelihood update that allocates first-order gain to the tangential subspace.Source: §3 · p. 3

4. TAG: Tangential Amplifying Guidance

TAG reweights diffusion updates to amplify directions tangent to the current sample while preserving radial movement. Its analysis links this choice to higher-density manifold regions, and the section extends TAG to classifier-free guidance.

4. TAG: Tangential Amplifying Guidance

  • TAG reweights a base increment by amplifying its tangential component while leaving the component parallel to the current state unchanged.The method uses a sample-based projection to separate the components and requires no additional denoising evaluations or retraining.Source: §4 · pp. 3–4, §4.1 · p. 4

4.1. Why does TAG improve Image Quality?

  • Under a smooth log-density and deterministic base-step assumptions, increasing the tangential weight monotonically raises first-order Taylor gain, except when the weight is one or the tangential score is zero.The resulting update steers sampling toward higher-density regions of the data manifold.Source: §4.1 · p. 5
  • Tangential scaling preserves the radial first-order term, whereas normal amplification strengthens radial contraction and leads to over-smoothing under the VP/DDIM schedule.The analysis connects the radial component to Tweedie’s correction.Source: §4.1 · p. 5
  • At 50 NFEs, TAG surpasses baseline sample quality at 250 NFEs, while normal amplification causes severe over-smoothing.The figure contrasts tangential guidance with +Normal.Source: Figure 5 · p. 5

4.2. TAG for Classifier-Free Guidance

  • For classifier-free guidance, TAG addresses a proposed tangential mismatch between conditional and unconditional scores by extracting the residual’s tangent and aligning the conditional prediction with it.The CFG formulation retains the usual guidance scale and adds a separate tangential emphasis scale.Source: §4.2 · pp. 5–6
  • Qualitative comparisons show enhanced detail and coherence in unconditional and conditional SDXL/3 samples, with TAG also applied on top of PAG and SEG.The cited table passages identify unconditional and conditional quantitative comparisons but do not provide their results.Source: Figure 6 · p. 6, Table 1 · p. 6, Table 2 · p. 6

5. Experiments

The experiments evaluate TAG across pretrained diffusion backbones, image-generation settings, and video generation. They report improvements in fidelity, alignment, compositional faithfulness, and quality, including when TAG is combined with existing guidance methods.

5.1. Improvements on Stable Diffusion Series

  • Across Stable Diffusion backbones, TAG improves unconditional FID and CMMD and conditional fidelity and prompt alignment without additional NFEs.Qualitative comparisons also report fewer artifacts and more coherent structures.Source: §5.1 · p. 6
  • Stacking TAG with PAG or SEG further improves sample quality at matched NFEs, with consistent gains across metrics, especially human-preference-oriented scores.The comparison reports fixes to action-binding and numeracy errors.Source: §5.1 · p. 6, Table 3 · p. 7, Table 4 · p. 7

5.2. Validation on Hallucination-oriented Benchmark

  • On hallucination-prone T2I-CompBench Spatial and Complex prompts, TAG improves key compositional metrics while maintaining comparable aesthetic and CLIP-Scores.The authors interpret this as mitigation of structural artifacts rather than a shift in perceptual style.Source: §5.2 · p. 7, Table 5 · p. 7

5.3. Comparison with geometry-aware guidance.

  • FID 19.29 for TAGcfg is lowest among geometry-aware CFG variants, while APG has the highest CLIPScore at 26.71.The authors describe tangential trajectory amplification as an effective, distinct form of CFG-signal correction.Source: §5.3 · p. 7, §5.4 · p. 7, Table 6 · p. 7

5.4. Improvements on Modern Image & Video Models

  • On Qwen-Image, TAG improves no-reference perceptual quality metrics alongside a marginal FID increase, suggesting detail refinement on a backbone already near the reference distribution.The authors characterize this as improved perceived quality with only a small FID trade-off on the base model.Source: §5.4 · p. 7, Table 7 · p. 7
  • On 100 VBench prompts with Wan2.2, TAG improves most reported metrics, including dynamic degree and quality measures, while preserving coherence in complex scenes.The authors note that increased dynamic degree alongside quality gains does not simply compress motion.Source: §5.4 · p. 7

6. Discussion

TAG is reported to improve sampling across ODE solvers, both alone and when added to stronger samplers, with marginal inference-time overhead. Its amplification strength and timing matter: excessive or poorly scaled corrections can reduce fidelity, motivating adaptive scaling.

6. Discussion

  • TAG alone substantially improves the baseline and is competitive with DPM++ and UniPC; adding TAG to these samplers yields further gains in both FID and IS.The results position TAG as both a standalone refinement and a plug-and-play addition to stronger solver variants.Source: §6 · pp. 7–8, Table 9 · p. 8
  • TAG matches baseline peak memory and adds 9.4% latency overhead, while PAG adds 112.8% overhead and substantially increases compute and memory.TAG uses the same UNet pass and lightweight vector projections; reported latency is 1.005 s for TAG versus 0.919 s for baseline.Source: §6 · p. 8, Table 10 · p. 8

6.1. η Selection and Adaptive Scaling

  • Moderate amplification improves detail and coherence, but larger η reduces fidelity and can produce over-smoothed or distorted outputs.The paper treats η as a sensitivity parameter rather than a simple improvement knob.Source: §6.1 · p. 8, Figure 8 · p. 8
  • Restricting TAG to a timestep window, especially the high-to-intermediate noise regime, can avoid over-amplifying late denoising steps but remains a manually tuned heuristic.The windowing is model- and sampler-dependent.Source: §6.1 · p. 8, Table 11 · p. 8
  • Adaptive scaling normalizes the tangential correction relative to the original guidance scale and maintains stable performance across a wider η range than fixed scaling.Fixed scaling degrades rapidly at larger values; δ is included for numerical stability.Source: §6.1 · pp. 8–9, Table 12 · p. 9

7. Conclusion, Limitations, and Implications

The conclusion frames hallucination reduction as a geometric guidance problem: TAG amplifies tangential sampling updates to steer trajectories toward higher-density data-manifold regions. The authors also state that their analysis and approximation leave aspects of the full trajectory and complex manifold geometry uncharacterized.

7. Conclusion, Limitations, and Implications

  • TAG decomposes sampling updates into normal and tangential components and amplifies the tangential component to steer trajectories toward higher-density data-manifold regions.The paper reports fewer hallucinations and improved fidelity.Source: §7 · p. 9
  • The theoretical analysis establishes local first-order likelihood gain but does not fully characterize the reverse trajectory; the radial–tangential approximation may miss complex manifold geometry.Source: §7 · p. 9
  • The results motivate viewing diffusion guidance as geometric shaping of the sampling trajectory and further analysis of trajectory-level behavior and local manifold geometry.Source: §7 · p. 9

Impact Statement

The impact statement notes possible uses for TAG alongside risks associated with higher-fidelity generation and inherited model biases.

Impact Statement

  • TAG may be useful in creative tools, prototyping, and education.Source: Impact Statement · p. 9
  • Higher-fidelity generation can also increase misuse risks, including deceptive content.Source: Impact Statement · p. 9
  • Because TAG builds on existing models, it may inherit their biases and limitations; deployments should use appropriate safety, legal, and licensing safeguards.Source: Impact Statement · p. 9

A. Proof & Derivation

The derivation establishes that amplifying the tangential component yields a non-decreasing first-order log-likelihood gain relative to the base update. The result is stated for a deterministic base step with a nonpositive sampler coefficient.

Proof for Theorem 4.1

  • The analysis defines a first-order Taylor gain for the TAG update at tk+1 and evaluates it using the model score as an approximation to the true score.The base step is written as ∆k+1 = ˜αk ϵθ(xk+1, tk+1) + βk xk+1, with ˜αk ≤ 0.Source: Proof for Theorem 4.1 · p. 13
  • The derivative of G(η) is non-negative, so the first-order gain is monotonically non-decreasing as tangential amplification increases.The argument uses ˜αk ≤ 0 and the non-negativity of σk+1 and the squared L2-norm.Source: Proof for Theorem 4.1 · p. 13
  • For η ≥ 1, TAG’s first-order gain is at least the base gain, with equality only when η = 1 or the tangential score component is zero.The proof’s monotonicity claim is for a deterministic base step with the DDIM coefficient ˜αk ≤ 0.Source: Proof for Theorem 4.1 · pp. 13–14

B. Broader Experiments

Broader evaluations report compatibility with Self-Swap Guidance, gains in unconditional generation, and qualitative extensions to image-to-3D, video, and other guidance combinations. The appendix also provides implementation details for applying TAG within a scheduler pipeline.

B.1. Application to Self-Swap Guidance (SSG, Section 5.1)

  • FID fell from 49.02 to 48.53 and CMMD from 0.654 to 0.648 when TAG was added to unconditional SSG; conditional TAGcfg improved three metrics while maintaining PickScore.The conditional improvements were in FID, ImageReward, and CLIPScore.Source: §B.1 · p. 15

B.2. Unconditional Generation on ImageNet.

  • FID-50K on ImageNet 64×64 fell from 11.04 to 10.37 with TAG on the EDM2 baseline.The evaluation tests unconditional generation without explicit conditioning signals.Source: Table 14 · p. 15, §B.2 · p. 15

B.3. Beyond 2D Image: Image-to-3D shape generation

  • TAG yields modest qualitative improvements on SV3D, with fewer view-dependent artifacts and slightly more consistent structure across viewpoints.A systematic quantitative study of 3D generation is left for future work.Source: Figure 9 · p. 15, §B.3 · p. 15

C.1. Image Generation: unconditional

  • Across SD v1.5, 2.1, XL, and 3, unconditional samples with TAG show sharper, more coherent structure and fewer artifacts while preserving diversity.Figure 10 compares baseline and TAG at matched NFEs.Source: Figure 10 · p. 16

C.2. Video Generation (Section 5.4)

  • The Wan 2.2 video comparison covers five prompts, showing six uniformly sampled frames from each 49-frame clip for baseline and TAG.The caption describes the comparison setup but does not state a qualitative outcome.Source: Figure 11 · p. 17

C.3. Image Generation: Plug-and-Play (for Section 5.1)

  • PAG, SEG, and SSG paired with TAGcfg more faithfully reflect highlighted prompt details, including missing objects and corrected actions, while preserving overall scene structure.These are qualitative comparisons against each guidance method alone.Source: Figure 12 · p. 18, Figure 13 · p. 19, Figure 14 · p. 20

C.4. Image Generation: Geometric Guidance (for Section 5.3)

  • Compared with TCFG and APG, TAGcfg more faithfully reflects highlighted prompt concepts, attributes, or relations while preserving the overall scene structure.The figure captions give qualitative comparisons and do not report quantitative measurements.Source: Figure 15 · p. 21, Figure 16 · p. 22

D. Implementation of the Tangential Amplifying Guidance (Section 4.2)

  • The implementation decomposes the post-scheduler update into radial and tangential components, then rescales those components before continuing sampling.The displayed code also includes a conditional TAG variant that forms aligned tangential guidance from conditional and unconditional scores.Source: D. Implementation of the Tangential… · p. 23

E.1.4. TASK-SPECIFIC MANIFOLD CONSTRAINTS

The discussion distinguishes task-specific manifold guidance from TAG’s general diffusion trajectory decomposition. It also specifies the unconditional evaluation setup and the implementation and hardware context.

E.1.4. TASK-SPECIFIC MANIFOLD CONSTRAINTS

  • Task-specific methods project external guidance gradients onto clean-data manifold tangents using an auxiliary autoencoder, and are tailored to observation-conditioned tasks rather than generic generation.For conditional image-to-image translation, reference-induced manifolds make tangential refinement and normal transport meaningful; the cited discussion contrasts this with standard diffusion.Source: §E.1.4 · pp. 24–25

E.2. TAG: Intrinsic Tangential Amplification for General Diffusion Sampling

  • TAG decomposes each single-step solver update into radial and tangential components relative to the iso-noise manifold; the tangential part captures semantic refinement, and amplifying it promotes local log-likelihood ascent.The method acts on the base solver update without auxiliary models or extra network passes and applies to unconditional and standard conditional generation.Source: §E.2 · p. 25

F.1. Implementation Details

  • The implementation injects TAG decomposition after the scheduler step by overriding each corresponding diffusers pipeline’s call method; experiments used RTX 4090 GPUs with 24 GB VRAM.The reported software stack includes PyTorch 2.5.1 with CUDA 12.1 and diffusers 0.35.1.Source: §F.1 · p. 26

F.2. Unconditional Generation

  • Unconditional experiments used four Stable Diffusion variants, 30,000 seeds per setting, 50 denoising steps, no classifier-free guidance, and unchanged radial scaling.TAG ran across all timesteps; tangential scaling values were selected using preliminary sweeps.Source: §F.2 · p. 26, Table 16 · p. 26

F.3. TAG for Classifier-Free Guidance

  • Conditional text-to-image experiments used 10,000 captions randomly sampled from the MS-COCO 2014 validation set and a fixed seed of 0.The passage begins to specify a CFG scale but does not provide its value.Source: §F.3 · p. 26

F.4. Compatibility with Guidance Methods (PAG, SEG, and SSG)

The compatibility experiments apply PAG, SEG, or SSG guidance before TAG’s scheduler-step decomposition. The appendix also documents metric definitions and the software and hardware used for evaluation.

F.4. Compatibility with Guidance Methods (PAG, SEG, and SSG)

  • PAG, SEG, and SSG guidance is applied at the noise-prediction level, followed by TAG decomposition after the scheduler step.The stated PAG and SEG settings use scale 3.0; the experiment settings specify unconditional SD v1.5 generation with 30,000 samples and PNDM.Source: §F.4 · pp. 26–27, Table 17 · p. 26

F.5. Evaluation Metrics

  • Evaluation reports FID, IS, aesthetic score, CMMD, ImageReward, and CLIPScore, alongside CLIP-IQA and MUSIQ for no-reference image quality.The passages describe FID and IS computation, CLIP-feature MMD for CMMD, and CLIPScore as scaled image–text cosine similarity.Source: §F.5 · p. 27

F.6. Software and Hardware

  • Experiments used NVIDIA GeForce RTX 4090 GPUs with 24 GB VRAM; generation was data-parallel across seeds, with each image generated on one GPU.The listed software includes PyTorch 2.5.1, CUDA 12.1, and diffusers 0.35.1.Source: §F.6 · p. 27
Loading 2510.04533v2…