Source-linked AI summary
Re-imagine the Negative Prompt Algorithm: Transform 2D Diffusion into 3D, alleviate Janus problem and Beyond
Mohammadreza Armandpour, Ali Sadeghian, Huangjie Zheng, Amir Sadeghian, Mingyuan Zhou
TL;DR
Text-to-image diffusion models can favor training-data patterns over the requested text, and conventional negative prompts fail when positive and negative concepts overlap. The paper proposes training-free Perp-Neg, which uses perpendicular score guidance to remove unwanted concepts while preserving the main semantics. It improves view fidelity in 2D diffusion and, when integrated with DreamFusion, alleviates the Janus problem in text-to-3D generation.
Problem
Diffusion models may fail to follow requested concepts or viewpoints, while existing negative prompting is unsatisfactory when positive and negative prompts overlap.
Method
Perp-Neg uses a perpendicular denoising direction for negative prompts, requiring no training, and extends this view-conditioning approach to DreamFusion.
Results
Perp-Neg improves prompt fidelity for view generation and alleviates DreamFusion’s Janus problem.
Takeaways & Limitations
The method supports editing out unwanted concepts while preserving the main concept and conditioning diffusion models toward desired rather than canonical views.
Takeaways & Limitations
The paper reports only an application using SDS loss and identifies novel-view synthesis, conditional 3D generation, editing, and texture addition as future extensions.
Abstract
from arXiv · showhide
Although text-to-image diffusion models have made significant strides in generating images from text, they are sometimes more inclined to generate images like the data on which the model was trained rather than the provided text. This limitation has hindered their usage in both 2D and 3D applications. To address this problem, we explored the use of negative prompts but found that the current implementation fails to produce desired results, particularly when there is an overlap between the main and negative prompts. To overcome this issue, we propose Perp-Neg, a new algorithm that leverages the geometrical properties of the score space to address the shortcomings of the current negative prompts algorithm. Perp-Neg does not require any training or fine-tuning of the model. Moreover, we experimentally demonstrate that Perp-Neg provides greater flexibility in generating images by enabling users to edit out unwanted concepts from the initially generated images in 2D cases. Furthermore, to extend the application of Perp-Neg to 3D, we conducted a thorough exploration of how Perp-Neg can be used in 2D to condition the diffusion model to generate desired views, rather than being biased toward the canonical views. Finally, we applied our 2D intuition to integrate Perp-Neg with the state-of-the-art text-to-3D (DreamFusion) method, effectively addressing its Janus (multi-head) problem. Our project page is available at https://Perp-Neg.github.io/
1. Introduction
The paper identifies failures in conventional negative prompting when positive and negative prompts overlap, then introduces Perp-Neg as a training-free remedy. It applies the method to improve view conditioning in 2D diffusion and alleviate the Janus problem in text-to-3D generation.
- Text-to-image diffusion models may generate frequent training-data concepts instead of accurately representing requested text, including missing or unwanted attributes.
- Conventional negative prompting can produce unsatisfactory results when the main prompt overlaps semantically with the negative prompt.
- Perp-Neg uses a perpendicular denoising direction for the negative prompt, removing undesired attributes while preserving the main prompt’s semantics without training.
- Experiments show that Perp-Neg improves diffusion-model prompt fidelity for view generation and enables view conditioning beyond canonical viewpoints.
- Integrating Perp-Neg with DreamFusion mitigates the Janus problem by improving the 2D model’s adherence to view instructions during 3D scene training.
2. Perp-Neg: Novel negative prompt algorithm
This section introduces diffusion-model preliminaries, analyzes why positive and negative prompts overlap, and develops Perp-Neg by projecting negative denoising directions perpendicular to the main prompt.
- 2.1. Preliminary: Diffusion models iteratively corrupt and generate data through forward and reverse processes over T steps.Intermediate forward steps and reverse steps are modeled with Gaussian distributions.
- 2.1. Preliminary: Text-to-image diffusion models condition noise prediction on text prompts encoded by pretrained language models to generate photo-realistic images.Complex prompts can still cause models to miss key elements and create undesired images.
- 2.2.1 The problem of semantic overlap: Composing positive and negative prompts assumes conditional independence, but semantic overlap makes this assumption unrealistic for specifying related attributes.The overlap ratio is one only under conditional independence, whereas style, content, and relations commonly interact.
- 2.2.2 Perpendicular gradient: Directly fusing overlapping denoising components can harm the main concept, while equal-and-opposite weights can cancel the denoising gradient and impair generation.This motivates extracting an independent component of the additional prompt rather than directly combining the full score.
- 2.2.2 Perpendicular gradient: Perp-Neg adds the component of a negative prompt perpendicular to the main prompt’s denoising direction, preserving the main component while removing undesired content.The projection term identifies the component most correlated with the main prompt, and the perpendicular remainder is used for sampling.
- 2.2.3 Perp-Neg algorithm: The perpendicular-gradient sampler is generalized from one additional prompt to a set of negative text prompts.The generalized formulation separates the main positive prompt from each negative prompt and assigns positive weights to them.
- 2.2.3 Perp-Neg algorithm: Perp-Neg’s denoising step is illustrated as part of the complete 2D generation scheme, with implementation details provided in Algorithm 1.The overview corresponds to the denoising operation used by the generalized sampler.
3. 2D diffusion model for 3D generation
2D diffusion models serve as proxies for optimizing 3D scene representations from rendered views, but canonical-view bias can produce Janus artifacts. The paper conditions views with positive and negative prompts and integrates Perp-Neg into DreamFusion to improve view faithfulness.
- 3. 2D diffusion model for 3D generation: 2D diffusion-based 3D generation optimizes a scene representation whose rendered images receive likelihood guidance from a pretrained diffusion model.The pipeline combines a 3D scene parameterization, differentiable renderer, and pretrained 2D diffusion model.
- 3. 2D diffusion model for 3D generation: Score Distillation Sampling replaces direct diffusion-loss optimization with a more robust and computationally efficient gradient formulation.The paper uses DreamFusion’s SDS formulation because direct optimization of the diffusion loss does not provide realistic samples.
- 3.1. The Janus problem: The Janus problem occurs when a generated 3D scene presents multiple canonical faces instead of the requested viewpoint.View-dependent prompting does not fully solve this bias because the 2D diffusion model may fail to follow view instructions and favor frequent training-data views.
- 3.2. Perp-Neg to alleviate Janus problem and 2D view conditioning: Perp-Neg combines positive and negative prompts to condition 2D diffusion on desired views and extend view-faithful guidance to DreamFusion.The method uses view-specific prompts, omits the back view as a negative for side/front generation when canonical views are not back, and adjusts negative-prompt weights to control view avoidance.
- 3.2. Perp-Neg to alleviate Janus problem and 2D view conditioning: Perp-Neg’s interpolation design accounts for possible front-view bias when generating transitions between side, back, and front views.The formulation distinguishes side-to-back from front-to-side interpolation and states that it must be adjusted if the assumed front-view bias does not hold.
- 3.2. Perp-Neg to alleviate Janus problem and 2D view conditioning: The paper applies Perp-Neg SDS in Stable DreamFusion by interpolating negative-prompt weights according to the related direction between 3D and 2D renderings.The extension is presented as an application of Perp-Neg using SDS, with direction-dependent weights for negative prompts.
4. Experiments
The experiments evaluate Perp-Neg for semantic-aligned 2D view generation and DreamFusion-based 3D generation. Across these settings, Perp-Neg improves view conditioning over vanilla sampling and conventional negative prompting, including fewer Janus-problem outputs.
- 2D generation: The experiments test requested side and back views using Stable Diffusion, comparing vanilla sampling, CEBM, and Perp-Neg across fixed-seed generations.Each prompt uses 50 images from seeds 0–49, with acceptance based on object/view alignment and visual quality criteria.
- 2D generation: Perp-Neg substantially improves successful view generation over vanilla sampling and CEBM by removing incorrect views while preserving prompt-aligned details.CEBM can produce artifacts or lose content because it does not account for overlap between positive and negative prompts.
- Prompt combinations: Negative-prompt effectiveness depends on the positive-negative view combination: ambiguous or poorly matched negative views are less effective for side and back generation.For side views, back-view negatives are less effective than front-view or combined negatives; for back views, front-view negatives are less effective than side-view negatives.
- 3D generation: In 14 trials per prompt, Perp-Neg produced Janus-free outputs for several DreamFusion prompts, whereas the model without Perp-Neg produced none except some Lamborghini and Super Mario outputs.Perp-Neg successes were reported for corgi, westie, lion, Lamborghini, and Super Mario prompts, but not cute pig.
5. Conclusion
The paper presents Perp-Neg as a training-free way to use overlapping negative and positive prompts without damaging the main concept. It applies this view-conditioning approach to 2D generation and SDS-based text-to-3D models to alleviate the Janus problem.
- Core contribution: Perp-Neg enables negative prompts to overlap with positive prompts while preserving the main concept.The method is presented as a way to remove unwanted concepts without damaging the requested content.
- 2D applications: Perp-Neg improves prompt faithfulness by reducing biased samples and can edit unwanted concepts from initially generated images.The paper describes feeding a sentence that expresses model bias as the negative prompt to obtain desired solutions.
- 3D application: The method conditions 2D diffusion models toward views of interest rather than canonical views and extends this property to SDS-based text-to-3D generation.The reported 3D application alleviates DreamFusion’s Janus problem.
A.1. Implementation details
The implementation places Perp-Neg in the sampling pipeline and computes negative-prompt guidance perpendicular to the main prompt component. The appendix also specifies the experimental pipeline and image-rejection criteria.
- Sampling implementation: Perp-Neg requires no additional training or fine-tuning and is implemented during sampling at each diffusion timestep.Algorithm 1 provides the timestep-level pseudocode.
- Sampling implementation: The algorithm extracts the main prompt component, processes auxiliary negative prompts, removes their projection onto the main component, and accumulates the resulting negative gradients.The perpendicular component is computed by subtracting the cosine projection onto the reference gradient.
- Experimental settings: For 2D experiments, Stable Diffusion v1.4 uses 50 DDIM steps, guidance scale a = 7.5, positive weight w_pos = 1, and generally satisfactory negative weights in [-5, -0.5].The single-negative-prompt view experiments use w_neg = −1.5.
- Evaluation criteria: Samples are rejected when they omit the requested object or view, contain hallucinated details, or show unrealistic color or texture artifacts.Multiple-object images are rejected if any object is not positioned in the correct view.
B. Additional experiments
Additional analyses examine object- and view-specific acceptance rates and provide qualitative illustrations of Perp-Neg. Back views remain harder than side views, and the method’s examples compare sampling with and without perpendicular negative guidance.
- Case-by-case analysis: Back views are consistently more difficult to generate than side views across the case-by-case analysis.The paper relates this pattern to greater training information about front views and the additional knowledge needed for back views.
- Case-by-case analysis: Peacocks are less likely to receive faithful generation in side-view cases, according to the object-specific analysis.The supplied passage begins a further object-specific comparison but does not include its complete continuation.
- Qualitative analysis: The additional figures illustrate Perp-Neg, compare sampling variants with and without negative prompts, and show accepted versus rejected generations.These illustrations complement the quantitative acceptance-rate analyses.
B.2. Ablation study
The ablation study examines how negative prompt weights affect attribute elimination in Perp-Neg. Smaller weights preserve relevance to the requested text, while larger weights can disrupt the intended viewpoint.
- Perp-Neg generates results that become less positioned in the requested side view when the negative prompt weight is larger, such as w = −0.1.This behavior is observed across the ablation generations in Figures 10–12.
- Decreasing the negative prompt weight makes the generated image more relevant to the requested text.
- The ablation indicates that Perp-Neg offers controllability over negative-attribute elimination through the negative prompt weight.
B.3. Additional results
Additional results cover 2D generation, view interpolation, and visualizations of requested object views. The supplementary material provides further 3D generation results.
- Additional qualitative results: The supplementary results include additional qualitative examples of 2D generation and view interpolation.
- 3D results: The 3D generation results are provided in the supplementary video.
- Additional visualizations: Figures 10–12 provide analogous visualizations of negative-prompt-weight ablations for peacocks, lions, and pandas.
- Additional visualizations: Additional figures visualize side-view and back-view generations for pandas, peacocks, and lions.
- Additional visualizations: Figure 14 shows panda back-view generations from seeds 0–49 using Perp-Neg, including successful and failed samples; most exhibit panda back-view semantics.
- View interpolation: Figure 19 compares view interpolation with and without Perp-Neg using the same seed across images for each prompt, with interpolation without Perp-Neg on top and Perp-Neg on the bottom.