PaperlayerView exact revision on arXiv

Source-linked AI summary

Steering Fields: Adaptive Vector Fields for Safe Image Generation and Beyond

Simone Facchiano, Jan Eric Lenssen, Bernt Schiele, Wolfgang Stammer, Fabio Galasso, Jonas Fischer

arXiv:2609.39573v1cs.CVcs.AI

TL;DR

Fixed steering directions cannot adapt to changing generation trajectories, limiting control over generated content. Steering Fields re-estimate directions at each step and unify compositional control with image editing. The paper reports state-of-the-art safety steering and semantic adherence, while noting incomplete suppression, manual parameter tuning, and added inference cost.

  • Problem

    Conventional steering applies one fixed direction throughout generation, despite changing trajectory states and the risk of unintended global changes.

    Source: §1 · p. 2, §2 · p. 3

  • Method

    Steering Fields adapt steering to the current trajectory state in a model-agnostic framework that supports concept induction, inhibition, and image editing.

    Source: §1 · p. 2, §5 · p. 9

  • Results

    Steering Fields achieve state-of-the-art safety steering and lead semantic-adherence metrics on image-editing benchmarks.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    The same trajectory-adaptive framework extends to structure-preserving image editing without inversion or explicit spatial anchors.

    Source: §1 · p. 2

    Unsafe content is not perfectly suppressed in all cases, parameters require manual tuning per application, and each integration step incurs additional inference cost.

    Source: §5 · p. 9

Abstract

from arXiv · show

As state-of-the-art text-to-image flow models achieve near-photorealistic quality, controlling their outputs, e.g., suppressing harmful content while promoting benign alternatives, has become a central challenge. The current steering paradigm consists of adding a global steering vector to selected activations. While functional, a fixed and example-agnostic vector applied uniformly along the entire trajectory cannot adapt to the changing state of the generation and often causes unintended global changes. We introduce Steering Fields, a generalization of steering vectors that adaptively re-estimates the steering direction at each step of the generative process. Steering Fields operate on the noisy states of flow models, expose a continuous trade-off between steering strength and content preservation, and are compositional, enabling the simultaneous induction and inhibition of concepts, setting a new state of the art on safety steering benchmarks. Despite using no explicit spatial masks or object priors, the trajectory-adaptive estimation naturally preserves local structure, in a manner reminiscent of image editing. In fact, Steering Fields can serve as a structure-preserving image-editing technique that achieves state-of-the-art semantic fidelity (CLIP, VQAScore), while remaining model-agnostic and inversion-free.

1 Introduction and Background

As flow models improve, controlling generated content has become central, but fixed steering vectors can cause unintended global changes. Steering Fields adapt steering to each trajectory step, support compositional control, and achieve strong safety-steering and image-editing results.

1 Introduction and Background

  • Fixed, example-agnostic steering vectors cannot adapt to changing generation states and can cause global changes that damage composition and local structure.As generation moves from noise to coarse structure and fine detail, a constant shift may not match the local activation-manifold geometry.Source: §1 · p. 2
  • Steering Fields re-estimate steering direction from the current trajectory state, operating on latent states rather than a hand-picked layer or module.The framework is model-agnostic, and classical activation steering is its constant-field special case.Source: §1 · p. 2
  • Without spatial masks, object priors, or inversion, trajectory-adaptive steering naturally preserves local image structure and extends to image editing.The paper reports strong semantic adherence on editing benchmarks.Source: §1 · pp. 2–3
  • Steering Fields establish state-of-the-art safety steering and lead semantic-adherence metrics on image-editing benchmarks across SD3.5 and FLUX1.The reported evaluation finds substantially reduced unsafe generations while preserving prompt fidelity.Source: §1 · pp. 2–3
  • Complementary attraction and repulsion support concept induction and inhibition, with continuous user control over steering strength and structure preservation.Source: §1 · p. 3

2 Background and Related Works

Flow models generate images by following a conditional velocity field, while conventional activation steering applies a fixed direction. Steering Fields replace that direction with a trajectory-adaptive field and extend the same control framework to structure-preserving editing.

2 Background and Related Works

  • Flow Matching models a latent state’s evolution with a time-dependent vector field conditioned on the text prompt, integrating from noise to an image.Source: §2 · p. 3
  • Activation steering adds or subtracts a single concept direction, commonly computed as the difference between target and away sample means (Turner et al., 2023; Marks and Tegmark, 2024).Source: §2 · p. 3
  • Because the conditional velocity varies with latent state and time, a fixed direction may fail to match trajectory geometry and can cause off-target distortion; Steering Fields instead use r_delta(z, t).The text notes that flow trajectories are rarely straight and are most fragile early in generation.Source: §2 · p. 3
  • Unlike editing methods that often use inversion to anchor structure, Steering Fields can condition source velocity on an encoded image latent to edit without inversion, attention manipulation, or spatial masks.Source: §2 · pp. 3–4

3 Steering Vector Fields

Steering Fields replace fixed activation-space directions with conditional velocity fields re-estimated along a flow trajectory, supporting compositional control and an inversion-free image-editing extension.

3.1 Setup

  • The flow setup represents generation as integrating a prompt-conditioned velocity from noise to a clean latent, with optional target and negative prompts specifying concepts to induce or suppress.Source: §3.1 · p. 4

3.2 A unified objective for compositional control

  • A single objective anchors the steered velocity to the source, attracts it toward a target concept, and repels it from an undesired concept.The attraction and repulsion strengths provide a continuous trade-off against fidelity to the source trajectory.Source: §3.2 · pp. 4–5

3.3 Per-step adaptivity and the relation to activation steering

  • Steering Fields re-estimate the steering direction from the current latent and timestep, rather than using a fixed direction throughout the trajectory.The flow-matching velocity depends on state and time; the direction is obtained from model evaluations at each step.Source: §3.3 · p. 5
  • Applying the intervention to the flow model's latent state makes Steering Fields model-agnostic and allows the same operator to edit an encoded image without inversion or spatial masks.For editing, the source velocity is conditioned on the encoded image latent rather than noise.Source: §3.3 · p. 5
  • On PieBench, Steering Fields achieve state-of-the-art semantic adherence on CLIP and VQAScore and a state-of-the-art HPSv2 score.Source: Figure 3 · p. 6

4 Experimental Evalutations

Across safety steering and image editing evaluations, Steering Fields suppress unsafe generations while retaining benign-prompt semantics, lead on editing semantic metrics, and also support concept blending.

4.1 Experimental Setup

  • The evaluations use FLUX1 and Stable Diffusion 3.5, safety benchmarks Ring-a-Bell and P4D, and PieBench++ image editing, with results averaged across 64 seeds and reported with 95% confidence intervals.Safety comparisons include UCE, ESD, EraseAnything, and SAFREE; editing comparisons include FLUX Img2Img, RF-Inversion, StableFlow, and FlowEdit.Source: §4.1 · p. 6, §4.2 · p. 6

4.2 Safety Evaluation via Steering

  • Steering Fields achieve the lowest NudeNet detection rates and largest VQAScoredetect reductions across the reported safety settings, while matching baseline CLIP and VQAScore on benign COCO prompts.FID remains in a comparable range across methods.Source: Table 1 · p. 7, §4.2 · p. 7

4.3 Image Editing via Steering Fields

  • Steering Fields lead on CLIP-txt, CLIP-dir, and VQAScore for PieBench editing, while performing on par with FlowEdit on HPSv2.They score lower on CLIP-img, consistent with inversion-based methods preserving source features better at the cost of less successful edits; the qualitative comparisons also show inversion baselines can miss target edits.Source: Figure 4 · p. 7, Table 2 · p. 8, §4.3 · p. 8

4 Experimental Evalutations

  • Together, the experiments show a unified framework for T2I safety steering and I2I editing without architectural changes or task-specific components.The image-editing setup requires no fine-tuning, inversion, or task-specific adaptation.Source: §4.3 · p. 8

4.4 Concept Blending via Steering Fields

  • With λ = 0, Steering Fields blend source and target concepts by mixing their velocity fields, producing plausible interpolations while preserving compositional structure and visual coherence.The examples include semantically distant combinations such as dogs with spaghetti and clouds, without inversion or fine-tuning.Source: §4.4 · p. 8

5 Discussion and Limitations

Steering Fields expose controllable trade-offs across safety steering, editing, and concept blending, while using one model-agnostic framework. The discussion also notes incomplete suppression, manual tuning, and additional inference cost.

Concept blending

  • Setting λ = 0 in Eq. 6 enables Steering Fields to blend semantically distant concepts, with examples including dogs with spaghetti and clouds, as shown in Figure 5.Source: Figure 5 · p. 9

Safety-fidelity tradeoff

  • Steering Fields’ safety-fidelity frontier is reported as favorable and unreachable by competing methods.Increasing β moves farther from the unsafe concept but deviates more from the source; decreasing it preserves fidelity but weakens suppression.Source: §5 · pp. 8–9

Inversion tradeoff for editing

  • For editing, inversion-based methods favor source preservation, whereas inversion-free Steering Fields follow target prompts more freely, with higher CLIP-txt, CLIP-dir, and VQAScore.The preferred operating point depends on whether strict source preservation or target-prompt alignment matters more.Source: §5 · p. 9

Compositional control

  • A single objective combines attraction toward target concepts and repulsion from undesired ones, with α and β independently controlling steering strength and structure preservation.This contrasts with approaches that commit to fixed suppression at training time.Source: §5 · p. 9

A unified, model-agnostic framework

  • The same operator acts directly on flow-model latent velocity and supports safety steering, editing, and concept blending without architecture-specific representations.Any flow model exposing a conditional velocity can be steered without modification.Source: §5 · p. 9

Limitations

  • Unsafe content is not perfectly suppressed; intimacy prompts without explicit nudity are harder, parameters require per-application manual tuning, and each integration step incurs extra forward passes.The method uses two forward passes per integration step, compared with one for the unsteered baseline.Source: §5 · p. 9

6 Conclusion

The conclusion presents Steering Fields as a unified approach to controlled image generation that adapts steering directions to time and latent-space position. It reports state-of-the-art safety steering and extensions to image editing and concept blending, while identifying broader domains and inversion-assisted editing as future directions.

6 Conclusion

  • Steering Fields generalize steering vectors by adapting steering directions to time and latent-space position for controlled image generation.The conclusion frames the approach as a unified framework for tasks including safety steering and image editing.Source: §6 · p. 9
  • The framework achieves state-of-the-art safety steering and extends to image editing and concept blending.Source: §6 · p. 9
  • Future directions include extending the framework to other domains and integrating inversion into editing for stronger source preservation alongside effective edits.The conclusion anticipates model-agnostic application across tasks and domains.Source: §6 · p. 9

A.1 Blending Vector Fields: Full Derivation

The appendices derive blending and replacing velocity-field updates, show blending as a special case of replacing, and give the algorithm and flow-matching background. The derivations depend on state-conditioned fields, which are evaluated during the generation trajectory.

A.1 Blending Vector Fields: Full Derivation

  • The blending minimizer is a convex combination of source and target fields, with α = µ/(1 + µ); its field difference depends on the current latent and timestep.The state dependence means the difference cannot be precomputed offline.Source: §A.1 · p. 14

A.2 Replacing Vector Fields: Full Derivation

  • The replacing objective combines source, target, and away fields, and its derivation assumes 1 + µ − λ > 0.The derivation expands the objective, groups terms involving v, and solves its gradient condition.Source: §A.2 · p. 14

A.3 Decomposition of the Replacing Minimizer

  • The replacing minimizer decomposes into source-field corrections toward the target and away fields, and setting λ = 0 recovers the blending minimizer.The derived coefficients are α = µ/(1 + µ − λ) and β = λ/(1 + µ − λ).Source: §A.3 · p. 15

A.4 Full Steering Fields algorithm

  • The algorithm supports T2I and I2I pipelines and BLEND or REPLACE modes, evaluating source and target fields at each step before updating the latent.Its inputs include source and target conditions, optional away condition, and steering schedules; I2I additionally takes an image, noise level, and start step.Source: §A.4 · p. 16

B Preliminaries: Flow Matching

  • Flow Matching generates images by integrating a learned time-dependent velocity field from Gaussian noise toward a clean latent, which a pretrained VAE decodes.The stated inference procedure integrates backward from t = 1 to t = 0 using an Euler step.Source: B Preliminaries: Flow Matching · p. 17

C.1 Additional Quantitative Results: model-agnostic feature of Steering Fields, nudity category

The additional evaluations report that Steering Fields outperform activation steering across metrics, retain state-of-the-art results on violence benchmarks, and transfer to SD3 in the tested nudity setting.

C.1 Additional Quantitative Results: model-agnostic feature of Steering Fields, nudity category

  • On SD3, the authors report that findings from SD3.5 and FLUX1 also apply to nudity suppression and text-image alignment using SD3.5-tuned hyperparameters.Table 3 is described as evaluating nudity suppression and text-image alignment.Source: §C.1 · p. 17, Table 3 · p. 17

C.2 Additional Quantitative Results: violence category

  • Steering Fields remain state of the art on violence-category benchmarks, extending the reported results beyond nudity.Source: §C.2 · p. 17

C.3 Additional Quantitative Results: comparison and ablations with activation steering

  • Steering Fields consistently outperform activation steering across metrics, with better suppression and lower FID than tested activation-steering configurations.The comparison reports the best activation-steering results across tested layers and strengths.Source: §C.3 · p. 17, Table 5 · p. 18

D Additional Qualitative Results for Blending and Replacing

The appendix adds qualitative examples of blending and replacing, documents prompt and hyperparameter choices, and reports dataset prompt-length statistics.

D Additional Qualitative Results for Blending and Replacing

E.1 Prompts for ctar and caway: nudity category

  • Nudity steering uses averages from 50 positive and negative prompt pairs, whereas violence steering uses a single prompt because it performed better.The violence prompts are “violent, explosion, blood, weapons, war” and “happy, peaceful, calm.”Source: §E.1 · pp. 19–20

E.2 Hyperparameter settings and Computational resources

  • Hyperparameter searches generated over 80,000 images for steering and 100,000 for editing; computational resources are also tabulated.The appendix reports selected steering settings for FLUX and SD3.5 and an editing noise setting.Source: §E.2 · p. 20, Table 6 · p. 20

E.3 Word Count Statistics for the Datasets

F Qualitative Results for Problematic Categories

The qualitative examples show safety steering on violence and nudity prompts, with the nudity comparison also highlighting preservation of image structure, colors, and poses.

F Qualitative Results for Problematic Categories

Loading 2609.39573v1…