PaperlayerView exact revision on arXiv

Source-linked AI summary

AcFlow: Controlling Text-to-Image Diffusion Transformers via Learned Conditional Activation Flow

Junran Wang, Zehao Jin, Tianyu Luan, Xinjie Shen

arXiv:2609.10723v1cs.CVcs.AI

TL;DR

Direct prompting offers limited control over style intensity and may fail to suppress unwanted concepts in text-to-image diffusion transformers. ACFLOW addresses this by transporting intermediate image-token activations through a learned, concept-conditioned velocity field in a frozen DiT, with the integration horizon controlling intervention strength. It achieves the best reported style–content trade-off in the high-style-alignment regime and demonstrates suppression of diverse concepts, while composition preservation remains limited.

  • Problem

    Direct prompting provides limited control over style intensity and can fail to suppress unwanted concepts in text-to-image diffusion transformers.

  • Method

    ACFLOW transports intermediate image-token activations through a learned concept-conditioned velocity field in a frozen DiT, using textual concepts and integration horizon for control.

  • Results

    ACFLOW achieves the best style–content trade-off among evaluated methods in the high-style-alignment regime and demonstrates suppression of diverse concepts.

  • Takeaways & Limitations

    Token-varying, activation-dependent transport supports fine-grained descriptions, continuous intervention, and generalization to held-out concepts without per-concept fitting.

  • Takeaways & Limitations

    ACFLOW does not guarantee preservation of the original image composition, which can change through object positions, poses, and scene layout.

Abstract

from arXiv · show

Text-to-image diffusion transformers (DiTs) are powerful generators, yet direct prompting provides limited control interface for style intensity and can fail to suppress unwanted concepts. To enable these controls, we introduce AcFlow, an inference-time controller that transports intermediate layer image-token activations through a learned concept-conditioned velocity field while keeping the base DiT frozen. A textual concept description specifies the desired intervention, while the integration horizon provides a continuous control parameter. The field produces token-varying, activation-dependent updates. With parameters shared across concepts within each task family, the field supports fine-grained descriptions and generalizes to concepts unseen during training without per-concept fitting. On style control, AcFlow achieves the best style--content trade-off among the evaluated baselines in the high-style-alignment regime. At a fixed operating point, AcFlow attains style--content alignment of 0.5365/0.2860, compared with 0.4397/0.2684 for the baseline with the highest style alignment. Qualitative results demonstrate suppression of diverse concepts, including cases where direct prompting fails. Our analyses support the learned velocity field as an adaptive control mechanism, with update directions varying across tokens and depend on their activation states. Our code is available at https://github.com/Nove1yst/AcFlow.

1 INTRODUCTION

ACFLOW adds inference-time control to frozen DiTs by transporting image-token activations through a concept-conditioned velocity field. It supports continuous style modulation, concept suppression, fine-grained descriptions, and generalization to unseen concepts.

  • The controller transports image-token activations through a learned velocity field conditioned on a textual concept description while keeping the base DiT frozen.The field applies token-varying, activation-dependent updates at inference time.
  • The integration horizon provides a continuous control parameter for intervention strength, enabling progressive style modulation and partial concept suppression.T = 0 recovers the unmodified parameter, while larger horizons expose intermediate suppression states.
  • One field shares parameters across concepts within each task family, supports fine-grained descriptions, and generalizes to unseen concepts without per-concept fitting.This design addresses the limited flexibility and concept-specific fitting requirements of existing control mechanisms.
  • ACFLOW achieves the best style–content trade-off among evaluated methods in the high-style-alignment regime while also demonstrating concept suppression.Direct prompting can leave unwanted concepts present, whereas ACFLOW supports progressive intervention through its flow horizon.
  • Controlled experiments and trajectory analysis show that token-specific updates adapt their directions and energy allocation through denoising steps.These analyses distinguish the learned field from broadcasting one fixed concept direction across tokens.

2 RELATED WORKS

Prior work provides continuous control through concept-specific directions, adapters, or intervention magnitudes. Recent nonlinear approaches vary updates with activations, while ACFLOW uses a concept-conditioned velocity field for token-level transport.

  • Continuous-control methods use concept-specific directions or adapters and adjust their influence through inference-time scales.Examples include optimized text-embedding directions and low-rank adapters in the diffusion backbone or text encoder.
  • Activation-intervention methods modify intermediate representations using fixed attribute directions, fixed target-to-safe directions, or dynamically varying magnitudes.These methods differ in whether the direction or only its magnitude changes across activations.
  • Nonlinear intervention makes update directions depend on activation state, but CAT computes updates from mean-pooled activations and broadcasts the same update across tokens.ACFLOW instead applies a learned concept-conditioned velocity-field formulation to image-token activations.

3 METHOD

AcFlow controls frozen DiTs by transporting image-token activations through a learned concept-conditioned flow field during inference. The method uses a continuous intervention horizon, image-token-only updates, and velocity distillation to produce adaptive, token-varying control.

  • 3.1 CONTROL AS A FLOW FIELD: The intervention horizon T controls strength: T = 0 leaves activations unchanged, while forward Euler integration writes the endpoint back into the residual stream.The method uses N integration steps to approximate the flow endpoint.
  • 3.1 CONTROL AS A FLOW FIELD: The full field allows token velocities to vary in both direction and magnitude, unlike shared-token or fixed-direction interventions.Simpler variants use mean-pooled shared velocities, scalar scaling, or a fixed concept direction.
  • 3.2 INTERVENING THE IMAGE TOKENS: AcFlow intervenes only on image tokens because text and image states have different roles and need not share an activation geometry.The image state lies on the residual path consumed by the output head, whereas text-state intervention affects outputs indirectly through later mixing.
  • 3.3 MODEL ARCHITECTURE: The token-swap diagnostic finds that image-state replacement recovers most stylization, whereas text-state replacement recovers less.The diagnostic replaces target-prompt block outputs in a source trajectory using the same initial noise.
  • 3.4 VELOCITY DISTILLATION: Velocity distillation trains the controller by matching a source-prompt student with AcFlow active to a frozen target-prompt teacher under shared noisy inputs.The student and teacher use the same noisy latent, noise level, and guidance setting; training sets T = 1 and minimizes mean squared velocity error.

4 EXPERIMENTS

ACFLOW is evaluated for continuous fine-grained style control and concept suppression using frozen FLUX.1-dev, with a shared field tested on unseen style families. It achieves the strongest high-style-alignment trade-off among evaluated methods while supporting progressive stylization and qualitative suppression.

  • Continuous style control: ACFLOW is strongest in both style and content alignment in the high-style-alignment region at the fixed operating point.Table 2 compares ACFLOW at T = 2 with baselines at their maximum-style operating points.
  • Continuous style control: 0.5365/0.2860 style–content alignment for ACFLOW exceeds 0.4397/0.2684 for Mean-AcT and 0.5109/0.2786 for Concept Sliders.
  • Continuous style control: 0.442 style alignment and 0.281 content alignment are achieved on style families excluded from training at T = 2.The shared field transfers visible stylization and adjustable influence to unseen families.
  • Continuous style control: Increasing T raises style alignment while gradually reducing content alignment, providing a continuous style–content control coordinate.Fixed-noise sequences show gradual style changes as T varies.
  • Continuous style control: Extending T beyond the training horizon strengthens stylization on seen and unseen families, but gains eventually saturate while content alignment continues to decline.Large horizons can also alter composition.
  • Concept suppression: ACFLOW suppresses held-in fog, lighthouses, and rivers and held-out peonies, feathered hats, and daisies across intermediate horizons.Partial suppression appears at intermediate T, while appearance and composition can also change.

5 ABLATIONS

Ablations show that ACFLOW remains effective across tested intervention blocks and that Euler-3 provides a compact configuration for strong style control. Block location offers improvement room without being necessary for effectiveness.

  • Intervention block: 0.541/0.475 mean style scores for block 12 are highest on held-in/held-out styles, while block 20 improves held-out scores from 0.442 to 0.464 over block 16.ACFLOW maintains similar content scores across blocks 12, 16, and 20.
  • Intervention block: ACFLOW remains effective across blocks 12, 16, and 20, so tested intervention location is not critical to effectiveness.The ablation supports robustness to block selection within the tested range.
  • Euler integration steps: Euler-3 gives the highest mean style scores on both splits, whereas Euler-1 significantly reduces style alignment and Euler-5 or Euler-10 do not improve it monotonically.Euler-10 gives the highest content scores among configurations with competitive style alignment.

6 CHARACTERIZING THE FLOW FIELD

The analyses indicate that ACFLOW’s updates use token-specific directions and adapt their geometry and energy allocation over denoising. Residual directions retain effects beyond the dominant shared axis, and observed trajectories are inconsistent with simple rescaling of a fixed displacement.

  • Token-direction controls: Matching the dominant shared-axis component’s norm to the full update does not recover the full-field style gain, while the smaller original-norm residual retains a substantial effect.These controls support contributions from directions beyond the dominant shared axis.
  • Displacement field geometry: 12.3%/14.3% off-axis energy for held-in/held-out styles rises to 38.3%/41.4% before falling to 3.2%/3.8% late in denoising.The maxima occur at σ = 0.849/0.772 and the late values at σ = 0.065.
  • Token-direction controls: Late updates become more aligned across tokens after token-direction variation changes through the trajectory.Figure 7 reports alignment returning to about 6% near the end, while Figure 8 shows changing energy concentration.
  • Displacement field geometry: Update direction structure and token energy distribution change systematically across noise levels, beyond uniform rescaling of a fixed displacement.Under simple rescaling, off-axis energy fraction and top-token energy share would remain unchanged.

7 CONCLUSIONS AND LIMITATIONS

ACFLOW provides shared, concept-conditioned activation transport with continuous control and demonstrated style and suppression capabilities. Its key limitation is that interventions can alter image composition alongside the intended concept.

  • Conclusions: ACFLOW steers image-token activations in a frozen DiT using token-varying, activation-dependent updates shared across concepts, with the integration horizon providing continuous control.The method supports fine-grained descriptions and held-out concepts without per-concept fitting.
  • Limitations: ACFLOW does not guarantee preservation of original image composition, because intervention can change object positions, poses, and scene layout.Specific concept control remains an open challenge.

A PORTABILITY ACROSS BACKBONES

On Z-Image, AcFlow transfers style control and concept suppression using backbone-matched fields, with continuous horizon-based responses and qualitative removal of held-in and held-out targets.

  • Style control: Increasing T from 0 to 0.8 raises style alignment from 0.1868 to 0.5579 on held-in styles and from 0.1645 to 0.5178 on held-out styles.Figure 9 evaluates held-in and held-out style families with 95% pointwise intervals.
  • Qualitative transfer: Held-out style and suppression examples show visible interventions at larger horizons while retaining recognizable scene elements.The style and suppression figures vary horizon or show target removal with fixed prompts and seeds.
  • NSFW suppression: ACFLOW reduces NudeNet detections from 402 to 73 at s = 2 and 38 at s = 3, while s = 2 has higher DINO-I2I and lower LPIPS than CA, ESD-x, and negative prompting.Increasing strength to s = 3 further reduces detections but lowers DINO similarity from 0.612 to 0.534 and raises LPIPS from 0.509 to 0.563.
  • NSFW suppression: ACFLOW has the lowest clean ASR at 9.17% and the lowest attacked ASR at 25.7% for the primary attack and 43.1% for the Union.These results are reported for the aligned Ring-A-Bell suite.

C IMPLEMENTATION AND REPRODUCTION DETAILS

The implementation keeps the DiT and text encoders frozen while training concept-conditioned activation-flow modules, with task-specific datasets, fixed evaluation protocols, and reproduced baseline procedures.

  • Backbone-specific implementation: ACFLOW modifies image tokens at selected backbone sites while leaving the generator parameters frozen; the learned field supplies the intervention residual.FLUX uses block-specific modulation, while Z-Image updates the image-token slice after layer 19 or 25 depending on task.
  • Continuous style control: MegaStyle training groups fine-grained descriptions into style families, using 39 training families and six held-out families for family-disjoint evaluation.The target is a VAE encoding of the real MegaStyle image, with source and teacher prompts defining velocity distillation.
  • Evaluation: Style alignment uses fine-tuned MegaStyle SigLIP reference-image cosine similarity, while the direct styled-prompt reference appends the identical description to the content prompt.Evaluation uses full sets at T = 2 and balanced subsets across other horizons with bootstrap confidence bands.
  • Baseline protocols: Baseline reproduction includes per-style fitting for AcT, ActAdd, SHIFT, Concept Sliders, and Text Slider, with their reported sites, adapters, directions, or scaling procedures retained.The implementation details specify distinct fitting and reuse protocols for each baseline.
  • Suppression and diagnostics: Removal data pairs source captions containing unwanted concepts with edited captions specifying removal, and the target latent is generated by the frozen backbone under the edited caption.Adversarial evaluation uses matched prompts and seeds with independently generated no-attack controls and specified Ring-A-Bell settings.

E.1 TOKEN-DIRECTION AND ENERGY CONTROLS

Token-direction controls show that displacement energy and intervention effect are not equivalent: residual and off-axis directions contribute materially, with geometry varying across denoising noise levels.

  • Measurement protocol: The energy fraction statistic weights denoising steps by full-field energy while weighting cases equally, rather than averaging unweighted stepwise fractions.The controls use trajectory-level measurements across 28 denoising steps.
  • Energy controls: Rank one accounts for 76.2%/75.2% of displacement energy on held-in/held-out styles, but retains only 66.4%/31.7% of full-field style gain at matched norm.The original-norm residual retains 44.8%/67.7%, and full-norm residual rescaling raises these ratios to 75.9%/118.1%.
  • Energy controls: The held-out matched-residual versus matched-rank-one contrast is 0.864 ([0.543, 1.552]), supporting a contribution from off-axis directions.The held-in contrast is 0.096 (95% CI [−0.077, 0.268]) and does not establish an advantage.
  • Geometry across denoising noise levels: Off-axis fractions in suppression rise from 4.6%/5.3% to maxima of 36.2%/43.4% at σ = 0.824/0.799, then fall to 2.7%/2.5%.The reported held-in/held-out curves use full-field trajectories and pointwise bootstrap intervals.

F COMPLETE QUANTITATIVE RESULTS

The complete quantitative views retain all evaluated operating points and compare style-content behavior across methods, families, Euler settings, and intervention blocks.

  • Complete comparison: The complete comparison includes competitive-region and full-range views, with the default N = 3, T = 2 operating point marked by a blue star.The full-range panel includes 95% paired hierarchical-bootstrap intervals over families and contents.
  • Sensitivity sweeps: Quantitative sweeps compare held-in and held-out families across style and content alignment for six Euler and intervention-block settings.Shading denotes 95% within-family bootstrap intervals, and horizontal lines show full-set styled-prompt references.

G EXTENDED QUALITATIVE RESULTS

Extended sweeps vary the flow horizon while holding prompts, conditions, and initial noise fixed, testing style control and concept suppression across held-in and held-out families. They show both controllable responses and failures, including persistence, non-monotonic behavior, and content loss at larger horizons.

  • Style control: Style sweeps cover held-in and held-out families across multiple horizons for both FLUX Euler-3 and Z-Image.Rows hold prompts, conditions, and initial noise fixed while columns vary the flow horizon.
  • Concept suppression: Concept-suppression sweeps evaluate ten cases across eleven horizons on both FLUX and Z-Image, separating held-in from held-out cases.Prompts, conditions, and initial noise remain fixed within each row.
  • Limitations: Some targets persist or respond non-monotonically, while larger style horizons can replace scene structure with texture or remove recognizable content.The reported examples include FLUX umbrella and lemons, Z-Image plush Siamese cat and housefly, and minimalism and baroque rows.
Loading 2609.10723v1…