Source-linked AI summary
AffectDelta: Beyond Emotion Labels for Image Editing
Xingzu Zhan, Lin Gu, Ruogu Fang
TL;DR
Existing emotion editors often reduce mixed affective endpoints to single labels or express changes through imprecise operation-level language. AffectDelta instead uses signed transitions between eight-dimensional distributions, and experiments report improved affective alignment and content preservation against six baselines.
Problem
Existing scene-level editors cannot fully represent mixed source and target affect with a single category, while operation-level language does not precisely quantify emotion changes.
Method
AffectDelta uses a frozen Emotion Distribution Predictor, a signed source-to-target distribution difference, and source-aware diffusion to produce scene-dependent edits without explicit text conditioning.
Results
AffectDelta ranks first on all six metrics, including Top-1 0.899, JSD 0.056, NAP 0.199, ∆-Cos 0.744, CLIP-I 0.778, and LPIPS 0.382.
Takeaways & Limitations
Distribution-valued control supports fine-grained emotional image editing with improved affective alignment and content preservation.
Abstract
from arXiv · showhide
Emotion-driven image editing aims to evoke a specified target emotion by modifying emotion-relevant visual cues in a source image, while preserving the overall composition and semantic-structural coherence of the original scene. Existing scene-level editors typically specify the target with a single emotion category and often learn visual transformations from operation-level text instructions. A category collapses a mixed affective endpoint into one dominant label, while language cannot precisely quantify how coexisting emotions should increase, decrease, or remain stable. We introduce AffectDelta, a source-aware editor that treats editing as a transition between eight-dimensional emotion distributions. A frozen Emotion Distribution Predictor estimates the source state, and the signed source-to-target difference encodes the direction and magnitude of the requested transition. Within AffectDelta, an internal transition encoder and a source-aware diffusion backbone jointly translate this signal into context-dependent semantic and appearance changes. To train this formulation, we construct AffectPair-249K, comprising 248,841 source-target pairs with predicted eight-dimensional distributions and spanning both cross-category and within-category transitions. Experiments against six baselines, combining quantitative evaluation with qualitative comparisons, demonstrate improved affective alignment and content preservation, while ablations validate our design choices. Code and dataset will be made publicly available upon acceptance.
Introduction
Emotion-driven image editing must preserve scene content while controlling mixed, context-dependent affect. AffectDelta addresses this gap by representing edits as signed transitions between full eight-dimensional emotion distributions and learning visual realizations without editing-text supervision.
- Emotion-driven editing modifies emotion-relevant cues while preserving scene identity and unrelated content, supporting creative, communicative, and psychological-research applications.
- Single target labels cannot express mixed source emotions or the full affective transition, while existing methods may rely on target-only or instruction-derived control.
- AffectDelta represents emotional editing as a signed transition between full distributions over eight emotion categories, encoding which emotions change and by how much.
- AffectPair-249K contains 248,841 task-filtered source–target pairs spanning cross-category transitions and same-Top-1 distribution shifts.
- AffectDelta is a unified source-aware diffusion editor that learns transition encoding and scene-dependent visual realization without editing-text supervision during adaptation.
Related Work
Prior work progressed from categorical and low-level affect modeling toward distributional representations, and from appearance filtering toward semantic scene editing. AffectDelta differs by conditioning semantic editing directly on signed source-to-target distribution differences rather than target labels or operation-level language.
- Visual Emotion Representation: Visual emotion analysis evolved from single-label categorization to distributional modeling of affective ambiguity, subjectivity, and interpretable affective attributes.
- AffectDelta conditions semantic editing directly on the signed difference between source and target distributions.
- Earlier affective editors mainly transferred color and texture or mapped emotion-guided signals to appearance transformations, largely preserving emotion-relevant scene content.
- Scene-level generative editors expanded affective editing to objects and actions, but representative methods used one-hot destinations, constructed instructions, or emotional descriptions.
- Diffusion and instruction-following editors preserve source structure through denoising or cross-attention control, with training data ranging from manually annotated triplets to large real-image datasets.
Method
AffectDelta formulates image editing as a signed transition between eight-dimensional source and target emotion distributions, then realizes that transition through a source-aware diffusion editor. Its distribution-aware dataset and text-free paired training preserve mixed affective states while learning context-dependent visual changes.
- Affective Transition: AffectDelta represents editing as the signed difference between source and target distributions over eight emotions, encoding which emotions should increase or decrease and by how much.The representation preserves dominant and secondary emotions instead of reducing either endpoint to one category.
- Emotion Distribution Predictor: A frozen Emotion Distribution Predictor estimates image-evoked distributions, using single-label pretraining followed by distributional fine-tuning on Flickr-LDL and Twitter-LDL.The second stage preserves relative mass assigned by annotators to coexisting emotions.
- Pair Construction: AffectPair-249K contains 248,841 task-filtered source-target pairs spanning cross-category transitions and within-category distribution shifts selected using EDP predictions.Within-category pairs preserve the Top-1 emotion while exhibiting large Jensen–Shannon divergence, capturing changes categorical supervision would discard.
- Text-Free Adaptation: Training excludes editing instructions and uses paired images, the transition, and the target distribution so visual outcomes teach scene-specific realizations of affective changes.This avoids binding an affective transition to a particular linguistic description or predefined operation.
- AffectDelta Architecture: An internal transition encoder and source-aware diffusion backbone jointly condition denoising on transition tokens and the source-image latent.The source latent anchors scene structure, while transition tokens specify where affect should move; inference requires only the source image and desired endpoint.
- Training Objectives: The standard diffusion noise-prediction objective learns paired visual transformations, while an auxiliary emotion loss aligns a reconstructed prediction with the target affect distribution.The auxiliary loss is evaluated by the frozen EDP because diffusion training alone does not explicitly require target-affect matching.
Experiments
AffectDelta is evaluated on a disjoint 4,977-pair test set against six editors using independent affective and content-preservation metrics. It achieves the strongest quantitative and qualitative results, while ablations support distribution-valued, source-aware control and graded affective editing.
- Experimental Setup: The 4,977-pair disjoint test set approximately balances eight target-emotion categories and includes cross-category and same-Top-1 transitions.The remaining 243,864 pairs are used for training, and all comparisons use this test set.
- Experimental Setup: Evaluation uses independent prediction to measure target Top-1, full-distribution JSD, affective progress, transition direction, and source-content preservation.JSD compares complete target distributions, while NAP and ∆-Cos assess progress and transition alignment.
- Quantitative Comparison: 0.899 Top-1, 0.056 JSD, 0.199 NAP, and 0.744 ∆-Cos make AffectDelta first on all six metrics.The results indicate accurate dominant-emotion and full-distribution control, along with alignment to the signed source-to-target transition.
- Qualitative Comparison: AffectDelta obtains the lowest JSD in all four qualitative cases spanning Amusement, Fear, Awe, and Disgust.Its edits adapt semantic and appearance changes to each source scene rather than applying a fixed text-linked operation.
- Multi-Directional and Graded Affective Control: From one source, varying target-emotion directions and strengths produces distinct images with progressively stronger affective expression while preserving the underlying scene.The complete target distribution supplies both affective direction and fine-grained intensity control.
- Ablation Study: Replacing the target distribution with a one-hot category reduces Top-1 from 0.899 to 0.642, JSD from 0.056 to 0.122, and ∆-Cos from 0.744 to 0.293.Target-only conditioning also lowers NAP from 0.199 to 0.023 and ∆-Cos from 0.744 to 0.694, supporting source-state conditioning.
Conclusion
AffectDelta formulates emotional image editing as signed transitions between complete eight-dimensional emotion distributions and learns scene-dependent visual realizations from paired outcomes. Comparisons, ablations, and graded-control results show improved affective alignment and content preservation for fine-grained emotional editing.
- Conclusion: AffectDelta is a source-aware editor driven by signed transitions between complete eight-dimensional emotion distributions.It is trained on AffectPair-249K without editing-text supervision during adaptation.
- Conclusion: Comparisons with six baselines, ablations, and graded-control results show improved affective alignment and content preservation.The reported findings support distribution-valued control for fine-grained emotional image editing.
Evaluation Metrics
The evaluation framework uses an independent emotion-distribution evaluator to assess dominant-emotion accuracy, complete-distribution agreement, transition progress and direction, and content preservation. Metrics are averaged over successfully evaluated test records with specified normalization and edge-case handling.
- Evaluation Metrics: Target Top-1 measures whether the edited image’s dominant predicted emotion matches the target distribution’s dominant component.It evaluates agreement at the dominant-emotion level.
- Evaluation Metrics: Base-2 JSD measures discrepancy between the predicted output and target distributions, with lower values indicating better agreement.Zero-probability terms are omitted and the result is clipped to [0, 1] for numerical stability.
- Evaluation Metrics: NAP measures progress from the independently predicted source state toward the target, approaching 1 at the target, equaling 0 without progress, and becoming negative when moving away.The metric is not clipped, so small source-to-output distances can amplify negative values.
- Evaluation Metrics: ∆-Cos compares the requested reference transition with the transition predicted for the source and edited images.Higher values are better; zero-norm transitions receive a recorded score of 0.
- Evaluation Metrics: CLIP-I measures semantic-content preservation with higher values better, whereas LPIPS measures perceptual distance with lower values better.Both metrics use fixed model configurations and metric-specific preprocessing across methods.
Implementation Details
AffectDelta uses an eight-output ResNet-18 EDP and trains a transition encoder with a diffusion U-Net on source-group-disjoint paired images. The VAE and EDP remain frozen while the transition encoder and U-Net are jointly optimized.
- Implementation Details: The EDP is an eight-output ResNet-18 fine-tuned on Flickr-LDL and Twitter-LDL emotion-distribution data.Its final residual stage and prediction head are fine-tuned, with held-out images used for model selection.
- Implementation Details: AffectDelta trains on 243,864 source-group-disjoint image pairs at 224 × 224 resolution.The pairs are used to learn the editor from source-target affective transitions.
- Implementation Details: A four-layer transition encoder maps each eight-dimensional signed emotion difference to 77 tokens of width 768 for the InstructPix2Pix cross-attention interface.The transition encoder and U-Net are jointly optimized while the VAE and EDP are frozen.
- Implementation Details: Training optimizes Ldiff + 0.1Lemo with AdamW for 15 epochs using learning rate 10^-5, batch size 32 per GPU, and gradient clipping at norm 1.0.The implementation uses distributed data parallelism and FP16 automatic mixed precision.
EDP Fine-Tuning Analysis
EDP fine-tuning improves agreement with dominant emotions and fidelity to full emotion distributions while reducing single-label overconfidence. Example-level analyses show recovery of secondary emotions and correction of spurious sadness predictions.
- 0.2298 to 0.7964 Top-1 agreement accompanies KL reduction from 3.6682 to 0.3456 and MAE reduction from 0.1745 to 0.0522.These changes indicate improved dominant-class agreement and full-distribution fidelity.
- Mean maximum probability falls from 0.7626 to 0.4532, indicating reduced single-label overconfidence rather than indiscriminate smoothing.Distribution-level supervision recovers plausible non-dominant probability mass.
- Fine-tuning preserves correct Top-1 decisions while recovering secondary emotion mass in awe and sadness examples.Initialization assigns 0.9999 to awe versus a target distribution including amusement and contentment, and 0.7922 to sadness versus a target containing three secondary emotions.
- 0.7802 and 0.5348 spurious sadness predictions are replaced by amusement Top-1 predictions of 0.8436 and 0.4995 on grayscale examples.The corrected examples are consistent with reduced reliance on low-level appearance cues.
Dataset Construction and Splits
The dataset construction groups related sources and assigns complete groups to splits, then organizes a balanced test core across directed emotion transitions and stratified affective changes. The final test set contains 4,977 pairs, while exact balance claims apply to its 4,960-pair core.
- Source-grouped partition: Source grouping keeps shared sources, edit chains, duplicate hashes, and perceptual near-duplicates within one split.Complete groups are assigned wholly to training or test.
- Balanced transition core: 4,960 pairs form a strictly balanced test core across all 64 directed Top-1 transition units.The 56 offdiagonal units contain 60 pairs each, while the eight same-Top-1 units contain 200 pairs each.
- Balanced transition core: 620 pairs represent every source Top-1 emotion and every target Top-1 emotion in the balanced core.The core includes 1,600 same-Top-1 pairs, or 32.3% of the core.
- Balanced transition core: The final test set has 4,977 pairs and the training set has 243,864 pairs after 17 additional audit pairs are retained.Exact balance statements refer to the 4,960-pair core, whereas evaluation metrics use all 4,977 test pairs.
- Within-unit stratification: Within transition units, sampling stratifies JSD or ∥∆p∥1, target entropy, scene type, and same-Top-1 change patterns.Same-Top-1 patterns include strengthening, weakening, or redistributing secondary emotions.