Source-linked AI summary
Semantic Image Manipulation Using Scene Graphs
Helisa Dhamo, Azade Farshad, Iro Laina, Nassir Navab, Gregory D. Hager, Federico Tombari, Christian Rupprecht
TL;DR
High-level semantic editing remains less explored than pixel-level manipulation, despite progress in semantics-driven image tasks. This paper edits an existing image through scene-graph changes without paired before-and-after edit supervision, and reports competitive baseline results and qualitative real-image edits.
Problem
High-level image manipulation has received less attention than pixel-level editing, and learning edits that preserve object identity and scene content presents challenges.
Source: §1 · pp. 1–2Method
The method generates a scene graph from an image and uses user-modified graph nodes or edges to generate an edited image without paired edit examples.
Source: §1 · p. 2Results
The system is competitive with baselines built from existing image-synthesis methods and qualitatively supports real-world removal, replacement, and relationship changes.
Source: §5 · p. 8, §4.2 · p. 8Takeaways & Limitations
A user can change image content and relationships by interacting with scene-graph nodes and edges while the resulting image preserves original content.
Source: §1 · p. 2
Abstract
from arXiv · showhide
Image manipulation can be considered a special case of image generation where the image to be produced is a modification of an existing image. Image generation and manipulation have been, for the most part, tasks that operate on raw pixels. However, the remarkable progress in learning rich image and object representations has opened the way for tasks such as text-to-image or layout-to-image generation that are mainly driven by semantics. In our work, we address the novel problem of image manipulation from scene graphs, in which a user can edit images by merely applying changes in the nodes or edges of a semantic graph that is generated from the image. Our goal is to encode image information in a given constellation and from there on generate new constellations, such as replacing objects or even changing relationships between objects, while respecting the semantics and style from the original image. We introduce a spatio-semantic scene graph network that does not require direct supervision for constellation changes or image edits. This makes it possible to train the system from existing real-world datasets with no additional annotation effort.
1. Introduction
Semantic image editing remains less explored than pixel-level manipulation, while a scene graph offers a way to edit objects and their relationships directly. The paper introduces graph-based editing that supports multiple changes without fully supervised before-and-after image pairs.
1. Introduction
- High-level image manipulation has received less attention than recognition, image description, and semantics-driven image synthesis.Image manipulation is typically performed at the pixel level, while higher-level examples are usually object-centric.Source: §1 · p. 1
- The method edits an image by modifying its scene graph, which represents objects, attributes, and interactions.The user can change visual entities and how they interact, both spatially and semantically.Source: §1 · pp. 1–2
- Users can remove or replace objects, rearrange their spatial composition, and change semantic relationships between them.The paper gives examples including deleting person nodes, replacing clouds with sky, and changing a person from walking to jogging.Source: §1 · p. 2
- Learning edits that preserve an object's identity and the rest of the scene poses a challenge, while collecting supervised before-and-after edit pairs is difficult.The paper states that training can instead use existing image and scene-graph pairs.Source: §1 · p. 2
2. Related Work
Related work spans conditional image generation, object- and scene-level manipulation, and scene-graph methods for understanding or generating images. The paper distinguishes its approach by editing an existing image and supporting complex semantic relationship changes.
2. Related Work
- Conditional generation has used labels, attributes, low-resolution images, segmentation maps, language, layouts, and scene graphs as prior information.Scene-graph generation methods produce natural scenes from graphs [17].Source: §2 · p. 2
- Semantic image manipulation has often been object-centric, while scene-level methods include inpainting, object removal, and semantic-map editing.The paper contrasts these with a single general-purpose model that incorporates edits through scene graphs and handles object deformations.Source: §2 · p. 2
- Scene graphs represent image content, and prior work studies generating graphs from images as well as generating images from graphs [17].The paper builds on graph-to-image generation while adding information transfer from the source image for editing rather than free-form generation.Source: §2 · pp. 2–3
- Unlike graph- or layout-based methods that generate multiple variants or use simplified spatial relations [1] [44], this approach edits an existing image and handles complex semantic relationships.A related 3D-aware modification method is limited to street scenes and cars and requires CAD models [51].Source: §2 · pp. 2–3
3. Method
The method turns an image into a user-editable scene graph, then generates an image from the modified graph while conditioning on the original image. It learns edits through reconstruction rather than paired before-and-after data.
3. Method
- Users edit object categories, locations, or relations in a scene graph; the system generates an image from the modified graph and original image content.The pipeline comprises graph generation, graph editing at inference, and output-image generation.Source: §3 · p. 3
3.1. Graph Generation
- Objects encode category, bounding box, and visual features, while relation embeddings capture predicate classes, preserving appearance and spatial information in the graph.This image-feature and location augmentation is intended to preserve object identity and appearance when locations or relationships change.Source: §3.1 · p. 4
3.2. Spatio-semantic Scene Graph Network
- The spatio-semantic scene graph network passes information between objects along relationships and predicts each object’s box, mask, and node feature.The predicted object representations are reassembled into the scene layout; box prediction also supports adding nodes after graph edits.Source: §3.2 · p. 4
3.3. Scene Layout
- The scene layout projects each object’s predicted mask and combined visual and node features into its bounding box, then sums the object tensors into one image representation.The resulting intermediate representation is used to reconstruct an image.Source: §3.3 · p. 4
3.4. Image Synthesis
- Image synthesis conditions on the predicted layout and low-level features from the source image, with selected source regions occluded and filled with Gaussian noise.The method uses either cascaded refinement networks or SPADE as the decoder.Source: §3.4 · p. 4
3.5. Training
- Training reconstructs the original image from randomly masked object features or boxes, avoiding paired edited-image data while retaining full supervision for graph prediction.For edits expected to change spatial arrangement, object locations are discarded while visual features remain to preserve identity; losses include box L1, adversarial, and unchanged-region photometric terms.Source: §3 · p. 3, §3.5 · pp. 4–5
4. Experiments
Experiments evaluate semantic image manipulation on synthetic CLEVR and natural Visual Genome images, using quantitative reconstruction tests and qualitative graph-driven edits.
4. Experiments
- CLEVR supports quantitative evaluation with editing pairs, while Visual Genome tests a less constrained real-image setting using an in-painting proxy when modification pairs are unavailable.The evaluation reports SSIM, MAE, and LPIPS for reconstruction, plus inception score and FID for generation quality and diversity.Source: §4 · p. 5
4.1. Synthetic Data
- On CLEVR, the method is better or on par with a fully supervised baseline on reconstruction metrics, while FID suggests paired supervision could improve visual quality.The manipulation modes include replacement, removal, relationship change, and addition; Figure 3 illustrates scene changes.Source: Table 1 · p. 5, §4.1 · p. 6, Figure 3 · p. 6
4.2. Real Images
- On Visual Genome, adding object visual features improves MAE, SSIM, and LPIPS reconstruction, while inception score and FID remain similar; masking box properties shifts reconstructed location and size, and masking features can change identity.The feature-encoding evaluation covers ground-truth and predicted graphs; Table 2 reports results for both graph settings.Source: §4.2 · pp. 6, 8, Table 2 · p. 7
- In the full generative setting, reconstruction errors are comparable to ISG [1], while the method outperforms [1] when a source image is given.SPADE outperforms Pix2pixHD and CRN on inception score and FID; those metrics mostly depend on decoder architecture.Source: §4.2 · pp. 6–8, Table 2 · p. 7
- With visual features, the method transfers appearance from the original image, making object repositioning possible without changing identity.Figure 4 compares the baseline and method when an object is occluded while its graph and latent features remain active.Source: Figure 4 · p. 6, §4.2 · p. 8
- Qualitative edits include replacement, removal, and relationship changes; replacements adapt to image context, and relationship edits meaningfully rearrange interacting objects.Removal works well on uniform backgrounds and can also handle some more complex structures.Source: Figure 5 · p. 7, §4.2 · p. 8
5. Conclusion
The conclusion presents scene-graph-based semantic image manipulation without training pairs of original and modified images. The system changes scene content and relationships, performs competitively against synthesis baselines, and shows qualitative promise on real-world images.
5. Conclusion
- The system changes both scene content and relationships through edits to scene-graph nodes and edges.It does not require training pairs of original and modified image content.Source: §5 · p. 8
- The system is competitive with baselines built from existing image-synthesis methods.The conclusion also reports qualitative evidence of modification ability on real-world images.Source: §5 · p. 8
- The authors identify interactive editing and robotics applications as directions for future work.Source: §5 · p. 8
6. Appendix
The appendix reports qualitative behavior, CLEVR results, dataset details, implementation choices, and failure cases. Results include strong CLEVR relationship-change performance and qualitative evidence of spatial repositioning, alongside limitations in preserving appearance and adapting unchanged image regions.
6. Appendix
- The appendix provides additional results and implementation and training details.It states that code and benchmark data splits will be released on the project webpage.Source: §6 · p. 11
6.1. More Qualitative Results
- Relationship edits reposition objects to fit the changed relation, and predicted predicate-conditioned spatial distributions resemble ground truth for well-constrained relations.A user-interface video demonstrates multiple changes to an image; qualitative comparison reports generally better object removal than [1], while auto-encoding is similar.Source: §6.1 · p. 11, Figure 7 · p. 12
6.2. Ablation study on CLEVR
- The SPADE-decoder version outperforms other models on CLEVR reconstruction, while relationship changes clearly dominate and other manipulation changes perform similarly to the baseline.Source: §6.2 · p. 11, Table 3 · p. 13, Table 4 · p. 14
6.3. Datasets
- CLEVR contains 21,310 paired 128 × 128 images with 3–7 objects, generated changes, and graph, box, class, and attribute annotations.Visual Genome uses 178 object categories and 45 relationship types, with 62,565 training, 5,506 validation, and 5,088 test images; predicted-graph evaluation uses 3,874 test images.Source: §6.3 · p. 11
6.4.1 Image →scene graph
- For Visual Genome experiments, the method uses a state-of-the-art scene-graph prediction network trained on a preprocessed subset with 399 object and 24 predicate categories.The appendix identifies Tables 3 and 4 as CLEVR reconstruction and manipulation results, respectively.Source: §6.4.1 · p. 12, Table 3 · p. 13, Table 4 · p. 14
6.4.2 Scene graph →image
- The scene-graph network predicts node features, masks, and bounding boxes, while the image generator uses cascaded refinement modules conditioned on predicted layout and masked image features.The method also describes a SPADE decoder, an object discriminator applied to changed image areas, and training settings including Adam at 10^-4.Source: §6.4.2 · pp. 14–15
6.5. Failure cases
- Restricting feature encoding helps prevent copying regions when repositioning non-rigid objects, but some edited-object properties, such as material, may not be recovered.The appendix illustrates preserved color but lost material in a modified object.Source: §6.5 · p. 15
- Unchanged areas are not adapted after edits, so unmarked shadows or reflections may not follow repositioned objects.The authors also report limited quality for some close-up objects and human-like faces on animals with face nodes.Source: §6.5 · p. 15