PaperlayerView exact revision on arXiv

Source-linked AI summary

ReDesign: Recovering Editable Design Structures from Images via Agentic Decomposition

Jooyeol Yun, Jintae Park, Hyesu Lim, Junha Hyung, Hyungjin Chung, Jaegul Choo

arXiv:2607.25565v1cs.CV

TL;DR

Recovering editable design structures from raster images is difficult and often manual because existing tools address separate reconstruction subtasks unreliably. ReDesign grows a hierarchical editable representation with specialized tools and local verification, achieving strong visual fidelity and the highest editability across layout, color, and text edits.

  • Problem

    Recovering editable designs from raster images remains largely manual, while combining specialized reconstruction tools reliably across diverse designs remains unresolved.

  • Method

    ReDesign grows a partial editable layer hierarchy through specialized tool actions, using graceful verification to accept, prune, or retry each expansion.

  • Results

    ReDesign achieves strong visual fidelity and the highest editability across layout, color, and text edits on the Figma Edit Replay Benchmark.

  • Takeaways & Limitations

    The results support structurally constrained agentic workflows in which reconstruction failures remain local and repairable rather than accumulating into hard failures.

  • Takeaways & Limitations

    Reconstructions do not always match reference layer splits because the desired editable granularity depends on the intended edit and designer control needs.

Abstract

from arXiv · show

Recovering an editable design file from a raster image is a common and costly bottleneck in modern design workflows, yet remains challenging since editability depends on recovering multi-modal attributes, such as typography, vector geometry, colors, grouping, and layer ordering. We present ReDesign, an agentic framework that grows an editable layer hierarchy by selecting and composing specialized tools across modalities. To keep this long decision process reliable despite imperfect tool outputs, we introduce graceful verification at each expansion, which provides local accept, prune, or retry feedback that prevents error accumulation and avoids large scale reruns. To evaluate editability at scale, we introduce the Figma Edit Replay Benchmark, consisting of 909 raw Figma files and 14,796 controlled edit instructions that replay edits on reconstructed outputs. Across this benchmark and standard reconstruction metrics, ReDesign achieves strong visual fidelity while delivering the highest editability across layout, color, and text edits, outperforming layered decomposition baselines and serial tool use pipelines.

1 Introduction

ReDesign addresses the costly, error-prone task of reconstructing editable design structures from raster images by growing layer hierarchies through agentic tool composition. It adds local graceful verification and introduces a benchmark for evaluating visual fidelity and editability through controlled Figma edits.

  • Motivation: Raster-to-editable reconstruction is largely manual and error-prone because real-world design assets and handoff deliverables often exist only as raster exports or screenshots.The target editable representations include separate typography, color, and layout objects with parameters that expose editing controls.
  • ReDesign: ReDesign casts reconstruction as structured tree expansion, composing specialized tools to grow an editable layer hierarchy into atomic elements.The framework addresses unreliable fixed-sequence composition and cascading errors among tools.
  • Graceful verification: Graceful verification evaluates proposed children at every node expansion and returns accept, prune, or retry outcomes to repair only the offending branch.The verifier targets duplication across sibling nodes and incomplete coverage of the parent region, preventing errors from accumulating into hard failures.
  • Evaluation: The Figma Edit Replay Benchmark contains 14,796 controlled edit instructions spanning layout, color, and text, alongside diverse real-world designs from raw Figma files.The benchmark includes ground-truth layer hierarchies and attributes and evaluates both visual fidelity and editability.
  • Evaluation: Experiments report superior performance for ReDesign compared with strong baselines on the introduced evaluation.The supplied passage states that evaluations assess visual fidelity and editability, while the contribution passage describes extensive experiments.

2 Related Work

Related work spans layer-aware generation and decomposition, image vectorization, and tool-using agents for controllable and editable image reconstruction. ReDesign builds on these directions while selectively combining representations and addressing limitations of serial tool invocation.

  • Layer-aware generation and decomposition: Layer-aware generation and decomposition isolate design elements into editable layers, supporting controllable composition and localized post-hoc editing.Methods include raster-to-layer decomposition and PSD-like layered outputs aimed at inherent editability.
  • Image vectorization: Vectorization exposes direct controls for shape geometry, scale, and appearance through vector paths or SVG programs.The literature ranges from classical tracing pipelines to learned models that generate vector primitives or scalable vector graphics programs.
  • Image vectorization: Forcing complex textures or photographic content into paths can distort authored representations and reduce editability.Practical reconstruction must choose among vector, text, and raster layers according to the region being represented.
  • Tool-using agents: Tool-using agents enable multi-step problem solving by selecting and executing external modules, while image-editing models provide localized manipulation primitives.Existing vision-agent pipelines often invoke tools serially and pair them with validation at the end of the pipeline.

3 Method

ReDesign reconstructs an editable design as a progressively expanded JSON layer hierarchy whose nodes encode multimodal metadata. A controller selects modular decomposition tools, while graceful verification locally accepts, revises, or rejects expansions to limit error propagation.

  • Reconstruction representation: The target is a JSON hierarchy recovered from a raster root, with editable vector shapes, colors, text, fonts, groups, and z-orders.The partial reconstruction tree begins with incomplete metadata and is exported once sufficient metadata has been filled.
  • Controller and action space: A controller policy grows the hierarchy coarse-to-fine, selecting from modular actions that encapsulate tool chains for specific decomposition scenarios.The action space can be extended as new tools become available, and early decisions establish grouping and ordering before later refinement.
  • Specialized decomposition tools: Specialized tools recover text, layered structure, disjoint components, entangled objects, vector paths, photographic layers, and editable typography.The pipeline combines OCR, segmentation, inpainting, layered generation, connected-component labeling, detection, segmentation, vectorization, font recognition, and typographic fitting.
  • Execution and memory: Parallel expansion across current leaf nodes improves throughput because each expansion depends only on its parent and lineal state, not sibling states.The hierarchy preserves coarse-to-fine consistency while expanding multiple leaves simultaneously.
  • Graceful verification: Each proposed parent-to-children expansion receives graceful verification that can accept valid children, provide revision signals, or prevent invalid decompositions from propagating.Verification checks whether the candidate children jointly explain the parent region, addressing duplicates, unexplained areas, and hallucinated content.
  • Execution and memory: Per-node lineal memory records tool inputs, outputs, choices, hyperparameters, verification outcomes, and failure reasons without tracking sibling states.This compact path history avoids exponential growth and concurrency issues during parallel execution.

4 Experiments

Experiments evaluate ReDesign on real-world Figma designs using edit replay and visual reconstruction metrics, with comparisons against layered decomposition, tool-using, and vectorization baselines. ReDesign achieves the strongest editability and layout fidelity while preserving competitive visual quality across Figma and Crello.

  • Datasets and benchmark: The benchmark contains 909 raw Figma files and 14,796 controlled edit instructions averaging 15 edits per design.Each file includes ground-truth layer hierarchy, element types, text and typography, vector shapes, colors, and ordering.
  • Baselines: Baselines include Qwen-Image-Layered, LayerD, a linear tool-using agent, and VTracer, with OCR and vectorization used to make layered outputs editable.Recent vectorization models such as OmniSVG and StarVector performed poorly on design images because of a domain gap.
  • Edit replay protocol: Edit replay applies identical operations to ground-truth and reconstructed designs, matching target elements by highest IoU before rendering the edited outputs.SSIM measures rendered similarity, while text edits additionally use OCR-based text recall.
  • Editability results: ReDesign achieves the best editability across all edit types, particularly recolor and opacity, where baselines often suffer edits bleeding into backgrounds or nearby layers.The results indicate that structured reconstruction supports precise targeting, while graceful verification limits failures caused by incorrect grouping and z-order.
  • Visual reconstruction results: On Figma, ReDesign is best across object appearance, global fidelity, and layout metrics, preserving structurally coherent and cleanly editable decompositions.On Crello, it achieves the strongest layout fidelity and remains competitive on object-level and global metrics without explicit Crello training.

5 Analysis

The analysis shows that graceful local verification and parallel tree expansion improve reconstruction efficiency, while agentic hierarchy growth enables user-controlled editing granularity. Comparisons and limitations further motivate explicit editable structures over direct image editing and fixed annotation granularities.

  • Efficiency: Graceful verification is faster, more accurate, and less variable than terminal verification because local repairs prevent costly error cascades and restarts.The analysis contrasts many small local checks with a few large terminal checks.
  • Efficiency: 7.1× speedup over serial tool use results from parallel expansion of independent frontier nodes in the reconstruction tree.Tree dependencies reduce the critical path from the number of expansions to hierarchy depth.
  • Direct image editing: Nano Banana 2 often alters unintended regions and image sizes during local move and rotate edits, motivating explicit editable structure recovery.The comparison shows that visually plausible single-step edits do not guarantee detailed spatial accuracy.
  • Editing granularity: Editable granularity depends on the intended edit: vectorization may be unnecessary for layout changes, whereas fine-grained paths matter for icons and shapes.Reconstructions need not match Figma or Crello layer splits one to one if rendered appearance remains correct.
  • Editing granularity: Users can request finer or coarser decompositions, and the system expands the hierarchy accordingly while revealing low-level vector paths only when needed.This user-in-the-loop control is enabled by the agentic tree and its higher-level groups.

6 Conclusion … B Extended Related Work

ReDesign reconstructs editable design files through agentic layer-hierarchy growth and local verification, achieving strong visual fidelity and high editability. The broader discussion covers workflow benefits, misuse risks, and related layered and vector representations.

  • 6 Conclusion: ReDesign grows a layer hierarchy with specialized tools and verifies every node expansion, enabling local error detection and repair instead of accumulated hard failures.The framework is presented as an agentic reconstruction approach for editable design files from raster images.
  • 6 Conclusion: 14,796 edit replay instructions comprise the new benchmark, where ReDesign delivers strong visual fidelity and the highest editability across layout and color edits.The supplied passage also introduces the benchmark as raw Figma files with controlled edit replay instructions.
  • 6 Conclusion: Structural constraints on agent progress can improve controllability, accuracy, and efficiency by making failures local, repairable, predictable, and scalable.This conclusion frames structural representations as a practical direction for agentic systems and workflow design.
  • A Broader Impact and Ethical Considerations: Recovering editable structure can reduce the effort of recreating flattened assets and simplify revisions to text, colors, layout, accessibility properties, and localized content.These benefits are especially relevant when the original source file is unavailable.
  • A Broader Impact and Ethical Considerations: ReDesign may lower barriers to copying copyrighted graphics, branded assets, and commercial templates, so deployment should target legitimate editing rather than bypassing ownership or attribution.Suggested safeguards include pairing such systems with deployment protections, though the supplied passage truncates the specific list.
  • B Extended Related Work: Layer-wise image generation represents images with multiple transparent layers, offering a more editable intermediate representation than a single raster image.Prior work includes foreground–background separation, limited transparent-layer synthesis, and post-hoc decomposition pipelines.
  • B Extended Related Work: Vector graphics generation converts raster images into vector graphics through differentiable rendering, iterative path fitting, or structured SVG generation, including semantically grouped layered SVGs.These methods target logos, icons, illustrations, and other shape-dominant assets, but primarily target vector-form reconstruction.

C Extended Analysis · C.1 Action Selection Across Decomposition Tree Depth · C.2 Retry Rates during Graceful Verification

ReDesign’s controller uses decomposition depth as an organizational prior, moving from semantically important global operations to localized visual refinement while adapting decomposition granularity to node complexity. Graceful verification is action-dependent: extraction actions use accept-or-retry decisions, whereas generative actions require additional handling because they may hallucinate content.

  • C.1 Action Selection Across Decomposition Tree Depth: At shallow depths, the controller prioritizes Text Extraction to preserve high-value, semantically precise elements such as text.
  • C.1 Action Selection Across Decomposition Tree Depth: At depth 1, Qwen-Image-Layered frequently performs a broad structural split that produces coarse semantic layers exposing the design’s major organization.
  • C.1 Action Selection Across Decomposition Tree Depth: At depth 2, CCL becomes more common as coarse layers are refined.
  • C.1 Action Selection Across Decomposition Tree Depth: Deeper nodes increasingly favor Detect & Segment, indicating localized, fine-grained refinement rather than global restructuring.
  • C.1 Action Selection Across Decomposition Tree Depth: The decomposition tree divides labor across depth, with early levels handling semantically important global structure and later levels performing precise visual cleanup.
  • C.1 Action Selection Across Decomposition Tree Depth: The controller adjusts Qwen-Image-Layered’s layer-length hyperparameter to the visual entanglement of each node instead of applying fixed decomposition granularity.
  • C.2 Retry Rates during Graceful Verification: CCL and Text Extraction use binary accept-or-retry verification because pruning their extracted content would irreversibly discard information.
  • C.2 Retry Rates during Graceful Verification: Detect & Segment and Fork Layers rely on generative models that occasionally hallucinate absent content, making their verification behavior action-dependent.

C.3 Robustness to VLM Backbones

ReDesign remains robust across VLM backbones, with Gemini-3-flash and GPT-5 mini trading off coverage and fidelity through different verification strictness. GPT-5 mini favors stricter pruning and global fidelity, while Gemini-3-flash analyzes multi-image context more thoroughly and avoids over-pruning.

  • C.3 Robustness to VLM Backbones: GPT-5 mini applies stricter verification, retaining only high-confidence elements to improve global-level fidelity by removing less-certain reconstructions.This trade-off is reflected in improved PSNR and LPIPS for the final composite.
  • C.3 Robustness to VLM Backbones: Gemini-3-flash analyzes multi-image context more thoroughly and avoids over-pruning, favoring broader element coverage.Gemini-3-flash is the default controller and verifier backbone.
  • C.3 Robustness to VLM Backbones: The two VLM backbones exhibit a coverage–fidelity trade-off governed by verification strictness.The comparison uses Gemini-3-flash and GPT-5 mini as both controller and verifier.

C.4 Quantitative ablation study … D.2 System Configuration

The paper evaluates reconstruction and editability through depth and memory ablations, then defines an edit-replay matcher that accommodates differing decompositions. Its training-free system combines publicly available tools, commercial VLM APIs, and LangGraph orchestration.

  • C.4 Quantitative ablation study: C.4 ablations vary tree depth and memory hierarchy to study their effects on decomposition accuracy and editability.Shallow depth produces coarse decompositions with higher accuracy but lower editability, whereas full depth improves layout accuracy and editability.
  • D.1 Edit Replay Protocol: D.1 matching represents ground-truth and predicted elements as RGBA images and binary masks, computing visible masks in descending z-order.This formulation accounts for occlusion before matching predicted and ground-truth design elements.
  • D.1 Edit Replay Protocol: D.1 directional containment ratios generate many-to-many candidate groups that handle over- or under-segmentation between predicted and ground-truth structures.Candidate subsets are enumerated symmetrically on both sides so valid parses remain matchable when decompositions differ.
  • D.1 Edit Replay Protocol: D.1 group-pair costs combine RGB difference, mask IoU, and merge penalties, after which the Hungarian algorithm finds a globally optimal matching.The merge penalties discourage combining unrelated elements merely to reduce matching cost.
  • D.1 Edit Replay Protocol: D.1 editability is evaluated by applying six edits to matched ground-truth and predicted sets, including deletion, opacity, recoloring, rotation, and translation.The protocol specifies channel, HSV, angle, and displacement transformations for these edits.
  • D.2 System Configuration: D.2 uses a training-free agentic pipeline built from public tools and commercial VLM APIs, orchestrated as a LangGraph state machine.An NVIDIA A6000 hosts acceleration-dependent tools locally, while optimization libraries enable operation on commercial GPUs under 8GB of VRAM.

D.3 Additional Qualitative results · D.4 Prompts

ReDesign recovers clean boundaries and stacking order under heavy occlusion, preserving localized edits for attribute and spatial changes. The framework’s prompts define context-aware controller planning and sequential verifier checks for hallucinations, redundancy, and parent coverage.

  • D.3 Additional Qualitative results: Under heavy occlusion, ReDesign recovers clean element boundaries and correct stacking order, keeping edits localized to the target element.This qualitative benefit applies to both attribute changes and spatial transformations.
  • D.3 Additional Qualitative results: All tool configurations use default settings without additional fine-tuning.This implementation condition is specified in Table 5.
  • D.4 Prompts: The Controller prompt defines available tools, action types, and the expected JSON output schema for context-aware planning.Each node-specific prompt includes the root image, current layer image, prior decomposition history, and failed attempts with verification feedback.
  • D.4 Prompts: The Verifier uses a three-step sequential protocol to check hallucinations against the parent, detect cross-child redundancy, and assess parent coverage.Parent coverage confirms whether valid children collectively account for all visual content in the parent.
  • D.3 Additional Qualitative results: The additional qualitative comparisons evaluate decomposition accuracy against baseline approaches.These comparisons are presented in Figure 15.
  • D.3 Additional Qualitative results: The additional edit replay results compare ReDesign with baseline approaches.These results are presented in Figure 16.
Loading 2607.25565v1…