Detect Anything in Graphic Design: Element-Level Rewards for Autoregressive Detection

1 Introduction

• DAD predicts design elements in compositional order with amodal boxes that include regions occluded by elements above them.Lower-layer elements provide context for detecting higher-layer elements.Source: §1 · p. 2
• EleRPO extends GRPO from sequence-level to element-level optimization by decomposing sequence rewards into unbiased, history-conditioned element rewards.The decomposition requires no extra reward models or sampling passes.Source: §1 · pp. 2–3
• DAD-10M contains 10 million designs, combining 5 million designer-created works with 5 million additional designs, and the released 100,000-design subset includes original layer representations.Ground-truth amodal boxes and compositional order are extracted from original layered works; the additional designs broaden occlusion and style diversity.Source: §1 · pp. 2–3
• DAD beats proprietary and open-source baselines on DAD-10M, transfers gains to existing benchmarks, and reaches human-level accuracy for predicting occluded regions.EleRPO also consistently improves over GRPO and other fine-grained RL methods, generalizing across detection models and a wide range of benchmarks.Source: §1 · p. 3

Autoregressive Object Detection

• Existing detectors, including autoregressive and VLM-based models, represent objects as unordered sets and cannot recover the order and occluded regions needed for layer reconstruction.Human annotations for this information are difficult to scale and can be imprecise; DAD instead learns from exact ground truth extracted from original layered representations.Source: §2 · p. 3

Reward Design for RL Post-Training

• GRPO [14] assigns sequence-level rewards uniformly to tokens, while finer-grained policy-signal and re-rollout methods rely on proxy supervision not directly aligned with final reward.The cited approaches include policy-statistic methods [42] [43], reward-model methods [16] [45], and re-rollout methods [48].Source: §2 · p. 3
• EleRPO derives element-level rewards through martingale difference decomposition because each predicted detection element measurably contributes to the final reward.It needs neither additional reward models nor re-rollouts.Source: §2 · p. 3

3 Method

• DAD formulates detection as compositional deconstruction, introduces EleRPO for element-level optimization, and constructs DAD-10M with layer annotations.The dataset contains 10,416,655 designs.Source: §3 · p. 3

3.1 Task Formulation

• Elements are generated in compositional order, using previously predicted lower-layer elements as context for higher-layer predictions.The paper reports that this ordering performs better than alternative element orderings.Source: §3.1 · p. 4

3.2 Element Relative Policy Optimization

• GRPO assigns every token the same sequence-level advantage, so an incorrect element in a high-reward rollout can still receive positive reinforcement.EleRPO decomposes the sequence-level reward into element-level rewards and converts estimates into policy-update advantages.Source: §3.2 · p. 4

3.2.1 Element-Level Reward via Martingale Difference Decomposition

• Martingale-difference rewards capture each element's change to conditional expected F1 while satisfying summation, causality, and unbiasedness properties.The rewards sum to the total change in final F1 over its expectation before generation; each depends only on elements through k and has zero conditional expectation given earlier elements.Source: §3.2.1 · p. 4

3.2.2 Advantage Estimation from Sampled Rollouts

• Rollouts are clustered by F1(k−1) to estimate expected improvements without rerollouts, yielding a 0.81 Pearson correlation with a rerollout-based estimator.The same clusters estimate both expected improvements, enabling a paired comparison across rollouts.Source: §3.2.2 · p. 5
• Because reward magnitude scales as O(1/(k + n)), EleRPO normalizes by layer-wise standard deviation and combines element-level estimates with GRPO advantages.The element-level estimates may be noisy because they use a finite rollout group.Source: §3.2.2 · p. 5

3.3 Dataset Construction

• DAD-10M combines 5,367,586 layered designer works with augmented and annotated designs, then filters for granularity to produce 24,195 high-quality designs.The resulting set is split into 19,195 RL-training designs and 5,000 test designs; 100,000 designs with replaced elements are selected for public release.Source: §3.3 · pp. 5–6

3.4 Implementation Details

• DAD is built on Qwen3-VL-2B, receives supervised fine-tuning, then 5,000 EleRPO steps using 16 rollouts per image.Training uses separate IoU-weighted F1 rewards for text and visual elements and averages them as the final reward.Source: §3.4 · p. 6

4.1.1 Comparison with State-of-the-Art

• DAD achieves the best performance across all six datasets, improving overall detection, amodal detection, and ordering quality.It is more efficient than proprietary baselines and comparable to open-source ones.Source: Table 1 · p. 6, §4.1.1 · p. 7

4.1.2 Human Evaluation on Amodal Detection

• DAD stays within the inter-annotator IoU range across occlusion levels, indicating human-level amodal detection quality.GPT-5.5 and Gemini-3.1-Pro fall below that range across all occlusion levels.Source: Figure 3 · p. 7, §4.1.2 · p. 7

4.1.3 Application to Image-to-Layer Decomposition

• DAD consistently outperforms GPT-5.5 and Gemini-3.1-Pro on image-to-layer recovery across datasets, matching ground-truth-box results on image-level PSNR and SSIM.The predictions also support faithful recovery of layers, including occluded regions missed by GPT-5.5.Source: Table 2 · p. 7, §4.1.3 · p. 8

4.2.1 Comparison with Existing Methods

• EleRPO outperforms GRPO and all other baselines by directly decomposing sequence-level rewards into element-level rewards.Policy-signal methods do not improve over GRPO, while re-rollout methods perform worse on continuous, fine-grained detection rewards.Source: Table 3 · p. 8, §4.2.1 · p. 8

4.2.2 Ablation Studies

• Replacing the causal element-level reward with a non-causal IoU reward causes a significant performance drop.The result supports aligning element-level rewards with autoregressive detection.Source: Table 4 · p. 9, §4.2.2 · p. 9
• Using only the element-level advantage outperforms the sequence-level GRPO advantage, and combining both improves performance further.The sequence-level advantage provides stable supervision that complements the fine-grained element-level advantage.Source: §4.2.2 · p. 9
• Compositional order improves SFT-only performance, and its gain becomes large after EleRPO, indicating synergy between order and element-level rewards.The stated rationale is that this order preserves compositional structure so marginal contributions reflect design logic.Source: §4.2.2 · p. 9

4.2.3 Generalizability of EleRPO

• EleRPO consistently outperforms GRPO across all model-benchmark pairs, with the largest gains on graphic-layout benchmarks and gains also on natural-image benchmarks.The evaluated benchmarks span settings with and without explicit layer structure.Source: Table 5 · p. 9, §4.2.3 · p. 9

5 Conclusion

• DAD recovers complete element boundaries under occlusion, and experiments support its utility for image-to-layer decomposition.The conclusion identifies richer compositional structures beyond individual elements as a possible direction for downstream design editing, retrieval, and structured design analysis.Source: §5 · p. 10

A.1 Proof of the Martingale Difference Decomposition

• The martingale-difference construction follows by taking successive differences of conditional expected final scores given the input and generated elements.The proof defines the filtration from the input image and lower-layer elements, then identifies each conditional expectation as the corresponding martingale term.Source: §A.1 · pp. 15–16
• The element-level rewards are uniquely determined as martingale differences by the three desired properties.The proof checks completeness, causality, and unbiasedness: rewards depend only on the input and elements generated so far, and have zero conditional expectation given the previous elements.Source: §A.1 · pp. 15–16

A.2 Derivation of the Element-Level Reward Estimator and Justification for Cluster Reuse

• Using the same pre-element cluster for both residual terms avoids cluster-reassignment bias and makes the comparison paired across rollouts of similar quality.Estimating the terms over different clusters adds a membership-change term that is not part of the local effect of adding element e_k.Source: §A.2 · p. 17

A.3 Empirical Validation of the Prefix-Clustering Approximation

• 0.811 Pearson correlation with the re-rollout gold standard supports the prefix-clustering approximation across 36,018 rollouts.The 95% clustered-bootstrap confidence interval over 500 images is [0.787, 0.832].Source: §A.3 · p. 17

A.4 Magnitude Analysis of the Element-Level Reward

• The element-level reward magnitude is bounded by O(1/(k + n)), motivating layer-wise normalization.Here, n is the number of ground-truth elements, and the passage states that reward scale depends on layer k.Source: §A.4 · pp. 17–18
• The single-element F1 increment is bounded between zero and one because adding a prediction preserves the previous matching and can add at most one matched pair.The appendix uses this increment bound in its analysis of reward magnitude.Source: §A.4 · pp. 17–18

B DAD-10M Dataset Construction

• The dataset combines internal layered designs, element-replaced designs, Ideogram-generated designs, and internal flat designs.The accompanying figure presents the dataset distribution and examples.Source: B DAD-10M Dataset Construction · p. 18, Figure 4 · p. 19
• Flat designs without original layer annotations are annotated with amodal boxes and compositional order.The annotation instructions group text into logical blocks, separate text from its background shape, and request full object boxes including inferred occluded regions.Source: B DAD-10M Dataset Construction · p. 19
• The annotation output orders layers back-to-front and represents each layer as text or visual with normalized box coordinates.The format specifies coordinates normalized to [0,1].Source: B DAD-10M Dataset Construction · p. 19
• A two-step filtering process targets merged small elements and likely over-split annotations, yielding 24,195 high-quality designs.It selects hard examples with 5–10 small elements and low preliminary-detector recall, then filters overlap components with at least 10 boxes and fill ratio above 2.0; 19,195 designs are used for RL training and 5,000 for testing.Source: B DAD-10M Dataset Construction · pp. 19–20

C DAD Training Details

• Each element is encoded by a bounding box with coordinates normalized to [0, 1000] and a text-or-visual category label.The model outputs the elements as a semicolon-delimited sequence.Source: C DAD Training Details · p. 20
• Training uses supervised fine-tuning followed by EleRPO reinforcement learning.SFT runs for 650,000 steps, followed by 5,000 RL steps with 16 rollouts per image.Source: C DAD Training Details · p. 20

D.1 Dataset Statistics

• Evaluation uses the DAD-10M test split and five external graphic-design benchmarks, with annotations converted into text and visual elements.The DAD-10M held-out test split contains 5,000 designs.Source: §D.1 · p. 20
• The external evaluation set includes Design-Multi-Layer-Bench, Crello, PrismLayersPro, LICA, and ChartGalaxy.The benchmarks provide different dataset forms, including layered works, vector templates, transparent images, component-level annotations, and reconstructed infographic charts.Source: §D.1 · p. 20
• The external benchmark checks found no exact or near-duplicate designs against DAD-10M.The near-duplicate check manually inspected the 1,000 most similar cross-dataset image pairs for each benchmark.Source: §D.1 · p. 21

• The proprietary comparison includes five named models, and the open-source comparison includes seven models.The proprietary models are GPT-4o, GPT-5, GPT-5.4, GPT-5.5, and Gemini-3.1-Pro; the open-source set includes GLIP-L, Detic-B, Grounding DINO-B, Florence2-L, Qwen3-VL, InternVL3.5, and Rex-Omni.Source: §D.2 · p. 21
• Proprietary models are queried through their official APIs, while open-source vision-language models use vLLM for accelerated batch inference.Both receive the structured prompt described in Appendix B.Source: §D.2 · p. 21
• Traditional open-vocabulary detectors and Rex-Omni use official inference pipelines with text and visual element queries.GLIP-L predictions are filtered with a 0.3 confidence threshold.Source: §D.2 · p. 21
• All models are evaluated zero-shot, without task-specific fine-tuning or in-context examples.Source: §D.2 · p. 21

• Annotators draw an amodal box for an occluded element after seeing only its visible region, with model predictions withheld.They first receive a brief instruction session with examples; Figure 5 shows the annotation interface.Source: §D.3 · p. 21, Figure 5 · p. 21
• Participation is voluntary, and annotators may stop at any time or request deletion of their data.The consent form explains the study purpose, procedure, and estimated duration; analyzed data are anonymized and used only for research.Source: §D.3 · p. 21
• Annotators receive $15 for a study taking approximately 20 minutes.Source: §D.3 · p. 21

E Additional Results for DAD Evaluation

• DAD achieves the best performance across all six datasets on overall detection quality, amodal detection quality, and ordering accuracy.It is also more efficient than proprietary baselines and comparable to open-source ones; results are averaged over three runs except proprietary models, evaluated once.Source: E Additional Results for DAD Evaluation · p. 22
• In the qualitative comparison, DAD detects elements missed by GPT-5.5 and recovers their occluded regions.Source: E Additional Results for DAD Evaluation · p. 24, Figure 6 · p. 25

F.1 Hyperparameters of RL Baselines

• EleRPO is compared with GRPO and finer-grained methods spanning policy-signal-based and re-rollout-based approaches, using method-specific hyperparameters from the original papers.The compared methods include GRPO-λ, GTPO, PRIME, VinePPO, and SPO.Source: §F.1 · pp. 25–26

F.2 Element Order Experimental Setup

• The element-order experiment compares random, left-edge, and compositional order; the first two variants use an explicit order index because sequence position no longer preserves compositional order.All variants are trained from Qwen3-VL-2B for 31,250 SFT steps on eight H200 GPUs, with an effective batch size of 32 and approximately 1M samples.Source: §F.2 · p. 26

• The benchmark suite covers layered graphic layouts, layouts without explicit layer structure, and natural-image detection, with annotations converted into text and visual elements.Source: §F.3 · p. 26
• ChartGalaxy designs are reconstructed from structured chart specifications, with 5,000 images sampled for training and 2,000 for evaluation.Source: §F.3 · p. 26
• Crello uses its official train/test split, while Rico is split into 53K training and 13K testing layouts.Source: §F.3 · p. 26
• PosterLayout is randomly split into 8,974 training and 1,000 testing examples; DocLayNet uses 69,375 training pages and 6,489 validation pages.Source: §F.3 · p. 26
• COCO uses 118,287 train2017 images and 5,000 val2017 images; Objects365, LVIS, and BDD100K use official training splits and validation evaluation.Source: §F.3 · p. 26

G Additional Results for EleRPO Evaluation

• Silhouette-based prefix clustering performs best among the four tested strategies; separate clustering underperforms because it introduces cluster-reassignment bias.The no-clustering variant is also competitive, and adaptive clustering further refines the global-mean estimate.Source: Table 12 · p. 27, G Additional Results for EleRPO… · p. 27
• Removing per-step RMS normalization or replacing it with global RMS pooling degrades performance, supporting normalization for position-dependent reward variance.The reward variance scales differ across element positions.Source: Table 12 · p. 27, G Additional Results for EleRPO… · p. 27
• Replacing the per-step cluster-wise mean with a global mean across all element positions causes a performance drop, supporting conditioning on element position.The comparison is reported in Table 12(a).Source: Table 12 · p. 27, G Additional Results for EleRPO… · p. 27

NeurIPS Paper Checklist

• The checklist reports that the authors released their code and 100,000 designs from the dataset.Source: NeurIPS Paper Checklist · p. 30