Source-linked AI summary
Structured-Prior-Guided Diffusion Inpainting with Physical Consistency for Traffic Sign Augmentation
Luo Li, Chongchong Huang, Jun Jia, Qiang Gao, Xinlong Liu, Gui Yang, Liang Cao
TL;DR
Rare traffic-sign categories are difficult to augment because general-purpose inpainting can distort their digits, geometry, perspective, and colours, while conventional methods cannot create new semantic categories. This paper introduces structured-prior-guided diffusion inpainting that separately conditions semantics, appearance, and geometry and adds physical consistency losses. In cross-source zero-shot evaluation, it ranks first on all nine metrics, reaches 91.1% OCR exact match, and improves rare-class pooled AP50 by 1.23×–7.40×.
Problem
Rare traffic-sign categories are difficult to augment because their real-world frequency leaves few samples, while general-purpose inpainting can distort sign digits, geometry, perspective, and colours.
Method
The framework injects semantic JSON text, a measured-colour front-view vector template, and an affine-aligned vector template through text cross-attention, IP-Adapter, and ControlNet, with physical consistency losses.
Results
The method ranks first on all nine metrics in cross-source zero-shot TT100K evaluation, reaches 91.1% OCR exact match, and raises rare-class pooled AP50 by 1.23×–7.40%.
Takeaways & Limitations
Structured prior modelling links generation-quality evaluation to downstream rare-class detection gains within the reported traffic-sign setting.
Takeaways & Limitations
Template coverage excludes noncircular and non-triangular signs, geometric fitting can fail under large-angle occlusion, and downstream conclusions are limited to speed-limit classes and some warning signs.
Abstract
from arXiv · showhide
Traffic sign detection faces a long-tailed data distribution. Many rare signs matter as much as common ones from a regulatory standpoint, yet they have very few samples. Generative data augmentation is one way out. General-purpose inpainting models, however, distort digits, deform geometry and perspective, and shift colours when applied directly to sign regions. We trace this to a single gap: the conditioning signal is too abstract for the physical composition of a sign. We propose a structured-prior-guided diffusion inpainting framework with physical consistency. It injects the semantic, appearance and geometric priors of a sign through three orthogonal pathways: a JSON-formatted text prompt, a front-view vector template rendered with measured dominant colours (via IP-Adapter), and an affine-aligned vector template (via ControlNet). Two physical consistency losses constrain colour with a CIELAB chromaticity $L_1$ term and edge structure with a Sobel gradient term. We train by self-supervised reconstruction on a large set of images collected in-house at AMAP, then evaluate zero-shot on the public TT100K-2021 dataset, a different source. Our method uses a Stable Diffusion 1.5 backbone of about 1.4B parameters. It beats seven representative competitors on every metric of reconstruction fidelity, physical consistency and semantic controllability. Its OCR exact-match rate reaches 91.1\%, against 44.2\% for the 12B industrial model FLUX.1 Fill [dev], and it needs only $1/14$ of that model's inference time. Leave-one-out ablations confirm that each of the three prior pathways and both loss terms contribute on their own. In downstream detection, the synthetic data raises the group-pooled AP50 of rare classes by $1.23\times$ to $7.40\times$ over a real-data-only baseline. Code and pre-trained models are available at https://github.com/52hz-whale/TrafficSignInpaint.
1 Introduction
Traffic-sign detection is increasingly limited by rare tail categories, while conventional augmentation cannot create genuinely new sign semantics. The paper proposes matching semantic, appearance, and geometric priors to separate conditioning pathways so diffusion inpainting can repaint signs with controlled content.
- Motivation: Rare traffic-sign categories remain difficult because their scarcity reflects real-world frequency, and they are often small objects.Additional collection mileage provides little benefit for these tail categories.
- Motivation: Conventional transforms preserve semantic content, while Copy-Paste reuses existing instances and reweighting methods add no information.These methods therefore cannot create new categories.
- Problem: General-purpose diffusion inpainting can distort sign text, geometry and perspective, and colour inconsistency can create domain shift for downstream detection.Wrong digits can become label noise, while natural language does not specify the continuous geometric quantities required by perspective projection.
- Problem: The paper attributes these failures to conditioning information being too abstract for a highly structured, standardised traffic sign.Natural language leaves known semantic, colour and geometric quantities for the model to guess.
- Proposed framework: The framework assigns semantics to JSON text, appearance to a colour-infused front-view vector template via IP-Adapter, and geometry to an affine-aligned template via ControlNet.The three pathways are orthogonal and are intended to let the model use rather than infer the sign’s known priors.
- Contributions: The automated framework combines separate prior pathways, physical consistency losses, and self-supervised reconstruction, while reporting first place on nine TT100K metrics and 1.23×–7.40× rare-class AP50 gains.The reported downstream gain reaches 7.40× when only 10% of real data is available.
2 Related Work
Prior inpainting and controllable-generation methods provide texture continuation, text or spatial control, but do not explicitly combine traffic-sign semantics, appearance, and geometry with physical consistency. This paper positions its design around that combination and its connection to visual text generation.
- Image inpainting: Early GAN, Transformer, and diffusion inpainting methods primarily erase regions or continue surrounding texture rather than reconstructing target sign structure.Their edge scores reflect texture continuation instead of correct target-structure reconstruction.
- Conditional injection and reference-driven generation: ControlNet and T2I-Adapter provide spatial conditioning, whereas IP-Adapter injects non-spatial appearance through decoupled cross-attention.This distinction motivates using ControlNet for geometry and IP-Adapter for global appearance attributes.
- Visual text generation: Visual text-generation methods use character-level masks, glyph images, or combined glyph-position-mask conditions to improve rendered text.The paper adopts character accuracy and exact-match rate with PaddleOCR as its evaluator.
- Positioning: The paper claims prior work has not explicitly injected semantic, appearance, and geometric traffic-sign priors while also adding loss-level physical consistency constraints.It also identifies a missing evidence chain from generation quality to downstream detection.
3 Method
The method constructs three structured sign conditions from annotated road images and injects them into SD1.5 through text, IP-Adapter, and ControlNet pathways. Automated fitting, colour clustering, self-supervised reconstruction, and physical losses support controlled repainting while preserving scene context.
- Task definition: The task generates a masked sign region conditioned on a target structured description and three orthogonal conditions, while leaving pixels outside the mask unchanged.The output is intended to follow the target sign distribution inside the mask.
- Prior injection: The semantic JSON prompt, colour-infused front-view template, and pose-aligned template enter the denoising network through text cross-attention, IP-Adapter, and ControlNet respectively.Each prior uses its own conditioning route.
- Training and inference: Self-supervised reconstruction extracts conditions from the original sign and reconstructs that same sign, so manually paired data is unnecessary.Any annotated road image containing a sign can become a training sample.
- Training and inference: Changing type or info edits category and digits, changing clustered colours edits hue and lightness, and changing geometric parameters edits rotation and scale.Because the pathways are decoupled, these edits can be applied separately or together.
- Overall framework: The framework builds masks and conditions, drives SD1.5 Inpainting with ControlNet, and pastes the repainted crop back into the original-resolution image.Training and inference share the same pathway, with training drawing conditions from the original sign region.
- Data construction: The fully automated data pipeline performs cropping, SAM 2 segmentation, analytic geometric fitting, dual-mask construction, and Lab dominant-colour clustering.Analytic fitting uses ellipses for circular signs and line-triplet enumeration for triangular signs; samples with failed fits are discarded.
- Appearance prior: The appearance prior replaces standard SVG colours with measured dominant colours before rasterisation, so the reference follows scene illumination rather than a fixed category template.The reference image is encoded by CLIP and injected through IP-Adapter’s decoupled cross-attention.
- Geometric prior: The geometric prior affine-aligns the colour template to image pose and feeds the template image directly to ControlNet rather than using only an edge map.The template jointly carries edge positions, region-filling relations, and colour scheme.
4 Experiments
Experiments show that structured priors and physical losses produce strong, controllable sign reconstructions, while ablations expose trade-offs among text accuracy, structure, colour fidelity, and perceptual realism.
- Generation quality: 91.1% OCR-EM versus 44.2% for FLUX.1 Fill, with the method ranking first on all nine metrics.The comparison covers reconstruction fidelity, physical consistency, and semantic controllability.
- Generation quality: The method outperforms non-diffusion and general text-conditioned baselines by preserving sign semantics, geometry, and scene-consistent colour.Non-diffusion methods erase sign semantics, while text-conditioned diffusion models often distort geometry, glyphs, and illumination matching.
- Generation quality: Correct OCR samples average 0.988 confidence versus 0.328 for incorrect samples, providing a reference-free screening signal.The separation is reported as an evaluation aid for generated samples.
- Controllability: Independent condition edits change speed values, categories, colours, lightness, rotation, and scale while keeping the scene and mask fixed.The experiment tests pathway editability one component at a time.
- Ablation study: Removing the IP-Adapter appearance prior drops OCR-EM from 91.1% to 66.1% and raises FID from 4.22 to 16.94.The vector template supplies digit layout, glyph sizing, relative positions, white space, and colour scheme.
- Ablation study: Measured-colour injection improves realism, while ControlNet improves structure at a slight OCR/FID cost.Switching to original template colours raises FID from 4.22 to 9.65; removing ControlNet drops EdgeIoU by 17% and EdgeF1 by 10%.
- Ablation study: The physical losses improve text accuracy, with the Sobel term additionally yielding EdgeIoU +7.4%, EdgeF1 +5.1%, SSIM +0.017, and ΔE −0.73.The combined losses add OCR-EM +2.2 points relative to the no-loss variant, while incurring some structural-metric costs.
- Ablation study: PSNR alone rewards conservative low-contrast outputs: V1 and V2 score 16.44 and 16.47, but have much worse FID and OCR-EM than Full.The paper therefore evaluates generation jointly with perceptual, semantic, and physical-consistency metrics.
5 Discussion and Conclusion
The framework achieves strong zero-shot generation and downstream detection gains, while its discussion identifies failure modes, data biases, and scope limits that constrain deployment.
- Failure cases and biases: Pseudo-text can arise because real training signs sometimes contain noncompliant annotation text that the model learns as a valid sign-face pattern.The resulting samples may add extra text when generating compliant signs.
- Failure cases and biases: Geometry–semantics conflicts can flatten or drop digits when the target layout cannot fit an extremely flattened elliptical guidance mask.This exposes a trade-off between ControlNet geometry and text generation.
- Scope limits: Synthetic-data diversity inherits scene and pose factors from source images, while content diversity comes from changing target categories and digits.The number and distribution of source instances therefore constrain the generated set.
- Scope limits: 42% of generated objects are small, versus 24–34% in the natural distribution, creating an intentional scale-distribution mismatch with the test set.The shift was introduced to address near-zero AP for small-object rare classes under the real-data-only baseline.
- Experimental evidence: 91.1% OCR exact-match was achieved, 46.9 percentage points above FLUX.1 Fill, at 1/14 of its inference time.The evaluation covered the full 5,182-sample TT100K set and used a cross-source zero-shot protocol.
- Experimental evidence: Leave-one-out ablations attribute independent contributions to the three prior pathways and two physical consistency losses.The ablations highlight appearance priors for digit legibility, colour injection for distributional realism, and Sobel loss for structural gains without distributional cost.
- Scope limits: The method is limited to circular and triangular templates, can fail under large-angle occlusion, and edits single targets serially.Downstream conclusions cover speed-limit classes and some warning signs; the marginal advantage over Copy-Paste tends to vanish at the 10% data fraction.