PaperlayerView exact revision on arXiv

Source-linked AI summary

BEVControl: Accurately Controlling Street-view Elements with Multi-perspective Consistency via BEV Sketch Layout

Kairui Yang, Enhui Ma, Jibin Peng, Qing Guo, Di Lin, Kaicheng Yu

arXiv:2308.01661v4cs.CV

TL;DR

BEV perception relies on street-view images paired with BEV layouts, while collecting and annotating multi-view scenes is costly and existing layout-based generation has limited editability and visual-element control. BEVControl uses editable BEV sketches to control foreground and background content in generated street views. The authors report richer training data and state-of-the-art object detection results on nuScenes.

  • Problem

    Collecting and annotating multi-view street scenes is costly, and existing BEV-layout generation methods offer limited editability and control of diverse visual elements.

    Source: §1 · p. 2

  • Method

    BEVControl uses editable BEV sketches to control foreground and background content in generated street-view images.

    Source: §1 · p. 2

  • Results

    BEVControl yields richer data for BEV perception and achieves state-of-the-art object detection results on nuScenes.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    The authors present BEV sketches as a cost-effective way to produce large amounts of street-view images.

    Source: §1 · p. 2

    The method currently controls background and foreground elements; future work will investigate lighting, weather, and more general scenes.

    Source: §6 · p. 9

Abstract

from arXiv · show

Using synthesized images to boost the performance of perception models is a long-standing research challenge in computer vision. It becomes more eminent in visual-centric autonomous driving systems with multi-view cameras as some long-tail scenarios can never be collected. Guided by the BEV segmentation layouts, the existing generative networks seem to synthesize photo-realistic street-view images when evaluated solely on scene-level metrics. However, once zoom-in, they usually fail to produce accurate foreground and background details such as heading. To this end, we propose a two-stage generative method, dubbed BEVControl, that can generate accurate foreground and background contents. In contrast to segmentation-like input, it also supports sketch style input, which is more flexible for humans to edit. In addition, we propose a comprehensive multi-level evaluation protocol to fairly compare the quality of the generated scene, foreground object, and background geometry. Our extensive experiments show that our BEVControl surpasses the state-of-the-art method, BEVGen, by a significant margin, from 5.89 to 26.80 on foreground segmentation mIoU. In addition, we show that using images generated by BEVControl to train the downstream perception model, it achieves on average 1.29 improvement in NDS score.

1. Introduction

BEVControl addresses the difficulty of editing detailed BEV segmentation layouts and the need to control diverse visual elements consistently across street views. It uses editable BEV sketches to control foreground and background content, and the paper reports state-of-the-art object detection results on nuScenes.

1. Introduction

  • Existing BEV-layout generation methods offer limited editability and do not sufficiently control diverse visual elements such as road layout, lane lines, and vehicle position or heading.These elements are needed to enhance BEV perception models’ generalization power.Source: §1 · p. 2
  • BEVControl uses editable background and foreground sketches to control their appearances and geometry separately, while a coordinator preserves visual consistency across views.The sketches are easier to edit than pixel-wise segmentation annotations.Source: §1 · p. 2
  • The controller processes sketch hints into street-view feature maps, and the coordinator uses cross-view-cross-element attention to enhance consistency among visual elements.The controller’s hints represent the geometric shapes of background and foreground elements, mapped from the same BEV sketch layout.Source: §1 · p. 2
  • BEV sketches are presented as a cost-effective way to produce large amounts of street-view images.The paper motivates image synthesis as an alternative to collecting and annotating multi-view street scenes, which requires costly annotation and consistency checks.Source: §1 · p. 2
  • BEVControl achieves state-of-the-art object detection results on nuScenes.The authors report that the method yields richer data for BEV perception.Source: §1 · p. 2

2. Related Work

Related work covers conditional image generation and methods for maintaining consistency across multiple views. BEVControl targets precise control of foreground and background geometry while coordinating visual elements across street-view perspectives.

2. Related Work

  • Conditional image generation uses visual hints such as layouts, semantic segmentation maps, sketches, and depth images to control image contents.The related work notes that text-based information roughly specifies image contents rather than providing fine-grained control.Source: §2 · p. 3
  • Generating multiple street-view perspectives requires controlling foreground and background geometry as well as semantic categories.The paper motivates using appropriate modalities for these contents to enhance generative control.Source: §2 · p. 3
  • Prior multi-view approaches use cross-view attention for panoramas and street scenes, while video methods use temporal cross-frame attention to preserve visual consistency.BEVGen generates multi-view street images from BEV segmentation layouts using an autoregressive transformer with cross-view attention.Source: §2 · pp. 3–4
  • Methods that coordinate global appearances can be less effective when precise control of individual contents, such as car orientation, is required.BEVControl proposes cross-view cross-object attention to augment consistency across generated multi-view images.Source: §2 · p. 4

3. BEVControl

BEVControl projects editable BEV sketches into camera-specific foreground and background conditions, then uses a controller and coordinator to generate geometry- and appearance-consistent street-view images. Its diffusion training adapts newly added UNet layers while freezing the original layers.

3. BEVControl

  • The editable sketch S and text prompt condition noisy street-view images; S encodes road lines and foreground-object centers, labels, and headings within a 160 × 160-meter area around the ego car.At inference, the noisy images are sampled from N(0, I).Source: §3 · p. 4
  • BEVControl projects BEV sketches into camera conditions, produces geometry-consistent latent features with a controller, and uses a coordinator to improve cross-view appearance consistency.Text prompts control environmental factors such as weather and lighting; the repeated UNet outputs generated street-view images.Source: §3 · p. 4, Figure 3 · p. 4

3.1. Controller

  • Camera-specific foreground boxes, labels, and headings are paired with spatially aligned road-trend conditions, embedded, and injected into the UNet through trainable self-attention layers.The controller uses the spatial hints to produce latent features geometrically consistent with the foreground and background conditions.Source: §3.1 · pp. 4–5

3.2. Coordinator

  • The coordinator uses cross-view-cross-element attention to exchange global latent and local element context between adjacent views, reducing computation while improving appearance consistency.The exchanged context can transmit environmental conditions and weather as well as object appearance and identity.Source: §3.2 · p. 5

3.3. Training Objective

  • Training follows the latent diffusion objective by predicting added noise, while freezing the original diffusion-model layers and training only newly introduced UNet layers.The authors state this reduces memory consumption and avoids knowledge forgetting and model collapsing.Source: §3.3 · p. 5

4. Evaluation Metrics for Content Controlling

Scene-level metrics alone can miss differences in fine-grained control, so the paper evaluates realism, multi-view consistency, and foreground and background control separately. It also combines these measures into an overall controlling score.

4. Evaluation Metrics for Content Controlling

  • Similar FID scores can accompany vastly different fine-grained foreground and background control, motivating metrics beyond scene-level evaluation.Figure 4 compares detail evaluation metrics.Source: §4 · p. 5, Figure 4 · p. 6
  • FID measures realism and diversity by comparing Inception-V3 features from generated and real images at the same perspective; lower FID indicates better scores.The reported FID averages across perspectives.Source: §4 · pp. 5–6
  • CLIP scores measure visual consistency using latent features from overlaps between adjacent generated perspectives, and higher scores indicate satisfactory consistency.The score is computed across generated street-view images from multiple perspectives.Source: §4 · p. 6
  • Foreground control uses BEVFormer detection metrics and CVT foreground segmentation, while background control uses CVT road segmentation on the BEV layouts.Higher mAP, NDS, foreground mIoU, and background mIoU, and lower orientation error indicate better control.Source: §4 · p. 6
  • The overall controlling score (OCS) combines measures of control from separate aspects; its component and reference scores are computed from generated and authentic images, respectively.A higher OCS indicates stronger overall controlling power.Source: §4 · p. 6

5. Experimental Results

On nuScenes, BEVControl achieves a higher overall controlling score than the compared methods, with evaluations spanning image quality, element control, ablations, and detection-data augmentation. The experiments also examine the dataset setting and the effects of controller and coordinator design choices.

5.2. Visual Element Control

  • BEVControl achieves a higher overall controlling score (OCS) than the compared methods in the nuScenes validation comparison.Table 1 compares methods generating six street-view perspectives per BEV layout and reports image quality and foreground/background control.Source: Table 1 · p. 7, §5.2 · p. 7

5.1. Dataset

  • The nuScenes evaluation uses 600 validation frames, each associated with six street-view RGB images, and distinguishes ten detection foreground categories from road background.nuScenes contains 1,000 scenes split into 700/150/150 train/validation/test examples; segmentation merges vehicle classes into a vehicle category alongside road.Source: §5.1 · pp. 6–7

5.2. Visual Element Control

  • BEVControl reports 24.85 FID, while its generated foregrounds and roads show stronger control than other generative models in the reported evaluations.The paper also demonstrates generation for user-drawn vehicle orientations and user-edited road-traffic sketches.Source: §5.2 · p. 7

5.3. Ablation Study on Controller

  • Using only a foreground or background hint degrades realism, diversity, and the corresponding control score; the complete controller outperforms joint embedding of both hints.The complete controller uses separate network streams to focus control on foreground and background objects.Source: §5.3 · p. 8

5.4. Ablation Study on Coordinator

  • Removing the coordinator degrades visual consistency, while adding cross-view attention improves the CLIP score over the controller-only alternative.The comparison also evaluates a coordinator with cross-view attention but without cross-element attention.Source: §5.4 · p. 8, Table 3 · p. 8

5.5. Data Augmentation for Object Detection

  • BEVFormer trained with BEVControl-generated images yields better detection results than the counterpart trained without data augmentation.The generated street-view images are paired with their BEV layouts to augment nuScenes training data.Source: §5.5 · p. 8, Table 4 · p. 8

6. Conclusion

BEVControl uses BEV sketches to improve control of street-view background and foreground elements while maintaining visual consistency across perspectives. The paper presents richer data for BEV perception as a benefit and identifies broader visual-element control as future work.

6. Conclusion

  • Figures 5 and 6 show generated foreground objects matching bounding-box sketches, especially in orientation, and street views matching road sketches.Source: Figure 5 · p. 9, Figure 6 · p. 10
  • BEVControl’s stronger control allows it to yield richer data for BEV perception.The method synthesizes background and foreground elements from BEV sketches while maintaining cross-perspective visual consistency.Source: §6 · p. 9
  • Future work will extend control beyond background and foreground elements to lighting and weather, and explore creating more general scenes.Source: §6 · p. 9

Related papers

Loading 2308.01661v4…