PaperlayerView exact revision on arXiv

Source-linked AI summary

SENSE: Satellite-based ENergy Synthesis for Sustainable Environment

Kailai Sun, Mingyi He, Heye Huang, Can Rong, Alok Prakash, Baoshen Guo, Shenhao Wang, Jinhua Zhao

arXiv:2605.18101v1cs.CVcs.AI

TL;DR

Urban Building Energy Modeling remains limited by predictive approaches, missing functional generation, and scarce aligned energy data. SENSE addresses these gaps with a controllable generative framework and reduces energy prediction error by 3%-11% NMBE and 1%-9% CVRMSE.

  • Problem

    Existing UBEM methods are mainly predictive, satellite diffusion models lack urban functional layers, and aligned high-quality building energy data remain scarce.

  • Method

    SENSE uses a controllable diffusion framework conditioned on road networks and urban density metrics to jointly generate aligned satellite imagery, energy consumption, and height maps.

  • Results

    SENSE reduced energy prediction error by 3%-11% NMBE and 1%-9% CVRMSE across the evaluated settings.

  • Takeaways & Limitations

    The framework and MUSE dataset provide a foundation for AI-driven scientific discovery across urban, energy, and building sciences.

  • Takeaways & Limitations

    The study models annual total energy consumption without temporal dynamics and evaluates generalisation across only four metropolitan areas.

Abstract

from arXiv · show

Urban Building Energy Modeling plays a critical role in achieving the United Nations' Sustainable Development Goals 7 and 11. Although existing studies based on satellite imagery and deep learning have achieved remarkable progress, many challenges exist: most existing studies are inherently predictive, failing to reflect the generative nature of urban planning; although generative AI and diffusion models have seen explosive growth in satellite imagery, they lack the urban functional generation (e.g., energy layer); third, aligned high-quality high-resolution building energy data with satellite imagery is limited and scarce. Here we propose SENSE (Satellite-based ENergy Synthesis for Sustainable Environment), a unified generative UBEM framework that jointly synthesizes realistic urban satellite imagery and aligned high-quality building energy consumption and height maps. By conditioning on road networks and urban density metrics, SENSE, based on a controllable diffusion model, leverages the knowledge learned by large vision models to generate urban building energy consumption and height information (annotations) in the latent space. Experiments across four cities (New York City, Boston, Lyon, Busan) demonstrate that SENSE achieves high visual fidelity and strong physical consistency, satisfying the ASHRAE standard metric. Experiments demonstrate that SENSE can generate enough annotated synthetic data using less than 20% labeled energy data, boosting downstream prediction performance by 10% IoU. Compared to SOTA urban energy prediction methods, SENSE significantly reduced prediction error (reduced 3%-11% NMBE and 1%-9% CVRMSE). This study offers an energy-efficiency urban planning and physical generation solution for urban science, energy science and building science. The dataset and code: https://huggingface.co/datasets/skl24/MUSE and https://github.com/kailaisun/GenAI4Urban-Energy/.

1 Introduction

SENSE addresses the predictive and data limitations of urban building energy modeling by using controllable diffusion to jointly generate satellite imagery and aligned building energy and height maps. Conditioned on road networks and urban density metrics, it generates energy information in latent space and demonstrates reliable synthesis.

  • Motivation: Urban building energy modeling is important because cities consume nearly 70% of world energy, while buildings account for 32% of global energy demand and 34% of CO2 emissions.Urban residents are projected to increase from 55% of the global population to 68% by 2050.
  • Challenges: Existing UBEM studies are predominantly predictive, making it difficult to generate new energy-efficient urban morphologies from urban plans.Diffusion-based satellite-image generation also lacks urban functional outputs such as building energy layers, while aligned high-quality energy data remain scarce.
  • Framework: SENSE jointly generates realistic satellite imagery and aligned building energy consumption and height maps using controllable diffusion conditioned on road networks and urban density metrics.The framework integrates building energy and height decoders to produce corresponding urban modalities.
  • Method: SENSE leverages large vision models’ learned visual representations to generate building energy consumption and height information in latent space instead of training a joint generator from scratch.This latent-space strategy is integrated with the controllable diffusion model and specialized decoders.
  • Results: 89.25% and 85.75% accuracies are achieved by the energy and height decoders, while generated building energy reaches NMBE of 3.05% and CVRMSE of 14.62%.These results support reliable generation of urban building energy consumption and height from the latent space.

2 Multi-city Urban Satellite-Energy Dataset

This section introduces a global multi-city urban satellite-energy dataset spanning New York City, Boston, Lyon, and Busan. It aligns satellite imagery, geospatial constraints, building morphology, and municipal energy records into quality-controlled 2 km × 2 km samples.

  • Dataset scope and structure: The dataset spans four cities across North America, Western Europe, and East Asia, with each sample defined as a 2 km × 2 km tile.Samples pair urban satellite imagery with aligned building-related annotations and prompts.
  • Data construction: Satellite imagery is cropped and mosaicked into 512×512 pixel tiles, while building attributes come from GHSL P2023A and urban density metrics are computed per 2 km×2 km cell.The reported density metrics include Building Volume Density (BVD) and Building Coverage Ratio (BCR).
  • Geospatial constraints: OSM-derived water bodies, railway infrastructure, and major roads provide rasterized geospatial constraints, while minor streets are excluded to avoid overconstraining local generation details.Extracted road networks range from motorways to tertiary roads.
  • Quality control: 2,788 tiles comprise the final high-quality dataset after spatial alignment, missing-label removal, and expert review of energy annotation maps.Three urban domain specialists rejected tiles containing large contiguous blocks of missing or null energy values.

3 Method

SENSE is a unified multimodal generative framework that jointly generates controllable urban satellite imagery, building energy consumption maps, and building height maps from textual and spatial urban constraints. Its two-stage design combines controllable latent diffusion with lightweight decoders that extract height and energy layers from visual latent features.

  • Unified framework: The framework models satellite imagery, energy maps, and height maps jointly, conditioning generation on textual descriptions, urban density metrics, and road networks.It decouples generation into visual latent-feature synthesis followed by building height and energy decoding.
  • Controllable imagery: ControlNet injects geospatial environmental constraints into latent diffusion, ensuring generated urban imagery respects topological boundaries and avoids physically invalid building locations.Constraints can include road networks and water features, while zero-initialized convolutions preserve pretrained visual knowledge at training onset.
  • Latent-space decoders: Lightweight plug-and-play decoders reuse visual-generation weights to extract building height and energy features from the diffusion model’s latent space.The approach exploits correlations between semantic visual features and urban physical attributes rather than training separate generative models from scratch.
  • Height generation: The H-Decoder formulates building-height recovery as generative segmentation with SegFormer and MiT encoders, using five discrete categories from background through four increasing height tiers.Its output is a five-channel probability map, trained with combined Cross-Entropy and Dice losses.
  • Energy generation: The energy decoder discretizes consumption into four classes—background plus Low, Medium, and High—and combines class-weighted cross-entropy with Dice loss to address rare high-consumption buildings.Class weights are inversely proportional to class frequency, penalizing errors on minority classes.

4 Experiments

SENSE generates visually realistic, spatially aligned urban imagery with building height and energy maps, while meeting ASHRAE calibration criteria. Under limited real data, mixed synthetic-real training improves downstream energy and height prediction.

  • Image generation: SENSE achieves PSNR 14.5, LPIPS 0.430, SSIM 0.234, and FSIM 0.669 across fidelity, perceptual similarity, and diversity assessments.These metrics indicate that the generated urban satellite imagery simulates real-image statistical properties while retaining diversity.
  • Physical-map generation: The H-Decoder reaches overall accuracy 85.75% and mIoU 0.6005, while the E-Decoder reaches overall accuracy 89.25% and mIoU 0.7093.Class 0 achieves IoU 0.8443 for non-building regions, and Class 4 achieves IoU 0.6664 with Recall 0.8099 for tall buildings.
  • Cross-city generation: Across NYC, Boston, Lyon, and Busan, generated imagery captures city-specific morphology, including Manhattan grids and irregular European and mountainous Asian street patterns.The generated satellite imagery, height maps, and energy maps are reported as spatially aligned.
  • Physical calibration: The generated energy maps achieve NMBE 3.05% and CVRMSE 14.62%, satisfying the stated ASHRAE calibration tolerance of ±10% and standard range of 30%.The framework infers energy values from generated-image latent features without explicit weather, material, building-type, or HVAC information.
  • Few-shot energy prediction: About 10% gains from mixed data training improve energy prediction under limited data, while IoU improvements often exceed 10% across Low, Medium, and High energy categories.The experiments train with 20% real data and evaluate on a fixed real-world test set; Low-energy precision improves 15.75% for Streltsov et al..
  • Few-shot height prediction: Mixed data training yields 4%- 7% gains for building height prediction under limited data, and consistently outperforms the Real Only baseline when real data is scarce (< 40%).Purely synthetic training causes a slight performance drop, whereas increased training data improves height prediction across training strategies.

5 Discussion

SENSE positions building energy consumption as an active urban-planning design parameter, enabling scenario simulation and comparative analysis in data-scarce regions. Its synthetic data generation improves downstream prediction while leaving temporal resolution and geographic diversity as key future challenges.

  • Contributions: SENSE enables planners to treat building energy consumption as an active design parameter by varying road networks, coverage ratios, and volume densities.This supports simulation of alternative urban development scenarios and integrates energy-map generation into urban science and planning.
  • Data efficiency: Less than 20% manually labelled energy data enables virtually infinite annotated-data generation, while mixing synthetic and real data improves existing state-of-the-art prediction models.The reported real-world mixing example uses 20% real data.
  • Applications: The framework supports government agencies, urban and energy scientists, and developers in envisioning scenarios and conducting comparative analyses, particularly in data-scarce developing regions.Synthetic data augment real training datasets for downstream building energy consumption prediction.
  • Applications: Open-access release aims to democratize energy-efficient urban planning by empowering policymakers and communities to customize planning aligned with SDGs 7 and 11.The framework is presented as supporting sustainable development through improved prediction performance.
  • Limitations: The study models annual total energy consumption without temporal dynamics, motivating future synthesis of hourly energy profiles despite the difficulty of obtaining fine-grained data.Future work should also expand geographic diversity beyond four big metropolitan areas.
  • Data governance: MUSE uses publicly available OpenStreetMap, Mapbox, and municipal energy disclosure records without private or non-consensual household data.Synthetic spatially aligned energy annotations help mitigate confidentiality constraints on sharing fine-grained urban energy information.

6 Conclusion

SENSE is a unified multimodal generative framework that synthesizes spatially aligned satellite imagery, building-height maps, and energy-consumption maps. The study also establishes the global multi-city MUSE dataset and demonstrates reliable generation of building height and energy consumption in existing GenAI latent spaces.

  • Framework: SENSE jointly synthesizes spatially aligned satellite imagery, building height, and energy consumption maps.The framework is presented as unified and multimodal.
  • Dataset: The study establishes MUSE as a global Multi-city Urban Satellite-Energy Dataset.MUSE provides the dataset foundation for the proposed framework.
  • Generation: Experiments show that existing GenAI models can reliably generate urban building energy consumption and height from latent space.This addresses the generation of physical urban information within GenAI models.

A Appendix · A.1 Implementation Details · A.1.1 Building Height Data.

The appendix formulates building-height and energy co-generation as categorical segmentation tasks, using quantile-based discretization to improve statistical validity and address class imbalance. Building heights use five classes, while energy consumption uses four log-transformed, tertile-based classes.

  • A.1.1 Building Height Data.: Building-height co-generation is formulated as segmentation rather than continuous regression because satellite-derived estimates and urban data are inherently noisy and highly variable.Continuous ground-truth heights are discretized into categorical labels.
  • A.1.1 Building Height Data.: Quantile-based discretization is applied to building heights to support statistical validity and alleviate class imbalance during training.The method converts continuous height values into categories before model training.
  • A.1.1 Building Height Data.: Energy consumption is divided into 4 classes, with Class 0 representing non-energy background.The remaining building pixels are assigned to three energy-level categories.
  • A.1.1 Building Height Data.: The three building-energy classes are defined as Class 1 (Low Energy), Class 2 (Medium Energy), and Class 3 (High Energy).These categories apply to building pixels after excluding the non-energy background class.
  • A.1.1 Building Height Data.: Energy classes are separated using the 33rd and 66th percentiles of the logarithmic energy-consumption distribution.The remaining building pixels are divided into three intervals based on tertiles.
  • A.1.1 Building Height Data.: Log-transformation is applied before energy discretization to handle the long-tail distribution typical of urban energy-consumption data.The transformation precedes the tertile-based class assignment.

A.1.2 Building Energy Data.

The framework fine-tunes ControlNet on 512 × 512 satellite imagery using full FP32 precision and distributed training with batch size 16.

  • ControlNet is fine-tuned with high-resolution satellite imagery at 512 × 512 pixels.
  • Full FP32 precision is used throughout training to maintain numerical stability and model performance.
  • Training uses batch size 16 with Distributed Data Parallel (DDP) for efficient, synchronized gradient updates across computation nodes.

A.1.3 Hyperparameters. · A.2 Segmentation performance for H-Decoder and E-decoder · A.3 Data filtering

The appendix reports class-wise segmentation behavior for H-Decoder and E-decoder, alongside MUSE city-level statistics and filtering of samples lacking building energy labels.

  • A.2 Segmentation performance for H-Decoder and E-decoder: Table 4 reports class-wise segmentation performance for H-Decoder and E-decoder.
  • A.2 Segmentation performance for H-Decoder and E-decoder: The models effectively capture extreme-value classes in the segmentation task.
  • A.2 Segmentation performance for H-Decoder and E-decoder: Background is identified as one of the extreme-value classes captured effectively by the models.
  • A.2 Segmentation performance for H-Decoder and E-decoder: High-consumption/Tall buildings are also captured effectively as an extreme-value class.
  • A.3 Data filtering: Table 5 summarizes city-level statistics data in MUSE.
  • A.3 Data filtering: Data filtering removed samples that clearly lacked building energy labels.
Loading 2605.18101v1…