Source-linked AI summary
Vision-centric generative AI models: A software-hardware perspective
Eleni Tselepi, Cristian Sestito, Shady Agwa, Themis Prodromakis
TL;DR
Vision generative models increasingly serve hardware-constrained edge applications, but quality-driven progress has raised their parameter, memory, and inference costs. The Perspective quantifies model and accelerator trade-offs and maps four model families to seven domains. It finds that dominant diffusion and transformer models fit at most two domains, while GANs appear viable across all seven, motivating software-hardware co-design.
Problem
Quality-driven model scaling has made leading vision generative models difficult to deploy on platforms constrained by size, weight, power, and cost.
Method
The Perspective traces model and hardware evolution, quantifies parameter and energy costs, and maps four model families against seven real-world application domains.
Results
Diffusion and transformer architectures are well-matched to at most two deployment contexts, whereas GANs appear as an option across all seven domains.
Takeaways & Limitations
Matching the right model to the right hardware and application can broaden generative AI deployment beyond the platforms favored by benchmark rankings.
Takeaways & Limitations
The present trajectory independently scales models and hardware, requiring substantially larger datacentres and increasing energy consumption.
Abstract
from arXiv · showhide
Vision generative artificial intelligence (AI) has emerged as one of the most rapidly advancing areas of deep learning. The explosion of multimodal models has made them widely associated with text-to-image applications running on large datacentres. However, vision generative models are equally needed in applications that operate under strict hardware constraints at the edge, including autonomous vehicles, agricultural sensors, and mobile devices. In this Perspective, we argue that progress in vision generative AI has been driven by output quality, with hardware evolving reactively to accommodate growing model demands. We quantify the parameter cost and energy efficiency of these models across a range of accelerator platforms, and map four generative model families against seven real-world application domains. Finally, we advocate a software-hardware co-design approach, where deployment constraints are considered from the start of the design process, ensuring that the "right model" runs on the "right hardware" to serve the "right application", making generative AI deployment sustainable and accessible across a much broader range of platforms.
Introduction
Vision generative AI spans datacentre and hardware-constrained edge applications, but rising output quality has increased parameter, memory, and inference demands. The Perspective maps these trade-offs and advocates software-hardware co-design for broader, more sustainable deployment.
- Vision generative AI creates and enhances visual content for datacentre, medical, agricultural, autonomous-driving, mobile, and AR/VR applications.
- Models that achieve the best benchmark results can require more than 10 GB of memory and several seconds per image, limiting deployment on tightly constrained platforms.
- Hardware evolved after model classes were established, producing a growing gap between dominant models and the platforms expected to run them.
- The Perspective traces model and hardware evolution, quantifies computational and hardware trade-offs, maps four model families to seven domains, and advocates software-hardware co-design.
Evolution of vision generative AI models
Vision generative AI progressed from compact latent models through adversarial, diffusion, and transformer-based families, with successive transitions primarily pursuing better quality and realism. These transitions increased compute, memory, and data-movement demands.
- Four dominant families—VAEs, GANs, diffusion models, and autoregressive transformers—define the major architectural transitions in vision generative AI.
- VAEs compress images into latent representations and reconstruct them in a single forward pass, providing computational efficiency but limited perceptual quality.
- GANs improved sharpness through generator-discriminator competition and retained single-pass generation, although adversarial optimization can be unstable and cause mode collapse.
- Diffusion models replaced adversarial competition with iterative denoising, offering more stable and scalable training and surpassing GANs on quality and benchmark diversity.
- Autoregressive transformers generate images token by token, making training expensive and inference slow, while later diffusion transformers used transformer backbones during denoising.
Evolution of AI hardware accelerators
AI accelerators evolved largely in reaction to established model demands rather than through software-hardware co-design. The resulting platforms target throughput, energy efficiency, memory movement, and transformer workloads in succession.
- Hardware platforms were adapted only after new model classes became established, while available hardware constraints were rarely considered before architectural decisions.
- GPUs replaced CPUs for deep-learning workloads because their arithmetic throughput and memory bandwidth suited convolutions and matrix multiplications.
- Neuromorphic processors, FPGAs, and compute-in-memory architectures pursued energy efficiency, specialized processing, and reduced data movement as model demands grew.
- NVIDIA Volta added Tensor Cores and delivered 125 TFLOPS of FP16 throughput, making large-scale GAN training more tractable.
- Diffusion introduced tens to hundreds of sequential evaluations per sample, while transformers increased pressure on memory capacity, bandwidth, and data movement.
- Hardware repeatedly absorbed model-driven demands, but two- to three-year accelerator development cycles remain structurally incompatible with rapidly evolving architectures.
The cost of unconstrained quality optimization
Quality gains do not scale proportionally with parameter count, and deployment feasibility depends on energy efficiency as well as model size. GANs offer a compact quality frontier, whereas specialized hardware can substantially improve efficiency across model families.
- FID and parameter counts across CIFAR-10, ImageNet 128×128, and ImageNet 256×256 expose the quality-complexity trade-off among generative model families.
- A CIFAR-10 GAN achieves FID 2.64 with 9.4 million parameters, comparable to diffusion models requiring three to twelve times more parameters.
- On ImageNet 256×256, the best diffusion model improves on the GAN frontier by 0.22 FID points but requires 3 billion parameters.
- GANs typically require 100 to 200 million parameters, compared with 400 million to 700 million for diffusion and 1 billion to 4 billion for transformers at comparable or marginally better results.
- ASIC deployments reach 135.1 TOPS/W for GANs, close to 100 TOPS/W for diffusion, and approximately 40 TOPS/W for transformers.
- Figure 4 compares each model family with application limits for output quality, latency, and memory across seven deployment domains.
Matching generative models to application domains
Seven application domains impose different limits on output quality, inference latency, and memory capacity, but dominant diffusion and transformer architectures fit only a small minority. GANs and VAEs remain practical across more constrained settings, with GANs spanning all seven domains.
- Deployment constraints: Seven deployment domains are governed by different combinations of output quality, inference latency, and memory capacity.The analysis evaluates feasibility across cloud content generation, medical imaging, autonomous vehicles, AR/VR, agriculture, mobile devices, and edge platforms.
- Output quality: Cloud content generation and medical imaging prioritize output quality and can accommodate diffusion and transformer models on GPU-class infrastructure.Cloud platforms have abundant power and memory, while medical imaging requires preservation of pathological and anatomical detail.
- Inference latency: Autonomous vehicles, AR/VR, and agriculture are latency-bound, requiring approximately 150ms, below 20ms, and approximately 500ms, respectively.GANs suit autonomous driving and AR/VR latency thresholds; agriculture’s additional memory limits make GANs and VAEs more practical.
- Memory capacity: Mobile and edge platforms typically provide 4–16 GB of DRAM, making GAN and VAE architectures more practical than diffusion and transformer models.Diffusion and transformer parameter counts exceed this tier’s capacity, and their memory-pressure behavior is difficult to manage in deployment.
- Cross-domain fit: Diffusion and transformer architectures are well-matched to at most two deployment contexts, whereas GANs appear as an option across all seven domains.For the remaining five domains, architectures largely abandoned by benchmark-driven research can still meet perceptual sufficiency requirements.
Outlook
The paper contrasts three deployment trajectories: continued model growth with rising datacentre costs, software–hardware co-design, and an ambient future based on redesigned models and beyond-CMOS hardware. Co-design broadens edge deployment while reducing, but not eliminating, reliance on remote datacentres; the most ambitious trajectory targets fully local operation.
- Outlook: Across four model families, output quality and visual realism improved while parameter counts, memory requirements, and inference times grew at each transition.Hardware evolved after model generations were established, creating a gap between model demands and real-world deployment platforms.
- Present trajectory: The present trajectory improves models independently of hardware, requiring larger datacentres and increasing energy consumption.Global datacentre electricity consumption is projected to double by 2030 from approximately 1.5% of global electricity consumption in 2024.
- Co-design trajectory: The co-design trajectory matches the right model to the right hardware for the right application, enabling broader edge deployment while reducing energy and infrastructure costs.Some applications run locally, but dependence on remote datacentres persists.
- Ambient future: The ambient-future trajectory requires foundational model redesign and beyond-CMOS technologies, including neuromorphic architectures, to run generative vision AI locally.The stated aim is operation without dependence on large datacentres; commercial in-memory computing applications are expected from 2030 onwards.
A software-hardware perspective Supplementary Information
The supplementary information provides tables covering generative-model quality and size, accelerator performance and energy metrics, and deployment requirements. It also identifies concrete edge-system constraints, including latency, reaction time, spraying inference time, and DRAM budgets.
- Table S1 reports FID scores and parameter counts across CIFAR-10, ImageNet 128×128, and ImageNet 256×256.
- Table S2 compares hardware accelerators using throughput, power consumption, and energy efficiency.
- Table S3 links model metrics with application deployment requirements for Figures 4a–d.
- Real-time AR/VR requires latency <16 ms per frame (>60 fps), while autonomous vehicles, precision spraying, and mobile or edge devices impose reaction-time, inference-time, and DRAM constraints.