Vision-centric generative AI models: A software-hardware perspective

• Vision generative AI creates and enhances visual content for datacentre, medical, agricultural, autonomous-driving, mobile, and AR/VR applications.Source: Introduction · p. 1
• Models that achieve the best benchmark results can require more than 10 GB of memory and several seconds per image, limiting deployment on tightly constrained platforms.Source: Introduction · p. 1
• Hardware evolved after model classes were established, producing a growing gap between dominant models and the platforms expected to run them.Source: Introduction · pp. 1–2
• The Perspective traces model and hardware evolution, quantifies computational and hardware trade-offs, maps four model families to seven domains, and advocates software-hardware co-design.Source: Introduction · p. 2

• Four dominant families—VAEs, GANs, diffusion models, and autoregressive transformers—define the major architectural transitions in vision generative AI.Source: Evolution of vision generative AI models · p. 2
• VAEs compress images into latent representations and reconstruct them in a single forward pass, providing computational efficiency but limited perceptual quality.Source: Evolution of vision generative AI models · p. 2
• GANs improved sharpness through generator-discriminator competition and retained single-pass generation, although adversarial optimization can be unstable and cause mode collapse.Source: Evolution of vision generative AI models · p. 2
• Diffusion models replaced adversarial competition with iterative denoising, offering more stable and scalable training and surpassing GANs on quality and benchmark diversity.Source: Evolution of vision generative AI models · p. 3
• Autoregressive transformers generate images token by token, making training expensive and inference slow, while later diffusion transformers used transformer backbones during denoising.Source: Evolution of vision generative AI models · pp. 3–4

• Hardware platforms were adapted only after new model classes became established, while available hardware constraints were rarely considered before architectural decisions.Source: Evolution of AI hardware accelerators · pp. 4–5
• GPUs replaced CPUs for deep-learning workloads because their arithmetic throughput and memory bandwidth suited convolutions and matrix multiplications.Source: Evolution of AI hardware accelerators · p. 5
• Neuromorphic processors, FPGAs, and compute-in-memory architectures pursued energy efficiency, specialized processing, and reduced data movement as model demands grew.Source: Evolution of AI hardware accelerators · p. 5
• NVIDIA Volta added Tensor Cores and delivered 125 TFLOPS of FP16 throughput, making large-scale GAN training more tractable.Source: Evolution of AI hardware accelerators · p. 5
• Diffusion introduced tens to hundreds of sequential evaluations per sample, while transformers increased pressure on memory capacity, bandwidth, and data movement.Source: Evolution of AI hardware accelerators · pp. 5–6
• Hardware repeatedly absorbed model-driven demands, but two- to three-year accelerator development cycles remain structurally incompatible with rapidly evolving architectures.Source: Evolution of AI hardware accelerators · p. 6

• FID and parameter counts across CIFAR-10, ImageNet 128×128, and ImageNet 256×256 expose the quality-complexity trade-off among generative model families.Source: The cost of unconstrained quality… · p. 7
• A CIFAR-10 GAN achieves FID 2.64 with 9.4 million parameters, comparable to diffusion models requiring three to twelve times more parameters.Source: The cost of unconstrained quality… · p. 7
• On ImageNet 256×256, the best diffusion model improves on the GAN frontier by 0.22 FID points but requires 3 billion parameters.Source: The cost of unconstrained quality… · p. 7
• GANs typically require 100 to 200 million parameters, compared with 400 million to 700 million for diffusion and 1 billion to 4 billion for transformers at comparable or marginally better results.Source: The cost of unconstrained quality… · p. 7
• ASIC deployments reach 135.1 TOPS/W for GANs, close to 100 TOPS/W for diffusion, and approximately 40 TOPS/W for transformers.Source: The cost of unconstrained quality… · p. 7
• Figure 4 compares each model family with application limits for output quality, latency, and memory across seven deployment domains.Source: Figure 4 · p. 8

Deployment constraints

• Seven deployment domains are governed by different combinations of output quality, inference latency, and memory capacity.The analysis evaluates feasibility across cloud content generation, medical imaging, autonomous vehicles, AR/VR, agriculture, mobile devices, and edge platforms.Source: Matching generative models to… · p. 8

Output quality

• Cloud content generation and medical imaging prioritize output quality and can accommodate diffusion and transformer models on GPU-class infrastructure.Cloud platforms have abundant power and memory, while medical imaging requires preservation of pathological and anatomical detail.Source: Matching generative models to… · pp. 8–9

Inference latency

• Autonomous vehicles, AR/VR, and agriculture are latency-bound, requiring approximately 150ms, below 20ms, and approximately 500ms, respectively.GANs suit autonomous driving and AR/VR latency thresholds; agriculture’s additional memory limits make GANs and VAEs more practical.Source: Matching generative models to… · p. 9

Memory capacity

• Mobile and edge platforms typically provide 4–16 GB of DRAM, making GAN and VAE architectures more practical than diffusion and transformer models.Diffusion and transformer parameter counts exceed this tier’s capacity, and their memory-pressure behavior is difficult to manage in deployment.Source: Matching generative models to… · p. 9

Cross-domain fit

• Diffusion and transformer architectures are well-matched to at most two deployment contexts, whereas GANs appear as an option across all seven domains.For the remaining five domains, architectures largely abandoned by benchmark-driven research can still meet perceptual sufficiency requirements.Source: Matching generative models to… · p. 9

Outlook

• Across four model families, output quality and visual realism improved while parameter counts, memory requirements, and inference times grew at each transition.Hardware evolved after model generations were established, creating a gap between model demands and real-world deployment platforms.Source: Outlook · p. 9

Present trajectory

• The present trajectory improves models independently of hardware, requiring larger datacentres and increasing energy consumption.Global datacentre electricity consumption is projected to double by 2030 from approximately 1.5% of global electricity consumption in 2024.Source: Outlook · pp. 9–10

Co-design trajectory

• The co-design trajectory matches the right model to the right hardware for the right application, enabling broader edge deployment while reducing energy and infrastructure costs.Some applications run locally, but dependence on remote datacentres persists.Source: Outlook · p. 10, Figure 5 · p. 10

Ambient future

• The ambient-future trajectory requires foundational model redesign and beyond-CMOS technologies, including neuromorphic architectures, to run generative vision AI locally.The stated aim is operation without dependence on large datacentres; commercial in-memory computing applications are expected from 2030 onwards.Source: Outlook · pp. 10–11, Figure 5 · p. 10

• Table S1 reports FID scores and parameter counts across CIFAR-10, ImageNet 128×128, and ImageNet 256×256.Source: Table S1 · p. 19
• Table S2 compares hardware accelerators using throughput, power consumption, and energy efficiency.Source: Table S2 · p. 21
• Table S3 links model metrics with application deployment requirements for Figures 4a–d.Source: Table S3 · p. 22
• Real-time AR/VR requires latency <16 ms per frame (>60 fps), while autonomous vehicles, precision spraying, and mobile or edge devices impose reaction-time, inference-time, and DRAM constraints.Source: A software-hardware perspective… · p. 22