PaperlayerView exact revision on arXiv

Source-linked AI summary

Soft Anisotropic Diagrams for Differentiable Image Representation

Laki Iinbor, Zhiyang Dou, Wojciech Matusik

arXiv:2604.21984v2cs.CV

TL;DR

Compact differentiable image representations often lack explicit spatial ownership and efficient local budget control. SAD addresses this with a soft anisotropic partition over adaptive sites, achieving strong quality–efficiency results, including 46.00 dB on Kodak while reducing encoding time to 2.2 s from 28 s for Image-GS.

  • Problem

    Existing image representations provide limited explicit spatial ownership, localized budget control, and sharp boundary modeling without costly kernel overlap.

  • Method

    SAD represents images with adaptive anisotropic sites and renders pixels using a temperature-controlled softmax over fixed-size top-K neighborhoods induced by a soft weighted Voronoi partition.

  • Results

    Across benchmarks, SAD outperforms Image-GS and Instant-NGP at matched bitrate; on Kodak, it achieves 46.00 dB PSNR and cuts encoding time to 2.2 s from 28 s for Image-GS.

  • Takeaways & Limitations

    SAD improves the quality–efficiency trade-off of compact image representations while providing explicit ownership and adjacency for differentiable pipelines and compression.

  • Takeaways & Limitations

    SAD’s rendering efficiency depends on accurate cached top-K candidate sets, whose refresh can dominate rendering time after large edits or at high resolutions.

Abstract

from arXiv · show

We introduce Soft Anisotropic Diagrams (SAD), an explicit and differentiable image representation parameterized by a set of adaptive sites in the image plane. In SAD, each site specifies an anisotropic metric and an additively weighted distance score, and we compute pixel colors as a softmax blend over a small per-pixel top-K subset of sites. We induce a soft anisotropic additively weighted Voronoi partition (i.e., an Apollonius diagram) with learnable per-site temperatures, preserving informative gradients while allowing clear, content-aligned boundaries and explicit ownership. Such a formulation enables efficient rendering by maintaining a per-query top-K map that approximates nearest neighbors under the same shading score, allowing GPU-friendly, fixed-size local computation. We update this list using our top-K propagation scheme inspired by jump flooding, augmented with stochastic injection to provide probabilistic global coverage. Training follows a GPU-first pipeline with gradient-weighted initialization, Adam optimization, and adaptive budget control through densification and pruning. Across standard benchmarks, SAD consistently outperforms Image-GS and Instant-NGP at matched bitrate. On Kodak, SAD reaches 46.0 dB PSNR with 2.2 s encoding time (vs. 28 s for Image-GS), and delivers 4-19 times end-to-end training speedups over state-of-the-art baselines. We demonstrate the effectiveness of SAD by showcasing the seamless integration with differentiable pipelines for forward and inverse problems, efficiency of fast random access, and compact storage.

1 Introduction

SAD is an explicit, differentiable image representation that uses adaptive anisotropic sites and a learnable soft partition to provide local, content-aligned ownership with fixed-size pixel queries. Its GPU-first optimization and budget-control pipeline improves compact-representation quality and encoding efficiency over stated baselines.

  • Motivation: Implicit representations lack explicit spatial ownership, while SAD provides localized, interpretable neighborhoods that support predictable queries, pruning, densification, and content-aligned discontinuities.These design goals address indirect budget control and smeared boundaries caused by kernel overlap.
  • Representation: SAD parameterizes the image with adaptive anisotropic sites and renders each pixel through a top-K softmax blend with learnable per-site temperatures.The temperatures can sharpen the soft partition into crisp, content-aligned boundaries and expose adjacency structure.
  • Geometric formulation: SAD uses an additively weighted anisotropic Voronoi construction, equivalently an Apollonius diagram, giving each site an interpretable influence radius.The additive per-site radius offset integrates with the anisotropic norm and the softmax relaxation.
  • Efficient rendering: K=8 top-K site lists and reuse, four-neighbor jump-flood propagation, and stochastic injection reduce rendering to O(P·K) fixed per-pixel work.The maintained list uses the same score as shading and adds probabilistic global coverage through stochastic injection.
  • Results: 46.00 dB on Kodak with N=50,000 primitives cuts encoding time from 28 s to 2.2 s, while SAD outperforms Image-GS and Instant-NGP at matched bitrate.On Image-GS, one reported matched-bitrate result is 37.86 dB at 0.5 BPP.

2 Related Work

Prior image representation work spans learned transform coding, implicit coordinate-based models, and point- or splat-based primitives. SAD relates to these lines through adaptive primitives, differentiable ownership, anisotropic rendering, and Voronoi-family partitions.

  • Image representation and compression: Neural image representation and compression includes learned transform coding, end-to-end optimization, implicit coordinate-based signal models, and adaptive primitives with learnable soft-to-sharp ownership.The related-work overview positions point- and splat-based primitives alongside these earlier approaches.
  • Implicit neural signal representations: Implicit coordinate-based networks model images or scenes as continuous functions of coordinates and use encodings such as Fourier features, periodic activations, adaptive coordinates, vector-quantized auto-decoders, and multiresolution hash grids.Instant-NGP is identified as a multiresolution hash-grid representation.
  • Point-based and splat-based rendering: Point- and splat-based rendering builds on surfels and surface splatting, including anisotropic filtering and efficient evaluation.Recent work adapts 2D Gaussian Splatting for compact image representations, including Image-GS.
  • Geometric relationships: Point-based methods are closely related to Voronoi diagrams, their variants, and dual graphs such as Delaunay triangulation.These geometric structures have been extensively explored in related areas.

3 Preliminaries

This section reviews Voronoi, Apollonius, power, and anisotropic diagram constructions, emphasizing their distance comparisons, influence structures, and ownership properties. It then situates SAD as a differentiable anisotropic Apollonius-style representation with softmax-based partitions.

  • Voronoi diagrams: Euclidean Voronoi diagrams partition a domain into cells of points closest to each generator, producing explicit ownership and adjacency through shared boundaries.In 2D, the cells have polygonal boundaries.
  • Additively weighted Voronoi (Apollonius) diagrams: Apollonius diagrams compare distances after subtracting per-site radii, so larger radii expand a site’s interpretable influence region.This construction is equivalent to comparing distances to weighted balls.
  • Power diagrams / Laguerre tessellations: Power diagrams add weights to squared distances, yielding a quadratic comparison distinct from Apollonius diagrams and a dual relationship to weighted Delaunay triangulations.They are also known as Laguerre diagrams or tessellations.
  • Anisotropic Voronoi and centroidal tessellations: Anisotropic Voronoi constructions replace the Euclidean norm with a site-dependent metric, commonly represented by an SPD matrix G_i.Anisotropic and centroidal variants support mesh generation and approximation under such metrics.
  • Connection to SAD: SAD combines an SPD metric G_i with an additive radius r_i, then replaces hard arg min ownership with a temperature-controlled softmax partition.This produces an anisotropic Apollonius-style comparison rather than a quadratic power/Laguerre score, while enabling stable gradients and sharp transitions.

4 Method

SAD represents images with anisotropic, additively weighted sites and renders each pixel through a differentiable softmax blend over a small top-K neighborhood. Its GPU-first method maintains these candidate neighborhoods efficiently, initializes sites from image gradients, and uses fixed-cost local computation for scalable optimization and rendering.

  • Site representation: SAD models images with anisotropic sites carrying positions, colors, radii, temperatures, and directional metrics, then renders pixels using a differentiable soft partition of unity over top-K sites.Unlike hard Voronoi ownership, multiple nearby sites can jointly contribute while preserving explicit spatial locality.
  • Site representation: The anisotropic additively weighted score separates geometric reach, directional stretch, and transition sharpness through radius r_i, metric G_i, and temperature τ_i.The metric satisfies det(G_i) = 1, changing aspect without area scaling; larger τ_i sharpens transitions.
  • Candidate maintenance: The maintained per-pixel top-K set approximates the strongest sites under the same rendering score, serving both as a computational device and a locality prior.Each pixel evaluates only a small explicit neighborhood of competitive sites rather than all sites.
  • Candidate maintenance: JFA provides coarse-to-fine long-range warm-up, after which temporal reuse and self-plus-4-neighbor propagation update candidate fields with a small nonlocal probe set.The jump schedule uses B/2, B/4, ..., 1, then remains at 1 for later single-pass immediate-neighbor updates.
  • Optimization: Gradient-weighted initialization concentrates sites in high-frequency regions while a uniform prior preserves coverage of soft areas, with sampled pixel colors assigned to sites.This initialization accelerates convergence but is not presented as a novel contribution.
  • GPU implementation: O(P·K) candidate maintenance and O(P·K) rendering keep per-pixel computation independent of total site count, unlike Image-GS’s O(P·G) overlap scan.GPU execution uses regular, coalesced fixed-size work; localized reductions replace O(P·K) scattered global atomics, improving cache locality and reducing contention.

5 Implementation Details

SAD uses a GPU-resident implementation with hand-derived forward and backward kernels across Metal, CUDA, and WebGPU. Its threadgroup-local hash accumulation reduces global atomic updates before flushing gradients to global memory.

  • Gradient accumulation: If the threadgroup hash table overflows, the implementation falls back to accumulating those sites through global atomics.This provides a fallback path for sites that cannot fit in the shared hash table.
  • Gradient accumulation: Threadgroup hash reduction accumulates per-site gradients in shared memory, then flushes them to global buffers with one atomic per site per tile.The hash table is keyed by siteID and uses bounded probing with open addressing for collisions.
  • GPU implementation: The full training and rendering pipeline runs as a single GPU-resident renderer using compute shaders, with Metal, CUDA, and WebGPU backends.Forward rendering, backward differentiation, candidate propagation, and Adam optimization execute without CPU work during training iterations.
  • GPU implementation: All forward kernels and corresponding gradients are hand-derived and implemented directly without automatic differentiation frameworks.The implementation avoids relying on automatic differentiation frameworks for the rendering pipeline.

6 Evaluation

SAD is evaluated against Instant-NGP and Image-GS using reconstruction quality, bitrate, encoding, rendering, and training-time metrics. Across datasets, resolutions, and ablations, SAD achieves higher quality and substantially faster processing, while its parameter-space BPP measures representational compactness rather than codec replacement.

  • Evaluation Protocol: Evaluation uses PSNR, SSIM, LPIPS, training time, and rendering performance, with model sizes matched to target parameter-space bitrates.PSNR and SSIM are higher-is-better metrics, whereas LPIPS is lower-is-better; all metrics use linear color space.
  • Rate–Distortion Results: 46.00 dB PSNR at approximately 16.0 BPP is achieved on Kodak with 0.9871 SSIM, 0.0032 LPIPS, and 2.2s encoding.This improves over Fast 2DGS by +2.87 dB and reduces encoding time from 10s to 2.2s.
  • Rate–Distortion Results: +2.55–3.29 dB over Image-GS is obtained on its benchmark at 0.2–0.5 BPP, with additional gains on DIV2K and CLIC.The reported DIV2K gains are +1.52/+2.58 dB, and the CLIC gains are +1.17/+1.98 dB at 0.5/2.0 BPP.
  • Efficiency: 1.75–3.36× speedup over Instant-NGP and 4.08–15.10× over Image-GS is achieved per epoch across image resolutions.At 2k resolution, SAD processes one epoch in 3.6 ms versus 12.1 ms for Instant-NGP and 14.7 ms for Image-GS.
  • Efficiency: 4–8× end-to-end speedup over Instant-NGP and 5–19× over Image-GS is achieved under equal-sample training, combining faster epochs with the stated step budgets.SAD reaches 37.5 dB PSNR on 2k images at 0.5 BPP in 2000 iterations.
  • Ablations: +2.30 dB over the best fixed-temperature baseline and +1.26 dB from learnable radius demonstrate the value of adaptive temperature and spatial coverage.Higher fixed temperatures sharpen boundaries while lower temperatures soften blending, but no single fixed value performs optimally across all regions.

7 More Applications

SAD supports differentiable physics simulation on irregular domains and extends unchanged to low-dimensional signal fitting. Its explicit sites enable hard boundary constraints, while its 1D parameterization supports comparisons with Gaussian splatting and SIREN at matched budgets.

  • Differentiable physics simulation: Freezing explicit boundary sites after initialization enforces hard Dirichlet conditions by excluding them from gradient updates.This avoids specialized distance-function constructions or penalty-based soft constraints typically needed with implicit MLP representations.
  • 1D signal fitting: In 1D, SAD uses four parameters per site—position, value, log-temperature, and radius—versus nine parameters in 2D.The comparison uses 64 primitives and 256 parameters for SAD and Gaussian splatting, approximately matching the SIREN budget.
  • 1D signal fitting: 79.6 dB is achieved by Gaussian splatting on a smooth sine, but its fixed-shape kernels round off a sharp step to 47.3 dB.Gaussian bases are near-optimal for band-limited interpolation but cannot directly model discontinuities.
  • 1D signal fitting: 43.1 dB is achieved by SIREN on the sharp-step target, with Gibbs-like ringing caused by truncating the discontinuity’s frequency content.SIREN represents continuous variation through a global sinusoidal basis.

8 Conclusion · A Appendix

SAD concludes as an explicit, differentiable image representation with temperature-controlled local blending and explicit spatial ownership, while identifying candidate-maintenance limitations, implementation dependencies, and several future applications. The paper also notes privacy, intellectual-property, and deceptive-editing risks associated with misuse of image representations.

  • 8 Conclusion: SAD uses a soft anisotropic additively weighted Voronoi partition and temperature-controlled softmax blending over per-pixel top-K adaptive sites.This formulation keeps optimization well-conditioned, makes spatial ownership explicit, and allows content-dependent boundary sharpening.
  • 8 Conclusion: Imperfect or stale top-K candidate lists can reduce quality at very low budgets, after aggressive densification or pruning, or during abrupt parameter changes.The limitation arises because SAD relies on accurate per-pixel candidate sets maintained through propagation.
  • 8 Conclusion: Cached top-K lists support the fastest rendering regime, while hardware and kernel configurations can affect absolute performance and preferred settings.Refreshing the candidate map can change the rendering cost, and the implementation is optimized for GPU execution and bandwidth-efficient kernels.
  • 8 Conclusion: Future work includes hierarchical and adaptive candidate maintenance, richer per-site appearance models, and improved anisotropic distance models.Examples include multi-resolution caches, resolution-aware reuse, streaming-friendly refresh schedules, local appearance models, learned decoders, and material cues.
  • 8 Conclusion: SAD could extend beyond 2D images to 3D representations and 2D/3D inverse problems, while its explicit sites and adjacency may support structured image tokenization and decoding.The paper also suggests generative-domain uses as alternatives to implicit coordinate MLPs.
  • 8 Conclusion: SAD currently adds no new content-generation or identity-inference capabilities beyond standard neural primitive-based codecs.Potential misuse still includes storing or transmitting sensitive imagery and enabling downstream manipulation at scale, raising privacy, IP, and deceptive-editing concerns.

B Implementation Details · B.1 Parameters and Constants

Section B.1 organizes implementation configuration into runtime parameters and fixed constants, with separate tables documenting each category. The constants are shared across backends, while runtime parameters are specific to implementation use.

  • B.1 Parameters and Constants: Table B11 lists the runtime parameters used by the implementation.It is the dedicated runtime-parameter reference for Section B.1.
  • B.1 Parameters and Constants: Table B12 lists fixed constants shared across backends.These constants provide a common configuration reference across backend implementations.
  • B.1 Parameters and Constants: The implementation details distinguish runtime parameters from fixed constants.The distinction is reflected in the separate Table B11 and Table B12 references.
  • B.1 Parameters and Constants: Runtime parameters are documented separately from constants shared across backends.Table B11 covers runtime parameters, whereas Table B12 covers shared fixed constants.
  • B.1 Parameters and Constants: Table B11 serves as the runtime-configuration reference for the implementation.The passage explicitly identifies its contents as runtime parameters used by the implementation.
  • B.1 Parameters and Constants: Table B12 serves as the cross-backend reference for fixed implementation constants.Its stated scope is limited to fixed constants shared across backends.

B.2 Data Layout and Packing

SAD uses a shared float32 site layout for training, a 16-byte packed format for evaluation, and compact top-K textures and candidate buffers for GPU-efficient rendering and updates.

  • Site buffer: Training stores each site as 10 float32 semantic values, with inactive sites marked by x < 0 and skipped by all kernels.The layout is shared across Metal, CUDA, and WebGPU.
  • Packed storage for evaluation: Evaluation packs all 10 semantic values into four 32-bit words totaling 16 bytes, using quantized position, color, temperature, radius, direction, and opacity fields.Per-image min/scale values are used for the packed logτ and radius fields.
  • Top-K candidate list: Per-pixel top-K site IDs are stored in two four-channel 32-bit unsigned integer textures, with K = 8 and 0xffffffff as the invalid sentinel.The arrangement provides coalesced loads for rendering and gradient computation.
  • Packed candidate sites: Candidate-update kernels use a compact 16-byte per-site buffer based on float16 pairs, regenerated after densification to reduce bandwidth.The packed fields include position, (logτ, r), anisotropy direction, and (a, 0).

B.3 Candidate Field and Jump Schedule

The candidate field uses a downscaled, bias-aware grid and supports GPU-friendly updates through configurable schedules. Optional JFA prepasses accelerate reinitialization, while infrequent updates preserve final PSNR and reduce training time.

  • Candidate field: The candidate grid has dimensions ⌈W/sgrid⌉×⌈H/sgrid⌉, uses integer coordinate mapping, and evaluates candidates at centered cell UVs to avoid systematic bias.This design maintains a downscaled candidate representation without interpolation while correcting cell-centering bias during evaluation.
  • Jump schedule: The jump schedule packs the step index and jump distance into a single 32-bit parameter for compact buffering and deterministic per-pass randomization.The lower and upper 16-bit halves store the step index and jump distance, respectively; xorshift state uses the step index and pixel ID.
  • Jump schedule: ⌈log2 max(W, H)⌉ flood passes in the optional JFA prepass sample 3×3 neighborhoods and write four closest sites into the first candidate texture.The prepass begins with a seed pass and is mainly useful for reinitialization or after large edits; VPT updates can build the candidate field alone.
  • Candidate field: 8–16 iterations between candidate-field updates did not measurably change final PSNR while reducing training time.The implementation exposes both update frequency and the number of passes per update.

B.4 Gradient Computation and Reduction … B.8 Backend Notes

The backend pipeline computes stable top-K softmax gradients, reduces them efficiently on GPU, and adapts site budgets through statistics-driven splitting and pruning. Shared GPU-resident implementations preserve consistent mathematics across Metal, CUDA, and WebGPU while supporting configurable dispatch and optional temperature diffusion.

  • B.4 Gradient Computation and Reduction: Top-K softmax gradients use max-subtracted normalization and cover position, logτ, r, color, anisotropy direction, and anisotropy scale while skipping invalid sites and guarding NaNs.Gradients are computed from the softmax weights of each pixel’s top-K list.
  • B.4 Gradient Computation and Reduction: WebGPU accumulates scaled 32-bit integer gradients, whereas Metal and CUDA use float atomics; Adam converts accumulations to floating point and normalizes by pixel count.This accommodates WebGPU’s lack of universally available floating-point atomics.
  • B.4 Gradient Computation and Reduction: 256-slot per-tile hash tables reduce 11 accumulators per site using multiplicative hashing, linear probing capped at 8 probes, and atomic compare-and-swap insertion.The 11 accumulators comprise 10 gradients plus a removal-delta.
  • B.5 Densification and Pruning: Densification scores combine per-site mass, energy, and weighted second moments, then radix sorting selects the highest-ranked candidates for splitting.The score is defined as energy/mass^α.
  • B.5 Densification and Pruning: Selected sites split into two children offset along a covariance- or gradient-estimated axis, inherit target-image colors, reduce logτ and r, and reset Adam states.When covariance statistics are unavailable, the method uses a local Sobel-like gradient and may apply r←0.85r.
  • B.5 Densification and Pruning: Pruning ranks sites by removal score and deactivates the lowest-ranked ones by setting their positions to (−1, −1), after which all kernels ignore them.Removal scores come from the removal-delta accumulator.
  • B.6 Tau Diffusion: Optional tau diffusion applies a per-site Jacobi update that mixes each raw logτ gradient with neighboring gradients using scalar λ.The averaging neighborhood is the local candidate neighborhood at each site.
  • B.7 Dispatch Configuration / B.8 Backend Notes: Image-space kernels use 16×16 groups, while per-site kernels use 64 or 256 threads; Metal, CUDA, and WebGPU share math, layouts, candidate updates, and GPU-resident buffers.WebGPU uses shared WGSL source, while Metal and CUDA translate the same kernels into MSL and CUDA C++; only site I/O crosses the host boundary.
Loading 2604.21984v2…