PaperlayerView exact revision on arXiv

Source-linked AI summary

Heat Kernel Textures: the Geodesic Gaussians That Do Not Splat

Simone Foti, Caner Korkmaz, Stefanos Zafeiriou, Tolga Birdal

arXiv:2609.07557v1cs.CVcs.GRcs.LG

TL;DR

UV mapping can waste texture storage and produce seams, distortions, and resolution variation. HKTex represents textures with anisotropic heat kernels intrinsic to the mesh surface and supports optimisation through differentiable rendering. The reported evaluations show competitive reconstruction quality with low storage for UV-texture fitting and consistent metric improvements over specified baselines for multi-view fitting.

  • Problem

    UV mapping can produce seams, distortions, varying resolution, and storage waste, motivating a texture representation defined on the surface with lower memory use.

    Source: §1 · p. 2

  • Method

    HKTex uses anisotropic heat kernels defined on the mesh surface, with kernel parameters optimised through a differentiable physically based renderer.

    Source: §1 · p. 2, §4 · pp. 5–6

  • Results

    HKTex has the lowest storage footprint after GT VTex with competitive reconstruction quality, and consistently outperforms high-resolution vertex colours and neural baselines across all evaluation metrics in multi-view fitting.

    Source: §5.4 · p. 13, §5.5 · p. 14

  • Takeaways & Limitations

    The conclusion reports that operating in intrinsic mesh geometry eliminates seams, distortions, and redundant storage associated with UV parametrisations.

    Source: §6 · p. 14

    Kernel aggregation and geodesic-neighbourhood queries add evaluation overhead, partly amortized in multi-view fitting where deeper ray tracing dominates rendering time.

    Source: §5.5 · p. 14

Abstract

from arXiv · show

3D Gaussian Splatting has recently revolutionised novel view synthesis as well as many other 3D vision methods and applications. Drawing inspiration from this representation, we now rethink textures to overcome the main issues of UV mapping while considerably lowering their memory footprint. Heat Kernel Textures (HKTex) eliminate UV unwrapping as well as their persistent issues of wasted UV space, seams, distortions, vertex-duplication, and varying resolution. Grounded in discrete Riemannian geometry and intrinsically defined on any manifold surface discretised as a triangular mesh, HKTex uses anisotropic heat kernels as geodesic equivalents to Gaussians. Like our kernels, also the optimisation of their position and the adaptive densification strategies were redefined to operate on the surface of the object to be textureised. Our novel representation is also fully integrated with a physically based renderer and can be optimised either from existing textures or multi-view images. Our project page and code are available at circle-group.github.io/research/HeatKernelTextures.

1 Introduction

UV mapping introduces seams, distortions, resolution variation, and storage waste; HKTex replaces it with a geodesic texture representation defined directly on the mesh surface. The paper also develops surface-constrained optimisation and density control and integrates HKTex into differentiable physically based rendering.

1 Introduction

  • UV mapping can cause seams, varying resolutions, distortions, vertex duplication, and wasted space, motivating a surface-defined representation with lower memory use.The proposed representation is intended to benefit games, virtual and augmented reality, and visual effects.Source: §1 · p. 2
  • HKTex replaces Euclidean Gaussians with anisotropic heat kernels and optimises their positions and density directly on the manifold surface.Kernel splits use the exponential map, and continuous evaluation supports differentiable ray tracing in physically based rendering.Source: §1 · p. 2
  • HKTex replaces UV-mapped textures with an intrinsic representation grounded in discrete Riemannian geometry, avoiding UV unwrapping.Its kernels are defined directly on the mesh surface.Source: §1 · p. 2
  • The paper integrates HKTex into a differentiable physically based rendering pipeline for distortion-free texture optimisation from multi-view images.The representation can also compress existing UV textures and support inverse rendering from 2D photos.Source: §1 · pp. 2–3

2 Related Work

Prior methods address UV-related texture problems with neural representations, point clouds, or Gaussian-based approaches, but retain limitations tied to memory, topology, UV mappings, or ambient-space geometry. HKTex instead evaluates manifold-bound heat kernels continuously at ray intersections in a physically based renderer.

2 Related Work

  • Neural texture methods reduce reliance on manual UV unwrapping but remain tied to ambient 3D coordinates or intermediate 2D mappings, and MLP evaluation can be computationally heavy.Intrinsic Neural Fields use Laplace-Beltrami eigenvectors but lack the local, adaptive expressivity of explicit primitives.Source: §2 · p. 3
  • Existing Gaussian-based mesh texturing methods rely on UV maps or ambient Euclidean space, so their primitives cannot intrinsically conform to general manifold topology.Other approaches store colors in multi-resolution hash grids or use Gaussians for local geometry.Source: §2 · p. 3
  • HKTex integrates manifold-bound heat kernels as continuous, differentiable textures evaluated at ray intersections in a physically based renderer.Prior Gaussian integrations into PBR frameworks still use volumetric clouds of Euclidean primitives.Source: §2 · p. 3

3 Background & Notation

The background formalizes a triangular mesh and continuous surface points, then introduces isotropic and anisotropic Laplace-Beltrami operators and their spectral decompositions. Heat kernels describe diffusion from a point source, with a spectral formulation used for evaluation.

3 Background & Notation

  • A mesh manifold is represented by vertex positions, face connectivity, and edges, while a surface point is specified by a face and its barycentric coordinates.Barycentric interpolation also extends vertex attributes to surface points.Source: §3 · p. 4
  • The Laplace-Beltrami operator is formed from a mass matrix and a stiffness matrix, whose weights can be modified to define anisotropic diffusion.When the angle and anisotropy parameters are zero, the anisotropic operator reduces to the isotropic LBO.Source: §3 · p. 4
  • The generalized eigendecomposition supplies mesh eigenvectors and eigenvalues, with corresponding parameter-dependent spectral properties for the anisotropic operator.These properties support the spectral heat-kernel formulation.Source: §3 · p. 4
  • A heat kernel represents how heat from a single point source diffuses over the surface, and in Euclidean space it has the explicit form of a Gaussian.The spectral approach is reported to be an order of magnitude faster than solving a sparse linear system and to scale to high-resolution meshes.Source: §3 · p. 5

4 Heat Kernel Textures

HKTex represents a mesh texture with optimisable anisotropic heat kernels and combines their local contributions over the surface. The method interpolates spectral bases and evaluates kernels continuously, while constraining optimisation and density changes to the manifold and supporting texture fitting through differentiable rendering.

4 Heat Kernel Textures

  • HKTex represents a texture as optimisable heat kernels whose source position, diffusion angle and anisotropy, scale, sharpness, and colour are jointly parameterized.The continuous anisotropic kernel is derived from a truncated spectral heat diffusion formulation and normalized by its source temperature.Source: §4 · pp. 5–6
  • Biharmonic-distance Gaussian weighting suppresses distant spectral-ringing contributions, while filtering controls kernel scale and border sharpness.The filtered kernel can retain a fading profile or become a crisp geodesic ellipsoid.Source: §4 · p. 6
  • The texture colour combines the top-contributing kernels through an alpha-weighted partition of unity added as a residual to a global base colour.Overlapping kernels blend in a controlled way, and empty regions default to the base colour.Source: §4 · p. 6

4.1 Parametric Interpolation of the Spectral Bases

  • For anisotropy and angle queries, the method interpolates precomputed spectral eigenproperties on a log-scaled parameter grid after aligning eigenvectors to resolve permutation, rotation, and sign ambiguities.The query uses bilinear interpolation within the grid cell bounding its parameters.Source: §4.1 · p. 7

4.2 Spatial Evaluation of Spectral Properties and Masses

  • Continuous barycentric evaluation and biharmonic KNN search restrict kernel evaluation to nearby sources, bounding computation by O(P × Ks) rather than evaluating all P points against S kernels.The method estimates arbitrary source masses with KDE, but reports that unit source masses are faster and do not deteriorate performance.Source: §4.2 · p. 8

4.3 Optimisation & Density Control

  • Kernel positions are updated with projected Riemannian gradients and geodesic exponential maps, with momentum parallel-transported to the new position.Pruning removes kernels with low hit counts or energy; densification clones small kernels or splits large ones along their principal axis using the exponential map.Source: §4.3 · p. 9

4.4 Rendering & Applications

  • HKTex is evaluated at ray intersections in a differentiable physically based renderer and can be optimised against sampled points from a UV texture or pixels from multi-view images.For multi-view fitting, sampled camera rays hitting the mesh provide the rendering-error supervision.Source: §4.4 · pp. 9–10

5 Experiments

Across UV-texture fitting and multi-view reconstruction, HKTex combines compact storage with competitive or stronger reconstruction quality. The experiments evaluate it on filtered Objaverse meshes using image-quality and storage metrics.

5.1 Data

  • The curated Objaverse subset contains about 3,000 filtered models, excluding disconnected, non-manifold, complex, or failed-LBO meshes and limiting vertex counts to 60,000.The authors note that some filtering criteria can be relaxed, with lower dataset quality or separate processing for certain meshes.Source: §5.1 · p. 10

5.2 Implementation details

  • Kernel locations use geodesic Riemannian SGD, while other parameters use Adam; the pipeline precomputes eigendecompositions and differentiates through Mitsuba3 rendering.The implementation uses 64-dimensional isotropic-LBO decompositions for biharmonic distances and 256-dimensional ALBO decompositions over a 7 × 7 angle-anisotropy grid.Source: §5.2 · pp. 10–11

5.3 Evaluation metrics

  • Evaluation renders five views per object and measures MSE, PSNR, SSIM, MS-SSIM, LPIPS, and total storage, including intrinsic parameters, network weights, and baseline topology overhead.All methods use the same formats, without method-specific quantization, and identical zip compression.Source: §5.3 · p. 11

5.4 Fitting UV-textures

  • Under tight budgets, UV-coordinate overhead can dominate storage on high-vertex meshes, forcing downsampled textures to 4 × 4 pixels or ImageGS to 10 Gaussians with catastrophic performance penalties.Both image-based baselines rely on UV mapping, whose per-vertex coordinates consume part of the texture budget.Source: §5.4 · p. 13
  • On 313 meshes, HKTex has the lowest storage footprint after GT VTex and the best SSIM, MS-SSIM, and MSE, with second-best LPIPS.LR UV-textures achieve the highest PSNR; neural baselines and HR GT VTex need significantly larger budgets for almost comparable accuracy.Source: §5.4 · p. 13, Table 1 · p. 14
  • Qualitatively, HKTex preserves sharper texture boundaries and more faithful colour distributions, while neural fields oversmooth and ImageGS can produce artifacts under storage constraints.Vertex colours, including subdivided versions, can miss texture detail because their positions follow geometry rather than texture content.Source: §5.4 · p. 13

5.5 Fitting multi-view images

  • The multi-view setup fits rendered ground-truth texture images using rays sampled from random views and compares HKTex with positional-encoding and hash-encoding neural baselines.The NVDiffRec adaptation fixes the mesh and environment map to ground truth and outputs only albedo.Source: §5.5 · pp. 13–14
  • Across 162 meshes, HKTex outperforms high-resolution vertex colours and neural baselines across all metrics while using substantially less storage.The multi-view results include improvements in LPIPS and SSIM.Source: §5.5 · p. 14

6 Conclusion

HKTex replaces UV-mapped textures with anisotropic heat kernels defined intrinsically on the mesh surface. It supports continuous appearance evaluation in a differentiable physically based renderer and fitting from textures or multi-view images, while the evaluation focuses on albedo.

6 Conclusion

  • HKTex replaces UV-mapped textures with anisotropic heat kernels defined directly on the surface manifold.The conclusion states that this intrinsic formulation eliminates seams, distortions, and redundant storage associated with UV parametrisations.Source: §6 · p. 14
  • Geodesic kernels and Riemannian optimisation keep the representation aligned with the surface while enabling continuous evaluation in a differentiable physically based renderer.The representation supports compressing existing UV textures and reconstructing appearance from multi-view imagery without a global surface parametrisation.Source: §6 · p. 15
  • The evaluation focuses on albedo textures; extending kernels to spatially varying BSDF parameters such as roughness or specular components is future work.The authors propose optimizing those additional material parameters through differentiable rendering with Mitsuba3.Source: §6 · p. 15

S.1 Additional Implementation Details

The implementation combines surface-aware kernel initialization and splitting with renderer-based optimization, sampling, and mass estimation. It also reports practical scope boundaries and computational details of the representation.

S.1 Additional Implementation Details

  • Kernel initialization mixes uniform and UV-gradient-based positions for texture fitting, while power-skewed sharpness and threshold initializations bias kernels toward localized starts.For UV fitting, half the positions are uniformly sampled and half follow image gradients; multi-view positions are uniformly sampled over the surface.Source: §S.1 · pp. 20–21
  • Importance sampling records reconstruction errors after 250 uniform-sampling steps, while kernel mass estimation assigns inverse local density and normalizes masses to surface area.The error weights use an exponential moving average with decay 0.99; KDE density is computed from pairwise biharmonic distances with an adaptive median bandwidth.Source: §S.1 · pp. 21–22
  • Kernel splitting replaces a kernel with two geodesically displaced kernels along its major axis, then shrinks them and resets their optimizer states.The exponential-map displacement uses the kernel threshold as a spatial-radius proxy; new kernel extents are reduced by a factor of 1.6.Source: §S.1 · p. 22
  • Multi-view fitting uses a differentiable Mitsuba3 texture on a Principled BSDF, with sampled intersecting rays, mini-batches of 1,024, and an L1 photometric loss.Training uses 8 gradient samples per pixel; predictions and targets are clipped and transformed to gamma space before loss, with HDR clipping optional.Source: §S.1 · p. 23
  • Real-photo fitting is not used for physically accurate reconstruction because the current model optimizes albedo alone, which would bake lighting, shadows, and reflections into diffuse colour.The authors identify joint optimization of additional material properties as a feasible extension.Source: §S.1 · p. 24
  • The 7 × 7 ALBO eigendecompositions are cached per mesh and excluded from timing; recomputation takes 97.8 ± 286.9 seconds, and GPU parallelization is proposed to reduce sequential CPU overhead.HKTex depends on eigenmode count, with K = 256 reported as sufficient in the experiments; learnable thresholds can compensate for larger kernels when K is reduced.Source: §S.1 · pp. 24, 26
  • With K = 3 on a two-triangle rectangle, HKTex represented texture above mesh resolution; test meshes used about 4.3k kernels versus 9.6k vertices on average.On average, 49.8% of faces had no kernels, while kernel centers concentrated unevenly where they benefited texture quality.Source: §S.1 · p. 26
  • Direct editing is non-trivial, but a target region can be baked to a dense point cloud, painted, and incorporated through brief parameter optimization.Source: §S.1 · p. 26

S.2 Additional Experiments

HKTex produces sharper, more detailed fits than other methods while closely matching ground-truth texture quality. The ablations identify error-based kernel insertion as the main density-control mechanism and show differing roles for other components.

S.2 Additional Experiments

  • HKTex produces sharper, more detailed results than other methods and generally matches the visual quality of ground-truth textures.The reported comparisons cover UV-texture and multi-view image fitting.Source: §S.2 · p. 26
  • Removing error-based densification degrades quality, while removing pruning leaves it almost unchanged; density control without either does not recover the loss.The ablation compares component removals against the tuned full model on UV-texture fitting.Source: Table 4 · p. 26, §S.2 · p. 27
  • Removing colour-formation KNN produces results effectively tied with the full model, though the stage can offer small computational benefits.The authors leave its use as an implementation choice.Source: Table 4 · p. 26, §S.2 · p. 27
  • Removing biharmonic weighting degrades all image metrics, while removing importance sampling preserves competitive PSNR but worsens MSE.The authors attribute the latter pattern to uniform sampling overfitting easier regions and missing harder, high-frequency areas.Source: Table 4 · p. 26, §S.2 · p. 27
  • Removing PGD lowers performance, supporting explicit constraints on kernel parameters during optimisation.The ablation instead uses sigmoid activations for bounded geometric parameters and tanh for kernel colours.Source: Table 4 · p. 26, §S.2 · p. 27
  • Approximately 10× less storage is required for HKTex textures than for the original ground-truth textures in UV-texture fitting.The figure caption describes this as approximately one order of magnitude less storage.Source: Figure 7 · p. 28

Related papers

Loading 2609.07557v1…