PaperlayerView exact revision on arXiv

Source-linked AI summary

PointGT: Simultaneous Geometry and Texture Editing for Point-Based Representations

Yanshu Zhang, George Shramko, Pratul P. Srinivasan, Ke Li

arXiv:2609.03341v1cs.CVcs.GR

TL;DR

Existing volumetric and Gaussian-based representations are difficult to edit, especially when fine texture editing must coexist with geometry deformation. PointGT combines an attention-based point representation, learned UV mapping, geometry regularization, and deformation-aware correspondence to support simultaneous editing. The method preserves texture edits under large deformations and outperforms competitive baselines on novel view synthesis, while attention-weighted surface points remain sensitive to point geometry and weight concentration.

  • Problem

    Existing reconstructed representations, including 3DGS, couple geometry and appearance or lose valid texture mappings under deformation, limiting simultaneous fine texture and geometry editing.

  • Method

    PointGT extends PAPR with learned UV mapping, geometry regularizers, and deformation-aware canonical correspondence for editable geometry and appearance.

  • Results

    PointGT preserves texture edits under large deformations and outperforms competitive baselines on novel view synthesis.

  • Takeaways & Limitations

    PointGT enables simultaneous geometry and texture editing in point-based neural representations while maintaining high rendering quality.

  • Takeaways & Limitations

    Attention-weighted surface points are not true geometric intersections and depend on concentrated weights and well-placed supporting points.

Abstract

from arXiv · show

We present PointGT, a point-based 3D representation that enables simultaneous editing of object geometry and appearance. Existing reconstruction and view synthesis techniques produce volumetric 3D representations that are high-quality and photorealistic, but are difficult to edit. In particular, recent efforts to enable texture editing for 3D Gaussian Splatting representations are not compatible with geometry edits and deformations. Our method combines a point-based representation that is well-suited for geometry deformations with a learned UV mapping technique that enables high-resolution texture editing. We show that PointGT enables fine-grained editing of both geometry and texture in point-based neural representations with high rendering quality.

1 Introduction

PointGT extends attention-based point rendering with learned UV mapping and deformation-aware correspondence, enabling simultaneous geometry and appearance editing despite limitations of Gaussian-based representations.

  • 3DGS couples geometry and appearance resolution, making fine texture edits and non-rigid Gaussian transformations difficult.
  • Attention-based point renderers support geometry editing because interpolation weights remain valid after deforming scene points.
  • PointGT extends PAPR with learned UV mapping from 3D points to a learned 2D texture map for appearance editing.
  • PointGT introduces geometry regularizers that improve PAPR geometry fidelity and are crucial for effective UV mapping.
  • Deformation-aware correspondence maps deformed-space ray intersections back to canonical space, preserving texture attachment during non-rigid edits.

2 Related Work

Related work spans editable mesh and neural representations, but point-based neural methods still struggle to preserve high-resolution textures during deformation because their neighborhoods and weights can shift.

  • NeRF editing commonly uses editable proxy geometry, but conversions can introduce approximation errors and constrain possible edits.
  • Mesh representations support appearance edits through explicit UV coordinates, whereas unstructured point clouds lack connectivity and stable texture-supporting neighborhoods.
  • PointGT targets canonical correspondence within dynamic attention fields to enable robust texture persistence in deformable point-based representations.
  • 3DGS offers straightforward particle transformations but requires many primitives and remains inconvenient for fine appearance edits and free-form deformation.
  • Global and per-primitive Gaussian texture maps address appearance decoupling but do not support simultaneous geometry deformation and texture editing.

3 Preliminaries

Attention-based point renderers represent scenes with unstructured point clouds and interpolate neighboring point features and positions to render rays and estimate surface intersections. Neural UV mappings then associate canonical surface points with texture coordinates, but deformations require correspondence because direct UV lookup becomes invalid.

  • 3.1 Attention-Based Point Renderers: Attention-based renderers represent scenes as learnable point clouds and select the K nearest points to each ray using perpendicular distance.A learned cross-attention mechanism assigns softmax weights to the selected neighbors.
  • 3.1 Attention-Based Point Renderers: Ray features are computed as weighted sums of neighboring features, while surface intersections are estimated by applying the same attention weights to point positions.The aggregated feature map is decoded to predict RGB colors.
  • 3.1 Attention-Based Point Renderers: The estimated surface point is an attention-weighted average rather than a true geometric intersection, so its quality depends on concentrated weights and well-placed supporting points.This sensitivity motivates the geometry regularizers introduced later.
  • 3.2 Neural UV Mapping: Canonical point positions and intersections provide the training domain for UV mapping, but deformation moves points into a space where direct UV lookup is no longer valid.PointGT therefore targets canonical correspondence within dynamic attention fields.
  • 3.2 Neural UV Mapping: Neural UV methods learn continuous mappings from 3D surface points to 2D texture atlases using cycle-consistency and distortion regularizers.PointGT uses a normal-free distortion loss because raw attention-interpolated points do not explicitly provide surface normals.
  • 3.2 Neural UV Mapping: PointGT maintains surface continuity after deformation by predicting ray–surface points through attention-based interpolation on a point cloud.The comparison contrasts this behavior with textured Gaussian splatting under large non-rigid deformations.

4 Method

PointGT combines geometry-regularized attention-based point rendering with learned UV mapping and deformation-aware canonical correspondence to support persistent, simultaneous geometry and texture editing.

  • 4 Method: Attention-based point rendering provides smooth interpolation that remains suitable for geometry deformation, unlike representations whose texture resolution is tied to volumetric primitives.PAPR interpolates features from K nearby points, while Gaussian-based methods struggle with fine deformations and persistent texture mapping.
  • 4.1 Learning PAPR with Geometry Regularization: Geometry regularizers improve ray–surface intersections by enforcing proximity to rays and encouraging selected neighbors toward predicted surface points.The combined objective adds close-to-ray and close-to-surface terms to PAPR’s rendering loss; the surface term is introduced after coarse geometry is learned.
  • 4.2 Learning UV Mapping: PointGT learns a multi-chart UV atlas and texture map from canonical ray–surface intersections for subsequent texture editing.The atlas is optimized with UV-mapping losses and Jacobian-based distortion regularization, while texture colors are supervised through bilinear atlas sampling.
  • 4.2 Learning UV Mapping: Jacobian-based distortion losses reduce anisotropic scaling and prevent degenerate texture mappings without requiring explicit surface normals.The distortion objective combines scaling and area penalties, with the area term discouraging shrinkage, blow-up, and near-zero regions.
  • 4.3 Geometry and Texture Editing: PointGT maps deformed-space ray–surface intersections back to canonical space through displacement fusion, preserving texture attachment under non-rigid edits.The correspondence mechanism transfers deformed intersections to canonical coordinates for consistent UV lookup.
  • 4.3 Geometry and Texture Editing: PointGT permits geometry and texture edits in either order, supporting iterative workflows in which both edit types are applied repeatedly.Texture edits replace sampled colors in regions identified by comparing the original and edited texture atlases.

5 Experiments

PointGT is evaluated for simultaneous geometry and texture editing under non-rigid deformation, novel view synthesis, and component ablations. It preserves attached edits, outperforms or matches GSTex across VBench dimensions, outperforms novel-view baselines, and benefits from its regularizers and deformation-aware correspondence.

  • Simultaneous Geometry and Texture Editing: PointGT preserves sharp texture edits attached to deforming surfaces under non-rigid motion, including local and global edits.Qualitative results report surface continuity without noticeable drifting or popping.
  • Simultaneous Geometry and Texture Editing: PointGT maintains sharper local edit details than GSTex across tested point budgets, while GSTex becomes blurry at some budgets.The comparison pastes a rocket icon into the same butterfly-scene region for both methods.
  • Simultaneous Geometry and Texture Editing: PointGT either outperforms or matches GSTex across all five VBench dimensions for orbit videos of edited, animated scenes.The evaluation uses five Objaverse scenes with artist-created motions and edited textures.
  • Novel View Synthesis: PointGT outperforms novel-view synthesis baselines both with uncapped primitives and with all methods capped at 5,000 primitives.The comparison covers Blender and DTU against PAPR, Texture-GS, GSTex, NeST-Splatting, and Textured Gaussians.
  • Ablation Study: On-ray regularization produces continuous surface points and clearer texture details, while on-surface regularization reduces noisy points, holes, and depth-map artifacts.The ablations connect improved surface sampling and geometry quality to sharper renderings and smoother depth maps.
  • Ablation Study: Increasing distortion-loss weight reduces UV distortion, while deformation-aware canonical transfer avoids UV jumps, texture drift, and popping.The correspondence ablation compares the proposed transfer with naïve interpolation using deformed-space attention weights.

6 Conclusion and Limitation

PointGT combines a PAPR-based point representation with learned UV mapping, geometry regularizers, and deformation-aware correspondence for simultaneous geometry and texture editing. It preserves edits under large deformations and improves novel-view synthesis with fewer primitives, while complex topology remains a limitation for its optimization-based UV parameterization.

  • Conclusion: PointGT enables simultaneous geometry and texture editing through learned UV mapping, geometry regularizers, and deformation-aware correspondence.The representation is based on PAPR and maps 3D points to a learned 2D texture map.
  • Conclusion: PointGT preserves texture edits under large deformations and outperforms competitive baselines for novel-view synthesis with fewer primitives.The conclusion also reports fewer artifacts and surface discontinuities than the baseline.
  • Limitation: The optimization-based UV parameterization can struggle to produce clean, low-distortion charts for objects with complex topology.The paper identifies learning-based UV mapping as a possible future direction.

A Multi-View Editing Results

Across five Objaverse scenes and 360° orbit renders, PointGT keeps edited textures sharp and attached to objects undergoing non-rigid deformation. The multi-view results show consistent appearance without visible seams, drifting, or loss of resolution.

  • Multi-View Editing Results: Across five Objaverse scenes, edited textures remain sharp and firmly attached across 360° views of non-rigidly deformed objects.The displayed views are evenly spaced samples from the same orbit renderings used for VBench evaluation.
  • Multi-View Editing Results: The orbit renders show no visible seams, drifting, or loss of resolution, supporting view consistency and edit attachment under large deformation.Figure 10 separately illustrates sequential part removal with learned UV charts and rendered texture maps.

A.1 Part Removal and Duplication

PointGT supports topology-changing edits while keeping textures attached through canonical UV mapping. On the Blossom scene, part removal and duplication are demonstrated with corresponding UV visualizations, rendered results, and texture maps.

  • Part Removal and Duplication: Canonical UV mapping lets PointGT keep edited textures attached when geometry is changed by removing or duplicating parts.The Blossom demonstrations cover sequential petal removal and duplicated geometry under texture edits.
  • Part Removal and Duplication: For part removal, petals are sequentially detached while the learned charts, rendered result, and 2D texture map show the editable remainder.The figure presents the removal sequence from top to bottom.
  • Part Removal and Duplication: For part duplication, the duplicated geometry is shown under two texture edits, with texture remaining correctly attached through canonical UV mapping.The figure pairs checkerboard UV charts with edited renders and corresponding texture maps.

B Additional Ablation Results

The ablations show that both the on-ray regularizer and deformation-aware correspondence improve texture-editing fidelity, with the regularizer reducing artifacts.

  • The on-ray regularizer significantly reduces texture-editing artifacts, while deformation-aware correspondence boosts editing fidelity.

C Discussion of Other Gaussian Splatting Baselines

Additional comparisons show that PointGT preserves sharp, continuous texture edits under deformation, addressing limitations of global and per-Gaussian texture representations.

  • PointGT keeps texture edits sharp and continuous after deformation, unlike Texture-GS, which suffers broken surface continuity and resolution loss.
  • Texture-GS may fail to learn a texture map suitable for reliable texture editing on complex geometries.
  • Textured Gaussians and GSTex attach learnable per-Gaussian texture maps queried through ray–Gaussian intersections, whereas NeST-Splatting uses a global world-space hash-grid texture.

D Additional Novel View Synthesis Results

PointGT achieves rendering quality comparable to or better than textured Gaussian baselines while using 30,000 primitives, with additional evaluations on MipNeRF360.

  • PointGT achieves rendering quality comparable to or better than the baselines while using only 30,000 primitives.The qualitative comparisons use either 5,000 primitives or each method’s originally suggested count.
  • The MipNeRF360 evaluation reports per-scene PSNR, SSIM, and LPIPS comparisons against Gaussian splatting, textured Gaussian baselines, and PAPR.PAPR and PointGT were evaluated at half resolution because full-resolution training exceeded available GPU memory.
  • The supplementary comparisons include simultaneous geometry-and-texture editing on the Butterfly scene and qualitative novel-view synthesis with 5,000 primitives.
Loading 2609.03341v1…