Source-linked AI summary
PlenoCI: Plenoptic CharacterIstics for View Dependence Aware Change Classification
Jason Lai, Chamuditha Jayanga Galappaththige, Niko Suenderhauf, Dimity Miller, Donald G. Dansereau
TL;DR
Radiance fields capture view-dependent phenomena, but are underconstrained, and SCD methods often suppress view-dependent cues as inconsistencies. PlenoCI detects plenoptic structures using analytic derivatives of a 3DGS-regressed field, within an instance-aware multiview change pipeline. It reports substantially fewer false positives on unchanged reconstructions and strong benchmark performance on CL-Splats and PASLCD.
Problem
3DGS captures view-dependent phenomena but is underconstrained, while recent SCD methods treat view-dependent effects as inconsistencies to suppress.
Source: §1 · p. 1Method
PlenoCI uses closed-form analytic derivatives of the 3DGS-regressed plenoptic field to detect 5D structures within an instance-aware pipeline that aggregates change cues across views.
Source: §1 · pp. 1–2Results
Two orders of magnitude fewer false positives are reported for unchanged independently optimized scenes; on CL-Splats, the method improves mIoU by 25.7% over the strongest competitor and closely matches the strongest PASLCD performance.
Source: §1 · p. 1, §5.1 · p. 6Takeaways & Limitations
The paper demonstrates PlenoCI's utility for distinguishing geometric from appearance-based changes.
Source: §6 · p. 8Under severe coverage asymmetry, sparse views can degrade reconstruction and produce spurious ray-difference cues, increasing the false-positive rate.
Source: §8.3 · p. 14
Abstract
from arXiv · showhide
Radiance field representations such as 3D Gaussian Splatting (3DGS) natively encode complex visual phenomena such as occlusions and view dependence, but they are inherently underconstrained. Independently optimized reconstructions converge to different primitive configurations, even in unchanged regions. We introduce Plenoptic CharacterIstics (PlenoCI), a novel feature built from the plenoptic field these representations approximate. PlenoCI directly captures rich visual behaviors while ignoring Lambertian textures. By deriving closed-form analytic plenoptic derivatives from a 3DGS representation, we efficiently detect these 5D structures. Our approach is robust to underconstrained representations by construction, reporting two orders of magnitude fewer false positives between independent reconstructions of unchanged scenes than concurrent work. We demonstrate PlenoCI's utility on change classification. First, we detect changes with an instance-aware 3DGS pipeline, achieving state-of-the-art results on CL-Splats with a 25.7% mIoU gain over the strongest competitor, while remaining competitive on the more challenging PASLCD benchmark. Leveraging PlenoCI, we classify changes as geometric or appearance-based with a balanced accuracy of 0.735, comparable to the best performing baseline. We believe plenoptic derivatives and PlenoCI open new directions for view dependence aware understanding in visually complex environments. Code and data are available at https://js0n-lai.github.io/plenoci.
1. Introduction
Radiance-field methods capture complex, view-dependent phenomena, but scene-change detection (SCD) methods often treat them as inconsistencies to suppress. PlenoCI instead uses structures in the plenoptic field for change detection and classification, with robustness to independently optimized 3DGS representations.
1. Introduction
- SCD and change classification support autonomous applications including damage assessment, environmental monitoring, and map updating.For long-term robotic monitoring, distinguishing genuine geometric and appearance changes from lighting fluctuations supports richer scene understanding and advanced reasoning.Source: §1 · p. 1
- Existing radiance-field SCD methods treat view-dependent effects as inconsistencies to suppress, although these representations capture occlusion edges, specularity, and reflections.The paper instead treats view dependence as a signal of complex visual behavior.Source: §1 · p. 1
- PlenoCI derives closed-form analytic plenoptic derivatives from 3DGS and uses them to detect view-dependent structures while bypassing differences between independently optimized primitives.For unchanged scenes, it reports a false-positive rate two orders of magnitude lower than concurrent primitive-space comparison.Source: §1 · pp. 1–2
- An instance-aware 3DGS pipeline aggregates change cues across views, then uses PlenoCI as a geometry proxy to classify changes as geometric or appearance-based.The pipeline first detects scene changes and then classifies them using view-dependent plenoptic structures, such as occlusion-edge evidence.Source: §1 · p. 1, Figure 1 · p. 2
2. Related Work
Prior work spans image-based SCD, radiance-field methods, and plenoptic analysis. This paper distinguishes its approach by retaining view-dependent structures and deriving analytic plenoptic derivatives from 3DGS.
2.1. 2D Scene Change Detection
- Early 2D SCD relies on aligned image pairs and costly annotations; foundation-model methods enable zero-shot detection but remain brittle to viewpoint discrepancies and view-dependent inconsistencies.The latter approaches can inherit backbone failure modes.Source: §2.1 · p. 2
2.2. 2D-3D Scene Change Detection
- Radiance-field SCD synthesizes pre-change scenes from post-change viewpoints and aggregates multiview cues, but existing methods suppress view-dependent evidence that this work analyzes directly.Concurrent primitive-space change detection models reconstruction drift but discards higher-order spherical harmonics encoding view dependence.Source: §2.2 · pp. 2–3
2.3. Leveraging the Plenoptic Field
- Earlier plenoptic derivatives assume Lambertian scenes or use dense numerical sampling; 3DGS regression provides closed-form analytic derivatives and supports efficient detection of 5D structures.The approach generalizes beyond light fields.Source: §2.3 · p. 3
3. PlenoCI: A New Plenoptic Feature
PlenoCI identifies plenoptic structures associated with complex visual behavior by measuring how ray color changes across position and direction. The paper derives analytic derivatives from 3DGS, uses a structure tensor to identify these patterns, and generalizes the feature from flatland to 5D scenes.
3. PlenoCI: A New Plenoptic Feature
- PlenoCI captures rays whose color changes across multiple position and direction axes, signaling complex behaviors such as occlusion edges and specularity.It is designed to summarize informative plenoptic structures while ignoring Lambertian textures.Source: §3 · p. 3
3.1. Toy Example: Flatland
- In flatland, the plenoptic field varies over two spatial dimensions and in-plane rotation, and is learned from posed 1D images using a radiance-field representation.The representation contains 2D Gaussian primitives with view-dependent color.Source: §3.1 · p. 3
3.1.1. Extracting 3D Plenoptic Derivatives
- Plenoptic derivatives measure ray-intensity sensitivity to origin or direction shifts, and 3DGS enables closed-form analytic derivatives of the plenoptic field.For the geometric term, the method evaluates intersecting Gaussians at each ray's point of maximum response.Source: §3.1.1 · p. 3
3.1.2. The Plenoptic Structure Tensor
- The plenoptic structure tensor smooths gradients and uses eigenvalues to distinguish flat regions, one-axis-sensitive textures or edges, and multi-axis-sensitive complex behaviors.Two large eigenvalues indicate behaviors such as occlusion edges and specularity; three large eigenvalues are highly unlikely in free space.Source: §3.1.2 · p. 4
3.1.3. PlenoCI in Flatland
- In flatland, PlenoCI selects rays where the structure tensor has two large eigenvalues, using the threshold λ1(r) ≥ τ1.The paper illustrates these rays at an occlusion edge.Source: §3.1.3 · p. 4
3.2. Generalizing to 5D
- For real 3D scenes, the plenoptic field has five dimensions: three for position and two for ray direction.The paper generalizes the flatland PlenoCI pipeline to this 5D field.Source: §3.2 · p. 4
- To avoid sampling 5D volumes, the method samples analytic local-light-field derivatives: one pinhole camera provides (ϕ, θ), while a fixed-spacing 3×3 camera array provides (x, y, z).It uses 2D Gaussian kernels for spatial and angular windows and scales spatial L* by the median rendered depth before constructing ∇L*.Source: §3.2 · p. 4
- In 5D, PlenoCI thresholds the third eigenvalue λ2 and masks the result with a Canny edge map; λ2 responds to occlusion edges and specularity.The paper also reports sensitivity of λ0 to edges and λ1 to corners.Source: §3.2 · p. 4
4. Change Detection and Classification
The section builds an instance-aware 3DGS pipeline that aggregates change evidence into sparse anchors and dense masks, then uses PlenoCI to distinguish geometric from appearance-based changes.
4. Change Detection and Classification
- After SfM registration, instance-aware 3DGS accumulates per-ray change evidence into sparse anchors and propagates it through instance cues to render dense change masks.The reconstructions share a coordinate frame.Source: §4 · p. 4
- PlenoCI is formed by masking the third structure-tensor eigenvalue λ2 with an edge image; repeated extraction across views yields the feature set R.The analytic plenoptic derivatives are sampled over a local light field, and λ2 captures complex visual behavior while ignoring Lambertian textures.Source: Figure 3 · p. 5
4.1. Finding Changes
- For each pose, edge rays compare RGB and DINOv2 features across times, with boundary pixels shifted foregroundward and both discrepancy terms equally weighted.The RGB discrepancy is L1 error, while feature discrepancy is cosine dissimilarity; both weights are 0.5.Source: §4.1 · p. 5
- The pipeline assigns ray evidence to dominant primitives and pools evidence from both times, resolving ambiguity when changed-object edges or disocclusions affect multiple primitives.This produces sparse anchors for later propagation across the changed instance.Source: §4.1 · pp. 5–6
4.2. Classifying Changes with PlenoCI
- PlenoCI weights occlusion-edge changes as geometric evidence; other changed primitives are labeled appearance-based, although specularity can also trigger its signal.Geometric labels additionally require instance-level mean and anchor-fraction thresholds and minimum geometric support.Source: §4.2 · p. 6
5. Experiments
The experiments evaluate binary change detection and geometric-versus-appearance classification on CL-Splats and PASLCD. Results show a strong CL-Splats advantage, competitive PASLCD performance, and lower false-positive rates across independent reconstructions of unchanged scenes.
5. Experiments
- Evaluation used CL-Splats (five simple-change scenes) and PASLCD (ten scenes with concurrent geometry and appearance changes); SCD tested held-out frames, while PlenoCI used 20 post-change views.The 20 views were all PASLCD training views and a farthest-point-sampled subset for CL-Splats.Source: §5 · p. 6
5.1. Binary Change Detection Results
- 25.7% mIoU improvement over the strongest competitor on CL-Splats; the method closely matches the strongest PASLCD performance.The comparison averages binary SCD results over both benchmarks.Source: §5.1 · p. 6, Table 1 · p. 7
5.2. Ablations
- On PASLCD, combining all change-scoring elements performs best; occupancy gating limits propagation from a few false-positive anchors while preserving sufficiently supported large changed instances.The ablation uses identical reconstructions and rays.Source: §5.2 · p. 6, Table 2 · p. 7
- Replacing learned instance IDs with ground-truth oracles raises recall by 20% while precision stays stable, consistent with learned undersegmentation of small changed objects.The oracle instances are formed by lifting ground-truth mask pixels to primitives and grouping nearby primitives.Source: Table 3 · p. 7, §5.2 · p. 7
- Querying all rays instead of only edge rays drops mIoU by 0.021 while considerably increasing compute costs.Source: §5.2 · p. 7
5.3. Multi-class Change Classification Results
- On PASLCD, PlenoCI-based classification is competitive with GS-Diff, but 84% Geo pixels amplify App errors and occlusion/specularity cues or object replacements can cause misclassification.Oracle instances improve recall for both classes.Source: Table 4 · p. 8, §5.3 · p. 8
5.4. Robustness to Reconstruction Ambiguity
- Unlike GS-Diff's primitive comparison with drift modeling and an accurate-geometry assumption, PlenoCI is robust to underconstrained primitives; false-positive rates are consistently lower across independent unchanged-scene reconstructions.The false-positive-rate comparison uses independently optimized 3DGS models; scenes with real changes average 3.51% change pixels, ranging from 0.17–20.12%.Source: Table 5 · p. 8, §5.4 · p. 8
6. Conclusion
PlenoCI captures complex visual behaviors while remaining robust to underconstrained representations. The paper derives closed-form analytic plenoptic derivatives from 3DGS and demonstrates PlenoCI for distinguishing geometric from appearance-based changes.
6. Conclusion
- PlenoCI explicitly captures complex visual behaviors discarded by existing methods while remaining robust to underconstrained representations.Source: §6 · p. 8
- Regressing the plenoptic field with 3DGS yields closed-form analytic derivatives.Source: §6 · p. 8
- The paper demonstrates PlenoCI for distinguishing geometric from appearance-based changes and suggests applications including localization, navigation, map stitching, visual odometry, and uncertainty estimation.These applications are proposed as possibilities for future use.Source: §6 · pp. 8–9
7. Plenoptic Gradients Derivation
The section derives plenoptic derivatives by differentiating volumetric ray color and then differentiating Gaussian fall-off with respect to ray-Gaussian geometry. It gives spatial and angular derivative routes and describes how the terms are accumulated and stabilized.
7.1. Volumetric Rendering
- Volumetric ray color is differentiated with respect to the ray, using the product rule and a logarithmic derivative of transmittance before substitution into the resulting expressions.The derivation defines C_i = c_iα_i, T_i as accumulated transmittance, and L_i = C_iT_i.Source: §7.1 · p. 12
7.2. Ray-Gaussian Intersection
- The geometric fall-off derivative starts from the ray’s projection into canonical Gaussian space, where the Mahalanobis distance describes its perpendicular distance.The ray is transformed using the Gaussian center, scale matrix, and rotation matrix.Source: §7.2 · p. 12
- Spatial derivatives are evaluated directly, while angular derivatives use the chain rule for normalization and canonical projection with spherical-coordinate directions.Azimuth ϕ spans [−π, π] and elevation θ spans [−π/2, π/2].Source: §7.2 · p. 12
- Per-Gaussian derivative terms are alpha-blended using standard 3DGS rasterization, and ε = 10^-3 is added to the scale matrix to stabilize flat primitives.The scale lower bound causes a small drop in render quality.Source: §7.2 · p. 12
8. Additional Visualizations
These visualizations show how plenoptic eigenvalues respond to scene structure and how instance quality, ray selection, and reconstruction coverage affect change detection. The robustness comparison finds fewer false positives than GS-Diff, while severe coverage asymmetry still increases false positives.
8. Additional Visualizations
- In the plenoptic structure tensor, λ0 responds to edges, λ1 to corners, and λ2 to occlusion edges, specularity, and reflections.The figure visualizes these pixelwise eigenvalue signals alongside the input RGB view.Source: §8 · p. 12, Figure 7 · p. 13
8.1. Instance Segmentation
- Oracle instances mitigate false negatives when undersegmentation prevents localized change anchors from propagating across changed objects.The examples show that the anchors can correctly localize changes even when the instance representation is undersegmented.Source: §8.1 · pp. 12–13, §8.3 · p. 14
8.2. Ray Selection Comparison
- Edge rays occupy 3.6% of the frame and retain comparable change-detection performance while substantially reducing downstream change-scoring compute.The extraction comparison averages PASLCD scenes using 20 sample views; extraction FLOPs exclude shared rasterization.Source: §8.2 · p. 13, Table 6 · p. 14
8.3. Robustness to Reconstruction Ambiguity
- Across unchanged Cantina reconstructions, PlenoCI reports fewer false positives than GS-Diff, though severe coverage asymmetry raises its false-positive rate.With only five of 20 training views, poor reconstruction quality produces false positives, but at a lower rate than GS-Diff.Source: §8.3 · p. 14, Table 7 · p. 14, Figure 9 · p. 15