Source-linked AI summary
PIE-PS: Photometric Stereo from Physical Irradiance Event Streams
Xiangze Meng, Guangyu Li, Jing Li, Di Mei, Songchen Ma, Mingkun Xu, Rui Ma
TL;DR
Event cameras provide normal cues under moving light, but the direct physics-only solver requires a fixed event contrast threshold and enough event pairs per pixel. PIE-PS builds graph-based context from paired event observations and grades their reliability; on DiLiGenT-Sim, it achieves lower average MAE than the reported baselines.
Problem
Event-based photometric stereo uses normal cues in moving-light event streams, but the direct solver requires a fixed contrast threshold and enough event-pair observations at each pixel.
Source: §1 · p. 2, §3.1 · p. 3Method
PIE-PS pairs adjacent same-pixel events with their light directions, represents each pair with a signed event-rate feature, and uses PIE-GNN and reliability grades to aggregate observations into dense normals.
Source: §1 · p. 2, §3.2 · p. 4, §3.3 · p. 5Results
On DiLiGenT-Sim, PIE-PS achieves 8.70° average MAE, 5.44° lower than EventUPS and 6.90° lower than the direct solver.
Source: §4.3 · p. 7Takeaways & Limitations
PIE-PS recovers dense surface normals from event streams using PIE observations and reliability grades before pixel aggregation.
Source: §3.2 · p. 4, §6 · p. 8PIE-PS assumes calibrated light directions, and strong specular highlights or complex inter-reflections can reduce PIE reliability and degrade normal estimation.
Source: §6 · p. 8
Abstract
from arXiv · showhide
Event cameras record asynchronous log-image-irradiance changes with microsecond latency and high dynamic range. These properties are useful for photometric stereo under moving illumination, but raw events are sparse and depend on an unknown contrast threshold. We start from the event trigger model and derive a physical relation between adjacent events, light motion, and surface normals. This relation gives a direct physics-only solver, but the solver needs the threshold, enough events at each pixel, and independent per-pixel optimization. To address these limits, we introduce PIE-PS, a learning-based framework for dense surface normal reconstruction from raw event streams and known lighting. We form Physical Irradiance Events (PIEs) by pairing two adjacent events at the same pixel with their corresponding light directions. Each PIE provides a Physical Irradiance Event Feature (PIEF), defined as the signed event rate. PIEF does not require the unknown contrast threshold. To share spatial and temporal context across nearby PIEs, we introduce PIE-GNN, which treats each PIE as a graph node and encodes it with its light-pair geometry. Since the reliability of PIE observations can vary with local appearance, illumination geometry, and sensor noise, Reliability-Grading Attention (RGA) predicts reliability weights to down-weight unreliable PIEs. Pixel aggregation then produces dense normals. Experiments on synthetic and real data show that PIE-PS outperforms prior event-based photometric stereo methods and the direct solver baseline.
1 Introduction
Event cameras’ microsecond latency and high dynamic range provide a useful sensing model for photometric stereo under moving light. PIE-PS builds on adjacent-event observations and combines physical cues with learned context and reliability weighting to predict dense normals.
1 Introduction
- Event cameras record asynchronous irradiance changes with microsecond latency and high dynamic range, providing normal cues under moving illumination.Source: §1 · p. 2
- Earlier event-based methods use per-pixel constraints, with sensitivity to contrast-threshold calibration, event density, or look-ahead events and without event polarity.EventPS depends on a fixed contrast threshold; PS-EIP uses three events, requires a look-ahead event, and does not use polarity.Source: §1 · p. 2
- The adjacent-event physical constraint supports a direct numerical solver, but it requires a known threshold, enough event pairs per pixel, and independent per-pixel optimization.The paper retains this solver as a physics-only baseline that makes the observation model explicit.Source: §1 · p. 2
- PIE-PS pairs adjacent same-pixel events with their light directions as Physical Irradiance Events, then uses signed event rate as a threshold-free feature.PIE-GNN encodes light-pair geometry and shares context across nearby observations; Reliability-Grading Attention weights PIEs before pixel aggregation predicts dense normals.Source: §1 · p. 2
- PIE-PS separates event-formation physics from learning: the physical model supplies event-pair and light-pair cues, while learning handles threshold, density, reliability, and spatial-context challenges.Its stated contributions include pairwise event photometry and a direct solver baseline, alongside PIEF, PIE-GNN, and RGA.Source: §1 · p. 2
2 Related Work
Prior photometric-stereo work addresses non-Lambertian effects and learns normals from calibrated or uncalibrated images, while event-based 3D vision spans several reconstruction tasks. Event-based photometric stereo is closer to PIE-PS, but prior approaches differ in sensor inputs, lighting assumptions, and event representations.
2.1 Photometric Stereo
- Photometric-stereo methods address shadows and specularities with robust optimization, sparse regression, low-rank modeling, and learning-based normal estimation.Related reflectance-acquisition methods jointly recover shape or normals with polarimetric SVBRDFs, whereas this work focuses on calibrated event-based dense normal recovery under moving illumination.Source: §2.1 · p. 2
2.2 Event-based 3D Vision
- Event-based 3D vision includes sparse depth, multi-view stereo, structured light, visual odometry, visual hulls, and implicit or Gaussian-splatting representations.These methods mainly target depth, tracking, novel-view synthesis, or 3D reconstruction rather than controlled-light pixel-wise normal estimation.Source: §2.2 · p. 2
2.3 Event-based Photometric Stereo
- EventPS uses a per-pixel differential constraint and depends on threshold and event density; PS-EIP uses a three-event interval profile but requires look-ahead and omits polarity.EventUPS extends event-based photometric stereo to uncalibrated illumination; PIE-PS instead uses adjacent-event pairs, retains timing, polarity, and light-pair geometry without a test-time contrast threshold, then applies graph context and reliability weighting.Source: §2.3 · p. 3
3 Methodology
PIE-PS derives event-pair photometric constraints and builds a learning-based pipeline for dense normal estimation from PIE observations. PIE-GNN and reliability grading handle threshold, event-density, and observation-reliability limits of direct per-pixel optimization.
3.1 Physical Modeling of Event Photometry
- Matching adjacent-event trigger measurements to light-motion irradiance yields a direct normal solver; PIEs package these pairs without C to address its threshold and event-density limits.Source: §3.1 · p. 3
3.2 Physical Irradiance Event Graph Neural Network
- PIEF is the signed event rate; PIE-GNN combines it with light-pair geometry and nearby-PIE context, then reliability-weighted pooling and pixel refinement predict dense normals.Source: §3.2 · pp. 3–4
3.3 Reliability-Grading Attention
- RGA scores time-ordered PIE embeddings at each pixel, assigning soft per-PIE weights that reduce unreliable observations' contribution during aggregation.Source: §3.3 · pp. 4–5
3.4 Training Strategy
- The network is trained with direct normal supervision using mean angular error on the final normal; at test time it needs only event pairs and light directions.Source: §3.4 · p. 5
4 Experiment
Across synthetic and real evaluations, PIE-PS reports lower MAE than the compared baselines and produces smoother, more accurate normals. The experiments also assess the reliability weights assigned by RGA.
4.1 Implementation Details
- Implementation uses a CeleX-V camera and programmable ring light; PIE-GNN uses 7-D nodes, 64-D embeddings, and 128-D hidden features, while RGA is a two-layer, four-head Transformer.Network training takes about 3.4 hours for 200 epochs on one NVIDIA H100 PCIe GPU.Source: §4.1 · p. 5
4.2 Datasets
- PIE-Sim has 77 training and 30 object-disjoint test scenes; DiLiGenT-Sim adds nine test scenes, and real evaluation uses three EventPS scenes plus three CAD-aligned captures.Source: §4.2 · p. 7
4.3 Quantitative Evaluation
- On DiLiGenT-Sim, PIE-PS averages 8.70° MAE, 5.44° below EventUPS and 6.90° below the direct solver.PS-EIP is omitted because its code and benchmark data are unavailable.Source: §4.3 · p. 7, Table 1 · p. 7
- On PIE-Sim, PIE-PS gives the lowest error in every group and averages 10.66° MAE, 11.22° below the direct solver and 15.18° below EventPS-FCN.Source: §4.3 · p. 7, Table 2 · p. 7
- PIE-PS has the best result on all six real scenes, averaging 10.38° MAE—10.53° below EventPS-FCN and 13.52° below the direct solver.Source: §4.3 · p. 7, Table 3 · p. 7
4.4 Qualitative Evaluation
- Across four settings, PIE-PS produces smoother, more accurate normals; competing event networks show local errors, while independent pixel solving yields noisy normals.Source: Figure 5 · p. 6, §4.4 · p. 7
- Low-RGA regions overlap GT-identified unreliable PIE regions more than reliable regions, supporting RGA as a soft reliability-weighting module.Source: Figure 6 · p. 6, §4.4 · p. 7
5 Ablation Study
The ablations show that both the signed event-rate feature and learned reliability weighting contribute to PIE-PS on PIE-Sim. RGA also assigns lower average weights to ground-truth-identified unreliable PIEs.
5 Ablation Study
- Removing PIEF increases average MAE from 10.66° to 14.52°, showing that light-pair geometry alone lacks the signed event response to light changes.The ablation removes the signed event-rate input while keeping light-pair geometry and the rest of the network unchanged.Source: §5 · pp. 7–8, Table 4 · p. 8
- Removing RGA increases average MAE to 11.99°, while ground-truth-identified unreliable PIEs receive lower weights on average.The weight distribution supports RGA as a soft reliability-weighting module.Source: §5 · p. 8, Figure 7 · p. 8, Table 4 · p. 8
6 Conclusion and Discussion
PIE-PS uses PIEs and PIE-GNN to recover dense surface normals from event streams, with RGA grading observations before pixel aggregation. The method assumes calibrated light directions; strong specular highlights or complex inter-reflections can reduce PIE reliability and degrade normal estimation.
6 Conclusion and Discussion
- PIE-PS uses PIEs and PIE-GNN to recover dense surface normals from event streams, with RGA assigning soft reliability grades before pixel aggregation.The authors identify extension to uncalibrated lighting and more general reflectance as future work.Source: §6 · p. 8
- Calibrated light directions remain an assumption, while strong specular highlights or complex inter-reflections can reduce PIE reliability and degrade normal estimation.The authors identify uncalibrated lighting and more general reflectance as future extensions.Source: §6 · p. 8