Source-linked AI summary
Differentiable Mesh State Estimation via Factor Graph Inference for Deformable Object Reconstruction
Lidia Al-Zogbi, Fangjie Li, Samuel Tobin, James Ferguson, Nithesh Kumar, Alejandro Chara, Kuan-I Chung, Mingxing Rao, Ayberk Acar, Susheela Sharma Stern, Robert Webster, Daniel Moyer, Alan Kuntz, Caleb Rucker, Tucker Hermans, Jie Ying Wu
TL;DR
Deformable-object reconstruction must estimate high-dimensional mesh states from partial, noisy, and nonlinear observations while maintaining physical plausibility. The paper proposes a factor-graph estimator that jointly combines physics priors, measurements, and temporal consistency in nonlinear least squares, achieving reliable reconstruction in simulations and ex vivo central-airway-obstruction experiments.
Problem
Updating time-varying tetrahedral meshes from point-cloud observations is a challenging inverse problem involving high-dimensional states, partial observability, nonlinear objectives, and measurement–physics trade-offs.
Method
A factor graph directly estimates tetrahedral-mesh vertex states by combining physics-based priors, sensor likelihoods, and temporal consistency in a nonlinear least-squares formulation solved with Levenberg–Marquardt.
Results
The method closely tracks ground-truth deformations in simulations and ex vivo central-airway-obstruction experiments, with the physics prior identified as the principal contributor in the reported simulation experiment.
Takeaways & Limitations
The probabilistic formulation supports measurement-driven and physically constrained mesh state estimation for deformable-object reconstruction.
Abstract
from arXiv · showhide
Estimating deformable object states remains a fundamental challenge in robotics and simulation. We propose a novel factor graph-based framework for probabilistic mesh state estimation of deformable objects. The method directly updates a tetrahedral mesh, a rich and physically-grounded representation of an environment, by combining physics priors, noisy sensor measurements, and temporal smoothness constraints within a unified probabilistic formulation. The estimation problem is posed as a nonlinear least-squares optimization and solved using Levenberg-Marquardt. Ex vivo central-airway obstruction experiments and simulations on deforming cube models demonstrate reliable and accurate reconstruction under both rigid motion and deformation, highlighting the potential of this probabilistic approach for principled, measurement-driven mesh state estimation in deformable object reconstruction.
I. INTRODUCTION
Updating time-varying tetrahedral meshes from point-cloud observations is a challenging inverse problem because states are high-dimensional, partially observed, nonlinear, and constrained by competing measurement and physics requirements. The proposed factor-graph formulation addresses these challenges through joint spatiotemporal optimization that combines physics, measurements, and temporal consistency.
- Tetrahedral-mesh state updating supports compliant manipulation, predictive simulation, and real-time digital twinning but involves high-dimensional, partially observed, nonconvex estimation.
- Existing methods separately address uncertainty propagation, incremental filtering, physical consistency, or tractability but lack a unified solution.
- The proposed factor graph performs joint spatiotemporal optimization by representing mesh vertices as states and encoding physics priors, measurement likelihoods, and temporal consistency as factors.
- Sparse matrix structures arising from mesh connectivity support computational efficiency and systematic fusion of multiple data modalities.
II. RELATED WORK
The paper connects factor-graph inference, differentiable geometric optimization, deformable modeling, and physics-informed estimation into a mesh-state framework. Its factors combine differentiable surface measurements, stiffness-based physical priors, and temporal evolution over a tetrahedral mesh.
- Differentiable residuals let point-to-surface measurements, temporal regularization, and stiffness-based priors contribute gradients to one mesh-state optimization.
- The approach combines observation likelihoods with mechanical regularization and temporal consistency for sparse, partial, noisy, or locally ambiguous observations.
- The mesh state consists of 3D coordinates for all vertices, while connectivity is assumed known and unchanged over time.
- The factor graph uses physics, measurement, and temporal factors to impose probabilistic constraints on selected mesh variables.
- A simulator predicts the next mesh state, and the stiffness matrix supplies physics-based information for the prior factor.
- Closest-point projection associates each measurement with a surface triangle and is differentiable with respect to that triangle’s vertex positions while the selected triangle remains fixed.
B. Optimization
The MAP estimator is written as a nonlinear least-squares problem over all mesh states and factor residuals, with new states appended as time advances. The resulting damped objective is solved iteratively using Levenberg–Marquardt in a Python/PyTorch prototype.
- The MAP problem optimizes jointly over the sequence of mesh states X = {X0, . . . , Xt} using a nonlinear least-squares objective.
- As time advances, new states and factors are appended while preserving information from past states, increasing the optimization dimension.
- Each factor contributes a weighted quadratic residual cost, with residuals ri(X) scaled by information matrices Λi.
- The loss is solved iteratively with the damped Levenberg–Marquardt algorithm in a Python 3.12 implementation using PyTorch automatic differentiation and accelerated numerical computation.
1) Ablation Studies:
The study evaluates factor-graph mesh estimation in simulated deforming cubes and an ex vivo airway-intervention model. It tests whether physics, measurements, and temporal structure support reconstruction under partial observations and deformation.
- Simulation ablations: The ablation compares temporal-only, measurement-only, combined temporal-measurement, simulation-measurement, and full simulation-measurement-temporal configurations.The study uses rigid translation, SPD-prior deformation, and non-SPD-prior deformation with noisy synthetic surface measurements.
- Simulation ablations: Across simulated cases, reconstruction remained stable with relatively small error even in poorly conditioned settings, although deformable and non-SPD-prior experiments increased overall RMSE.Errors were evaluated on all vertices, external vertices, and unobserved external vertices.
- Clinical motivation: Accurate airway-state estimation matters because CAO intervention occurs in narrow, deformable, partially visible anatomy where bleeding, perforation, hypoxia, or loss of airway control can be fatal.These conditions motivate maintaining airway and tumor geometry estimates during manipulation and cutting.
- Ex vivo evaluation: The ex vivo experiment used a Virtuoso robot, sheep-trachea CAO phantoms with approximately 50% tumor occlusion, robotic deformation, registered CT ground truth, and nine pushes across three phantoms.The factor graph was run for ten time steps using inputs captured at 1 Hz, with the final output evaluated.
- Ex vivo evaluation: The ex vivo factor graph represents the CAO as a tetrahedral mesh and combines an XPBD simulation factor with an MDE point-cloud measurement factor.The objective is to infer volumetric deformation so unobserved regions remain registered during intervention.
3) Factors:
The measurement factor uses a metric monocular-depth-estimation network and tumor segmentation to provide point-cloud observations for the tetrahedral mesh. Its uncertainty is calibrated empirically from error evaluation.
- MDE factor: The metric MDE factor builds a point cloud from DepthAnything-v2 and fine-tunes it using CT-derived depth supervision registered to the robotic viewport.Training uses 6000 video frames and an ℓ2 Euclidean loss in two stages.
- Segmentation factor: Tumor segmentation removes depth predictions from the tool and trachea so the measurement factor focuses on tumor geometry.The segmentation network combines a frozen SAM2 Hiera encoder with adapters, multiscale feature refinement, and a U-Net-style decoder.
- Reported results: Table I reports simulation results as mean ± standard deviation in millimeters.
- Uncertainty: The MDE factor's uncertainty is set empirically according to its measured error.
4) Quantitative Evaluation:
Quantitative evaluation compares reconstructed and reference point clouds using nearest-neighbor, Chamfer, RMS, and 95th-percentile distances. Errors are also separated into MDE-observable and unobserved regions while restricting computation to the lesion surface.
- Point-cloud comparison: The optimized tetrahedral mesh is sampled into an estimated point cloud and compared with a reference point cloud in the robot base frame.
- Metrics: The evaluation reports average reconstruction accuracy and localized geometric disagreement using multiple complementary distance metrics.These include one-sided and reverse nearest-neighbor errors, symmetric Chamfer distance, RMS nearest-neighbor error, and the 95th-percentile distance.
- Surface evaluation: Table II summarizes surface reconstruction errors over the full and unobserved surfaces as mean ± standard deviation in millimeters.
- Surface evaluation: Table III summarizes surface reconstruction errors within the MDE-observable region as mean ± standard deviation in millimeters.
- Evaluation scope: Only the lesion surface is used for metric computation because the lesion-trachea interface is treated as fixed and sub-surface vertices lack ground-truth CT correspondence.
- Observability analysis: The analysis separates regions that are directly observable by MDE from regions that are not, using camera-center ray tracing to define the observable proxy subset.
A. Simulation
In deformable-cube shear simulation, the full factor-graph model achieved sub-millimeter reconstruction errors, with the simulation prior providing the main accuracy benefit. Temporal smoothness helped measurement-only estimation but added little once the physics prior was included.
- A. Simulation: Adding temporal smoothness to measurement-only estimation reduced all-vertex RMSE from 280 ± 220 mm to 120 ± 9.0 mm and unobserved-external-vertex RMSE from 350 ± 280 mm to 110 ± 11 mm.The temporally smoothed configuration nevertheless remained poorly constrained, with several optimization steps reaching the maximum iteration limit.
- A. Simulation: Temporal-only estimation achieved 5.5 ± 3.4 mm all-vertex error and 5.3 ± 3.2 mm unobserved-external-vertex error, but this result reflects near-static initialization over a short trajectory.Without an absolute factor, copyforward initialization leaves the mesh at its initial state, which remains close to ground truth during this interval.
- A. Simulation: 0.73 ± 0.22 mm all-vertex RMSE and 0.67 ± 0.23 mm unobserved-external-vertex RMSE were achieved by both simulation-and-measurement and full models.The full model’s external-vertex error was 0.76 ± 0.21 mm, versus 0.77 ± 0.21 mm for simulation and measurement.
- A. Simulation: Figure 2 compares factor-graph inputs with ground truth and shows the factor-graph output superimposed.The figure provides a visual view of the reconstructed mesh relative to the reference state.
- A. Simulation: The simulation prior was the principal contributor to accurate reconstruction, while the temporal factor provided no measurable improvement once that prior was present.The full model and simulation-and-measurement configuration produced nearly identical errors.
B. Ex Vivo Studies
Ex vivo central-airway obstruction studies show that factor-graph estimation improves tumor-surface alignment beyond tissue simulation alone, including unobserved regions. The results also indicate complementary roles for precise local measurements and global mechanical priors.
- B. Ex Vivo Studies: The full factor graph achieved lower Chamfer distance than tissue simulation alone, including in tumor-surface regions not directly observed by MDE.The stiffness-matrix prior can propagate information from observed regions to constrain the unobserved mesh state.
- B. Ex Vivo Studies: In the MDE-observable region, factor graph had the lowest dCD and dH95, whereas MDE had the lowest dRMSE.The results associate MDE with higher local precision and factor graph with better tumor-surface coverage.
- B. Ex Vivo Studies: Near grazing-angle views, sparse MDE point clouds and small image-space or pose misalignments can produce larger surface displacements, disproportionately affecting bidirectional metrics.This provides context for interpreting the observable-region comparison.
- B. Ex Vivo Studies: MDE provides locally precise but spatially limited measurements, while tissue simulation supplies a global tissue-mechanics prior constraining the overall deformation state.Joint use combines accurate local observations with mechanical propagation through regions lacking direct MDE observations.
- B. Ex Vivo Studies: Preliminary simulation and ex vivo CAO experiments showed that the proposed factor graph closely tracked ground-truth deformations.Future work includes richer sensor-noise models, additional physical validation, and mesh-topology changes during cutting.