Source-linked AI summary
Understanding Temporal Semantic Stability in Open-Vocabulary UAV Perception through Metric 3D Fusion
Saurbh Singh Jamwal
TL;DR
Open-vocabulary UAV predictions can fluctuate across repeated observations, while frame-wise and image-space evaluations provide limited evidence about semantic evolution at persistent physical locations. The paper uses metric 3D fusion and voxel-level measures to study that evolution. On UAVid-3D, recurrent voxels reveal disagreement obscured by aggregate agreement, and belief drift decreases as additional evidence accumulates.
Problem
Frame-wise and image-space evaluations provide limited understanding of how semantic predictions evolve when repeated UAV observations are associated with the same physical locations.
Source: §1 · p. 1Method
Results
On UAVid-3D, recurrent voxels expose greater semantic disagreement while belief drift decreases as additional evidence accumulates, across backbones and tested analysis variations.
Source: §6 · p. 9Takeaways & Limitations
Semantic consistency should be interpreted together with observation support because reduced world-space recurrence can increase aggregate stability scores.
Source: §6 · p. 9Experiments cover a small set of UAVid-3D trajectories in predominantly static outdoor environments, and dynamic objects challenge identity preservation.
Source: §5 · p. 8
Abstract
from arXiv · showhide
Recent open-vocabulary segmentation models have advanced semantic perception for UAVs, but predictions from moving aerial platforms can remain temporally inconsistent across repeated observations of the same physical scene. We investigate temporal semantic stability by associating frame-wise predictions with persistent world-space locations through metric 3D fusion. We introduce a voxel-level evaluation framework that jointly characterises final semantic agreement, Semantic Belief Drift (SBD), Observation Persistence (OP), and semantic uncertainty. Experiments on UAVid-3D reveal substantial frame-wise semantic flicker and show that high aggregate world-space agreement can overstate temporal stability when locations have limited repeated-observation support. Persistence-stratified analysis shows that recurrent voxels expose greater semantic disagreement, while belief drift decreases as additional evidence accumulates. This behaviour is observed across two segmentation backbones and remains consistent under variations in voxel resolution, geometric association, and temporal sampling density. Conditions that reduce world-space recurrence can increase apparent aggregate stability, demonstrating that semantic consistency must be interpreted together with observation support. Our findings highlight observation persistence as an essential conditioning variable for evaluating long-horizon semantic reliability.
1 Introduction
The paper studies how open-vocabulary UAV predictions evolve when repeated observations are associated with persistent physical locations. Its voxel-level analysis shows that apparent semantic agreement depends on observation support and spatial association, not just final labels.
1 Introduction
- Frame-wise metrics and image-space temporal coherence provide limited insight into how predictions evolve at the same physical locations over long-term UAV observation.Camera motion, viewpoint and scale changes, and occlusions complicate comparisons across frames.Source: §1 · p. 1
- Metric 3D fusion associates predictions from changing UAV viewpoints with persistent voxel locations to analyse their temporal behaviour.The paper uses fusion as an evaluation mechanism, not as a proposed semantic mapping system.Source: §1 · pp. 1–2
- The analysis focuses primarily on static scene regions; dynamic entities require motion reasoning or object-level tracking to preserve identity.Voxel-level stability alone is insufficient for modelling persistent identity when objects move.Source: §1 · p. 2
- The framework combines Voxel Semantic Stability, Semantic Belief Drift, Observation Persistence, and semantic entropy to measure agreement, belief evolution, repeated observation, and uncertainty.These measures distinguish final semantic agreement from how evidence changes over time.Source: §1 · pp. 1–2
- Sparse or degraded world-space associations can inflate aggregate agreement, while recurrent voxels reveal persistent semantic disagreement.The study examines persistence, voxel resolution, geometric perturbation, and temporal sampling.Source: §1 · p. 2
2 Related Work
Prior work advances aerial semantic segmentation, image-space temporal consistency, and metric-semantic mapping. This paper uses metric 3D fusion to evaluate temporal reliability at persistent world-space locations, complementing frame-level and image-space measures.
Semantic Segmentation for UAV and Remote-Sensing Perception
Temporal Consistency in Sequential Perception
- Video segmentation methods such as mask propagation [7], STCN [8], and Video K-Net [9] primarily improve temporal coherence in image space.Image-space coherence does not directly measure stability at the same world-space locations under UAV camera motion.Source: §2 · pp. 2–3
Metric-Semantic Mapping and World-Space Representations
Related-work gap
- The paper addresses a gap in characterising open-vocabulary UAV semantic instability after geometric alignment, using persistent voxels to quantify agreement, belief drift, persistence, and uncertainty.This provides a complementary evaluation perspective beyond conventional image-level accuracy metrics.Source: §2 · pp. 2–3
3 Semantic Stability Analysis Framework
The framework evaluates semantic stability after associating repeated predictions with persistent world-space locations. It combines measures of final agreement, belief evolution, observation support, and residual uncertainty.
3.1 Temporal Semantic Stability
- Stability is the consistency of semantic predictions assigned to the same physical location under repeated observations, assessed after geometric association in world space.Metric 3D fusion is an evaluation substrate, not a proposed semantic mapping system.Source: §3.1 · p. 3
3.2 Semantic Evidence and World-Space Association
- Per-pixel predictions become one-hot class-evidence vectors or soft scores, then depth and camera geometry associate observations with world-space locations.The association uses depth, camera intrinsics, and the camera-to-world transformation.Source: §3.2 · p. 3
3.3 Persistent Voxel Evidence
- Within each voxel, observations from one frame are averaged so that voxel contributes at most one evidence vector per frame; evidence across T_q frames forms its temporal history.This prevents multiple pixels from a single frame being treated as repeated temporal evidence.Source: §3.3 · pp. 3–4
3.4 Voxel-Level Stability Metrics
- VSS measures concentration of accumulated evidence on the dominant class, while SBD measures average belief change between successive observations and OP captures repeated-observation support.OP = 1 cannot establish temporal consistency; higher OP provides progressively stronger temporal evidence.Source: §3.4 · p. 4
- Normalized semantic entropy measures residual uncertainty: low entropy indicates concentrated evidence, whereas high entropy indicates competing interpretations.The four measures characterize temporal behaviour, not absolute correctness; repeated observations may consistently support an incorrect class.Source: §3.4 · p. 4
4 Experiments
The experiments evaluate frame-wise flicker and world-space stability on UAVid-3D using several segmentation approaches and a multi-scene protocol. All semantic models show frame-wise variation, with SegEarth-OV the most temporally consistent of those evaluated.
4.1 Experimental Setup
- The main multi-scene evaluation samples 20 frames per scene across five UAVid-3D scenes, with scene0 and scene1 sampled at a stride of four frames.Other scenes use larger temporal spacing according to the available semantic predictions.Source: §4.1 · p. 4
- SegEarth-OV and SegFormer predictions are mapped to a common taxonomy for matched world-space analysis, while SAM with heuristic assignment and raw SAM proposals are included in frame-wise analysis.World-space analysis associates predictions using depth, camera intrinsics, and poses, then evaluates persistent voxels at 0.2 m resolution unless otherwise stated.Source: §4.1 · p. 4
4.2 Frame-wise Temporal Semantic Flicker
- 29.47% for SAM with heuristic semantic assignment, 25.39% for SegFormer, and 15.73% for SegEarth-OV are the reported mean frame-wise flicker rates; SegEarth-OV is most consistent among the evaluated semantic models.Flicker is the percentage of valid pixels whose labels change between consecutive sampled frames.Source: §4.2 · pp. 4–5
4.3 World-Space Evidence Fusion
Metric 3D fusion associates frame-wise predictions with persistent world-space voxels, enabling stability analysis across five UAVid-3D scenes. Aggregate agreement is high, but limited repeated-observation support means it cannot alone establish temporal stability.
4.3 World-Space Evidence Fusion
- Frame-wise predictions are projected using depth, camera intrinsics, and poses, then accumulated into persistent voxels that associate changing viewpoints with shared world-space locations.All 20 sampled frames were processed in each of the five scenes without skipped frames.Source: §4.3 · p. 5
4.4 Multi-Scene Voxel-Level Stability
- Macro-average VSS is 0.957 across 705,902 persistent voxels, while mean OP is only 1.51, so aggregate agreement has limited repeated-observation support.The voxels were constructed from 1,353,903 sampled world-space semantic points.Source: §4.4 · p. 5
- VSS exceeds 0.90 in every scene, but single-observation voxels have maximal VSS by construction and contribute substantially to aggregate statistics.The authors therefore distinguish apparent stability from stability supported by repeated observations.Source: §4.4 · p. 5
- SBD remains non-zero across scenes, and scene5 and scene8 show higher SBD than scene0 despite VSS values of 0.978 and 0.966.Final agreement and accumulated-belief evolution capture distinct aspects of semantic behaviour.Source: §4.4 · p. 5
- Scene0 and scene1 have more sampled world-space points and higher mean OP than scene3, scene5, and scene8, motivating persistence-stratified analysis.The latter scenes have mean OP values closer to one.Source: §4.4 · p. 5
4.5 Analysis of Semantic Stability Metrics
Persistence-stratified analysis shows that recurrent voxels expose semantic disagreement hidden by aggregate agreement, while accumulated beliefs drift less as evidence grows. Uncertainty and instability also vary across semantic classes.
4.5 Analysis of Semantic Stability Metrics
- The analysis distinguishes final agreement, belief evolution, repeated observation, and residual uncertainty using VSS, SBD, OP, and semantic entropy.The framework examines how these measures vary with persistence, uncertainty, and semantic class.Source: §4.5 · p. 5, Table 2 · p. 6, Table 3 · p. 6
4.5.1 Semantic Agreement and Belief Drift
- VSS measures final concentration on the dominant class, whereas SBD measures accumulated-belief change; Figure 2 shows aggregate disagreement and drift concentrated near zero.A single observation necessarily yields VSS=1 and zero entropy, while SBD is undefined, so aggregate agreement requires persistence-aware interpretation.Source: §4.5.1 · p. 6, Figure 2 · p. 7
4.5.2 Observation Persistence
- Approximately 57.9% of voxels are observed once and attain VSS=1 and zero entropy by construction, while recurrent voxels have VSS near 0.87.Repeated observations expose semantic disagreement invisible at OP=1.Source: §4.5.2 · p. 6
- Mean SBD decreases from 0.0790 at OP=2 to 0.0454 for 3 ≤ OP ≤ 5 and 0.0216 for OP>5, even as recurrent observations reveal disagreement.Belief trajectories become progressively less changeable as evidence accumulates, while disagreement among observations can remain.Source: §4.5.2 · p. 6
4.5.3 Semantic Uncertainty
- Entropy rises from zero for singly observed voxels to 0.1602 at OP=2, 0.2285 for 3 ≤ OP ≤ 5, and 0.2530 for OP>5.This reflects disagreement exposed by additional observations, not evidence that repeated observation makes perception less stable.Source: §4.5.3 · p. 6
4.5.4 Class-wise Stability
- Road and tree exhibit higher SBD and entropy than low vegetation, while rare static-car and human classes lack enough voxels for reliable comparison.The class-wise statistics should be interpreted jointly with observation persistence because categories may differ in spatial coverage and recurrence.Source: §4.5.4 · p. 7
4.6 Cross-Backbone Temporal Stability
Under a shared five-scene, three-class protocol, SegFormer shows higher aggregate stability than SegEarth-OV but fewer recurrent voxels. The difference remains when comparison is conditioned on recurrence.
4.6 Cross-Backbone Temporal Stability
- SegFormer has aggregate VSS 0.9848 and SBD 0.0380, versus SegEarth-OV at VSS 0.9657 and SBD 0.0661, but fewer voxels have OP≥3.The recurrent-voxel fractions are 15.29% for SegFormer and 21.79% for SegEarth-OV.Source: §4.6 · p. 7
- For OP≥3, SegFormer achieves VSS 0.9410 and SBD 0.0186, compared with 0.8692 and 0.0413 for SegEarth-OV.The difference remains after conditioning on recurrence, rather than being explained solely by lower observation persistence.Source: §4.6 · p. 7
- Aggregate temporal metrics can mislead model comparisons when a backbone appears more stable partly because it produces fewer recurrent world-space associations.Conditioning on OP separates temporal behaviour from differences in recurrent semantic coverage.Source: §4.6 · pp. 7–8
4.7 Sensitivity to World-Space Association
Across changes to voxel resolution, geometric association, and temporal sampling, reduced recurrence can raise apparent aggregate stability. Repeated-observation support exposes disagreement, so stability measures require joint interpretation with OP.
4.7 Sensitivity to World-Space Association
- Increasing voxel resolution from 0.1 to 0.4 m raises OP≥3 from 3.90% to 42.61% while aggregate VSS falls from 0.9824 to 0.9274.Fine discretisation fragments repeated observations across voxels; coarser discretisation increases recurrence but also semantic mixing, with 0.2 m an intermediate trade-off.Source: §4.7 · p. 8
- With 0.3% relative depth noise, OP≥3 falls from 22.12% to 9.81% as aggregate VSS rises from 0.9569 to 0.9777.Recurrent-voxel SBD changes only from 0.0497 to 0.0541; similar behaviour occurs under mild pose perturbations.Source: §4.7 · p. 8
- Sampling 20 to 80 frames raises OP≥3 from 1.61% to 18.99% and lowers aggregate VSS from 0.9805 to 0.9400.Recurrent-voxel VSS changes from 0.8544 to 0.8446, while SBD decreases from 0.0579 to 0.0512.Source: §4.7 · p. 8
- Across all three studies, reduced recurrence can increase aggregate apparent stability, whereas stronger repeated-observation support exposes semantic disagreement.VSS and related stability measures therefore require joint interpretation with OP.Source: §4.7 · p. 8
4.8 Qualitative Analysis
The visualisations show spatially localised semantic drift and uncertainty, and distinguish recurrent locations from areas with too few observations for reliable temporal assessment.
4.8 Qualitative Analysis
- The visualisations illustrate voxel-level stability, belief drift, and semantic uncertainty after frame-wise predictions are associated in world space.Their purpose is to show these behaviours, not semantic mapping performance.Source: §4.8 · p. 8
- Drift and uncertainty are spatially localised rather than uniform across the reconstructed scene.The visualisations categorise voxels as stable, drifting, uncertain, or low-persistence based on final agreement, belief drift, accumulated evidence, and repeated observations.Source: §4.8 · p. 8
- Low-persistence regions must be distinguished from recurrent locations that show consistent evidence or localised disagreement and belief evolution.The visualisation supports interpreting VSS and SBD jointly with OP and entropy.Source: §4.8 · p. 8
5 Limitations and Future Work
The framework measures temporal consistency, not semantic correctness, and its world-space results depend on geometric association and the evaluated trajectories and environments.
5 Limitations and Future Work
- High VSS and low SBD can occur even when a model repeatedly assigns an incorrect label, so the framework complements rather than replaces ground-truth accuracy.The analysis characterises consistency of accumulated semantic evidence, not absolute correctness.Source: §5 · p. 8
- Pose and depth noise can reduce observation recurrence while making aggregate VSS appear higher, altering both voxel correspondence and the measured population.More realistic localization, calibration, and depth-error models remain to be studied.Source: §5 · p. 8
- Experiments cover a small set of UAVid-3D trajectories and predominantly static outdoor environments; recurrence may differ with flight patterns, overlap, or sensing configurations.Persistent spatial association alone cannot preserve the identity of moving objects.Source: §5 · p. 8
- Future work targets broader models, datasets, modalities, and trajectories, with motion-aware or object-level association for dynamic environments.The authors also propose studying correctness, stability, geometric uncertainty, and observation recurrence jointly.Source: §5 · p. 8
6 Conclusion
The conclusion presents a voxel-level framework for analysing temporal semantic stability in open-vocabulary UAV perception. Its results show why aggregate agreement must be read alongside observation support.
6 Conclusion
- The framework associates frame-wise predictions with persistent voxel locations and jointly characterises final agreement, belief evolution, observation persistence, and residual uncertainty.It uses VSS, SBD, OP, and semantic entropy.Source: §6 · p. 9
- High aggregate semantic agreement can overstate temporal stability when voxels are sparsely observed, while conditions reducing recurrence can increase aggregate stability scores.The authors identify observation persistence as an essential conditioning variable for temporal semantic evaluation.Source: §6 · p. 9
- Recurrent voxels expose greater semantic disagreement, while belief drift decreases as additional evidence accumulates.This persistence-aware behaviour remains consistent across segmentation backbones and variations in voxel resolution, geometric association, and temporal sampling density.Source: §6 · p. 9