Source-linked AI summary
LangStreet: Persistent Language Fields for Anchor-Decoded Street Gaussians
Runyi Yang, Deheng Zhang, Xiaoye Wang, Mengjiao Ma, Lei Sun, Kanzhi Wu, Ajad Chhatkuli, Luc Van Gool, Danda Pani Paudel
TL;DR
View-conditioned children in anchor-decoded Gaussian scenes lack persistent semantic ownership, challenging stable language querying and compact storage. LangStreet assigns ownership to persistent slots and anchors, aggregates renderer responsibilities, completes sparse slots, and encodes detail with low-rank residuals. Across three driving benchmarks, its base variant nearly matches max accuracy while using substantially less feature storage, with completion helping most under sparse slot evidence.
Problem
Anchor-decoded representations generate view-conditioned children whose geometry and appearance vary with the camera, breaking the persistent primitive-semantic correspondence assumed by conventional language Gaussian fields.
Method
LangStreet routes child-rendering observations to persistent slots, exactly marginalizes additive evidence to anchors, completes weak slots with anchor-aligned evidence, and stores compact anchor-relative residuals.
Results
Across KITTI-360, vKITTI2, and Waymo, LangStreet base nearly matches max while using substantially less feature storage; on KITTI-360 it reaches 34.19 2D mIoU with 2.72 GiB versus max’s 34.20 mIoU with 12.90 GiB.
Takeaways & Limitations
Language fields for view-conditioned splats can balance stable semantic ownership, conserved evidence, local detail, and representation cost through the reconstruction’s persistent hierarchy.
Takeaways & Limitations
Fixed-taxonomy mIoU can miss coherent within-class material, boundary, and surface variation, so PCA remains a diagnostic rather than a replacement for mIoU.
Abstract
from arXiv · showhide
Language Gaussian fields implicitly assume that the primitive carrying semantics remains identifiable across views. This assumption breaks in scalable anchor-decoded representations, where persistent anchors generate view-conditioned child Gaussians whose geometry and appearance vary with the camera. We introduce Ours, a persistent language field for such structured Gaussian scenes. Our key idea is semantic ownership: transient children route observations, while persistent decoder slots and their parent anchors own the language field. We use alpha-compositing responsibilities to accumulate additive directional evidence at slots; these statistics marginalize exactly to anchors. We then complete weakly supported slots with anchor-aligned evidence while preserving the anchor direction, and represent slot detail through low-rank residuals in anchor-relative semantic coordinates. Our primary model, Ours (base), stores anchor features together with compact slot residuals. Ours (light) retains only anchor features, whereas Ours (max) stores the full-dimensional completed slot features explicitly. Without scene-specific semantic optimization, Ours (base) nearly matches Ours (max) across KITTI, Virtual KITTI, and Waymo. On KITTI, it achieves 34.19 2D mIoU with a 2.72 GiB effective feature footprint, compared with 34.20 mIoU and 12.90 GiB for Ours (max). The same accuracy-storage trend holds on Virtual KITTI and Waymo. These results show that language fields on view-conditioned splats require persistent semantic ownership, conserved evidence, and a hierarchy that balances stability, detail, and representation cost. Our code, checkpoints, and benchmark suite will be publicly available.
1. Introduction
LangStreet makes street-scale Gaussian reconstructions persistent and language-queryable by assigning semantics to persistent slots and anchors rather than view-conditioned children. It aggregates renderer-routed evidence hierarchically and uses compact slot residuals to balance detail with storage.
- Motivation: Anchor-decoded scenes break conventional semantic ownership because view-conditioned children vary with the camera, while slots and anchors remain persistent addresses.Children route rendered observations but cannot themselves own persistent semantics; slot content is induced by observations assigned to the same address.
- Method: LangStreet routes observations through children to persistent slots, whose additive directional evidence sums exactly to parent-anchor evidence.Weakly supported slots receive anchor-aligned evidence while preserving the coarse anchor direction.
- Method: Anchor-relative low-rank residuals retain sub-anchor semantic detail without storing a full-dimensional feature at every slot.This hierarchy supports fine slot-level detail while exploiting shared anchor structure for compactness.
- Method: LangStreet offers base, light, and max variants, with base storing anchor features plus compact slot codes and max storing completed slot features explicitly.All variants derive from the same renderer-routed evidence without scene-specific semantic optimization, and their embeddings support new text queries.
- Evaluation: Across matched evaluations, base nearly retains max accuracy with substantially less feature storage on KITTI-360 and Waymo.The introduction previews this trend across KITTI-360, Waymo, and vKITTI2, with completion especially benefiting sparse slot-level queries.
2. Related Work
Prior work develops structured urban Gaussian representations and language fields, but LangStreet focuses on persistent semantic ownership for anchor-decoded scenes. Related systems target reconstruction, compression, representation learning, open-vocabulary understanding, or scene querying rather than this ownership problem.
- Structured urban Gaussian representations: Structured urban Gaussian methods scale reconstruction through decomposition, partitioning, levels of detail, graph pruning, or decoding children from persistent anchors and slots.These works primarily address reconstruction, compression, or representation learning.
- Language fields and compact semantics: Language-field methods lift image features into radiance fields, aggregate multi-view evidence, attach semantics to explicit Gaussians, or learn hierarchical affinity fields.Training-free estimators construct semantics through rendering, diffusion, solving, or visibility-aware aggregation.
- Semantic street scenes: Semantic street-scene systems support persistent radiance fields, continual mapping, outdoor language understanding, generation, or question answering over existing fields.The cited comparison distinguishes these goals from LangStreet’s focus on semantic ownership in anchor-decoded representations.
3. Preliminaries and Problem Setup
This setup distinguishes persistent Gaussian addresses from view-conditioned child realizations and defines a hierarchical field with slots as fine addresses and anchors as coarser parents. Frozen reconstruction components and renderer responsibilities provide the basis for assigning observations to persistent semantics.
- Persistent Gaussian addresses: Conventional Gaussian scenes store indexed primitives that remain identifiable across cameras, enabling each primitive to carry a persistent normalized language feature.Text relevance is measured by the cosine score between the primitive feature and a normalized text embedding.
- Anchor-decoded Gaussian scenes: Anchor-decoded scenes store persistent anchors with indexed slots, while a frozen decoder predicts each child’s covariance, opacity, and appearance from anchor state, slot index, and camera condition.The slot’s canonical center is persistent, but the complete child Gaussian is view-conditioned.
- Anchor-decoded Gaussian scenes: A slot is the anchor-local index forming a persistent decoder address, whereas its child may change geometry, appearance, opacity, and activity with the camera.Slots do not have semantics a priori; observations assigned to their persistent addresses induce semantic content.
- Rendering responsibility: Rendering responsibilities measure each active child’s alpha-compositing contribution to a pixel, with inactive slots assigned zero responsibility.These responsibilities allow observations to be routed from transient children to persistent slots.
- Persistent language-field objective: The construction keeps the reconstruction, decoder, and 2D engine frozen while routing observations, summarizing anchors, recovering sparsely supported slots, and avoiding full-dimensional features at every slot.The field uses a unit observation feature and supports text-query scoring at both slot and anchor levels.
4. LangStreet
LangStreet assigns language evidence to persistent slots and anchors rather than view-conditioned child Gaussians, then completes sparse slots and compresses slot detail with anchor-relative residuals. Its renderer-native construction supports multiple storage variants and query readout without scene-specific semantic optimization.
- 4.1. Renderer-Routed Additive Evidence: Additive renderer responsibilities route observations to persistent slots, whose unnormalized evidence and support statistics marginalize exactly to parent anchors.The weights account for projected footprint, opacity, and occlusion; normalized directions would not preserve exact marginalization.
- 4.2. Direction-Preserving Hierarchical Completion: Weakly supported slots receive anchor-aligned completion that preserves the anchor’s normalized direction while diminishing as their own evidence grows.Completion is concentrated using sibling agreement and target-slot support, while unsupported slots inherit the anchor direction.
- 4.3. Anchor-Relative Coding: Anchor-relative residual coding uses a shared low-rank basis for recurring within-anchor variation and requires no semantic labels or feature-field optimization.The anchor acts as a local semantic origin, while slot-specific coefficients retain local detail.
- 4.3. Anchor-Relative Coding: Since r ≪D, LangStreet (base) retains slot-level variation with substantially less storage than LangStreet (max).Base stores anchor features plus low-rank residual codes, whereas light omits slot codes and max stores completed slot directions explicitly.
- 4.4. Construction and Query Readout: A standard renderer backward pass accumulates slot evidence and support, after which completion and coding are applied without a semantic CUDA kernel or scene-specific semantic optimization.Feature channels are processed in blocks using the stock gsplat rasterizer.
- 4.4. Construction and Query Readout: Rendered 2D queries composite text scores with the same responsibilities, while 3D queries access declared persistent anchor or canonical slot addresses.Changing the text bank changes dot products without rebuilding the scene or language field.
5. Experiments
Experiments use controlled benchmarks and matched estimation protocols across KITTI-360, vKITTI2, and Waymo. LangStreet base preserves near-max accuracy with substantially lower storage, while completion improves sparse-support 3D access and qualitative detail.
- Benchmarking Protocol: The evaluation covers eight KITTI-360 windows, all five vKITTI2 clones with held-out camera evaluation, and an eleven-segment Waymo core.Waymo reports matched 2D results but no comparable 3D headline.
- Quantitative Results: 20.56 versus 18.47 3D mIoU on KITTI-360 and 25.30 versus 23.12 on vKITTI2 shows that completion benefits sparse spatial access while leaving 2D accuracy essentially unchanged.The canonical-slot control reports a larger field-level gain under sparse vKITTI2 support.
- Quantitative Results: LangStreet base gives the strongest KITTI-360 3D result and near-max accuracy across all three datasets, while the controlled study compares estimation principles rather than original end-to-end systems.All methods share geometry, views, observations, prompts, renderer, and evaluator.
- Qualitative Results: Base and max preserve thin structures and object boundaries that light smooths, while completion fills sparse slot holes; the same ordering holds on Waymo.PCA visualizations compare light, base, and max features within each scene.
6. Ablation Study
The ablations separate the effects of completion, hierarchical readout, and anchor-relative coding, showing that sparse-support recovery and compact residual coding preserve performance across benchmarks. Fixed-taxonomy mIoU can miss finer semantic structure and remains bounded by observation, geometry, and label-alignment bottlenecks.
- Completion: Support-aware completion improves 3D mIoU from 18.47 to 20.56 on KITTI-360 and from 23.12 to 25.30 on vKITTI2 while retaining 2D accuracy.Sparse-support gains are especially visible under vKITTI2’s held-out-camera setting.
- Coding: Rank-8 compression lowers KITTI-360 2D/3D mIoU from 34.20/20.56 to 33.53/19.88, while the loss is smaller on vKITTI2 at 42.80/25.06 versus 42.87/25.30.Rank-128 is therefore used as the default base setting, with rank-8 reported as a more aggressive compression option.
- Completion versus readout: Completion repairs weak semantic evidence, whereas hierarchical fallback improves coverage of persistent spatial addresses, so the raw-to-max progression combines two distinct effects.The controlled canonical-slot test attributes 0.31 and 2.18 3D points to completion on KITTI-360 and vKITTI2, respectively.
- Evaluation limits: Fixed-taxonomy mIoU rewards only hard labels from 17 text embeddings, while PCA can reveal coherent within-class material, boundary, and surface variation.PCA is diagnostic rather than a replacement for mIoU.
- Observation ceiling: Ground-truth regions improve 2D and slot-3D mIoU by 4.39 and 4.40 points, while adding ground-truth labels gives a further 21.59 points in 2D but 9.71 in 3D.The remaining ceiling reflects distinct upstream semantics, geometry, and sparse-support bottlenecks.
7. Conclusion
LangStreet makes anchor-decoded street Gaussians language-queryable by assigning semantic ownership to persistent slots and anchors rather than transient view-conditioned children. Across three driving benchmarks, its base representation nearly matches the max field with substantially less storage, while completion helps most under sparse evidence.
- Conclusion: LangStreet separates transient routing from persistent semantic ownership, accumulating renderer evidence at slots, marginalizing it to anchors, completing sparse slots, and coding local detail compactly.The reconstruction, decoder, and 2D engine remain frozen during field construction.
- Conclusion: Across three driving benchmarks, LangStreet (base) nearly matches LangStreet (max) while using substantially less feature storage, with completion helping most when slot evidence is sparse.The conclusion also recommends reporting the stored field separately from its readout.