Source-linked AI summary
Pattern-Affinitive Propagation across Depth, Surface Normal and Semantic Segmentation
Zhenyu Zhang, Zhen Cui, Chunyan Xu, Yan Yan, Nicu Sebe, Jian Yang
TL;DR
The paper asks whether recurring affinity patterns across depth, surface normal, and semantic segmentation can support joint prediction more explicitly than conventional task interaction. It proposes PAP, combining non-local cross-task affinity propagation with task-specific feature diffusion, and reports state-of-the-art or competitive results on NYUD-v2, SUN-RGBD, and KITTI.
Problem
Existing joint-task methods mainly fuse features or share parameters, leaving explicit common patterns across tasks insufficiently modeled.
Method
PAP learns non-local affinity matrices for the tasks, adaptively integrates cross-task affinities, and iteratively diffuses them within each task's feature space.
Results
State-of-the-art or competitive results are reported for depth estimation, surface normal prediction, and semantic segmentation on NYUD-v2, SUN-RGBD, and KITTI.
Takeaways & Limitations
Cross-task complementary affinities can be modeled and used to boost and regularize joint learning across the three tasks.
Takeaways & Limitations
The diffusion stability statement assumes a normalized affinity matrix with finite Frobenius norm, and the paper identifies future work on improving efficiency and extending the method to more vision tasks.
Abstract
from arXiv · showhide
In this paper, we propose a novel Pattern-Affinitive Propagation (PAP) framework to jointly predict depth, surface normal and semantic segmentation. The motivation behind it comes from the statistic observation that pattern-affinitive pairs recur much frequently across different tasks as well as within a task. Thus, we can conduct two types of propagations, cross-task propagation and task-specific propagation, to adaptively diffuse those similar patterns. The former integrates cross-task affinity patterns to adapt to each task therein through the calculation on non-local relationships. Next the latter performs an iterative diffusion in the feature space so that the cross-task affinity patterns can be widely-spread within the task. Accordingly, the learning of each task can be regularized and boosted by the complementary task-level affinities. Extensive experiments demonstrate the effectiveness and the superiority of our method on the joint three tasks. Meanwhile, we achieve the state-of-the-art or competitive results on the three related datasets, NYUD-v2, SUN-RGBD and KITTI.
1. Introduction
The paper addresses joint prediction of depth, surface normal, and semantic segmentation by seeking explicit cross-task affinity patterns rather than relying mainly on feature fusion or parameter sharing. It proposes PAP, which combines non-local cross-task affinities with task-specific diffusion to improve the three predictions.
- Joint prediction of depth, surface normal, and semantic segmentation is motivated by their importance and potential applications in autonomous driving, SLAM, and socially interactive robotics.
- Existing joint-task methods mainly use feature fusion or parameter sharing, which may create information ambiguity and do not explicitly model what task-level information is transmitted.
- Statistical analysis on NYUD-v2 and SUN-RGBD examines recurring second-order affinity and dissimilarity patterns across the three tasks using task-specific error or consistency metrics.
- PAP uses non-local similarities and two propagation stages to integrate cross-task affinity patterns and diffuse them within each task.
- Extensive experiments report competitive or superior performance for depth estimation, surface normal prediction, and semantic segmentation on NYUDv2, SUN-RGBD, and KITTI.
2. Related Works
Related work covers single-task depth, surface-normal, and RGBD segmentation methods, alongside affinity-learning approaches. The paper distinguishes PAP through data-driven, unsupervised, non-local, and cross-task affinity learning.
- Depth Estimation: Depth-estimation research includes multi-scale continuous CRFs and ordinal regression losses for monocular depth prediction.
- RGBD Semantic Segmentation: RGBD semantic-segmentation methods fuse depth information, including approaches based on 3D graph neural networks and depth-aware location computation.
- Surface Normal Estimation: Surface-normal estimation methods use deep networks incorporating local, global, vanishing-point, skip-connected, or 3D geometric information.
- Affinity Learning: Affinity-learning research spans physical priors, supervised convolutional random walks, non-local networks, and local pixel-wise affinities.
- Affinity Learning: PAP differs by learning data-driven non-local cross-task affinities without prior knowledge or task-specific supervision.
3. Non-Local Affinities
The method represents pair-wise relationships across all pixel positions with normalized, non-local similarity matrices for depth, segmentation, and surface normal tasks. These matrices provide the affinity information integrated for task-specific propagation.
- PAP models similarities between feature vectors at arbitrary pixel positions, rather than restricting affinities to local neighbors.
- Exponential similarity functions make pair-wise affinities non-negative and assign larger values to similar than dissimilar pairs.
- Row normalization reduces scale effects, while the resulting similarity matrix is symmetric, non-negative, and has finite Frobenius norm.
- Separate affinity matrices are computed for depth, semantic segmentation, and surface normal before their integration for a specific task.
4. Pattern-Affinitive Propagation
PAP is implemented as an end-to-end network that learns cross-task affinity patterns and spreads them through task-specific non-local diffusion. Its architecture combines shared and task-specific features, adaptive affinity integration, iterative propagation, reconstruction, and pair-wise supervision.
- The Network Architecture: The network uses a shared encoder followed by three task-specific networks for depth, surface normal, and semantic segmentation.Hierarchical encoder features are integrated with each task-specific network.
- Cross-Task Propagation: Affinity learning forms task-level similarity matrices from task-specific features and adaptively combines the three task matrices for each target task.The matrices encode pair-wise relationships, while learned weights α_Ti combine cross-task information.
- Cross-Task Propagation: Affinity learning is applied at 1/16, 1/8, and 1/4 input scales, thereby learning non-local patch-level relationships.The affinity matrices are computed during decoding at multiple resolutions.
- Task-Specific Propagation: Task-specific propagation performs iterative non-local diffusion on features or initial predictions using the combined affinity matrices.The diffusion spreads affinitive patterns within each task and is described both iteratively and through a PDE formulation.
- Task-Specific Propagation: Weighted accumulation with the initial feature or prediction limits deviation during diffusion, with the output formed after t* steps.The weighting parameter β controls the accumulation of the diffused state and the initial state.
- The Loss Function: The training objective combines pair-wise affinity constraints with berHu, L1, and cross-entropy losses for the three tasks.Randomly sampled pairs reduce the memory burden of pair-wise loss computation while covering varied-distance relationships.
5. Experiment
Experiments evaluate PAP across NYUD-v2, SUN-RGBD, and KITTI, including joint-task gains, module and diffusion ablations, affinity visualizations, and comparisons with prior methods. Results show improved or competitive predictions, while more diffusion iterations trade higher performance for longer testing time.
- Datasets and metrics: NYUD-v2, SUN-RGBD, and KITTI provide evaluation settings spanning joint prediction, RGBD benchmarks, and task distillation.NYUD-v2 includes depth, normals, and semantic labels; SUN-RGBD evaluates depth and segmentation; KITTI lacks or has limited labels for other tasks.
- Joint-task learning: Jointly predicting three tasks with PAP outperforms single-task models, with three-task learning obtaining the best results on NYUD-v2.The study evaluates depth, surface normal, and semantic segmentation using a shared ResNet-18 backbone and affinity matrices at 1/8 input scale.
- Network settings and baselines: PAP, the reconstruction network, and pair-wise loss each contribute to performance, while cross-stich units and convolutional spatial propagation layers perform more weakly.The analysis also examines affinity-matrix scale and similarity functions; larger scales can help, whereas the tested similarity functions differ little.
- Diffusion iterations: More diffusion iterations improve all task performances within the tested range, but increase testing time and create a performance–time trade-off.The experiments use ResNet-18 on NYUD-v2 with affinity matrices learned at 1/8 input scale and 480×640 test inputs.
- Affinity visualization: Cross-task affinity maps produce closer relationships to pixels sharing depth, normal direction, and semantic labels than single-task maps.The learned affinity matrices are non-local dense graphs that refine single-task similarities without direct affinity supervision.
- Benchmark comparisons: PAP achieves competitive depth results on NYUD-v2, superior surface-normal performance across all reported metrics, and superior or competitive segmentation results on RGBD datasets.On KITTI, PAP outperforms the single-task and cross-stich models and is faster than DORN, although slightly weaker than DORN.
6. Conclusion
The paper concludes that PAP jointly predicts depth, surface normal, and semantic segmentation by learning pair-wise affinity relationships and propagating complementary information across and within tasks. Extensive experiments report state-of-the-art or competitive results on the three tasks, while future work targets broader task generalization and improved efficiency.
- PAP jointly predicts depth, surface normal and semantic segmentation using learned pair-wise affinity relationships.
- Cross-task and task-specific propagation use complementary affinity information to boost and regularize joint-task learning.
- Extensive experiments obtain state-of-the-art or competitive results on depth, surface normal and semantic segmentation.
- Future work may generalize PAP to more vision tasks and improve its efficiency.