PaperlayerView exact revision on arXiv

Source-linked AI summary

Pattern-Affinitive Propagation across Depth, Surface Normal and Semantic Segmentation

Zhenyu Zhang, Zhen Cui, Chunyan Xu, Yan Yan, Nicu Sebe, Jian Yang

arXiv:1906.03525v1cs.CV

TL;DR

The paper asks whether recurring affinity patterns across depth, surface normal, and semantic segmentation can support joint prediction more explicitly than conventional task interaction. It proposes PAP, combining non-local cross-task affinity propagation with task-specific feature diffusion, and reports state-of-the-art or competitive results on NYUD-v2, SUN-RGBD, and KITTI.

  • Problem

    Existing joint-task methods mainly fuse features or share parameters, leaving explicit common patterns across tasks insufficiently modeled.

    Source: §1 · p. 2

  • Method

    PAP learns non-local affinity matrices for the tasks, adaptively integrates cross-task affinities, and iteratively diffuses them within each task's feature space.

    Source: §3 · p. 3, §4.2 · p. 4

  • Results

    State-of-the-art or competitive results are reported for depth estimation, surface normal prediction, and semantic segmentation on NYUD-v2, SUN-RGBD, and KITTI.

    Source: §1 · p. 2, §6 · p. 8

  • Takeaways & Limitations

    Cross-task complementary affinities can be modeled and used to boost and regularize joint learning across the three tasks.

    Source: §1 · p. 2, §6 · p. 8

    The diffusion stability statement assumes a normalized affinity matrix with finite Frobenius norm, and the paper identifies future work on improving efficiency and extending the method to more vision tasks.

    Source: §4.3 · p. 5, §6 · p. 8

Abstract

from arXiv · show

In this paper, we propose a novel Pattern-Affinitive Propagation (PAP) framework to jointly predict depth, surface normal and semantic segmentation. The motivation behind it comes from the statistic observation that pattern-affinitive pairs recur much frequently across different tasks as well as within a task. Thus, we can conduct two types of propagations, cross-task propagation and task-specific propagation, to adaptively diffuse those similar patterns. The former integrates cross-task affinity patterns to adapt to each task therein through the calculation on non-local relationships. Next the latter performs an iterative diffusion in the feature space so that the cross-task affinity patterns can be widely-spread within the task. Accordingly, the learning of each task can be regularized and boosted by the complementary task-level affinities. Extensive experiments demonstrate the effectiveness and the superiority of our method on the joint three tasks. Meanwhile, we achieve the state-of-the-art or competitive results on the three related datasets, NYUD-v2, SUN-RGBD and KITTI.

1. Introduction

The paper addresses joint prediction of depth, surface normal, and semantic segmentation by seeking explicit cross-task affinity patterns rather than relying mainly on feature fusion or parameter sharing. It proposes PAP, which combines non-local cross-task affinities with task-specific diffusion to improve the three predictions.

  • Joint prediction of depth, surface normal, and semantic segmentation is motivated by their importance and potential applications in autonomous driving, SLAM, and socially interactive robotics.Source: §1 · p. 1
  • Existing joint-task methods mainly use feature fusion or parameter sharing, which may create information ambiguity and do not explicitly model what task-level information is transmitted.Source: §1 · pp. 1–2
  • Statistical analysis on NYUD-v2 and SUN-RGBD examines recurring second-order affinity and dissimilarity patterns across the three tasks using task-specific error or consistency metrics.Source: §1 · p. 2
  • PAP uses non-local similarities and two propagation stages to integrate cross-task affinity patterns and diffuse them within each task.Source: §1 · p. 2
  • Extensive experiments report competitive or superior performance for depth estimation, surface normal prediction, and semantic segmentation on NYUDv2, SUN-RGBD, and KITTI.Source: §1 · p. 2

2. Related Works

Related work covers single-task depth, surface-normal, and RGBD segmentation methods, alongside affinity-learning approaches. The paper distinguishes PAP through data-driven, unsupervised, non-local, and cross-task affinity learning.

Depth Estimation

  • Depth-estimation research includes multi-scale continuous CRFs and ordinal regression losses for monocular depth prediction.Source: §2 · p. 2

RGBD Semantic Segmentation

  • RGBD semantic-segmentation methods fuse depth information, including approaches based on 3D graph neural networks and depth-aware location computation.Source: §2 · p. 2

Surface Normal Estimation

  • Surface-normal estimation methods use deep networks incorporating local, global, vanishing-point, skip-connected, or 3D geometric information.Source: §2 · p. 2

Affinity Learning

  • Affinity-learning research spans physical priors, supervised convolutional random walks, non-local networks, and local pixel-wise affinities.Source: §2 · p. 2
  • PAP differs by learning data-driven non-local cross-task affinities without prior knowledge or task-specific supervision.Source: §2 · p. 3

3. Non-Local Affinities

The method represents pair-wise relationships across all pixel positions with normalized, non-local similarity matrices for depth, segmentation, and surface normal tasks. These matrices provide the affinity information integrated for task-specific propagation.

  • PAP models similarities between feature vectors at arbitrary pixel positions, rather than restricting affinities to local neighbors.Source: §3 · p. 3
  • Exponential similarity functions make pair-wise affinities non-negative and assign larger values to similar than dissimilar pairs.Source: §3 · p. 3
  • Row normalization reduces scale effects, while the resulting similarity matrix is symmetric, non-negative, and has finite Frobenius norm.Source: §3 · p. 3
  • Separate affinity matrices are computed for depth, semantic segmentation, and surface normal before their integration for a specific task.Source: §3 · p. 3

4. Pattern-Affinitive Propagation

PAP is implemented as an end-to-end network that learns cross-task affinity patterns and spreads them through task-specific non-local diffusion. Its architecture combines shared and task-specific features, adaptive affinity integration, iterative propagation, reconstruction, and pair-wise supervision.

The Network Architecture

  • The network uses a shared encoder followed by three task-specific networks for depth, surface normal, and semantic segmentation.Hierarchical encoder features are integrated with each task-specific network.Source: §4.1 · p. 3

Cross-Task Propagation

  • Affinity learning forms task-level similarity matrices from task-specific features and adaptively combines the three task matrices for each target task.The matrices encode pair-wise relationships, while learned weights α_Ti combine cross-task information.Source: §4.2 · p. 4
  • Affinity learning is applied at 1/16, 1/8, and 1/4 input scales, thereby learning non-local patch-level relationships.The affinity matrices are computed during decoding at multiple resolutions.Source: §4.2 · p. 4

Task-Specific Propagation

  • Task-specific propagation performs iterative non-local diffusion on features or initial predictions using the combined affinity matrices.The diffusion spreads affinitive patterns within each task and is described both iteratively and through a PDE formulation.Source: §4.3 · pp. 4–5
  • Weighted accumulation with the initial feature or prediction limits deviation during diffusion, with the output formed after t* steps.The weighting parameter β controls the accumulation of the diffused state and the initial state.Source: §4.3 · p. 5

The Loss Function

  • The training objective combines pair-wise affinity constraints with berHu, L1, and cross-entropy losses for the three tasks.Randomly sampled pairs reduce the memory burden of pair-wise loss computation while covering varied-distance relationships.Source: §4.4 · p. 5

5. Experiment

Experiments evaluate PAP across NYUD-v2, SUN-RGBD, and KITTI, including joint-task gains, module and diffusion ablations, affinity visualizations, and comparisons with prior methods. Results show improved or competitive predictions, while more diffusion iterations trade higher performance for longer testing time.

Datasets and metrics

  • NYUD-v2, SUN-RGBD, and KITTI provide evaluation settings spanning joint prediction, RGBD benchmarks, and task distillation.NYUD-v2 includes depth, normals, and semantic labels; SUN-RGBD evaluates depth and segmentation; KITTI lacks or has limited labels for other tasks.Source: §5.1 · p. 5

Joint-task learning

  • Jointly predicting three tasks with PAP outperforms single-task models, with three-task learning obtaining the best results on NYUD-v2.The study evaluates depth, surface normal, and semantic segmentation using a shared ResNet-18 backbone and affinity matrices at 1/8 input scale.Source: §5.3 · p. 6

Network settings and baselines

  • PAP, the reconstruction network, and pair-wise loss each contribute to performance, while cross-stich units and convolutional spatial propagation layers perform more weakly.The analysis also examines affinity-matrix scale and similarity functions; larger scales can help, whereas the tested similarity functions differ little.Source: §5.3 · p. 6

Diffusion iterations

  • More diffusion iterations improve all task performances within the tested range, but increase testing time and create a performance–time trade-off.The experiments use ResNet-18 on NYUD-v2 with affinity matrices learned at 1/8 input scale and 480×640 test inputs.Source: §5.3 · p. 6, Figure 4 · p. 6

Affinity visualization

  • Cross-task affinity maps produce closer relationships to pixels sharing depth, normal direction, and semantic labels than single-task maps.The learned affinity matrices are non-local dense graphs that refine single-task similarities without direct affinity supervision.Source: §5.3 · p. 6, Figure 5 · p. 7

Benchmark comparisons

  • PAP achieves competitive depth results on NYUD-v2, superior surface-normal performance across all reported metrics, and superior or competitive segmentation results on RGBD datasets.On KITTI, PAP outperforms the single-task and cross-stich models and is faster than DORN, although slightly weaker than DORN.Source: §5.4 · pp. 7–8, §5.5 · p. 8

6. Conclusion

The paper concludes that PAP jointly predicts depth, surface normal, and semantic segmentation by learning pair-wise affinity relationships and propagating complementary information across and within tasks. Extensive experiments report state-of-the-art or competitive results on the three tasks, while future work targets broader task generalization and improved efficiency.

  • PAP jointly predicts depth, surface normal and semantic segmentation using learned pair-wise affinity relationships.Source: §6 · p. 8
  • Cross-task and task-specific propagation use complementary affinity information to boost and regularize joint-task learning.Source: §6 · p. 8
  • Extensive experiments obtain state-of-the-art or competitive results on depth, surface normal and semantic segmentation.Source: §6 · p. 8
  • Future work may generalize PAP to more vision tasks and improve its efficiency.Source: §6 · p. 8

Related papers

Loading 1906.03525v1…