Region Mutual Information Loss for Semantic Segmentation

• Pixel-wise losses ignore inter-pixel dependencies that carry object-structure information, potentially limiting recognition of weakly evidenced or spatially small objects.Source: §1 · pp. 1–2
• CRF methods can require slow iterative inference, while pixel-affinity methods can require extra branches or memory for affinity matrices.Source: §1 · p. 2
• RMI represents each pixel with itself and its eight neighbors, forming a 9-D point that encodes a 3×3 region's pixel relationships.Source: §1 · p. 2
• RMI maximizes a lower bound on mutual information between prediction and ground-truth multidimensional distributions because their actual MI is infeasible to calculate.Source: §1 · p. 2
• Downsampling limits RMI's added computational resources to training, allows integration without base-model changes, and adds no inference steps at test time.Source: §1 · p. 2
• RMI achieves substantial, consistent improvements on PASCAL VOC 2012 and CamVid, and outperforms other evaluated pixel-relationship methods on both datasets.Source: §1 · p. 3

• Entropy measures the uncertainty of a random variable.Source: §2 · p. 3
• Joint entropy and conditional entropy are defined for a pair of random variables with joint and conditional distributions.Source: §2 · p. 3
• The entropy of a pair equals the entropy of one variable plus the conditional entropy of the other.Source: §2 · p. 3
• Mutual information measures how much information two random variables contain about each other.Source: §2 · p. 3
• Mutual information can also be viewed as the reduction in one variable's uncertainty given knowledge of the other.Source: §2 · p. 3

3 Methodology

• For an R × R region, P and Y are d-dimensional vectors of predicted probabilities and ground-truth labels, with d = R × R.The method defines their distributions and joint distribution to formulate mutual information between the two vectors.Source: §3 · p. 3
• Pixel dependence makes the joint density difficult to analyze; when normality requires R ≥ 30, d exceeds 900 and memory consumption becomes extremely large.The paper therefore derives a lower bound of I(Y; P) and maximizes that bound to maximize the actual mutual information.Source: §3 · p. 3

3.1 A lower bound of mutual information

• A normal distribution maximizes entropy among distributions with the same covariance, supporting a lower bound that is simplified by ignoring parameter-independent constants.The bound depends on the posterior covariance of Y given P.Source: §3.1 · p. 4

3.2 An approximation of posterior variance

• The posterior covariance is approximated under a second-order independence assumption, which equates conditional and unconditional means and variances without requiring strict independence.The cited approximation also assumes the conditional variance of the residual does not depend on the value of P.Source: §3.2 · p. 4

P )T Cov(Y , P )T . (12)

• Under quadratic loss, µy + Ayp(P − µp) is the Bayes linear estimator if and only if (Y − AypP) is second-order independent of P.The paper says this theorem supports its assumption for approximating posterior variance.Source: P )T Cov(Y , P )T . (12) · p. 4
• The posterior-variance approximation differs from the real posterior variance by a restricted amount under the stated assumption; otherwise, the conditional mean is not the linear MMSE estimator.The paper grounds this qualification in Bayes linear optimality.Source: P )T Cov(Y , P )T . (12) · p. 4
• M is positive semidefinite because it is the covariance matrix of the residual Y − AypP.Source: P )T Cov(Y , P )T . (12) · p. 4

4 Implementation details

• Downsampling the ground truth and predicted probabilities reduces the extra memory and computation required to form R × R regions, with little sacrifice of performance.For R = 3, the example tensor's memory use rises from about 0.33GB to 2.97GB without downsampling.Source: §4 · p. 5
• The RMI value is normalized by d because its magnitude is likely related to the number of eigenvalues of M.Source: §4 · p. 5
• To address determinant underflow, the implementation adds ξI with ξ = 1e −6, uses Cholesky decomposition, and computes RMI in double precision.The text says the small diagonal addition has little effect on the optima and makes M positive definite for the decomposition.Source: §4 · p. 5
• λ = 0.5 weights pixel similarity and structural similarity equally in the combined cross-entropy and RMI objective.The paper describes cross-entropy as measuring pixel-intensity similarity and RMI as measuring structural similarity.Source: §4 · p. 5
• The paper uses sigmoid rather than softmax probabilities to avoid interference between channels, while reporting roughly the same performance for the two loss choices.Source: §4 · p. 5

5.1 Experimental setup

• Experiments use DeepLabv3 and DeepLabv3+ with ResNet-101 backbones, evaluate on PASCAL VOC 2012 and CamVid using mIoU, and train with dataset-specific schedules.The models use ImageNet pretraining; PASCAL VOC training uses trainaug, while CamVid training uses its training and validation sets.Source: §5.1 · pp. 5–6

5.2 Methods of comparison

• CRF is applied as post-processing and can add time-consuming inference, while affinity field loss groups same-class neighboring pixels and separates different-class pairs, requiring substantial memory.For affinity field loss, downsampling labels can reduce memory use but may hurt performance.Source: §5.2 · p. 6

5.3.1 Effectiveness of RMI

• On PASCAL VOC 2012, RMI raises validation mIoU by 1.57% with DeepLabv3 and 1.49% with DeepLabv3+, with test-set improvements of 2.00% and 1.93%, respectively.On validation, DeepLabv3 with RMI also catches DeepLabv3+ without RMI.Source: Table 1 · p. 7, §5.3.1 · p. 7, Table 2 · p. 7

5.3.2 Ablation study

• Average pooling performs best among the tested downsampling methods; larger downsampling factors can hurt performance, while larger regions and smaller factors tend to improve it at greater computational cost.The study did not test RMI without downsampling because of GPU-memory limits; DeepLabv3 with DF = 2 and R = 3 reached 79.23% on PASCAL VOC 2012 validation, versus 78.85% reported for an improved DeepLabv3+ model with similar settings.Source: Table 3 · p. 8, §5.3.2 · p. 8

5.4 Results on CamVid dataset

• On the CamVid test set, RMI improves mIoU by 2.61% with DeepLabv3 and 2.24% with DeepLabv3+; the evaluation uses DF = 4 and R = 3.Source: §5.4 · p. 8, Table 4 · p. 8

6 Conclusion

• RMI achieved substantial and consistent improvements on standard benchmark datasets.The conclusion does not specify the datasets or numerical gains.Source: §6 · p. 9
• RMI represents a pixel with its neighbours, forming multidimensional points and distributions for prediction and ground truth, then maximizes a lower bound on their mutual information.The lower bound is used because the actual mutual information is hard to calculate directly.Source: §6 · p. 9
• RMI requires only a few additional training-stage memory resources and makes no changes to the base segmentation model.The conclusion also states that RMI needs no extra inference steps during testing.Source: §6 · p. 9
• On PASCAL VOC 2012 validation examples, DeepLabv3+ with RMI shows richer details than DeepLabv3+ with cross-entropy.Examples include airplane-wing bumps, plant branches, and animal limbs.Source: Figure 2 · p. 9