PaperlayerView exact revision on arXiv

Source-linked AI summary

Variational Positive-incentive Noise: How Noise Benefits Models

Hongyuan Zhang, Sida Huang, Yubin Guo, Xuelong Li

arXiv:2306.07651v2cs.LGcs.CV

TL;DR

The paper asks how random noise can benefit models despite the usual assumption that noise is harmful, and proposes VPN, which optimizes a variational bound on the Pi-Noise objective. The generator improves base models in most reported cases and tends to disturb irrelevant regions in complicated images.

  • Problem

    The positive or negative impact of noise is difficult to assess because mutual-information integrals are intractable and precise p(y|x) is unavailable for single-label samples.

    Source: §1 · p. 1

  • Method

    VPN optimizes a variational bound and uses Monte Carlo sampling to train a neural π-noise generator that can be paired with existing base models without changing their architectures.

    Source: §1 · p. 1, §3.3 · p. 4

  • Results

    Accuracy increases by over 4% on Tiny ImageNet with hϕ=ResNet-34 and Gθ=ResNet-18, while the generator promotes the base model in most cases and is more stable than random Gaussian noise.

    Source: §4.2 · p. 6

  • Takeaways & Limitations

    The generator tends to inject noise into irrelevant regions of complicated images so the base model can pay more attention to the objects.

    Source: §4.3 · p. 7, §6 · p. 8

    With single-label data, each sample has only one label, which leads to biased estimates of expectations even when Monte Carlo sampling avoids requiring exact p(y|x).

    Source: §5 · p. 7

Abstract

from arXiv · show

A large number of works aim to alleviate the impact of noise due to an underlying conventional assumption of the negative role of noise. However, some existing works show that the assumption does not always hold. In this paper, we investigate how to benefit the classical models by random noise under the framework of Positive-incentive Noise (Pi-Noise). Since the ideal objective of Pi-Noise is intractable, we propose to optimize its variational bound instead, namely variational Pi-Noise (VPN). With the variational inference, a VPN generator implemented by neural networks is designed for enhancing base models and simplifying the inference of base models, without changing the architecture of base models. Benefiting from the independent design of base models and VPN generators, the VPN generator can work with most existing models. From the experiments, it is shown that the proposed VPN generator can improve the base models. It is appealing that the trained variational VPN generator prefers to blur the irrelevant ingredients in complicated images, which meets our expectations.

1 INTRODUCTION

Although noise is often treated as harmful, Pi-Noise frames noise that simplifies a task as potentially beneficial. The paper introduces VPN to approximate this objective and apply a learned noise generator to existing models.

1 INTRODUCTION

  • Pi-Noise defines noise that simplifies a task as beneficial, contrasting with the conventional focus on removing noise and the possibility of pure noise.Pure noise has zero mutual information with the task.Source: §1 · p. 1
  • For single-label classification, label uncertainty in p(y|x) is treated as a measure of task complexity, while complex backgrounds may warrant non-one-hot label probabilities.Source: §1 · p. 1
  • The paper identifies two challenges: mutual-information integrals are usually intractable, and precise p(y|x) is unavailable when each sample has one ground-truth label.Source: §1 · p. 1
  • VPN optimizes a variational bound and uses Monte Carlo sampling to train a neural noise-density generator without requiring precise p(y|x).This addresses the intractable computation and partially addresses the unavailable-distribution challenge.Source: §1 · p. 1
  • The generator can produce noise for samples in training and inference, and visualizations show it disturbing irrelevant regions, including a flower resembling a butterfly.Source: §1 · p. 2, Figure 1 · p. 2

2 RELATED WORK

VPN differs from conventional augmentation in its use of generated noise at inference, and from adversarial training in the purpose of its perturbations. It adds an independent generator module rather than a new representation-learning principle like VIB.

2.1 Difference from Conventional Augmentation

  • Unlike conventional augmentation, which is used during training, VPN can generate noise for unseen samples during both training and inference.Source: §2.1 · p. 2

2.2 Learnable Augmentation

  • Adversarial training perturbs inputs to induce high-confidence incorrect predictions, whereas π-noise is intended to enhance base models by minimizing loss.Source: §2.2 · p. 2

2.3 Variational Inference

  • VIB changes the representation-learning objective, while VPN adds an independent generator that can be trained jointly with or after the base model.The base model can predict p(y|x).Source: §2.3 · pp. 2–3, Figure 2 · p. 3

3 VARIATIONAL POSITIVE-INCENTIVE NOISE

VPN replaces the intractable positive-incentive noise objective with a variational bound and uses a separate generator to produce noise for base models. Its Gaussian noise design and variational training preserve base-model architectures while constraining the noise distribution.

3 VARIATIONAL POSITIVE-INCENTIVE NOISE

  • VPN trains a noise generator alongside a task base model to produce π-noise that benefits the model without changing its architecture.The generator can also be trained with a frozen base model.Source: §3 · p. 3, §3.2 · p. 4

3.1 Define Task Entropy over Distribution DX

  • Task entropy is defined over the data distribution DX, rather than only over a given dataset, to provide a more reasonable setting and a concise Monte Carlo formulation.Source: §3.1 · p. 3

3.2 Variational Approximation

  • VPN optimizes a variational lower bound using a tractable approximation q(y|x, ε) of p(y|x, ε), with Monte Carlo estimation and the reparameterization trick.The paper does not assume y → x → ε, since that Markov condition would imply p(y|x, ε) = p(y|x), which conflicts with π-noise.Source: §3.2 · pp. 3–4

3.3 Positive-incentive Gaussian π-Noise

  • The Gaussian π-noise design uses a zero-mean, uncorrelated distribution with diagonal covariance, reducing parameters and computation from O(d^2) to O(d).Constraining the mean to zero prevents the stated trivial solution; bounds on covariance magnitude can further control noise intensity.Source: §3.3 · p. 4
  • The generator estimates the noise distribution parameters from both x and y, then samples noise; the base model processes x + ε without an architecture change.The described implementation feeds x and ε jointly using the simpler x + ε scheme.Source: §3.3 · pp. 4–5
  • Table 1 reports test accuracy for jointly trained base models and VPN generators; VPN outperforms the strong augmentations except on Fashion-MNIST with ResNet18 and ResNet34.Source: Table 1 · p. 5

4 EXPERIMENTS

The experiments test whether VPN improves base models, what task-simplifying noise looks like, and how generator architecture affects performance. Results report gains across many settings and visualizations in which learned noise targets irrelevant image content.

4.2 Performace Analysis

  • With frozen base models, accuracy improvement is even more stable than Table 1, although ResNet-50 degrades on Tiny ImageNet; joint-trained bases often need no inference noise.Source: Table 4 · p. 5, §4.2 · p. 7

4.1 Implementation Details

  • Experiments span Fashion-MNIST, CIFAR-10, Tiny ImageNet, and ILSVRC2012; they test linear through deep base models and compare Baseline, Random, and No Noise.Source: §4.1 · pp. 5–6

4.2 Performace Analysis

  • With m=1, curves oscillate possibly, while training is fast without apparent performance loss; larger m yields similar final accuracies and faster convergence.Source: Figure 4 · p. 6, §4.2 · p. 7
  • VPN promotes the base model in most joint-training cases, with accuracy increasing by over 4% on Tiny ImageNet for ResNet-34 plus a ResNet-18 generator.Source: §4.2 · p. 6
  • Random Gaussian augmentation is unstable compared with VPN, while DNN3 is more stable than ResNet-18, which the authors suspect may overfit.Source: §4.2 · p. 6

4.3 Visualization

  • On grayscale Fashion-MNIST, VPN targets object pixels; on colorful datasets it disturbs background, sometimes precisely distinguishing irrelevant ingredients in Tiny ImageNet images.Source: §4.3 · p. 7
  • Only part of the background is detected, possibly because Tiny ImageNet training uses only 500 images.Source: §4.3 · p. 7
  • The visualizations are reported to align with task-entropy expectations and to show VPN helping enhance base models.Source: §4.3 · p. 7

5 FUTURE WORKS

The paper identifies biased Monte Carlo estimates when single-label data lack class preferences for each sample, and points to suitable data collection as a principal future task. It also notes that simple two-input processing may constrain performance and suggests broader applications for π-noise.

5 FUTURE WORKS

  • Single-label datasets provide no alternate labels for each sample, biasing Monte Carlo expectations; collecting data that supplies class preferences is identified as principal work.Even multi-label datasets may not provide the needed preferences, and a uniform label distribution is not considered suitable.Source: §5 · p. 7
  • Simple summation of the two inputs is intended to prevent overfitting, but may limit performance and introduce bias; more sophisticated processing, especially for p(ε|x, y), remains to be studied.The paper proposes a module φ: X × Y → X′ with dim X′ = dim X as an ideal direction.Source: §5 · pp. 7–8
  • The authors identify stochastic systems, contrastive learning [32], and multimodal learning as potential settings for extending π-noise.They note that offers a framework for system-stability analysis and suggests an approach to modeling p(ε|x, y).Source: §5 · p. 8

6 CONCLUSION

The paper proposes VPN, a variational-inference approximation for learning π-noise, and reports that it promotes base models whether or not they are trained with the generator. Its visualizations suggest that learned noise targets irrelevant regions of complicated images.

6 CONCLUSION

  • VPN approximates learning π-noise through variational inference and is presented as a module independent of existing classifiers.The conclusion describes it as an approximate method following the π-noise framework.Source: §6 · p. 8
  • The π-noise generator promotes the base model whether or not the base model is trained with the generator.Source: §6 · p. 8
  • Visualizations suggest a well-trained generator injects noise into irrelevant regions of complicated images, allowing the base model to focus on the right objects.Source: §6 · p. 8

Related papers

Loading 2306.07651v2…