Adapting Auxiliary Losses Using Gradient Similarity

Yunshu Du; Wojciech M. Czarnecki; Siddhant M. Jayakumar; Mehrdad Farajtabar; Razvan Pascanu; Balaji Lakshminarayanan

Source-linked AI summary

Adapting Auxiliary Losses Using Gradient Similarity

Yunshu Du, Wojciech M. Czarnecki, Siddhant M. Jayakumar, Mehrdad Farajtabar, Razvan Pascanu, Balaji Lakshminarayanan

arXiv:1812.02224v2Dec 5, 2018stat.MLcs.LG

TL;DR

Auxiliary losses can improve data efficiency but may become harmful as training progresses, and their usefulness is difficult to determine in advance. The paper adapts auxiliary updates using gradient cosine similarity, applying them when they are descent directions for the main task. It proves convergence to main-task critical points and reports practical benefits across supervised and reinforcement-learning settings.

Problem
Auxiliary tasks can help main-task learning, but their alignment and usefulness may be difficult to determine and can change during training.
Method
The method uses cosine similarity between main and auxiliary gradients as an adaptive weight, applying auxiliary updates when they are descent directions for the main task.
Results
The approach is guaranteed to converge on the main task and empirically blocks negative transfer while sometimes providing positive transfer in supervised learning and reinforcement learning.
Takeaways & Limitations
Gradient similarity provides a heuristic for adapting auxiliary-task transfer to the main task during learning.
Takeaways & Limitations
Gradient-cosine estimation may be expensive or noisy, fixed thresholds may require improvement, and local similarity does not support global conclusions.

Abstract

from arXiv · show

One approach to deal with the statistical inefficiency of neural networks is to rely on auxiliary losses that help to build useful representations. However, it is not always trivial to know if an auxiliary task will be helpful for the main task and when it could start hurting. We propose to use the cosine similarity between gradients of tasks as an adaptive weight to detect when an auxiliary loss is helpful to the main loss. We show that our approach is guaranteed to converge to critical points of the main task and demonstrate the practical usefulness of the proposed algorithm in a few domains: multi-task supervised learning on subsets of ImageNet, reinforcement learning on gridworld, and reinforcement learning on Atari games.

1 INTRODUCTION

Neural networks remain data-inefficient and expensive to train, motivating methods that transfer knowledge from auxiliary tasks. The paper focuses on using gradient information to adapt this transfer as task usefulness changes.

Neural networks require large labeled datasets or millions of reinforcement-learning frames, making training data-inefficient and expensive.
The paper studies transferring knowledge from one or more auxiliary tasks to a main task.
Auxiliary-task success depends on alignment with the main task, which can be difficult to know in advance and can change during training.
The proposed heuristic measures task similarity using gradient information at the current parameter values.

2 NOTATION AND PROBLEM DESCRIPTION

The paper seeks an adaptive way to use an auxiliary task when it helps the main task and suppress it when it causes negative transfer. It replaces fixed loss weighting with an iteration-wise heuristic based on task usefulness.

The objective is to leverage an auxiliary task while helpful and block its negative transfer when it becomes harmful.
The shared parameters θ connect main and auxiliary networks, while φ_main and φ_aux remain task-specific.
Fixed λ cannot adapt when auxiliary-task usefulness changes, and task relatedness is often ambiguous to quantify.
The ideal objective modulates λ at each iteration to improve main-task performance or convergence speed.
The proposed approximation avoids solving an expensive nonlinear maximization problem and does not require hyperparameter tuning or differentiating through learning.

3 GRADIENT COSINE SIMILARITY

Gradient cosine similarity determines whether auxiliary updates are descent directions for the main loss, enabling adaptive transfer. The method has convergence guarantees for the main task, while its speed benefits and global behavior remain limited.

Gradient cosine similarity: Positive cosine similarity indicates aligned task gradients, whereas negative similarity means the auxiliary update can hinder minimizing the main loss.The synthetic example gives cosine values 1 at θ = −20 and −1 at θ = 5.
Gradient cosine similarity: The method applies the auxiliary update only when its gradient has non-negative cosine similarity with the main gradient.
Convergence guarantees: For sufficiently small α(t), the proposed update converges to a local minimum of the main loss, including when the auxiliary contribution is an arbitrary vector field.
Convergence guarantees: The guarantee prevents divergence but does not ensure faster convergence or positive transfer; empirical speed effects can be positive or negative.
Toy experiments: In toy experiments, the method converges in all cases, speeding learning in positive-transfer settings and preserving convergence when naive addition diverges.
Implementation: The unweighted variant uses a binary sign of cosine similarity and is used in experiments because its sign can be estimated more robustly from noisy mini-batches.
Limitations: The update rule can produce a non-conservative vector field, so it need not correspond to the gradient of any loss function.

4 APPLICATIONS

Across supervised and reinforcement-learning applications, gradient cosine similarity adapts auxiliary-task use: it leverages positive transfer and blocks negative transfer. The method improves or preserves main-task performance across ImageNet, RotatedMNIST, gridworld, Breakout, and Ms. PacMan, while gradient estimates can be noisy.

Binary classification tasks: Gradient cosine similarity distinguishes helpful from harmful auxiliary tasks in ImageNet class-pair experiments.Near classes produce higher similarity, whereas far classes produce mostly negative similarity.
Binary classification tasks: When the auxiliary task helps, the method gives an initial accuracy boost and later behaves similarly to multi-task training.When the auxiliary task could hinder, it switches off the auxiliary loss and recovers single-task performance, although blocking occurred only around step 15,000.
Multi-class classification tasks: On RotatedMNIST, the method mitigates negative interference from rotated auxiliary inputs and surpasses both multi-task and single-task training at 45 degrees.At 90 and 135 degrees, performance still degrades, but the method avoids a significant amount of the multi-task baseline’s negative interference.
Gridworld tasks: In gridworld, cosine-weighted distillation reaches baseline performance after one-third as many steps and averages 3 points after 10,000 steps.The method ignores teacher signals when they disagree with the policy-gradient estimator and uses teacher logits when they agree.
Single and multi-task Atari games: With a sub-optimal Breakout teacher, adaptive KL distillation learns quickly initially and then continues with pure reinforcement learning after the auxiliary loss is turned off.The method scales the KL penalty by gradient cosine similarity and turns it off below a fixed threshold to prevent negative transfer.
Single and multi-task Atari games: In multi-task Breakout and Ms. PacMan, the method turns off unhelpful distillation, learns Ms. PacMan without forgetting Breakout, and achieves the best average performance.The evolving gradient cosine similarity provides a cue for the usefulness of the auxiliary loss.

5 RELATED WORK

The paper situates its method within transfer learning, task-similarity estimation, auxiliary-task learning, and adaptive multi-task loss scaling. Its distinctive contribution is using gradient-based similarity to adapt updates according to current task relatedness.

The work addresses transfer learning by seeking a general measure of task similarity rather than relying on relatedness mappings supplied by human experts.
The method exploits gradient information to form a heuristic similarity measure at current parameters and adaptively weight model updates.
Auxiliary tasks have been used across reinforcement-learning settings including unsupervised control, navigation, terminal prediction, and multiagent policy transfer.
The approach differs from adaptive multi-task methods that scale losses using gradient magnitude, task uncertainty, or multi-objective formulations.

6 SHORTCOMINGS

The proposed method has practical and theoretical limitations: gradient-similarity estimates may be noisy or costly, optimizer dynamics may become less smooth, convergence speed is not guaranteed, and local harm may not predict global effects.

Gradient-cosine estimation can be expensive or noisy, while the current method uses a fixed threshold that may require smoothing or tuning.
The changing loss surface can disrupt gradient-statistics or second-order optimizers such as Adam and RMSprop, although this behavior was not observed empirically.
The method guarantees convergence of the main task but not convergence speed, and counterexamples show it can slow learning relative to optimizing the main task alone.
The approach assumes that auxiliary harm to the main loss locally implies globally slower convergence, although better representations could theoretically reverse that outcome.

7 CONCLUSIONS

The paper concludes that gradient-based adaptation can use auxiliary losses while preventing them from hurting the main task, with empirical promise for selecting suitable auxiliaries. It also identifies settings where an auxiliary task initially hurts but later helps as an open direction.

The method applies auxiliary gradients only when they form a descent direction for the main task, aiming to prevent harmful auxiliary transfer.
The authors report empirical potential for using the hypothesis as a way to select suitable auxiliary tasks.
Future work could examine auxiliaries that hurt initially but help later, including annealed β-VAE and confidence-penalty settings.

Supplementary Material

The supplementary analysis establishes that the proposed update rule converges to critical points of the main loss under sufficiently small step sizes. The proof treats auxiliary updates as general vector fields, not only gradients of losses.

The convergence result applies to an update rule combining the main gradient field G(θ) with an arbitrary vector field V(θ).
The update rule converges to a local minimum of the main loss when α(t) is sufficiently small.
The derived expression combines the main-gradient norm with an auxiliary contribution gated by nonnegative inner-product alignment.
The descent quantity reaches zero only when the main gradient is zero, so convergence occurs at a critical point of the main loss.

A.2 PROOF FOR PROPOSITION 3

The counterexample shows that the proposed merged update field need not be conservative or path invariant, even though the main loss gradient is conservative. The proof uses two paths between the same points and Green’s Theorem to establish this result.

A.2 PROOF FOR PROPOSITION 3: Because the merged vector field is not path invariant, it is not conservative.The proof concludes this using Green’s Theorem.
A.2 PROOF FOR PROPOSITION 3: Figure 7 visualizes the main-loss gradient field and the merged field along the two paths.Blue arrows denote the main gradient, violet arrows the merged field, and dotted or dashed lines denote paths A and B.
A.2 PROOF FOR PROPOSITION 3: The broader method can still converge on the main task despite this non-conservative-field property.The paper discusses positive examples, while also noting that convergence speed is not guaranteed.

C WEIGHTED VERSION OF OUR METHOD

The weighted method updates shared parameters with the main gradient plus the auxiliary gradient scaled by its positive cosine similarity. The section also situates this procedure among the paper’s experimental evaluations.

C WEIGHTED VERSION OF OUR METHOD: Algorithm 2 computes both task gradients and updates shared parameters using only positively aligned auxiliary gradients.The update is ∇θLmain + max(0, cos(∇θLmain, ∇θLaux))∇θLaux.
C WEIGHTED VERSION OF OUR METHOD: Task-specific parameters are updated with their corresponding gradients before the shared parameters are updated.The algorithm initializes shared and task-specific parameters, computes four gradients, and separates task-specific from shared updates.
C WEIGHTED VERSION OF OUR METHOD: The paper reports experimental details for ImageNet classification, gridworld reinforcement learning, and Atari games.These experiments evaluate the proposed method across supervised and reinforcement-learning settings.

D.1 IDENTIFYING NEAR AND FAR CLASSES IN IMAGENET

The ImageNet analysis compares hierarchical class distance and embedding-based FID to identify near and far classes, finding a loose relationship that sometimes matches human judgments of similarity. The broader experiments also illustrate both benefits and limitations of gradient-similarity weighting across supervised and reinforcement-learning settings.

D.1 IDENTIFYING NEAR AND FAR CLASSES IN IMAGENET: ImageNet class distance is defined by the tree level of the lowest common ancestor, with leaves at level 0 and the root at level 19.For example, trimaran and catamaran have class distance 1 because their lowest common ancestor is one level up.
D.1 IDENTIFYING NEAR AND FAR CLASSES IN IMAGENET: The toy optimization examples show that the proposed method can speed convergence when the auxiliary loss reinforces useful descent directions.Simple mixing can slow or prevent optimization, while weighted and unweighted proposed variants converge faster in the positive example.
D.1 IDENTIFYING NEAR AND FAR CLASSES IN IMAGENET: An auxiliary loss can also slow convergence by moving iterates through flat regions, even while the main loss continues to decrease.The paper presents this as a limitation of convergence speed rather than a failure of convergence itself.
D.1 IDENTIFYING NEAR AND FAR CLASSES IN IMAGENET: FID measures similarity using distances between class embeddings from the penultimate layer of a pretrained ResNetV2-50.The embeddings are summarized by their means and covariances.
D.1 IDENTIFYING NEAR AND FAR CLASSES IN IMAGENET: 400 class pairs sampled across eight LCA levels show that LCA and FID are loosely correlated and sometimes reflect human judgments of similarity.Trimaran and catamaran are similar visually and conceptually, whereas rock python and traffic light are dissimilar in both respects.
D.1 IDENTIFYING NEAR AND FAR CLASSES IN IMAGENET: In gridworld, gradient-cosine-driven distillation reaches an average score of 3 after 10,000 steps and baseline performance after one third of the steps.The auxiliary loss is a per-state cross-entropy between teacher and student action distributions.
D.1 IDENTIFYING NEAR AND FAR CLASSES IN IMAGENET: Figure 11 depicts the initial task, a uniformly explored Q-learning solution, the transformed auxiliary task, and the gradient-cosine-driven policy-gradient solution.The figure presents the task transformation and representative solutions side by side.

E GRADIENT COSINE SIMILARITY IN HIGH DIMENSIONS

The high-dimensional analysis asks whether cosine similarity remains informative when gradients are noisy. Simulations indicate that random-vector cosine similarity quickly approaches zero, while noisy observations can still preserve similarity when underlying gradients are aligned.

E GRADIENT COSINE SIMILARITY IN HIGH DIMENSIONS: For independent Gaussian vectors, cosine similarity drops to zero quickly as dimensionality increases.The left panel generates random vectors with zero mean and variance σ^2.
E GRADIENT COSINE SIMILARITY IN HIGH DIMENSIONS: When true main and auxiliary gradients are aligned, noisy gradient observations can still make cosine similarity a potential task-similarity measure in high dimensions.The right panel simulates noisy copies of an underlying aligned vector.
E GRADIENT COSINE SIMILARITY IN HIGH DIMENSIONS: The paper does not establish that locally ascending on the main loss harms convergence speed for non-convex neural-network losses.An auxiliary loss might move a model out of a suboptimal basin of attraction toward a better solution.
E GRADIENT COSINE SIMILARITY IN HIGH DIMENSIONS: The practical effectiveness of the heuristic is reported without a theoretical explanation of its efficiency.The paper hypothesizes that neural-network overparameterization may help explain why it works in practice.