Source-linked AI summary
Robust Learning Through Cross-Task Consistency
Amir Zamir, Alexander Sax, Teresa Yeo, Oğuzhan Kar, Nikhil Cheerla, Rohan Suri, Zhangjie Cao, Jitendra Malik, Leonidas Guibas
TL;DR
Visual tasks applied to the same image should produce mutually consistent predictions, but standard learning does not directly enforce that relationship. The paper introduces data-driven cross-task consistency through inference-path invariance over task graphs, yielding more accurate and better-generalizing predictions and an intrinsic energy useful for confidence and domain-shift detection.
Problem
Predictions for multiple visual tasks from one image are expected to be consistent, yet standard independent learning can fail to enforce this relationship in finite-data settings.
Method
The method augments supervised learning with data-driven consistency constraints requiring inference paths with the same endpoints to produce the same results over a task graph.
Results
Learning with cross-task consistency fits data better, improves prediction accuracy and generalization, and produces Consistency Energy that detects out-of-distribution images with ROC-AUC=0.95.
Takeaways & Limitations
Consistency Energy provides an intrinsic quantity for unsupervised confidence estimation and domain-shift detection.
Takeaways & Limitations
Improvements are bounded by optimization methods because adding consistency constraints can make optimization harder.
Abstract
from arXiv · showhide
Visual perception entails solving a wide set of tasks, e.g., object detection, depth estimation, etc. The predictions made for multiple tasks from the same image are not independent, and therefore, are expected to be consistent. We propose a broadly applicable and fully computational method for augmenting learning with Cross-Task Consistency. The proposed formulation is based on inference-path invariance over a graph of arbitrary tasks. We observe that learning with cross-task consistency leads to more accurate predictions and better generalization to out-of-distribution inputs. This framework also leads to an informative unsupervised quantity, called Consistency Energy, based on measuring the intrinsic consistency of the system. Consistency Energy correlates well with the supervised error (r=0.67), thus it can be employed as an unsupervised confidence metric as well as for detection of out-of-distribution inputs (ROC-AUC=0.95). The evaluations are performed on multiple datasets, including Taskonomy, Replica, CocoDoom, and ApolloScape, and they benchmark cross-task consistency versus various baselines including conventional multi-task learning, cycle consistency, and analytical consistency.
1. Introduction
Predictions for different tasks applied to the same image should satisfy cross-task consistency because they describe one underlying scene. The paper proposes data-driven constraints based on inference-path invariance to improve prediction quality and generalization.
- Cross-task inconsistency can reveal contradictory predictions, such as detecting a ball where depth estimation predicts a flat surface.The predictions constrain one another because object and depth domains are not independent.
- Learning with consistency produces higher-quality and more mutually consistent predictions than independent learning.The comparison is illustrated using baseline independent learning and consistency-augmented learning.
- Consistency constraints can improve data fitting, reduce sample complexity, and discourage reliance on superficial visual cues.The constraints are rooted in physical or geometric rules and are associated with improved generalization.
- The method augments learning for an arbitrary task dictionary with explicit, data-driven cross-task consistency constraints.It applies to task pairs whose outputs are statistically dependent even when their analytical relationship is unknown, difficult to program, or non-differentiable.
- Inference-path invariance requires predictions with the same endpoints to agree despite different intermediate task domains.For example, normals inferred through RGB, depth, or shading paths should produce the same result.
- A perceptual-loss formulation and a tractable training schedule make the associated optimization problem manageable while mitigating residual network errors and one-to-many mappings.The formulation addresses cases where one domain cannot always be inferred from another with certainty.
2. Related Work
The paper situates cross-task consistency among structured prediction, graphical models, functional maps, vector calculus, and prior consistency methods. Unlike conventional multi-task and transfer learning, it explicitly enforces consistency across arbitrary paths in a task graph.
- Consistency methods have been used in language, temporal vision, 3D geometry, recognition, and image translation.The paper contrasts prior cycle-based approaches, often limited to two or a few domains, with consistency over arbitrary paths of varied lengths.
- Conventional multi-task learning shares an encoder but does not necessarily produce cross-task-consistent predictions because consistency is not directly enforced.The paper reports this issue in prior work and quantifies it in its own evaluation.
- Inference-path invariance is conceptually related to conservative vector fields, where integration between endpoints is independent of the path.The analogy motivates requiring equal outputs for different inference routes with shared endpoints.
- Figure 2 develops consistency from the standard multi-task setup to triangle constraints and then to arbitrary paths in a larger task system.The generalized construction enforces agreement between paths with the same endpoints.
- Transfer learning also lacks a specific default mechanism for imposing cross-task consistency.The proposed method differs by learning general data-driven consistency constraints explicitly.
- Consistency Energy differs from ensemble averaging because its estimates come from different cues or inference paths rather than repeated versions of one network.Using multiple cues is expected to improve uncertainty capture.
3. Method
The method augments task prediction with learned cross-task constraints, requiring inference paths with shared endpoints to agree. It makes consistency training tractable through separable and perceptual losses, then extends it to arbitrary paths and globally consistent task graphs.
- Inference-path consistency: The framework learns task mappings and cross-task functions, then uses inference-path invariance so paths with the same endpoints produce matching outputs.The task graph can include arbitrary prediction domains and learned cross-task functions.
- Triangle consistency: Triangle consistency adds a term requiring prediction through an intermediate domain to match direct prediction of the target domain.The consistency term couples otherwise independent learning of the prediction networks.
- Separable optimization: The triangle objective is replaced by a separable upper bound whose terms can be optimized independently for each prediction network.This reduces the resource demands of simultaneous training while preserving the same minimizer.
- Perceptual loss: Perceptual consistency compares predictions through a cross-task network and remains robust when that network is noisy or the cross-task mapping is ill-posed.The formulation uses fY1Y2 to judge fXY1(x) against y1 through another task’s representation.
- Perceptual loss: The perceptual formulation removes the need for y2 in training, allowing pair data (x, y1) instead of triplet data (x, y1, y2).For multiple domains, it can use datasets containing pairs among the domains rather than one fully multi-annotated dataset.
- Globally consistent graphs: For arbitrary task graphs, global consistency requires all feasible paths to satisfy the consistency constraint, but direct optimization is intractable because of the many networks and paths.The proposed training schedule therefore approximately optimizes the global objective.
4. Consistency Energy
Consistency Energy measures intrinsic disagreement among inference paths without ground truth, and experiments show it tracks prediction reliability, domain shift, and improved consistency from explicit training constraints.
- Definition: Consistency Energy is the standardized average of pairwise inconsistencies among inference paths for a query and target domain.It can be computed per pixel or per image and requires no ground truth or supervision.
- Training behavior: Consistency Energy decreases during training and converges below independent-learning and multi-task baselines, showing consistency does not emerge automatically.The comparison is reported on test-set predictions.
- Accuracy: Cross-task consistency improved predictions across Replica and Taskonomy tasks, including depth, normals, reshading, and pixel-wise semantics.Reported evaluations used both Direct and Perceptual error metrics, with large improvements in most setups.
- Confidence: 0.67 Pearson correlation links Consistency Energy with supervised error, supporting its use for confidence and uncertainty quantification.This analysis used Taskonomy test images without domain shift from training data.
- Domain shift: ROC-AUC=0.95 was achieved for out-of-distribution detection using per-image energy, compared with 0.51 for OC-NN.Out-of-distribution samples had notably higher energy than Taskonomy in-distribution samples.
- Generalization: Consistency-constrained models generally showed greater robustness to domain shifts and better adaptation with little data.The paper also reports usefulness from consistency with unsupervised tasks such as 2D edges and keypoints.
6. Conclusion and Limitations
The framework improves prediction accuracy and generalization, while its current scope is limited by optimization difficulty, paired-data requirements, and limited experimentation beyond pixel-wise tasks.
- Conclusion: The paper concludes that data-driven cross-task consistency yields more accurate predictions and improved generalization.It also identifies Consistency Energy as useful for confidence estimation and domain-shift detection.
- Future work: The study used inference paths only to enforce consistency, not to aggregate multiple paths into a stronger estimator.Probabilistic path aggregation is proposed as a promising direction because path errors were sufficiently uncorrelated.
- Limitations: The framework currently requires paired training data, leaving extension to unlabeled or unpaired data for future work.The authors describe such an extension as feasible but open.
- Limitations: Experiments primarily used pixel-wise tasks, while classification and other low-dimensional outputs remain to be explored.These tasks induce more severely ill-posed cross-task relationships.
- Limitations: Consistency gains are bounded by optimization methods because added constraints can make optimization harder.Neural-network implementations of cross-task functions can also produce output artifacts similar to those in image synthesis.
- Future work: The properties of Consistency Energy were analyzed post hoc, and active use for network modification or unsupervised domain adaptation requires further study.The authors identify this as an open direction for more focused analysis.