Robust Learning Through Cross-Task Consistency

Introduction

What is consistency: suppose an object detector detects a ball in a particular region of an image, while a depth estimator returns a flat surface for the same region. This presents an issue – at least one of them has to be wrong, because they are inconsistent. More concretely, the first prediction domain (objects) and the second prediction domain (depth) are not independent and consequently enforce some constraints on each other, often referred to as consistency constraints.

Why is it important to incorporate consistency in learning: first, desired learning tasks are usually predictions of different aspects of one underlying reality (the scene that underlies an image). Hence inconsistency among predictions implies contradiction and is inherently undesirable. Second, consistency constraints are informative and can be used to better fit the data or lower the sample complexity. Also, they may reduce the tendency of neural networks to learn “surface statistics” (superficial cues) , by enforcing constraints rooted in different physical or geometric rules. This is empirically supported by the improved generalization of models when trained with consistency constraints (Sec. 5).

How can we design a learning system that makes consistent predictions: this paper proposes a method which, given an arbitrary dictionary of tasks, augments the learning objective with explicit constraints for cross-task consistency. The constraints are learned from data rather than apriori given relationships.For instance, it is not necessary to encode that surface normals are the 3D derivative of depth or occlusion edges are discontinuities in depth. This makes the method applicable to any pairs of tasks as long as they are not statistically independent; even if their analytical relationship is unknown, hard to program, or non-differentiable. The primary concept behind the method is ‘inference-path invariance’. That is, the result of inferring an output domain from an input domain should be the same, regardless of the intermediate domains mediating the inference (e.g., RGB$\shortrightarrow$normals and RGB$\shortrightarrow$depth$\shortrightarrow$normals and RGB$\shortrightarrow$shading$\shortrightarrow$normals are expected to yield the same normals result). When inference paths with the same endpoints, but different intermediate domains, yield similar results, this implies the intermediate domain predictions did not conflict as far as the output was concerned. We apply this concept over paths in a graph of tasks, where the nodes and edges are prediction domains and neural network mappings between them, respectively (Fig. 2(d)). Satisfying this invariance constraint over all paths in the graph ensures the predictions for all domains are in global cross-task agreement.inference-path invariance was inspired by Conservative Vector Fields in vector calculus and physics that are (at a high level) fields in which integration along different paths yield the same results, as long as their endpoints are the same . Many key concepts in physics are ‘conservative’, e.g., gravitational force: the work done against gravity when moving between two points is independent of the path taken.

To make the associated large optimization job manageable, we reduce the problem to a ‘separable’ one, devise a tractable training schedule, and use a ‘perceptual loss’ based formulation. The last enables mitigating residual errors in networks and potential ill-posed/one-to-many mappings between domains; this is crucial as one may not be able to always infer one domain from another with certainty (Sec. 3).

Interactive visualizations, trained models, code, and a live demo are available at http://consistency.epfl.ch/.

Related Work

The concept of consistency and methods for enforcing it are related to various topics, including structured prediction, graphical models , functional maps , and certain topics in vector calculus and differential topology . We review the most relevant ones in context of computer vision.

Utilizing consistency: Various consistency constraints have been commonly found beneficial across different fields, e.g., in language as ‘back-translation’ or in vision over the temporal domain , 3D geometry , and in recognition and (conditional/unconditional) image translation . In computer vision, consistency has been extensively utilized in the cycle form and often between two or few domains . In contrast, we consider consistency in the more general form of arbitrary paths with varied-lengths over a large task set, rather than the special cases of short cyclic paths. Also, the proposed approach needs no prior explicit knowledge about task relationships .

Multi-task learning: In the most conventional form, multi-task learning predicts multiple output domains out of a shared encoder/representation for an input. It has been speculated that the predictions of a multi-task network may be automatically cross-task consistent as the representation from which the predictions are made are shared. This has been observed to not be necessarily true in several works , as consistency is not directly enforced during training. We also make the same observation (see visuals here) and quantify it (see Fig. 8(a)), which signifies the need for explicit augmentation of consistency in learning.

Transfer learning predicts the output of a target task given another task’s solution as a source. The predictions made using transfer learning are sometimes assumed to be cross-task consistent, which is often found to not be the case , as transfer learning does not have a specific mechanism to impose consistency by default. Unlike basic multi-task learning and transfer learning, the proposed method includes explicit mechanisms for learning with general data-driven consistency constraints.

Uncertainty metrics: Among the existing approaches to measuring prediction uncertainty, the proposed Consistency Energy (Sec. 4) is most related to Ensemble Averaging , with the key difference that the estimations in our ensemble are from different cues/paths, rather than retraining/reevaluating the same network with different random initializations or parameters. Using multiple cues is expected to make the ensemble more effective at capturing uncertainty.

Method

We define the problem as follows: suppose ${\scriptstyle\mathcal{X}}$ denotes the query domain (e.g., RGB images) and ${\scriptstyle\mathcal{Y}}{=}\{{\scriptstyle\mathcal{Y}}_{1},$…$,{\scriptstyle\mathcal{Y}}_{n}\}$ is the set of $n$ desired prediction domains (e.g., normals, depth, objects, etc). An individual datapoint from domains $({\scriptstyle\mathcal{X}},{\scriptstyle\mathcal{Y}}_{1},$…$,{\scriptstyle\mathcal{Y}}_{n})$ is denoted by $(x,y_{1},$…$,y_{n})$. The goal is to learn functions that map the query domain onto the prediction domains, i.e. $\mathcal{F}_{\scriptscriptstyle\mathcal{X}\hskip 0.5pt}{=}\{f_{{\scriptscriptstyle\mathcal{X}\hskip 0.5pt}{\scriptscriptstyle\mathcal{Y}}_{j}}|{\scriptstyle\mathcal{Y}}_{j}{\in}{\scriptstyle\mathcal{Y}}\}$ where $f_{{\scriptscriptstyle\mathcal{X}\hskip 0.5pt}{\scriptscriptstyle\mathcal{Y}}_{j}}(x)$ outputs $y_{j}$ given $x$. We also define $\mathcal{F}_{\scriptscriptstyle\mathcal{Y}}{=}\{f_{{\scriptscriptstyle\mathcal{Y}}_{i}{\scriptscriptstyle\mathcal{Y}}_{j}}|{\scriptstyle\mathcal{Y}}_{i},{\scriptstyle\mathcal{Y}}_{j}{\in}{\scriptstyle\mathcal{Y}},i{\neq}j\}$, which is the set of ‘cross-task’ functions that map the prediction domains onto each other; we use them in the consistency constraints. For now assume $\mathcal{F}_{\scriptscriptstyle\mathcal{Y}}$ is given apriori and frozen; in Sec. 3.3 we discuss all functions $f$s are neural networks in this paper, and we learn $\mathcal{F}_{\scriptscriptstyle\mathcal{Y}}$ just like $\mathcal{F}_{\scriptscriptstyle\mathcal{X}\hskip 0.5pt}$.

The first and last terms are the standard direct losses for training $f_{{\scriptscriptstyle\mathcal{X}\hskip 0.5pt}{\scriptscriptstyle\mathcal{Y}}_{1}}$ and $f_{{\scriptscriptstyle\mathcal{X}\hskip 0.5pt}{\scriptscriptstyle\mathcal{Y}}_{2}}$. The middle term is the consistency term which enforces that predicting ${\scriptstyle\mathcal{Y}}_{2}$ out of the predicted ${\scriptstyle\mathcal{Y}}_{1}$ yields the same result as directly predicting ${\scriptstyle\mathcal{Y}}_{2}$ out of ${\scriptstyle\mathcal{X}}$ (done via the given cross-task function $f_{{\scriptscriptstyle\mathcal{Y}}_{1}{\scriptscriptstyle\mathcal{Y}}_{2}}$).Operator ${\circ}$ denotes function composition: g{\circ}h(x){\raisebox{-0.15pt}{\triangleq}}g(h(x)). Thus learning to predict ${\scriptstyle\mathcal{Y}}_{1}$ and ${\scriptstyle\mathcal{Y}}_{2}$ are not independent anymore.

The triangle loss 1 is the smallest unit of enforcing cross-task consistency. Below we make two improving modifications on it via function ‘separability’ and ‘perceptual losses’.

The loss $\mathcal{L}_{{\scriptscriptstyle\mathcal{X}\hskip 0.5pt}{\scriptscriptstyle\mathcal{Y}}_{1}{\scriptscriptstyle\mathcal{Y}}_{2}}^{\textit{\tiny{triangle}}}$ involves simultaneous training of two networks $f_{{\scriptscriptstyle\mathcal{X}\hskip 0.5pt}{\scriptscriptstyle\mathcal{Y}}_{1}}$ and $f_{{\scriptscriptstyle\mathcal{X}\hskip 0.5pt}{\scriptscriptstyle\mathcal{Y}}_{2}}$, thus it is resource demanding. We show $\mathcal{L}_{{\scriptscriptstyle\mathcal{X}\hskip 0.5pt}{\scriptscriptstyle\mathcal{Y}}_{1}{\scriptscriptstyle\mathcal{Y}}_{2}}^{\textit{\tiny{triangle}}}$ can be reduced to a ‘separable’ function resulting in two terms that can be optimized independently.

From the triangle inequality we can derive:

which after substitution in Eq. 1 yields:

The upper bound for $\mathcal{L}_{{\scriptscriptstyle\mathcal{X}\hskip 0.5pt}{\scriptscriptstyle\mathcal{Y}}_{1}{\scriptscriptstyle\mathcal{Y}}_{2}}^{\textit{\tiny{triangle}}}$ in inequality 2 can be optimized in lieu of $\mathcal{L}_{{\scriptscriptstyle\mathcal{X}\hskip 0.5pt}{\scriptscriptstyle\mathcal{Y}}_{1}{\scriptscriptstyle\mathcal{Y}}_{2}}^{\textit{\tiny{triangle}}}$ itself, as they both have the same minimizer.Both sides of inequality 2 are ${\geq}0$ and $\text{=}0$ for the minimizer $f_{{\scriptscriptstyle\mathcal{X}\hskip 0.5pt}{\scriptscriptstyle\mathcal{Y}}_{1}}(x)\text{=}{y}_{1}$ & $f_{{\scriptscriptstyle\mathcal{X}\hskip 0.5pt}{\scriptscriptstyle\mathcal{Y}}_{2}}(x)\text{=}{y}_{2}$. The terms of this bound include either $f_{{\scriptscriptstyle\mathcal{X}\hskip 0.5pt}{\scriptscriptstyle\mathcal{Y}}_{1}}$ or $f_{{\scriptscriptstyle\mathcal{X}\hskip 0.5pt}{\scriptscriptstyle\mathcal{Y}}_{2}}$, but not both, hence we now have a loss separable into functions of $f_{{\scriptscriptstyle\mathcal{X}\hskip 0.5pt}{\scriptscriptstyle\mathcal{Y}}_{1}}$ or $f_{{\scriptscriptstyle\mathcal{X}\hskip 0.5pt}{\scriptscriptstyle\mathcal{Y}}_{2}}$, and they can be optimized independently. The part pertinent to the network $f_{{\scriptscriptstyle\mathcal{X}\hskip 0.5pt}{\scriptscriptstyle\mathcal{Y}}_{1}}$ is:

named separate, as we reduced the closed triangle objective $\mathop{\vphantom{\triangle}}_{{\scriptstyle\mathcal{X}}}{\stackrel{{\scriptstyle{\scriptstyle\mathcal{Y}}_{1}}}{{\triangle}}}_{{\scriptstyle\mathcal{Y}}_{2}}$in Eq. 1 to two separate path objectives ${{\scriptstyle\mathcal{X}}}{\shortrightarrow}{{\scriptstyle\mathcal{Y}}_{1}}{\shortrightarrow}{{\scriptstyle\mathcal{Y}}_{2}}$ and ${{\scriptstyle\mathcal{X}}}{\shortrightarrow}{{\scriptstyle\mathcal{Y}}_{2}}$. The first term of Eq. 3 enforces the general correctness of predicting ${\scriptstyle\mathcal{Y}}_{1}$, and the second term enforces its consistency with ${\scriptstyle\mathcal{Y}}_{2}$ domain.

1.2 Reconfiguration into a “Perceptual Loss”

Training $f_{{\scriptscriptstyle\mathcal{X}\hskip 0.5pt}{\scriptscriptstyle\mathcal{Y}}_{1}}$ using the loss $\mathcal{L}_{{\scriptscriptstyle\mathcal{X}\hskip 0.5pt}{\scriptscriptstyle\mathcal{Y}}_{1}{\scriptscriptstyle\mathcal{Y}}_{2}}^{\textit{\tiny{separate}}}$ requires a training dataset with multi domain annotations for one input: $(x,{y_{1}},{y_{2}})$. It also relies on availability of a perfect function $f_{{\scriptscriptstyle\mathcal{Y}}_{1}{\scriptscriptstyle\mathcal{Y}}_{2}}$ for mapping ${\scriptstyle\mathcal{Y}}_{1}$ onto ${\scriptstyle\mathcal{Y}}_{2}$; i.e. it demands ${y_{2}}{=}f_{{\scriptscriptstyle\mathcal{Y}}_{1}{\scriptscriptstyle\mathcal{Y}}_{2}}({y_{1}})$. We show how these two requirements can be reduced.

Again, from triangle inequality we can derive:

which after substitution in Eq. 3 yields:

Similar to the discussion for inequality 2, the upper bound in inequality 5 can be optimized in lieu of $\mathcal{L}_{{\scriptscriptstyle\mathcal{X}\hskip 0.5pt}{\scriptscriptstyle\mathcal{Y}}_{1}{\scriptscriptstyle\mathcal{Y}}_{2}}^{\textit{\tiny{separate}}}$ as both have the same minimizer.Both sides of inequality 5 are ${\geq}0$ and ${=}0$ for the minimizer $f_{{\scriptscriptstyle\mathcal{X}\hskip 0.5pt}{\scriptscriptstyle\mathcal{Y}}_{1}}(x){=}{y}_{1}$. The term $|f_{{\scriptscriptstyle\mathcal{Y}}_{1}{\scriptscriptstyle\mathcal{Y}}_{2}}({y}_{1})-{y}_{2}|$ is a constant and ${\sim}0$, as it is exactly the training objective of $f_{{\scriptscriptstyle\mathcal{Y}}_{1}{\scriptscriptstyle\mathcal{Y}}_{2}}$. The non-zero residual should be ignored and assumed 0 as the non-zero part is irrelevant to $f_{{\scriptscriptstyle\mathcal{X}\hskip 0.5pt}{\scriptscriptstyle\mathcal{Y}}_{1}}$, but imperfections of $f_{{\scriptscriptstyle\mathcal{Y}}_{1}{\scriptscriptstyle\mathcal{Y}}_{2}}$. As the last term is a constant w.r.t. $f_{{\scriptscriptstyle\mathcal{X}\hskip 0.5pt}{\scriptscriptstyle\mathcal{Y}}_{1}}$, the final loss for training $f_{{\scriptscriptstyle\mathcal{X}\hskip 0.5pt}{\scriptscriptstyle\mathcal{Y}}_{1}}$ subject to consistency with domain ${\scriptstyle\mathcal{Y}}_{2}$ is:

The loss $\mathcal{L}_{{\scriptscriptstyle\mathcal{X}\hskip 0.5pt}{\scriptscriptstyle\mathcal{Y}}_{1}{\scriptscriptstyle\mathcal{Y}}_{2}}^{\textit{\tiny{perceptual}}}$ no longer includes ${y_{2}}$, hence it admits pair training data $(x,{y_{1}})$ rather than triplet $(x,{y_{1}},{y_{2}})$.Generally for $n$ domains, this formulation allows using datasets of pairs among $n$ domains, rather than one $n$-tuple multi annotated dataset. Comparing $\mathcal{L}_{{\scriptscriptstyle\mathcal{X}\hskip 0.5pt}{\scriptscriptstyle\mathcal{Y}}_{1}{\scriptscriptstyle\mathcal{Y}}_{2}}^{\textit{\tiny{perceptual}}}$ and $\mathcal{L}_{{\scriptscriptstyle\mathcal{X}\hskip 0.5pt}{\scriptscriptstyle\mathcal{Y}}_{1}{\scriptscriptstyle\mathcal{Y}}_{2}}^{\textit{\tiny{separate}}}$ shows the modification boiled down to replacing ${y_{2}}$ with $f_{{\scriptscriptstyle\mathcal{Y}}_{1}{\scriptscriptstyle\mathcal{Y}}_{2}}({{y_{1}}})$. This makes intuitive sense too, as ${y_{2}}$ is the match of ${y_{1}}$ in the ${\scriptstyle\mathcal{Y}}_{2}$ domain.

Ill-posed tasks and imperfect networks: If $f_{{\scriptscriptstyle\mathcal{Y}}_{1}{\scriptscriptstyle\mathcal{Y}}_{2}}$ is a noisy estimator, then $f_{{\scriptscriptstyle\mathcal{Y}}_{1}{\scriptscriptstyle\mathcal{Y}}_{2}}({y_{1}}){=}{y_{2}{+}{noise}}$ rather than $f_{{\scriptscriptstyle\mathcal{Y}}_{1}{\scriptscriptstyle\mathcal{Y}}_{2}}({y_{1}}){=}{y_{2}}$. Using a noisy $f_{{\scriptscriptstyle\mathcal{Y}}_{1}{\scriptscriptstyle\mathcal{Y}}_{2}}$ in $\mathcal{L}_{{\scriptscriptstyle\mathcal{X}\hskip 0.5pt}{\scriptscriptstyle\mathcal{Y}}_{1}{\scriptscriptstyle\mathcal{Y}}_{2}}^{\textit{\tiny{separate}}}$ corrupts the training of $f_{{\scriptscriptstyle\mathcal{X}\hskip 0.5pt}{\scriptscriptstyle\mathcal{Y}}_{1}}$ since the second loss term does not reach 0 if $f_{{\scriptscriptstyle\mathcal{X}\hskip 0.5pt}{\scriptscriptstyle\mathcal{Y}}_{1}}(x)$ correctly outputs ${y_{1}}$. That is in contrast to $\mathcal{L}_{{\scriptscriptstyle\mathcal{X}\hskip 0.5pt}{\scriptscriptstyle\mathcal{Y}}_{1}{\scriptscriptstyle\mathcal{Y}}_{2}}^{\textit{\tiny{perceptual}}}$ where both terms have the same global minimum and are always 0 if $f_{{\scriptscriptstyle\mathcal{X}\hskip 0.5pt}{\scriptscriptstyle\mathcal{Y}}_{1}}(x)$ outputs ${y_{1}}$ – even when $f_{{\scriptscriptstyle\mathcal{Y}}_{1}{\scriptscriptstyle\mathcal{Y}}_{2}}({y_{1}}){=}{y_{2}{+}{noise}}$. Thus $\mathcal{L}_{{\scriptscriptstyle\mathcal{X}\hskip 0.5pt}{\scriptscriptstyle\mathcal{Y}}_{1}{\scriptscriptstyle\mathcal{Y}}_{2}}^{\textit{\tiny{perceptual}}}$ enables a robust training of $f_{{\scriptscriptstyle\mathcal{X}\hskip 0.5pt}{\scriptscriptstyle\mathcal{Y}}_{1}}(x)$ w.r.t. imperfections in $f_{{\scriptscriptstyle\mathcal{Y}}_{1}{\scriptscriptstyle\mathcal{Y}}_{2}}$. This is crucial since neural networks are almost never perfect estimators, e.g., due to lacking an optimal training process for them or potential ill-posedness of the task y_{1}{\mathrel{\mathchoice{\mkern 2.0mu\clipbox{{.30pt}000}{\displaystyle\vphantom{+}{\shortrightarrow}}}{\mkern 2.0mu\clipbox{{.30pt}000}{\textstyle\vphantom{+}{\shortrightarrow}}}{\mkern 2.0mu\clipbox{{.30pt}000}{\scriptstyle\vphantom{+}{\shortrightarrow}}}{\mkern 2.0mu\clipbox{{.30pt}000}{\scriptscriptstyle\vphantom{+}{\shortrightarrow}}}}}y_{2}. Further discussion and experiments are available in supplementary material.

Perceptual Loss: The process that led to Eq. 6 can be generally seen as using the loss $|g{\circ}f(x){-}g(y)|$ instead of $|f(x){-}y|$. The latter compares $f(x)$ and $y$ in their explicit space, while the former compares them via the lens of function $g$. This is often referred to as “perceptual loss” in super-resolution and style transfer literature –where two images are compared in the representation space of a network pretrained on ImageNet, rather than in pixel space. Similarly, the consistency constraint between the domains ${\scriptstyle\mathcal{Y}}_{1}$ and ${\scriptstyle\mathcal{Y}}_{2}$ in Eq. 6 (second term) can be viewed as judging the prediction $f_{{\scriptscriptstyle\mathcal{X}\hskip 0.5pt}{\scriptscriptstyle\mathcal{Y}}_{1}}(x)$ against ${y_{1}}$ via the lens of the network $f_{{\scriptscriptstyle\mathcal{Y}}_{1}{\scriptscriptstyle\mathcal{Y}}_{2}}$; here $f_{{\scriptscriptstyle\mathcal{Y}}_{1}{\scriptscriptstyle\mathcal{Y}}_{2}}$ is a “perceptual loss” for training $f_{{\scriptscriptstyle\mathcal{X}\hskip 0.5pt}{\scriptscriptstyle\mathcal{Y}}_{1}}$. However, unlike the ImageNet-based perceptual loss , this function has the specific and interpretable job of enforcing consistency with another task. We also use multiple $f_{{\scriptscriptstyle\mathcal{Y}}_{1}{\scriptscriptstyle\mathcal{Y}}_{i}}$s simultaneously which enforces consistency of predicting ${\scriptstyle\mathcal{Y}}_{1}$ against multiple other domains (Sections 3.2 and 3.3).

The derived $\mathcal{L}_{{\scriptscriptstyle\mathcal{X}\hskip 0.5pt}{\scriptscriptstyle\mathcal{Y}}_{1}{\scriptscriptstyle\mathcal{Y}}_{2}}^{\textit{\tiny{perceptual}}}$ loss augments learning of $f_{{\scriptscriptstyle\mathcal{X}\hskip 0.5pt}{\scriptscriptstyle\mathcal{Y}}_{1}}$ with a consistency constraint against one domain ${\scriptstyle\mathcal{Y}}_{2}$. Straightforward extension of the same derivation to enforcing consistency of $f_{{\scriptscriptstyle\mathcal{X}\hskip 0.5pt}{\scriptscriptstyle\mathcal{Y}}_{1}}$ against multiple other domains (i.e. when $f_{{\scriptscriptstyle\mathcal{X}\hskip 0.5pt}{\scriptscriptstyle\mathcal{Y}}_{1}}$ is part of multiple simultaneous triangles) yields:

where $Y$ is the set of domains with which $f_{{\scriptscriptstyle\mathcal{X}\hskip 0.5pt}{\scriptscriptstyle\mathcal{Y}}_{1}}$ must be consistent, and $|Y|$ is the cardinality of $Y$. Notice that $\mathcal{L}_{{\scriptscriptstyle\mathcal{X}\hskip 0.5pt}{\scriptscriptstyle\mathcal{Y}}_{1}{\scriptscriptstyle\mathcal{Y}}_{2}}^{\textit{\tiny{perceptual}}}$ is a special case of $\mathcal{L}_{{\scriptscriptstyle\mathcal{X}\hskip 0.5pt}{\scriptscriptstyle\mathcal{Y}}_{1}{{\scriptscriptstyle{Y}}}}^{\textit{\tiny{perceptual}}}$ where $Y{=}\{{{\scriptstyle\mathcal{Y}}_{2}}\}$. Fig. 5 summarizes the derivation of losses for $f_{{\scriptscriptstyle\mathcal{X}\hskip 0.5pt}{\scriptscriptstyle\mathcal{Y}}_{1}}$.

Fig. 4 shows qualitative results of learning $f_{{\scriptscriptstyle\mathcal{X}\hskip 0.5pt}{\scriptscriptstyle\mathcal{Y}}_{1}}$ with and without cross-task consistency for a sample query.

3 Beyond Triangles: Globally Consistent Graphs

The discussion so far provided the loss for the cross-task consistent training of one function $f_{{\scriptscriptstyle\mathcal{X}\hskip 0.5pt}{\scriptscriptstyle\mathcal{Y}}_{1}}$ using elementary triangle based units. We also assumed the functions $\mathcal{F}_{\scriptscriptstyle\mathcal{Y}}$ were given apriori. The more general multi-task setup is: given a large set of domains, we are interested in learning functions that map the domains onto each other in a globally cross-task consistent manner. This objective can be formulated over a graph $\mathcal{G}{=}(\mathcal{D},\mathcal{F})$ with nodes representing all of the domains $\mathcal{D}{=}({\scriptstyle\mathcal{X}}\cup{\scriptstyle\mathcal{Y}}$) and edges being neural networks between them $\mathcal{F}{=}(\mathcal{F}_{\scriptstyle\mathcal{X}}\cup\mathcal{F}_{\scriptstyle\mathcal{Y}})$; see Fig.2(c).

Extension to Arbitrary Paths: The transition from three domains to a large graph $\mathcal{G}$ enables forming more general consistency constraints using arbitrary-paths. That is, two paths with same endpoint should yield the same results – an example is shown in Fig.2(d). The triangle constraint in Fig.2(b,c) is a special case of the more general constraint in Fig.2(d), if paths with lengths 1 and 2 are picked for the green and blue paths. Extending the derivations done for a triangle in Sec. 3.1 to paths yields:

which is the loss for training $f_{{\scriptscriptstyle\mathcal{X}\hskip 0.5pt}{\scriptscriptstyle\mathcal{Y}}_{1}}$ using the arbitrary consistency path ${\scriptstyle\mathcal{X}}{\shortrightarrow}{\scriptstyle\mathcal{Y}}_{1}{\shortrightarrow}{\scriptstyle\mathcal{Y}}_{2}...{\shortrightarrow}{\scriptstyle\mathcal{Y}}_{k}$ with length $k$ (full derivation provided in supplementary material). Notice that Eq. 6 is a special case of Eq. 8 if $k{=}2$. Equation 8 is particularly useful for incomplete graphs; if the function ${\scriptstyle\mathcal{Y}}_{1}{\shortrightarrow}{\scriptstyle\mathcal{Y}}_{k}$ is missing, consistency between domains ${\scriptstyle\mathcal{Y}}_{1}$ and ${\scriptstyle\mathcal{Y}}_{k}$ can still be enforced via transitivity through other domains using Eq. 8.

Also, extending Eq. 8 to multiple simultaneous paths (as in Eq. 7) by summing the path constraints is straightforward.

Global Consistency Objective: We define reaching global cross-task consistency for graph $\mathcal{G}$ as satisfying the consistency constraint for all feasible paths in $\mathcal{G}$. We can write the global consistency objective for $\mathcal{G}$ as $\mathcal{L}_{\mathcal{G}}=\sum_{p\in\mathcal{P}}\mathcal{L}_{p}^{\textit{\tiny{perceptual}}}$, where $p$ represents a path and $\mathcal{P}$ is the set of all feasible paths in $\mathcal{G}$.

Optimizing the objective $\mathcal{L}_{\mathcal{G}}$ directly is intractable as it would require simultaneous training of all networks in $\mathcal{F}$ with a massive number of consistency pathsFor example, a complete $\mathcal{G}$ with $n$ nodes includes $n(n-1)$ networks and $\sum_{k=2}^{L}{\binom{n}{k+1}}(k+1)!$ feasible paths, with path length capped at $L$.. In Alg.1 we devise a straightforward training schedule for an approximate optimization of $\mathcal{L}_{\mathcal{G}}$. This problem is similar to inference in graphical models, where one is interested in marginal distribution of unobserved nodes given some observed nodes by passing “messages” between them through the graph until convergence. As exact inference is usually intractable for unconstrained graphs, often an approximate message passing algorithm with various heuristics is used.

Instead of optimizing all terms in $\mathcal{L}_{\mathcal{G}}$, Alg.1 selects one network $f_{ij}{\in}\mathcal{F}$ to be trained, selects consistency path(s) $p{\in}\mathcal{P}$ for it, and trains $f_{ij}$ with $p$ for a fixed number of steps using loss 8 (or its multi path version if multiple paths selected). This is repeated until all networks in $\mathcal{F}$ satisfy a convergence criterion.

A number of choices for the selection criterion in SelectNetwork and SelectPath is possible, including round-robin and random selection. While we did not observe a significant difference in the final results, we achieved the best results using maximal violation criterion: at each step select the network and path with the largest lossSee supplementary material for an experimental comparison.. Also, Alg.1 starts from shorter paths and progressively opens up to longer ones (up to length $L$) only after shorter paths have converged. This is based on the observation that the benefit of short and long paths in terms of enforcing cross-task consistency overlap, while shorter paths are computationally cheaper9. For the same reason, all of the networks are initialized by training using the standard direct loss (Op.1 in Alg.1) before progressively adding consistency terms.

Finally, Alg.1 does not distinguish between $\mathcal{F}_{x}$ and $\mathcal{F}_{x}$ and can be used to train them all in the same pool. This means the selected path $p$ may include networks not fully converged yet. This is not an issue in practice, because, first, all networks are pre-trained with their direct loss (Op.1 in Alg.1) thus they are not wildly far from their convergence point. Second, the perceptual loss formulation makes training $f_{ij}$ robust to imperfections in functions in $p$ (Sec. 3.1.2). However, as practical applications primarily care about $\mathcal{F}_{x}$, rather than $\mathcal{F}_{y}$, one can first train $\mathcal{F}_{y}$ to convergence using Alg.1, then start the training of $\mathcal{F}_{x}$ with well trained and converged networks $\mathcal{F}_{y}$. We do the latter in our experiments.A further cheaper alternative is applying cross-task consistent learning only on $\mathcal{F}_{x}$ and training $\mathcal{F}_{y}$ using standard independent training. This is significantly cheaper and more convenient, but still improves $\mathcal{F}_{x}$ notably. Please see supplementary material for how to normalize and balance the direct and consistency loss terms, as they belong to different domains with distinct numerical properties.

Consistency Energy

We quantify the amount of cross-task consistency in the system using an energy-based quantity called Consistency Energy. For a single query $x$ and domain ${\scriptscriptstyle\mathcal{Y}}_{k}$, the consistency energy is defined to be the standardized average of pairwise inconsistencies:

where $\mu_{i}$ and $\sigma_{i}$ are the average and standard deviation of $|f_{{\scriptscriptstyle\mathcal{Y}}_{i}{\scriptscriptstyle\mathcal{Y}}_{k}}{\circ}f_{{\scriptscriptstyle\mathcal{X}\hskip 0.5pt}{\scriptscriptstyle\mathcal{Y}}_{i}}(x){-}f_{{\scriptscriptstyle\mathcal{X}\hskip 0.5pt}{\scriptscriptstyle\mathcal{Y}}_{k}}(x)|$ over the dataset. Eq. 9 can be computed per-pixel or per-image by average over its pixels. Intuitively, the energy can be thought of as the amount of variance in predictions in the lower row of Fig. 3 – the higher the variance, the higher the inconsistency, and the higher the energy. The consistency energy is an intrinsic quantity of the system and needs no ground truth or supervision.

In Sec. 5.3, we show this quantity turns out to be quite informative as it can indicate the reliability of predictions (useful as a confidence/uncertainty metric) or a shift in the input domain (useful for domain adaptation). This is based on the fact that if the query is from the same data distribution as the training and is unchallenging, all inference paths of a system trained with consistency path constraints work well and yield similar results (as they were trained to); whereas under a distribution shift or for a challenging query, different paths break in different ways resulting in dissimilar predictions, and therefore, creating a higher variance. In other words, usually correct predictions are consistent while mistakes are inconsistent. (Plots 8(b), 8(c), 8(d).)

Experiments

The evaluations are organized to demonstrate the proposed approach yields predictions that are I. more consistent (Sec.5.1), II. more accurate (Sec.5.2), and III. more generalizable to out-of-training-distribution data (Sec.5.4). We also IV. quantitatively analyze the Consistency Energy and report its utilities (Sec.5.3).

Datasets: We used the following datasets in the evaluations:

Taskonomy : We adopted Taskonomy as our main training dataset. It includes 4 million real images of indoor scenes with multi-task annotations for each image. The experiments were performed using the following 10 domains from the dataset: RGB images, surface normals, principal curvature, depth (zbuffer), reshading, 3D (occlusion) edges, 2D (Sobel) texture edges, 3D keypoints, 2D keypoints, and semantic segmentation. The tasks were selected to cover 2D, 3D, and semantic domains and have sensor-based/semantic ground truth. We report results on the test set. Also, as one of the out-of-domain tests, we use a version of Taskonomy images where they undergo distortions (e.g., blurring).

Replica has high resolution 3D ground truth and enables more reliable evaluations of fine-grained details. We test on 1227 images from Replica (no training), besides Taskonomy test data.

CocoDoom contains synthetic images from the Doom video game. We use it as one of the out-of-training-distribution datasets.

ApolloScape contains real images of outdoor driving scenes. We use it as another out-of-training-distribution dataset.

NYU : We also evaluated on NYUv2. The findings are similar to those on Taskonomy and Replica (in supplementary material).

Baselines: The main baseline categories are described below. To prevent confounding factors, our method and all baselines were implemented using the same UNet network when feasible and were re-trained on Taskonomy dataset.

Baseline UNet (standard independent learning) is the main baseline. It is identical to consistency models in all senses, except being trained with only the direct loss and no consistency terms.

Multi-task learning: A network with one shared encoder and multiple decoders each dedicated to a task, similar to . This baseline shows if consistency across tasks would emerge by sharing a representation without explicit consistency constraints.

Cycle-based consistency, e.g., is a way of enforcing consistency between two domains assuming a bijection between them. This assumption is violated between many domains (e.g. RGB$\leftrightarrow$3D, as texture cannot be recovered from 3D). This baseline is a special case of the triangle in Fig.2(b) by setting ${\scriptstyle\mathcal{Y}}_{2}{=}{\scriptstyle\mathcal{X}}$.

Baseline perceptual loss network uses frozen random (Gaussian weight) networks as $\mathcal{F}_{\scriptscriptstyle\mathcal{Y}}$, rather than training them to be cross-task functions. This baseline would show if the improvements were owed to the priors in the architecture of constraint networks, rather than them executing cross-task consistency constraints.

GAN-based image translation: We used Pix2Pix , which is conditional GAN based framework .

Blind guess: A query-agnostic statistically informed guess computed from data for each domain (visuals in supplementary). It shows what can be learned from general dataset regularities.

GeoNet is a task-specific consistency method analytically curated for depth and normals. This baseline shows how closely the task-specific consistency methods based on known analytical relationships perform vs the proposed generic data-driven method. The “original” and “updated” variants represent original authors’ released networks and our re-implemented and re-trained version.

Fig. 8(a) (blue) shows the amount of inconsistency in test set predictions (Consistency Energy) successfully decreases over the course of training. The convergence point of the network trained with consistency constraints is well below baseline independent learning (orange) and multi-task learning (green)–which shows consistency among predictions does not naturally emerge in either case without explicit constraining. Plots of individual loss terms similarly show minimizing the direct term does not lead to automatic minimization of consistency terms (provided in supplementary).

2 Accuracy of Predictions

Figures 6 and 7 compare the prediction results of networks trained with cross-task consistency against the baselines in different domains. The improvements are considerable particularly around the difficult fine-grained details.

The corresponding Standard Error for the reported numbers are provided in supplementary material, which show the trends are statistically significant. Tab. 1 also includes evaluation of the networks when trained with little data (0.25% subset of Taskonomy dataset), which shows the consistency constraints are useful under low-data regime as well.

We adopted normals as the canonical task for more extensive evaluations, due to its practical value and abundance of baselines. The conclusions remained the same regardless.

Using Consistency with Unsupervised Tasks: Unsupervised tasks can provide consistency constraints, too. Examples of such tasks are 2D Edges and 2D Keypoints (SURF), which are included in our dictionary. Such tasks have fixed operators that can be applied on any image to produce their respective domains without any additional supervision. Interestingly, we found enforcing consistency with these domains is still useful for gaining better results (see supplementary material for the experiment). The ability to utilize unsupervised tasks further extends the applicability of our method to single/few task datasets.

3 Utilities of Consistency Energy

Below we quantitatively analyze the Consistency Energy. The energy is shown (per-pixel) for sample queries in Fig. 6.

Consistency Energy as a Confidence Metric (Energy vs Error): Plot 8(b) shows the energy of predictions has a strong positive correlation with the error computed using ground truth (Pearson corr. 0.67). This suggests the energy can be adopted for confidence quantification and handling uncertainty. This experiment was done on Taskonomy test set thus images had no domain shift from the training data.

Consistency Energy as a Domain Shift Detector: Plot 8(c) shows the energy distribution of in-distribution (Taskonomy) and out-of-distribution datasets (ApolloScape, CocoDoom). Out-of-distribution datapoints have notably higher energy values, which suggests that energy can be used to detect anomalous samples or domain shifts. Using the per-image energy value to detect out-of-distribution images achieved $\text{ROC-AUC}\text{=}0.95$; the out-of-distribution detection method OC-NN scored $0.51$.

Plot 8(d) shows the same concept as 8(c) (energy vs domain shift), but when the shift away from the training data is smooth. The shift was done by applying a progressively stronger Gaussian blur with kernel size 6 on Taskonomy test images. The plot also shows the error computed using ground truth which has a pattern similar to the energy.

We find the reported utilities noteworthy as handling uncertainty, domains shifts, and measuring prediction confidence in neutral networks are open topics of research with critical values in, e.g., active learning , real-world decision making , and robotics .

4 Generalization & Adaptation to New Domains

To study: I. how well the networks generalize to new domains without any adaptation and quantify their resilience, and II. how efficiently they can adapt to a new domain given a few training examples by fine-tuning, we test the networks trained on Taskonomy dataset on various new domains. The experiment were conducted on smooth (blurring ) and discrete (Doom , ApolloScape ) shifts. For (II), we use a small number (16-128) of images from the new domain to fine-tune the networks with and without consistency constraints. The original training data (Taskonomy) is retained during fine-tuning so prevent the networks from forgetting the original domain .

Models trained with consistency constraints generally show more robustness against domain shifts (see Fig. 9 and pre-adaptation numbers in Table 2) and a better adaptation with little data (see post-adaptation numbers in Table 2 and Fig. 10). The challenging external queries shown in Figures 6&7&1 similarly denote a good generalization.

Supplementary Material: We defer additional discussions and experiments, particularly analyzing different aspects of the optimization, stability analysis of the experimental trends, and proving qualitative results at scale to the supplementary material and the project page.

Conclusion and Limitations

We presented a general and data-driven framework for augmenting standard supervised learning with cross-task consistency. The evaluations showed learning with cross-task consistency fits the data better yielding more accurate predictions and leads to models with improved generalization. The Consistency Energy was found to be an informative intrinsic quantity with utilities toward confidence estimation and domain shift detection. Below we briefly discuss some of the limitations and assumptions:

Path Ensembles: We used the various inference paths only as a way of enforcing consistency. Aggregation of multiple (comparably weak) inference paths into a single strong estimator (e.g., in a manner similar to boosting) is a promising direction that this paper did not address. Performing the aggregation in a probabilistic manner seems viable, as we found the errors of different paths are sufficiently uncorrelated, suggesting possibility of assembling a strong estimator.

Unlabeled/Unpaired Data: The current framework requires paired training data. Extending the concept to unlabeled/unpaired data, e.g., as in , appears feasible and remains open for future work.

Categorical/Low-Dimensional Tasks: We primarily experimented with pixel-wise tasks. Classification tasks, and generally tasks with low-dimensional outputs, will be interesting to experiment with, especially given the more severely ill-posed cross-task relationships they induce.

Optimization Limits: The improvements gained by incorporating consistency are bounded by the success of available optimization techniques, as addition of consistency constrains at times makes the optimization job harder. Also, implementing cross-task functions as neural networks makes them subject to certain output artifacts similar to those seen in image synthesis with neural networks.

Adversarial Robustness: Lastly, if learning with cross-task consistency indeed reduces the tendency of neural networks to learn surface statistics (Sec. 1), studying its implications in defence against adversarial attacks will be worthwhile.

Energy Analyses: We performed post-hoc analyses on the Consistency Energy. More concrete understanding of the properties of the energy and potentially using it actively for network modification, e.g, in unsupervised domain adaptation, requires further focused studies.

Acknowledgement: This work was supported by a grant from SAIL Toyota Center for AI ResearchToyota Research Institute (“TRI”) provided funds to assist the authors with their research but this article solely reflects the opinions and conclusions of its authors and not TRI or any other Toyota entity., a Vannevar Bush Faculty Fellowship, ONR MURI grant N00014-14-1-0671, an Amazon AWS Machine Learning Award, and Google Cloud.

References