DISCO Nets: DISsimilarity COefficient Networks

Introduction

Related Work

Deep neural networks, and in particular, Convolutional Neural Networks (CNNs) are comprised of several convolutional layers, followed by one or more fully connected (dense) layers, interleaved by non-linear function(s) and (optionally) pooling. Recent probabilistic models use CNNs to represent non-linear functions of the data. We observe that such models separate into two types. The first type of model does not explicitly compute the probability distribution of interest. Rather, these models allow the user to sample from this distribution by feeding the CNN with some noise $\bm{z}$. Among such models, Generative Adversarial Networks (GAN) presented in Goodfellow et al. are very popular and have been used in several computer vision applications, for example in Denton et al. , Springenberg , Radford et al. and Yan et al. . A GAN model consists of two networks, simultaneously trained in an adversarial manner. A generative model, referred as the Generator G, is trained to replicate the data from noise, while an adversarial discriminative model, referred as the Discriminator D, is trained to identify whether a sample comes from the true data or from G. The GAN training objective is based on a minimax game between the two networks and approximately optimizes a Jensen-Shannon divergence. However, as mentioned in Goodfellow et al. and Radford et al. , GAN models require very careful design of the networks’ architecture. Their training procedure is tedious and tends to oscillate. GAN models have been generalized to conditional GAN (cGAN) in Mirza and Osindero , where some additional input information can be fed to the Generator and the Discriminator. For example in Mirza and Osindero a cGAN model generates tags corresponding to an image. Gauthier applies cGAN to face generation. Reed et al. propose to generate images of flowers with a cGAN model, where the conditional information is a word description of the flower to generateAt the time writing, we do not have access to the full paper of Reed et al. and therefore cannot take advantage of this work in our experimental comparison.. While the application of cGAN is very promising, little quantitative evaluation has been done. Furthermore, cGAN models suffer from the same difficulties we mentioned for GAN. Another line of work has developed towards the use of statistical hypothesis testing to learn probabilistic models. In Dziugaite et al. and Li et al. , the authors propose to train generative deep networks with an objective function based on the Maximum Mean Discrepancy (MMD) criterion. The MMD method (see Gretton et al. ) is a statistical hypothesis test assessing if two probabilistic distributions are similar. As mentioned in Dziugaite et al. , the MMD test can been seen as playing the role of an adversary. The second type of model approximates intractable posterior distributions with use of variational inference. The Variational Auto-Encoders (VAE) presented in Kingma and Welling is composed of a probabilistic encoder and a probabilistic decoder. The probabilistic encoder is fed with the input $\bm{x}\in\mathcal{\bm{X}}$ and produces a posterior distribution $P(\bm{z}|\bm{x})$ over the possible values of noise $\bm{z}$ that could have generated $\bm{x}$. The probabilistic decoder learns to map the noise $\bm{z}$ back to the data space $\mathcal{\bm{X}}$. The training of VAE uses an objective function based on a Kullback-Leibler Divergence. VAE and GAN models have been combined in Makhzani et al. , where the authors propose to regularise autoencoders with an adversarial network. The adversarial network ensures that the posterior distribution $P(\bm{z}|\bm{x)}$ matches an arbitrary prior $P(\bm{z})$. In hand pose estimation, imagine the user wants to obtain accurate positions of the thumb and the index finger but does not need accurate locations of the other fingers. The task loss $\Delta_{\text{task}}$ might be based on a weighted L2-norm between the predicted and the ground-truth poses, with high weights on the thumb and the index. Existing probabilistic models cannot be tailored to task-specific losses and we propose the DISsimilarity COefficient Networks (DISCO Nets) to alleviate this deficiency.

DISCO Nets

We begin the description of our model by specifying how it can be used to generate samples from the posterior distribution, and how the samples can in turn be employed to provide a pointwise estimate. In the subsequent subsection, we describe how to estimate the parameters of the model.

A DISCO Net consists of several convolutional and dense layers (interleaved by non-linear function(s) and possibly pooling) and takes as input a pair $(\bm{x},\bm{z})\in\mathcal{\bm{X}}\times\mathcal{Z}$, where $\bm{x}$ is input data and $\bm{z}$ is some random noise. Given one pair $(\bm{x},\bm{z})$, the DISCO Net produces a value for the output $\bm{y}$. In the example of hand pose estimation, the input depth image $\bm{x}$ is fed to the convolutional layers. The output of the last convolutional layer is flattened and concatenated with a noise sample $\bm{z}$. The resulting vector is fed to several dense layers, and the last dense layer outputs a pose $\bm{y}$. From a single depth image $\bm{x}$, by using different noise samples, the DISCO Net produces different pose candidates for the depth image. This process is illustrated in Figure 2. Importantly, DISCO Nets are flexible in the choice of the architecture. For example, the noise could be concatenated at any stage of the network, including at the start.

Pointwise Prediction.

In order to obtain a single prediction $\bm{y}$ for a given input $\bm{x}$, DISCO Nets use the principle of Maximum Expected Utility (MEU), similarly to Premachandran et al. . The prediction $\bm{y}_{\Delta_{\text{task}}}$ maximises the expected utility, or rather minimises the expected task-specific loss $\Delta_{\text{task}}$, estimated using the sampled candidates. Formally, the prediction is made as follows:

where $(\bm{y_{1}},...,\bm{y}_{K})$ are the candidate outputs sampled for the single input $\bm{x}$. Details on the MEU method are in the supplementary material.

2 Learning DISCO Nets

We want DISCO Nets to accurately model the true probability $P(\bm{y}|\bm{x})$ via $Q(\bm{y}|\bm{x})$. In other words, $Q(\bm{y}|\bm{x})$ should be as similar as possible to $P(\bm{y}|\bm{x})$. This similarity is evaluated with respect to the loss specific to the task at hand. Given any non-negative symmetric loss function between two outputs $\Delta(\bm{y},\bm{y^{\prime}})$ with $(\bm{y},\bm{y^{\prime}})\in\mathcal{\bm{Y}}\times\mathcal{\bm{Y}}$, we employ a diversity coefficient that is the expected loss between two samples drawn randomly from the two distributions. Formally, the diversity coefficient is defined as:

Intuitively, we should minimise $\textrm{DIV}_{\Delta}(P,Q,D)$ so that $Q(\bm{y}|\bm{x})$ is as similar as possible to $P(\bm{y}|\bm{x})$. However there is uncertainty on the output $\bm{y}$ to predict for a given $\bm{x}$. In other words, $P(\bm{y}|\bm{x})$ is diverse and $Q(\bm{y}|\bm{x})$ should be diverse as well. Thus we encourage $Q(\bm{y}|\bm{x})$ to provide sample outputs, for a given $\bm{x}$, that are diverse by minimising the following dissimilarity coefficient:

with $\gamma\in$. The dissimilarity $\textrm{DISC}_{\Delta}(P,Q,D)$ is the difference between the diversity between $P$ and $Q$, and an affine combination of the diversity of each distribution, given $\bm{x}\sim D(\bm{x})$. These coefficients were introduced by Rao with $\gamma=\nicefrac{{1}}{{2}}$ and used for latent variable models by Kumar et al. . We do not need to consider the term $\textrm{DIV}_{\Delta}(P,P,D)$ as it is a constant in our problem, and thus the DISCO Nets objective function is defined as follows:

When minimising $F$, the term $\gamma\textrm{DIV}_{\Delta}(Q,Q,D)$ encourages $Q(\bm{y}|\bm{x})$ to be diverse. The value of $\gamma$ balances between the two goals of $Q(\bm{y}|\bm{x})$ that are providing accurate outputs while being diverse.

Optimisation.

Let us consider a training dataset composed of $N$ examples input-output pairs $D=\{(\bm{x}_{n},\bm{y}_{n}),n=1..N\}$. In order to train DISCO Nets, we need to compute the objective function of equation (4). We do not have knowledge of the true probability distributions $P(\bm{y},\bm{x})$ and $P(\bm{x})$. To overcome this deficiency, we construct estimators of each diversity term $\textrm{DIV}_{\Delta}(P,Q)$ and $\textrm{DIV}_{\Delta}(Q,Q)$. First, we take an empirical distribution of the data, that is, taking ground-truth pairs $(\bm{x}_{n},\bm{y}_{n})$. We then estimate each distribution $Q(\bm{y}|\bm{x}_{n})$ by sampling $K$ outputs from our model for each $\bm{x}_{n}$. This gives us an unbiased estimate of each diversity term, defined as:

We have an unbiased estimate of the DISCO Nets’ objective function of equation (4):

where $\bm{y}_{k}=G(\bm{z}_{k},\bm{x}_{n};\bm{\theta})$ is a candidate output sampled from DISCO Nets for ($\bm{x}_{n}$,$\bm{z}_{k}$), and $\bm{\theta}$ are the parameters of DISCO Nets. It is important to note that the second term of equation $\eqref{trainingobj}$ is summing over $k$ and $k^{\prime}\neq k$ to have an unbiased estimate, therefore we compute the loss between pairs of different samples $G(\bm{z}_{k},\bm{x}_{n};\bm{\theta})$ and $G(\bm{z}_{k^{\prime}},\bm{x}_{n};\bm{\theta})$. The parameters $\bm{\theta}$ are learned by Gradient Descent. Algorithm 1 shows the training of DISCO Nets. In steps 4 and 5 of Algorithm 1, we draw $K$ random noise vectors $(\bm{z}_{n,1},...\bm{z}_{n,k})$ per input example $\bm{x}_{n}$, and generate $K$ candidate outputs $G(\bm{z}_{n,k},\bm{x}_{n};\bm{\theta})$ per input. This allow us to compute an unbiased estimate of the gradient in step 7. For clarity, in the remainder of the paper we do not explicitely write the parameters $\bm{\theta}$ and write $G(\bm{z}_{k},\bm{x}_{n})$.

3 Strictly Proper Scoring Rules.

A scoring rule $S(Q,P)$, as defined in Gneiting and Raftery , evaluates the quality of a predictive distribution $Q$ with respect to a true distribution $P$. When using a scoring rule one should ensure that it is proper, which means it is maximised when $P=Q$. A scoring rule is said to be strictly proper if $P=Q$ is the unique maximiser of $S$. Hence maximising a proper scoring rule ensures that the model aims at predicting relevant forecast. Gneiting and Raftery define score divergences corresponding to a proper scoring rule S:

If $S$ is proper, $d$ is a valid non-negative divergence function, with value 0 if (and only if, in the case of strictly proper) $Q=P$. For example the MMD criterion (see Gretton et al. ) mentioned in Section 2 is an example of this type of divergence. In our case, any loss $\Delta$ expressed with an universal kernel will define the DISCO Nets’ objective function as such divergence (see Zawadzki and Lahaie ). For example, by Theorem 5 of Gneiting and Raftery , if we take as loss function $\Delta_{\beta}(\bm{y},\bm{y^{\prime}})=||\bm{y}-\bm{y^{\prime}}||_{2}^{\beta}=\sum_{i=1}^{d_{y}}|(y^{i}-y^{\prime i}|^{2})^{\nicefrac{{\beta}}{{2}}}$ with $\beta\in$ excluding 0 and 2, our training objective is (the negative of) a strictly proper scoring rule, that is:

This score has been referred in the litterature as the Energy Score in Gneiting et al. , Pinson and Tastu , Gneiting and Raftery . By employing a (strictly) proper scoring rule we ensure that our objective function is (only) minimised at the true distribution $P$, and expect DISCO Nets to generalise better on unseen data. We show below that strictly proper scoring rules are also relevant to assess the quality of the distribution $Q$ captured by the model.

Discriminative power of proper scoring rules.

As observed in Fukumizu et al. , kernel density estimation (KDE) fails in high dimensional output spaces. Our goal is to compare the quality of the distribution captured between two models, $Q_{1}$ and $Q_{2}$. In our setting $Q_{1}$ better models $P$ than $Q_{2}$ according to the scoring rule $S$ and its associated divergence $d$ if $d(Q_{1},P)<d(Q_{2},P)$. As noted in Pinson and Tastu , $S$ being proper does not ensure $d(Q_{1},\bm{y})<d(Q_{2},\bm{y})$ for all observations $\bm{y}$ drawn from $P$. However if the scoring rule is strictly proper scoring rule, this property should be ensured in the neighbourhood of the true distribution.

Experiments : Hand Pose Estimation

Given a depth image $\bm{x}$, which often contains occlusions and missing values, we wish to predict the hand pose $\bm{y}$. We use the NYU Hand Pose dataset of Tompson et al. to estimate the efficiency of DISCO Nets for this task.

The NYU Hand Pose dataset of Tompson et al. contains 8252 testing and 72,757 training frames of captured RGBD data with ground-truth hand pose information. The training set is composed of images of one person whereas the testing set gathers samples from two persons. For each frame, the RGBD data from 3 Kinects is provided: a frontal view and 2 side views. In our experiments we use only the depth data from the frontal view. While the ground truth contains J = 36 annotated joints, we follow the evaluation protocol of Oberweger et al. and use the same subset of J = 14 joints. We also perform the same data preprocessing as in Oberweger et al. , and extract a fixed-size metric cube around the hand from the depth image. We resize the depth values within the cube to a $128\times 128$ patch and normalized them in $$. Pixels deeper than the back of the cube and missing depth values are both set to a depth of 1.

Methods.

We employ loss functions between two outputs of the form of the Energy score (8), that is, $\Delta_{\textrm{training}}=\Delta_{\beta}(\bm{y},\bm{y^{\prime}})=||\bm{y}-\bm{y^{\prime}}||_{2}^{\beta}$. Our first goal is to assess the advantages of DISCO Nets with respect to non-probabilistic deep networks. One model, referred as $\text{DISCO}_{\beta,\gamma}$, is a DISCO Nets probabilistic model, with $\gamma\neq 0$ in the dissimilarity coefficient of equation (6). When taking $\gamma=0$, noise is injected and the model capacity is the same as $\text{DISCO}_{\beta,\gamma\neq 0}$. The model $\text{BASE}_{\beta}$, is a non-probabilistic model, by taking $\gamma=0$ in the objective function of equation (6) and no noise is concatenated. This corresponds to a classic deep network which for a given input $\bm{x}$ generates a single output $\bm{y}=G(\bm{x})$. Note that we write $G(\bm{x})$ and not $G(\bm{z},\bm{x})$ since no noise is concatenated.

Evaluation Metrics.

We report classic non-probabilistic metrics for hand pose estimation employed in Oberweger et al. and Taylor et al. , that are, the Mean Joint Euclidean Error (MeJEE), the Max Joint Euclidean Error (MaJEE) and the Fraction of Frames within distance (FF). We refer the reader to the supplementary material for detailed expression of these metrics. These metrics use the Euclidean distance between the prediction and the ground-truth and require a single pointwise prediction. This pointwise prediction is chosen with the MEU method among $K$ candidates. We added the probabilistic metric ProbLoss. ProbLoss is defined as in Equation 8 with the Euclidean norm and is the divergence associated with a strictly proper scoring rule. Thus, ProbLoss ranks the ability of the models to represent the true distribution. ProbLoss is computed using $K$ candidate poses for a given depth image. For the non-probabilistic model $\text{BASE}_{\beta}$, only a single pointwise predicted output $\bm{y}$ is available. We construct the $K$ candidates by adding some Gaussian random noise of mean and diagonal covariance $\Sigma=\sigma\mathds{1}$, with $\sigma\in\{1\textrm{mm},5\textrm{mm},10\textrm{mm}\}$ and refer to the model as $\text{BASE}_{\beta,\sigma}$. We also evaluate the non-probabilistic model $\text{BASE}_{\beta}$ using its pointwise prediction rather than the MEU method. Results are consistent and detailed in the supplementary material.

Loss functions.

As we employ standard evaluation metrics based on the Euclidean norm, we train with the Euclidean norm (that is, $\Delta_{\textrm{training}}(\bm{y},\bm{y^{\prime}})=||\bm{y}-\bm{y^{\prime}}||_{2}^{\beta}$ with $\beta=1$). When $\gamma=\frac{1}{2}$ our objective function coincides with ProbLoss.

Architecture.

Training.

We use 10,000 examples from the 72,757 training frames to construct a validation dataset and train only on 62,757 examples. Back-propagation is used with Stochastic Gradient Descent with a batchsize of $256$. The learning rate is fixed to $\lambda=0.01$ and we use a momentum of $m=0.9$ (see Polyak ). We also add L2-regularisation controlled by the parameter $C$. We use $C=[0.0001,0.001,0.01]$ which is a relevant range as the comparative model $\text{BASE}_{\beta}$ is best performing for $C=0.001$. Note that DISCO Nets report consistent performances across the different values $C$, contrarily to $\text{BASE}_{\beta}$. We use 3 different random seeds to initialize each model network parameters. We report the performance of each model with its best cross-validated seed and $C$. We train all models for 400 epochs as it results in a change of less than $3\%$ in the value of the loss on the validation dataset for $\text{BASE}_{\beta}$. We refer the reader to the supplementary material for details on the setting.

2 Results.

Table LABEL:tab1 reports performances on the test dataset, with parameters cross-validated on the validation set. All versions of the DISCO Net model outperform the $\text{BASE}_{\beta}$ model. Among the different values of $\gamma$, we see that $\gamma=0.5$ better captures the true distribution (lower ProbLoss) while retaining accurate performance on the standard pointwise metrics. Interestingly, using an all-zero noise at test-time gives similar performances on pointwise metrics. We link this to the observation that both the MEAN and the MEU method perform equivalently on these metrics (see supplementary material).

Qualitative Evaluation.

In Figure 3 we show candidate poses generated by $\text{DISCO}_{\beta=1,\gamma=0.5}$ for 3 testing examples. The left image shows the input depth image, and the right image shows the ground-truth pose (in grey) with 100 candidate outputs (superimposed in transparent red). The model predict the joint locations and we interpolate the joints with edges. If an edge is thinner and more opaque, it means the different predictions overlap and that the uncertainty on the location of the edge’s joints is low. We can see that $\text{DISCO}_{\beta=1,\gamma=0.5}$ captures relevant information on the structure of the hand.

Figure 4 shows the matrices of Pearson product-moment correlation coefficients between joints. We note that DISCO Net with $\gamma=0.5$ better captures the correlation between the joints of a finger and between the fingers.

3 Comparison with existing probabilistic models.

To the best of our knowledge the conditional Generative Adversarial Networks (cGAN) from Mirza and Osindero has not been applied to pose estimation. In order to compare cGAN to DISCO Nets, several issues must be overcome. First, we must design a network architecture for the Discriminator. This is a first disadvantage of cGAN compared to DISCO Nets which require no adversary. Second, as mentioned in Goodfellow et al. and Radford et al. , GAN (and thus cGAN) require very careful design of the networks’ architecture and training procedure. In order to do a fair comparison, we followed the work in Mirza and Osindero and practical advice for GAN presented in Larsen and Sønderby . We try (i) cGAN, initialising all layers of D and G randomly, and (ii) $\text{cGAN}_{\text{init, fixed}}$ initialising the convolutional layers of G and D with the trained best-performing $\text{DISCO}_{\beta=1,\gamma=0.5}$ of Section 4.2, and keeping these layers fixed. That is, the convolutional parts of G and D are fixed feature extractors for the depth image. This is a setting similar to the one employed for tag-annotation of images in Mirza and Osindero . Details on the setting can be found in the supplementary material. Table LABEL:cgantab shows that the cGAN model obtains relevant results only when the convolutional layers of G and D are initialised with our trained model and kept fixed, that is $\text{cGAN}_{\text{init, fixed}}$. These results are still worse than DISCO Nets performances. While there may be a better architecture for cGAN, our experiments demonstrate the difficulty of training cGAN over DISCO Nets.

4 Reference state-of-the-art values.

We train the best-performing $\text{DISCO}_{\beta=1,\gamma=0.5}$ of Section 4.2 on the entire dataset, and compare performances with state-of-the-art methods in Table 4 and Figure 5. These state-of-the-art methods are specifically designed for hand pose estimation. In Oberweger et al. a constrained prior hand model, referred as NYU-Prior, is refined on each hand joint position to increase accuracy, referred as NYU-Prior-Refined. In Oberweger et al. , the input depth image is fed to a first network NYU-Init, that outputs a pose used to synthesize an image with a second network. The synthesized image is used with the input depth image to derive a pose update. We refer to the whole model as NYU-Feedback. On the contrary, DISCO Nets uses a single network whose architecture is similar to NYU-Prior (without constraining on a pose prior). By accurately modeling the distribution of the pose given the depth image, DISCO Nets obtain comparable performances to NYU-Prior and NYU-Prior-Refined. Without any extra effort, DISCO Nets could be embedded in the presented refinement and feedback methods, possibly boosting state-of-the-art performances. Table 4: DISCO Nets compared to state-of-the-art performances $\pm$ SEM. Model MeJEE (mm) MaJEE (mm) FF (80mm) NYU-Prior 20.7$\pm$0.150 44.8$\pm$0.289 91.190 NYU-Prior-Refined 19.7$\pm$0.157 44.7$\pm$0.327 88.148 NYU-Init 27.4$\pm$0.152 55.4$\pm$0.265 86.537 NYU-Feedback 16.0$\pm$0.096 36.1$\pm$0.208 97.334 $\text{DISCO}_{\beta=1,\gamma=0.5}$ 20.7$\pm$0.121 45.1$\pm$0.246 93.250 Figure 5: Fractions of frames within distance $d~{}\textrm{in mm}$ (by 5 mm). Best viewed in color.

Discussion.

We presented DISCO Nets, a new family of probabilistic model based on deep networks. DISCO Nets employ a prediction and training procedure based on the minimisation of a dissimilarity coefficient. Theoretically, this ensures that DISCO Nets accurately capture uncertainty on the correct output to predict given an input. Experimental results on the task of hand pose estimation consistently support our theoretical hypothesis as DISCO Nets outperform non-probabilistic equivalent models, and existing probabilistic models. Furthermore, DISCO Nets can be tailored to the task to perform. This allows a possible user to train them to tackle different problems of interest. As their novelty resides mainly in their objective function, DISCO Nets do not require any specific architecture and can be easily applied to new problems. We contemplate several directions for future work. First, we will apply DISCO Nets to other prediction problems where there is uncertainty on the output. Second, we would like to extend DISCO Nets to latent variables models, allowing us to apply DISCO Nets to diverse dataset where ground-truth annotations are missing or incomplete.

Acknowlegements.

This work is funded by the Microsoft Research PhD Scholarship Programme. We would like to thank Pankaj Pansari, Leonard Berrada and Ondra Miksik for their useful discussions and insights.

References.