Additive Margin Softmax for Face Verification

Introduction

Face verification is widely used for identity authentication in enormous areas such as finance, military, public security and so on. Nowadays, most face verification models are built upon Deep Convolutional Neural Networks and supervised by classification loss functions , metric learning loss functions or both . Metric learning loss functions such as contrastive loss or triplet loss usually require carefully designed sample mining strategies and the final performance is very sensitive to these strategies, so increasingly more researchers shift their attentions to building deep face verification models based on improved classification loss functions .

Current prevailing classification loss functions for deep face recognition are mostly based on the widely-used softmax loss. The softmax loss is typically good at optimizing the inter-class difference (i.e., separating different classes), but not good at reducing the intra-class variation (i.e., making features of the same class compact). To address this, lots of new loss functions are proposed to minimize the intra-class variation. proposed to add a regularization term to penalize the feature-to-center distances. In , researchers proposed to use a scale parameter to control the ”temperature” of the softmax loss, producing higher gradients to the well-separated samples to further shrink the intra-class variance. In , the authors introduced an conceptually appealing angular margin to push the classification boundary closer to the weight vector of each class. also provided a theoretical guidance of training a deep model for metric learning tasks using the classification loss functions. also improved the softmax loss by incorporating differnet kinds of margins.

In this work, we propose a novel and more interpretable way to import the angular margin into the softmax loss. We formulate an additive margin via $\cos\theta-m$, which is simpler than and yields better performance. From Equation (3), we can see that $m$ is multiplied to the target angle $\theta_{y_{i}}$ in , so this type of margin is incorporated in a multiplicative manner. Since our margin is a scalar subtracted from $cos\theta$, we call our loss function Additive Margin Softmax (AM-Softmax).

Experiments on LFW BLUFR protocol and MegaFace show that our loss function with the same network architecture achieves better results than the current state-of-the-art approaches.

Preliminaries

To better understand the proposed AM-Softmax loss, we will first give a brief review of the original softmax and the A-softmax loss . The formulation of the original softmax loss is given by

where $\bm{f}$ is the input of the last fully connected layer ($\bm{f}_{i}$ denotes the the $i$-th sample), $W_{j}$ is the $j$-th column of the last fully connected layer. The $W_{y_{i}}^{T}\bm{f}_{i}$ is also called as the target logit of the $i$-th sample.

In the A-softmax loss, the authors proposed to normalize the weight vectors (making $\|W_{i}\|$ to be $1$) and generalize the target logit from $\|\bm{f}_{i}\|cos(\theta_{y_{i}})$ to $\|\bm{f}_{i}\|\psi(\theta_{y_{i}})$,

where the $\psi(\theta)$ is usually a piece-wise function defined as

where $m$ is usually an integer larger than $1$ and $\lambda$ is a hyper-parameter to control how hard the classification boundary should be pushed. During training, the $\lambda$ is annealing from $1,000$ to a small value to make the angular space of each class become more and more compact. In their experiments, they set the minimum value of $\lambda$ to be 5 and $m=4$, which is approximately equivalent to $m=1.5$ (Figure 2).

Additive Margin Softmax

In this section, we will first describe the definition of the proposed loss function. Then we will discuss about the intuition and interpretation of the loss function.

defines a general function $\psi(\theta)$ to introduce the large margin property. Motivated by that, we further propose a specific $\psi(\theta)$ that introduces an additive margin to the softmax loss function. The formulation is given by

Compared to the $\psi(\theta)$ defined in L-Softmax and A-softmax (Equation (3)), our definition is more simple and intuitive. During implementation, the input after normalizing both the feature and the weight is actually $x=cos\theta_{y_{i}}=\frac{W_{y_{i}}^{T}f_{i}}{\|W_{y_{i}}\|\|f_{i}\|}$, so in the forward propagation we only need to compute

In this margin scheme, we don’t need to calculate the gradient for back-propagation because $\Psi^{\prime}(x)=1$. It is much easier to implement compared with SphereFace .

Since we use cosine as the similarity to compare two face features, we follow to apply both feature normalization and weight normalization to the inner product layer in order to build a cosine layer. Then we scale the cosine values using a hyper-parameter $s$ as suggested in . Finally, the loss function becomes

In this paper, we assume that the norm of both $W_{i}$ and $\bm{f}$ are normalized to 1 if not specified. In , the authors propose to let the scaling factor $s$ to be learned through back-propagation. However, after the margin is introduced into the loss function, we find that the $s$ will not increase and the network converges very slowly if we let $s$ to be learned. Thus, we fix $s$ to be a large enough value, e.g. $30$, to accelerate and stablize the optimization.

As described in Section 2, propose to use an annealing strategy to set the hyper-parameter $\lambda$ to avoid network divergence. However, to set the annealing curve of $\lambda$, lots of extra parameters are introduced, which are more or less confusing for starters. Although properly tuning those hyper-parameters for $\lambda$ could lead to impressive results, the hyper-parameters are still quite difficult to tune. With our margin scheme, we find that we no longer need the help of the annealing strategy. The network can converge flexibly even if we fix the hyper-parameter $m$ from scratch. Compared to SphereFace , our additive margin scheme is more friendly to those who are not familiar with the effects of the hyper-parameters. Another recently proposed additive margin is also described in . Our AM-Softmax is different than in the sense that our feature and weight are normalized to a predefined constant $s$. The normalization is the key to the angular margin property. Without the normalization, the margin $m$ does not necessarily lead to large angular margin.

2 Discussion

Our additive margin scheme has a clear geometric interpretation on the hypersphere manifold. In Figure 3, we draw a schematic diagram to show the decision boundary of both conventional softmax loss and our AM-Softmax. For example, in Figure 3, the features are of $2$ dimensions. After normalization, the features are on a circle and the decision boundary of the traditional softmax loss is denoted as the vector $P_{0}$. In this case, we have $W_{1}^{T}P_{0}=W_{2}^{T}P_{0}$ at the decision boundary $P_{0}$.

For our AM-Softmax, the boundary becomes a marginal region instead of a single vector. At the new boundary $P_{1}$ for class $1$, we have $W_{1}^{T}P_{1}-m=W_{2}^{T}P_{1}$, which gives $m=(W_{1}-W_{2})^{T}P_{1}=\cos(\theta_{W_{1},P_{1}})-\cos(\theta_{W_{2},P_{1}})$. If we further assume that all the classes have the same intra-class variance and the boundary for class $2$ is at $P_{2}$, we can get $\cos(\theta_{W_{2},P_{1}})=\cos(\theta_{W_{1},P_{2}})$ (Fig. 3). Thus, $m=\cos(\theta_{W_{1},P_{1}})-\cos(\theta_{W_{1},P_{2}})$, which is the difference of the cosine scores for class $1$ between the two sides of the margin region.

2.2 Angular Margin or Cosine Margin

In SphereFace , the margin $m$ is multiplied to $\theta$, so the angular margin is incorporated into the loss in a multiplicative way. In our proposed loss function, the margin is enforced by subtracting $m$ from $\cos\theta$, so our margin is incorporated into the loss in an additive way, which is one of the most significant differences than . It is also worth mentioning that despite the difference of enforcing margin, these two types of margin formulations are also different in the base values. Specifically, one is $\theta$ and the other is $\cos\theta$. Although usually the cosine margin has an one-to-one mapping to the angular margin, there will still be some difference while optimizing them due to the non-linearity induced by the cosine function.

Whether we should use the cosine margin depends on which similarity measurement (or distance) the final loss function is optimizing. Obviously, our modified softmax loss function is optimizing the cosine similarity, not the angle. This may not be a problem if we are using the conventional softmax loss because the decision boundaries are the same in these two forms ($\cos\theta_{1}=\cos\theta_{2}\Rightarrow\theta_{1}=\theta_{2}$). However, when we are trying to push the boundary, we will face a problem that these two similarities (distances) have different densities. Cosine values are more dense when the angles are near or $\pi$. If we want to optimize the angle, an $\arccos$ operation may be required after the value of the inner product $W^{T}\bm{f}$ is obtained. It will potentially be more computationally expensive.

In general, angular margin is conceptually better than the cosine margin, but considering the computational cost, cosine margin is more appealing in the sense that it could achieve the same goal with less efforts.

2.3 Feature Normalization

In the SphereFace model , the authors added the weight normalization based on Large Margin Softmax , leaving the feature still not normalized. Our loss function, following , applies feature normalization and uses a global scale factor $s$ to replace the sample-dependent feature norm in SphereFace . One question arises: when should we add the feature normalization?

Our answer is that it depends on the image quality. In ’s Figure 1, we can see that the feature norm is highly correlated with the quality of the image. Note that back propagation has a property that,

Thus, after normalization, features with small norms will get much bigger gradient compared with features that have big norms (Figure 5). By back-propagation, the network will pay more attention to the low-quality face images, which usually have small norms. Its effect is very similar with hard sample mining . The advantages of feature normalization are also revealed in . As a conclusion, feature normalization is most suitable for tasks whose image quality is very low.

From Figure 5 we can see that the gradient norm may be extremely big when the feature norm is very small. This potentially increases the risk of gradient explosion, even though we may not come across many samples with very small feature norm. Maybe some re-weighting strategy whose feature-gradient norm curve is between the two curves in Figure 5 could potentially work better. This is an interesting topic to be studied in the future.

2.4 Feature Distribution Visualization

To better understand the effect of our loss function, we designed a toy experiment to visualize the feature distributions trained by several loss functions. We used Fashion MNIST (10 classes) to train several 7-layer CNN models which output 3-dimensional features. These networks are supervised by different loss functions. After we obtain the 3-dimensional features, we normalize and plot them on a hypersphere (ball) in the 3 dimensional space (Figure 4).

From the visualization, we can empirically show that our AM-Softmax performs similarly with the best SphereFace (A-Softmax) model when we set $s=10,m=0.2$. Moreover, our loss function can further shrink the intra-class variance by setting a larger $m$. Compared to A-Softmax , the AM-Softmax loss also converges easier with proper scaling factor $s$. The visualized 3D features well demonstrates that AM-Softmax could bring the large margin property to the features without tuning too many hyper-parameters.

Experiment

In this section, we will firstly describe the experimental settings. Then we will discuss the overlapping problem of the modern in-the-wild face datasets. Finally we will compare the performance of our loss function with several previous state-of-the-art loss functions.

Our loss function is implemented using Caffe framework . We follow all the experimental settings from , including the image resolution, preprocessing method and the network structure. Specifically speaking, we use MTCNN to detect faces and facial landmarks in images. Then the faces are aligned according to the detected landmarks. The aligned face images are of size $112\times 96$, and are normalized by subtracting 128 and dividing 128. Our network structure follows , which is a modified ResNet with 20 layers that is adapted to face recognition.

All the networks are trained from scratch. We set the weight decay parameter to be $510-4$. The batch size is 256 and the learning rate begins with 0.1 and is divided by 10 at the 16K, 24K and 28K iterations. The training is finished at 30K iterations. During training, we only use image mirror to augment the dataset.

In testing phase, We feed both frontal face images and mirror face images and extract the features from the output of the first inner-product layer. Then the two features are summed together as the representation of the face image. When comparing two face images, cosine similarity is utilized as the measurement.

2 Dataset Overlap Removal

The dataset we use for training is CASIA-Webface , which contains 494,414 training images from 10,575 identities. To perform open-set evaluations, we carefully remove the overlapped identities between training dataset (CASIA-Webface ) and testing datasets (LFW and MegaFace ). Finally, we find 17 overlapped identities between CASIA-Webface and LFW, and 42 overlapped identities between CASIA-Webface and MegaFace set1. Note that there are only 80 identities in MegaFace set1, i.e. over half of the identities are already in the training dataset. The effect of overlap removal is remarkable for MegaFace (Table 2). To be rigorous, all the experiments in this paper are based on the cleaned dataset. We have made our overlap checking code publicly availablehttps://github.com/happynear/FaceDatasets to encourage researchers to clean their training datasets before experiments.

In our paper, we re-train some of the previous loss functions on the cleaned dataset as the baselines for comparison. Note that, we make our experiments fair by using the same network architecture and training dataset for every compared methods.

3 Effect of Hyper-parameter m𝑚m

There are two hyper-parameters in our proposed loss function, one is the scale $s$ and another is the margin $m$. The scale $s$ has already been discussed sufficiently in several previous works . In this paper, we directly fixed it to $30$ and will not discuss its effect anymore.

The main hyper-parameter in our loss function is the margin $m$. In Table 1, we list the performance of our proposed AM-Softmax loss function with $m$ varies from 0.25 to 0.5. From the table we can see that from $m=0.25$ to $0.3$, the performance improves significantly, and the performance become the best when $m=0.35$ to $m=0.4$.

We also provide the result for the loss function without feature normalization (noted as w/o FN) and the scale $s$. As we explained before, feature normalization performs better on low quality images like MegaFace, and using the original feature norm performs better on high quality images like LFW .

In Figure 6, we draw both of the CMC curves to evaluate the performance of identification and ROC curves to evaluate the performance of verification. From this figure, we can show that our loss function performs much better than the other loss functions when the rank or false positive rate is very low.

Conclusion and Future Work

In this paper, we propose to impose an additive margin strategy to the target logit of softmax loss with feature and weights normalized. Our loss function is built upon the previous margin schemes, but it is more simple and interpretable. Comprehensive experiments show that our loss function performs better than A-Softmax on LFW BLUFR and MegaFace .

There is still lots of potentials for the research of the large margin strategies. There could be more creative way of specifying the function $\psi(\theta)$ other than multiplication and addition. In our AM-Softmax loss, the margin is a manually tuned global hyper-parameter. How to automatically determine the margin and how to incorporate class-specific or sample-specific margins remain open questions and are worth studying.

References