Mitigating Unwanted Biases with Adversarial Learning

Introduction

Machine learning leverages data to build models capable of assessing the labels and properties of novel data. Unfortunately, the available training data frequently contains biases with respect to things that we would rather not use for decision making. Machine learning builds models faithful to training data and can lead to perpetuating these undesirable biases. For example, systems designed to predict creditworthiness and systems designed to perform analogy completion have been demonstrated to be biased against racial minorities and women respectively. Ideally we would be able to build a model which captures exactly those generalizations from the data which are useful for performing some task which are not discriminatory in a way which the people building those models consider unfair.

Work on training machine learning systems that output fair decisions has defined several useful measurements for fairness: Demographic Parity, Equality of Odds, and Equality of Opportunity. These can be imposed as constraints or incorporated into a loss function in order to mitigate disproportional outcomes in the system’s output predictions regarding a protected demographic, such as sex.

In this paper, we examine these fairness measures in the context of adversarial debiasing. We consider supervised deep learning tasks in which the task is to predict an output variable $Y$ given an input variable $X$, while remaining unbiased with respect to some variable $Z$. We refer to $Z$ as the protected variable. For these learning systems, the predictor $\hat{Y}=f(X)$ can be constructed as (input, output, protected) tuples $(X,Y,Z)$. The predictor $f(X)$ is usually given access to the protected variable $Z$, though this is not strictly necessary. This construction allows the determination of which types of bias are considered undesirable for a particular application to be chosen through the specification of the protected variable.

We speak to the concept of mitigating bias using the known term debiasingNote that “debias” may not be quite the right word, as all bias is not necessarily removed., following definitions provided by ? (?) and refined by ? (?).

Demographic Parity. A predictor $\hat{Y}$ satisfies demographic parity if $\hat{Y}$ and $Z$ are independent.

This means that $P(\hat{Y}=\hat{y})$ is equal for all values of the protected variable $Z$: $P(\hat{Y}=\hat{y})=P(\hat{Y}=\hat{y}|Z=z)$.

Equality of Odds. A predictor $\hat{Y}$ satisfies equality of odds if $\hat{Y}$ and $Z$ are conditionally independent given $Y$.

This means that, for all possible values of the true label $Y$, $P(\hat{Y}=\hat{y})$ is the same for all values of the protected variable: $P(\hat{Y}=\hat{y}|Y=y)=P(\hat{Y}=\hat{y}|Z=z,Y=y)$

Equality of Opportunity. If the output variable $Y$ is discrete, a predictor $\hat{Y}$ satisfies equality of opportunity with respect to a class $y$ if $\hat{Y}$ and $Z$ are independent conditioned on $Y=y$.

This means that, for a particular value of the true label $Y$, $P(\hat{Y}=\hat{y})$ is the same for all values of the protected variable: $P(\hat{Y}=\hat{y}|Y=y)=P(\hat{Y}=\hat{y}|Z=z,Y=y)$

We present an adversarial technique for achieving whichever one of these definitions is desired.Achieving equality of odds and demographic parity are generally incongruent goals. See also ? (?) for incongruency between calibration and equalized odds. A predictor $f$ will be trained to model $Y$ as accurately as possible while satisfying one of the above equality constraints. Demographic parity will be achieved by introducing an adversary $g$ which will attempt to predict a value for $Z$ from $\hat{Y}$. The gradient of $g$ will then be incorporated into the weight update rule of $f$ so as to reduce the amount of information about $Z$ transmitted through $\hat{Y}$. Equality of odds will be achieved by also giving $g$ access to the true label $Y$, thereby limiting any information about $Z$ which $\hat{Y}$ contains beyond the information already contained in $Y$.

We consider the case where the protected variable is a discrete feature present in the training set as well as the case in which the protected variable must be inferred from latent semantics (in particular, gender from word embeddings). In order to accomplish the latter we adapt a technique presented by ? (?) to define a subspace capturing the semantics of the protected variable, and then train a model to perform a word analogies task accurately, while unbiased on this protected variable. A consequence of this technique is that the network learns “debiased” embeddings, embeddings that have the semantics of the protected variable removed. These embeddings are still able to perform the analogy task well, but are better at avoiding problematic examples such as those shown in ? (?).

Results on the UCI Adult Dataset demonstrate the technique we introduce allows us to train a model that achieves equality of odds to within 1% on both protected groups.

We also compare with the related previous work of ? (?), and find we are able to better equalize the differences between the two groups, measured by both False Positive Rate and False Negative Rate (1 - True Positive Rate), although note that the previous work performs better overall for False Negative Rate.

We provide some discussion on caveats pertaining to this approach, difficulties in training these models that are shared by many adversarial approaches, as well as some discussion on difficulties that the fairness constraints introduce.

Related Work

There has been significant work done in the area of debiasing various specific types of data or predictor.

Debiasing word embeddings: ? (?) devises a method to remove gender bias from word embeddings. The method relies on a lot of human input; namely, it needs a large “training set” of gender-specific words.

Simple models: ? (?) demonstrate that removing the protected variable from the training data fails to yield a debiased model (since other variables can be highly correlated with the protected variable), and devise a method for learning fair predictive models in cases when the learning model is simple (e.g. linear regression). ? (?) discuss the shortcomings of focusing solely on demographic parity, present alternate definitions of fairness, and devise a method for deriving an unbiased predictor from a biased one, in cases when both the output variable and the protected variable are discrete.

Adversarial training: ? (?) pioneered the technique of using multiple networks with competing goals to force the first network to “deceive” the second network, applying this method to the problem of creating real-life-like pictures. ? (?) apply an adversarial training method to achieve equality of opportunity in cases when the output variable is discrete. They also discuss the ability of the adversary to be powerful enough to enforce a fairness constraint even when it has access to a very small training sample.

Adversarial Debiasing

We begin with a model, which we call the predictor, trained to accomplish the task of predicting $Y$ given $X$. As in Figure 1, we assume that the model is trained by attempting to modify weights $W$ to minimize some loss $L_{P}(\hat{y},y)$, using a gradient-based method such as stochastic gradient descent.

The output layer of the predictor is then used as an input to another network called the adversary which attempts to predict $Z$. This is part of the network corresponds to the discriminator in a typical GAN (?). We will suppose the adversary has loss term $L_{A}(\hat{z},z)$ and weights $U$. Depending on the definition of fairness being achieved, the adversary may have other inputs.

For Demographic Parity, the adversary gets the predicted label $\hat{Y}$. Intuitively, this allows the adversary to try to predict the protected variable using nothing but the predicted label. The goal of the predictor is to prevent the adversary from doing this.

For Equality of Odds, the adversary gets $\hat{Y}$ and the true label $Y$.

For Equality of Opportunity on a given class $y$, we can restrict the training set of the adversary to training examples where $Y=y$.This last technique of restricting the training set is discussed at length by ? (?), so we only mention it here.

In order for gradients to propagate correctly, $\hat{Y}$ above refers to the output layer of the network, not to the discrete prediction; for example, for a classification problem, $\hat{Y}$ could refer to the output of the softmax layer.

We update $U$ to minimize $L_{A}$ at each training time step, according to the gradient $\nabla_{U}L_{A}$. We modify $W$ according to the expression:

where $\alpha$ is a tuneable hyperparameter that can vary at each time step and we define $\text{proj}_{v}x=0$ if $v=0$.

The middle term $\text{proj}_{\nabla_{W}L_{A}}\nabla_{W}L_{P}$ prevents the predictor from moving in a direction that helps the adversary decrease its loss while the last term, $\alpha\nabla_{W}L_{A}$, attempts to increase the adversary’s loss. Without the projection term, it is possible for the predictor to end up helping the adversary (see Fig. 2). Without the last term, the predictor will never try to hurt the adversary, and, due to the stochastic nature of many gradient-based methods, will likely end up helping the adversary anyway. The result is that when training is completed the desired definition of equality should be satisfied.

Notice that our definitions and method make no assumptions about the nature of the output and protected variables: in particular, they work with both regression and classification models, as well as with both discrete and continuous protected variables.

Properties

We note several properties of the above method that we believe distinguish it from past work.

Generality: The above method can be used to enforce demographic parity, equality of odds, or equality of opportunity as described in ? (?). Further, it applies without modification to the cases when the output variable and/or protected variable are continuous instead of discrete.

Model-agnostic: The adversarial approach described can be applied regardless of how simple or complex the predictor’s model is, as long as the model is trained using a gradient-based method, as many modern learning models are. Further, as we will discuss later, at least in some situations, we suggest that the adversary does not need to be nearly as complex as the predictor—a simple adversary can be used with a complex predictor.

Optimality: Under certain conditions, we show that if the predictor converges, it must converge to a model that satisfies the desired fairness definition. Since the predictor also attempts to decrease the prediction loss $L_{P}$, the predictor should still perform well on the target task.

Theoretical Guarantees

Let the predictor, the adversary, and their weights $W$, $U$ be defined according to Section 3 Let $L_{A}(W,U)$ be the adversary’s loss, convex in $U$, concave in $W$,We understand that these assumptions are not satisfied in most use cases involving neural networks; however, as with most theoretical analyses of machine learning models (see, for example, ? (?) or ? (?); the former makes even stronger assumptions), assumptions of concavity are necessary for any proofs to work and continuously differentiable everywhere.

When the predictor’s weights are $W_{0}$, the predictor gives the same output $\hat{Y}$ regardless of input $X$. (For example, when $W_{0}=0$).

There are some weights $U_{0}$ that minimize $L_{A}$ when the weights for $\hat{Y}$ have no effect on the output: For all $W$, $L_{A}(W,U_{0})=\min_{U}L_{A}(W_{0},U)$.

Predictor and adversary converge to $W^{*}$ and $U^{*}$ respectively.

Then, $L_{A}(W^{*},U^{*})=L_{A}(W^{*},U_{0})$. That is, the adversary gains no advantage from using the weights for $\hat{Y}$.

Since the adversary converges, $L_{A}(W^{*},U^{*})\leq L_{A}(W^{*},U_{0})$: otherwise, since $L_{A}$ is convex in $U$, the adversary’s weights would move toward $U_{0}$. In other words, the adversary’s minimum is the point at which the adversary gains an advantage from using $\hat{Y}$. Similarly, since the predictor converges, $L_{A}(W^{*},U^{*})\geq L_{A}(W_{0},U^{*})$: Otherwise, the predictor would be able to increase the adversary’s loss by moving toward $W_{0}$, and the projection term and negative weight on $\nabla_{W}L_{A}$ in Eqn. 1 would push the predictor to move towards . Then:

so we must have $L_{A}(W^{*},U^{*})=L_{A}(W^{*},U_{0})$. ∎

Note that, in this proof, the adversary can be operating in a few different ways, as long as it is given $\hat{Y}$ as one of its inputs; for example, for demographic parity, it could be given only $\hat{Y}$; for equality of odds, it can be given both $\hat{Y}$ and $Y$.

We will show in the next propositions that the adversary gaining no advantage from information about $\hat{Y}$ is exactly the condition needed to guarantee that desired definitions of equality are satisfied.

Let the training data be comprised of triples $(X,Y,Z)$ drawn according to some distribution $D$. Suppose:

The adversary is trained for demographic parity; i.e. the adversary is given only the prediction $\hat{y}$.

The predictor completely fools the adversary; in particular, the adversary achieves loss $H(Z)$, the entropy of $Z$.

Then the predictor satisfies demographic parity; i.e., $\hat{Y}\perp Z$.

Notice that if the adversary draws $A(\hat{y})$ according to the distribution $Z|\hat{Y}=\hat{y}$, then its loss is exactly the conditional entropy

where the expectation is taken over $(x,y,z)\sim D$. Now suppose for contradiction that $\hat{Y}$ is dependent on $Z$. Then $H(Z|\hat{Y})<H(Z)$, so the adversary can achieve loss less than $H(Z)$, contradicting assumption (4). ∎

If assumptions (2)-(4) above are replaced with the analogous equality of odds assumptions; in particular, that the adversary is given $\hat{y}$ and $y$, and the adversary cannot achieve loss better than $H(Z|Y)$ then the predictor will satisfy Equality of Odds; i.e., $(\hat{Y}\perp Z)|Y$

Analogous to the above. Notice that if the adversary draws $A(\hat{y},y)\sim(Z|\hat{Y}=\hat{y},Y=y)$, then its loss is exactly the conditional entropy

where the expectation is again taken over $(x,y,z)\sim D$. But if $\hat{Y}$ is conditionally dependent on $Z$ given $Y$, then $H(Z|\hat{Y},Y)<H(Z|Y)$, so the adversary can achieve loss less than $H(Z|Y)$. ∎

These claims together illustrate that a sufficiently powerful adversary trained on a sufficiently large training set can indeed accurately enforce the demographic parity or equality of odds constraints on the predictor, if the adversary and predictor converge. Guaranteed convergence is harder to achieve, both in theory and practice. In the practical scenarios below we discuss methods to encourage the training algorithm to converge, as well as reasonable choices of the adversary model that are both powerful and easy to train.

Experiments

All models were trained using the Adam optimizer (?) for both predictor and adversary.

We generate a training sample ${(x^{(i)},y^{(i)},z^{(i)})}_{i=1}^{n}$ (where $z$ is the protected variable) as follows. For each $i$, let $r\in{0,1}$ be picked uniformly at random, and let $v\sim N(r_{i},1)$. Let $u,w\sim N(v_{i},1)$ vary independently. Then $x^{(i)}=(r,u),y^{(i)}=[w>0],z^{(i)}=r$. (where $[~{}]$ denotes an indicator function). Intuitively, the variable that we are trying to predict, $y$, depends directly on $v$ and $r$. We are given as inputs the protected variable $r$, and a noisy measurement of $v$. The end goal would be to train a model that predicts $y$ while being unbiased on $r$, effectively removing the direct signal for $r$ from the learned model.

If one trains generically a logistic regression model to predict $y$ given $x$, it outputs something like $y=\sigma(0.7u+0.7r)$, which is a reasonable model, but heavily incorporates the protected variable $r$. To debias, We now train a model that achieves demographic parity. Note that removing the variable $r$ from the training data is insuffucient for debiasing: the model will still learn to use $u$ to predict $y$, and $u$ is correlated with $r$. If we use the described technique and add in another logistic model that tries to predict $z$ given $y$, we find that the predictor model outputs something like $y=\sigma(0.6u-0.6r+0.6)$. Notice that not only is $r$ not included with a positive weight anymore, the model actually learns to use a negative weight on $r$ in order to balance out the effect of $r$ on $u$ Notice that $u-r\sim N(0,2)$; i.e., it is not dependent on $r$, so we have successfully trained a model to predict $y$ independently of $r$.

Word Embeddings

We train a model to perform the analogy task (i.e., fill in the blank: man : woman :: he : ?).

It is known that word embeddings reflect or amplify problematic biases from the data they are trained on, for example, gender (?). We seek to train a model that can still solve analogies well, but is less prone to these gender biases. We first calculate a “gender direction” $g$ using a method based on ? (?) which gives a method for defining the protected variable. We will use this technique in the context of defining gender for word embeddings, but, as discussed in ? (?), the technique generalizes to other protected variables and other forms of embeddings. Following ? (?), we pick 10 (male, female) word pairs, and define the and define the bias subspace to be the space spanned by the top $k$ principal components of the differences, where $k$ is a tuneable parameter. In our experiments, we find that $k=1$ gives reasonable results, so we did not experiment further.

As a model, we use the following: let $v=x_{2}+x_{3}-x_{1}$, and output $\hat{y}=v-ww^{T}v$, where our model parameter is $w$. Intuitively, $v$ is the “generic” analogy vector as is commonlysee e.g. ? (?) used for the analogy task. If left to its own devices (i.e., if not told to be unbiased on anything), the model should either learn $w=0$ or else learn $w$ as a useless vector.

By contrast, if we add the adversarial discriminator network (here, simply $\hat{z}=w_{2}^{T}\hat{y}$), we expect the debiased prediction model to learn that $w$ should be something close to $g$ (or $-g$), so that the discriminator cannot predict $z=proj_{g}y$. Indeed, both of these expectations hold: Without debiasing, the trained vector $w$ is approximately a unit vector nearly perpendicular to $g:w^{T}g=0.08,||w||=0.82$; with debiasing, $w$ is approximately a unit vector pointing in a direction highly correlated with $g:w^{T}g=0.55,||w||=0.96$. Even after debiasing, gendered analogies such as man : woman :: he : she are still preserved; however, many biased analogies go away, suggesting that the adversarial training process was indeed successful. An example of the kinds of changes in analogy completions observed after debiasing are illustrated in Table 1The presence of nurse in the second position may seem worrying, but it should be noted that in this particular set of word embeddings, nurse is the nearest neighbor to doctor; no amount of debiasing will change this..

UCI Adult Dataset

To better align with the work in ? (?), we attempt to enforce equality of odds on a model for the task of predicting the income of a person – in particular, predicting whether the income is >\50k$ – given various attributes about the person, as made available in the UCI Adult dataset (?).

Details on the features that the dataset provides are available in Table 2. We use both categorical and continuous columns as given, with exception to the fnlwgt feature, which we discard. We convert the remaining columns into tensors where the categorical columns are sparse tensors, age is bucketized at boundaries $$, and the rest of the continuous columns are real-valued.

As discussed before, to enforce equality of odds, we give the adversary access to the true label $y$. The adversary will learn the relationship between $y$ and $z$ regardless of what the predictor does; further, if the predictor’s predictions $\hat{y}$ give more information about $z$ than is already contained in $y$, the adversary will be able to improve its loss. Thus, the predictor, in attempting to fool the adversary, will move toward making sure that $\hat{y}$ does not give such additional information; in other words, toward equality of odds.

Our protected variable $z$ is a binary-valued variable for the two sexes annotated, male and female.

Our predictor model is straightforward logistic regression: $\hat{y}=\sigma(w_{1}\cdot x+b)$, where $\sigma$ is the sigmoid function. Our adversary model takes the form of the following logistic-regression-like model:

where $c$ and $b$ are learnable scalars, $w_{2}$ is a learnable vector, and $\sigma^{-1}$ is the inverse of the sigmoid function (logit function) $\sigma^{-1}(t)=\log t-\log(1-t)$. Intuitively, we want our adversary to be able to learn functions of the form $\hat{z}=f(y,[\hat{y}>0.5])$ (i.e. dependent only on the boolean predicted value $[\hat{y}>0.5]$), and thus enforce equality of odds. Here, the adversary would learn such a function by making $c$ extremely large. We add 1 to $|c|$ to make sure the adversary never tries to ignore $\hat{y}$ by setting $c=0$, which could be a difficult local minimum for the adversary to escapeThis value added to $|c|$ is an adjustable hyperparameter; we found reasonable results using the value 1 and thus not feel the need to experiment further.. This adversary is both general enough to be used whenever $y$ and $z$ are both discreteIf $y$ and $z$ are multi-class, then the sigmoid becomes a softmax, but everything else remains the same., and powerful enough that deviation from true equality of odds should cause the adversary to be able to decrease its loss.

Without tweaking, this algorithm ran into issues with local minima, and the resulting models were often closer to demographic parity than equality of odds. We implemented a technique that helped: by increasing the hyperparameter $\alpha$ in Eqn. 1 over time, the predictor had a much easier time learning to deceive the adversary and therefore more strictly enforce equality of odds. We set $\alpha=\sqrt{t}$ (where $t$ is the step counter), and to avoid divergence we set the predictor’s step size to $\eta\propto 1/t$, so that $\alpha\eta\rightarrow 0$ as is preferred for stochastic gradient-based methods such as Adam.

We train the model twice, once with debiasing and once without, and present side-by-side confusion matrices on the test set for income bracket with respect to the protected variable values Male and Female, shown in Table 3, and we present the false positive rates (FPR) and false negative rates (FNR) in Table 4. Note that false negative rate is equal to $1-{}$ true positive rate, so the trade-offs are directly comparable to the $(x,y)$ values of an ROC curve.

We notice that debiasing has only a small effect on overall accuracy ($86.0\%$ vs $84.5\%$), and that the debiased model indeed (nearly) obeys equality of odds: as shown in Table 4, with debiasing, the FNR and FPR values are approximately equal across sex subgroups: $0.0647\approx 0.0701$ and $0.4458\approx 0.4349$.

Although the values don’t exactly reach equality, neither difference is statistically significant: a two-proportion two-tail large sample $z$-test yields $p$-values 0.25 for $y=0$ and 0.62 for $y=1$.

Conclusion

In this work, we demonstrate a general and powerful method for training unbiased machine learning models. We state and prove theoretical guarantees for our method under reasonable assumptions, demonstrating in theory that the method can enforce the constraints that we claim, across multiple definitions of fairness, regardless of the complexity of the predictor’s model, or the nature (discrete or continuous) of the predicted and protected variables in question. We apply the method in practice to two very different scenarios: a standard supervised learning task, and the task of debiasing word embeddings while still maintaining ability to perform a certain task (analogies). We demonstrate in both cases the ability to train a model that is demonstrably less biased than the original one, and yet still performs extremely well on the task at hand. We discuss difficulties in getting these models to converge. We propose, in the common case of discrete output and protected variables, a simple adversary that is usable regardless of the complexity of the underlying model.

Future Work

This process yields many questions that require further work to answer.

The debiased word embeddings we have trained are still useful in analogies. Are they still useful in other, more complex tasks?

The adversarial training method is hard to get right and often touchy, in that getting the hyperparameters wrong results in quick divergence of the algorithm. What ways can be used to stabilize training and ensure convergence, and thus ensure that the theoretical guarantees presented here can work?

There is a body of existing work for image recognition using adversarial networks. Image recognition in general can sometimes be subject to various biases such as being more or less successful at recognizing the faces of people of different races. Can multiple adversaries be combined to create high accuracy image recognition systems which do not exhibit such biases?

In general, do more complex predictors require more complex adversaries? It would appear that in the case of $y$ and $z$ discrete, a very simple adversary suffices no matter how complex the predictor. Does this also apply to continuous cases, or would a simple adversary be too easy to deceive for a complex predictor?

References