On Learning Fairness and Accuracy on Multiple Subgroups

Introduction

Machine learning has made rapid progress in sociotechnical systems such as automatic resume screening, video surveillance, and credit scoring for loan applications. Simultaneously, it has been observed that learning algorithms exhibited biased predictions on the subgroups of population . For example, the algorithm denies a loan application based on sensitive attributes such as gender, race, or disability, which has heightened public concerns.

To this end, fair learning is recently highlighted to mitigate prediction disparities. The high-level idea is quite straightforward: adding fair constraints during the training . As a result, fair learning principally gives rise to two desiderata. On the one hand, the fair predictor should be informative to ensure accurate predictions for the data. On the other hand, the predictor is required to guarantee fairness to avoid prediction disparities across subgroups. Therefore, it is crucial to understand the possibilities and then design provable approaches for achieving both informative and fair learning.

Clearly, achieving both objectives depends on predefined fair notations. Consider demographic parity as the fair criteria, which necessitates the independence between the predictor’s output $f(X)$ and the sensitive attribute (or subgourp index) $A$. Thus, if the sensitive attribute $A$ and the ground-truth label $Y$ are highly correlated, it is impossible to learn a both fair and informative predictor.

In summary, this work aims to propose a novel principled framework for ensuring group sufficiency, as well as preserving an informative prediction with a small generalization error. In particular, we focus on one challenge scenario: the data includes multiple or even a large number of subgroups, some with only limited samples, as often occurs in the real-world. For example, datasets for the self-driving car are collected from a wide range of geographical regions, each with a limited number of training samples . How can we ensure group sufficiency as well as accurate predictions? Specifically, our contributions are summarized as follows:

Controlling group sufficiency We adopted group sufficiency gap to measure fairness w.r.t. group sufficiency of a classifier $f$ (Sec.3), and then derive an upper bound of the group sufficiency gap (Theorem 4.1). Under proper assumptions, the upper bound is controlled by the discrepancy between the classifier $f$ and the subgroup Bayes predictors. Namely, minimizing the upper bound also encourages an informative classifier.

Algorithmic contribution Motivated by the upper bound of the group sufficiency gap, we develop a principled algorithm. Concretely, we adopt a randomized algorithm that produces a predictive-distribution $Q$ over the classifier ($f\sim Q$) to learn informative and fair classification. We further formulate the problem as a bilevel optimization (Sec. 5.3), as shown in Fig.1. (1) In the lower-level, the subgroup specific dataset $S_{a}$ and the fair predictive-distribution $Q$ are used to learn the subgroup specific predictive-distribution $\overline{Q}^{\star}_{a}$, where $Q$ is regarded as an informative prior for learning limited data within each subgroup. Theorem 5.1 formally demonstrates that under proper assumptions, the lower-level loss can effectively control the generalization error. (2) In the upper-level, the fair predictive-distribution $Q$ is then updated to be close to all subgroup specific predictive-distributions, in order to minimize the upper bound of the group sufficiency gap.

Empirical justifications The proposed algorithm is applicable to the general parametric and differentiable model, where we adopt the neural network in the implementation. We evaluate the proposed algorithm on two real-world NLP datasets that have shown prediction disparities w.r.t. group sufficiency. Compared with baselines, the results indicate that group sufficiency has been consistently improved, with almost no loss of accuracy. Code is available at https://github.com/xugezheng/FAMS.

Related Work

Algorithmic fairness Fairness has been attached great importance and widely studied in various applications, such as natural language processing , natural language generation , computer vision , and deep learning . Then various approaches have been proposed in algorithmic fairness. They typically add fair constraints during the training procedure, such as demographic parity or equalized odds . Apart from this, other fair notions are adopted such as accuracy parity , which requires each subgroup to attain the same accuracy; small prediction variance , which ensures small prediction variations among the subgroup; or small prediction loss for all the subgroups . Furthermore, based on the concept of Independence (e.g. demographic parity $A\perp\!\!\!\!\perp f(X)$) or conditional independence (e.g. equalized odds $A\perp\!\!\!\!\perp f(X)|Y$ or group sufficiency $A\perp\!\!\!\!\perp Y|f(X)$), another popular line in fair learning is then naturally integrated with information theoretical framework through adding mutual information constraints such as .

Understanding fairness-accuracy trade-off As for the theoretical aspect, further investigated the relation of fairness (demographic parity) and algorithmic stability. formally justified the inherent trade-off between fairness (w.r.t. demographic parity and equalized odds) and accuracy, whereas the analysis is conducted for the binary sensitive attribute with the population loss. studied the fair-accuracy trade-off in the multi-task learning.

Bi-level optimization in fairness Bi-level optimization seeks to solve problems with a hierarchical structure. Namely, two levels of optimization problems where one task is nested inside another . Several ideas related to bi-level optimization have been proposed in the context of fair-learning. For instance, we could design a min-max optimization to learn fair representation when considering demographic parity (DP) or equalized odds (EO) . In this context, a representation function aims to minimize the loss caused by the discriminator in the lower-level. Simultaneously, in the upper-level, a discriminator could be introduced to maximize the loss. Then fair representation could be enforced through the bi-level optimization. Besides, if the accuracy and its variants are tracked as the metrics for each subgroup , the bi-level objective could also be deployed in controlling the loss or the prediction variance , where the lower-level’s goal is to minimize the loss for each subgroup and the upper-level’s goal is to estimate the prediction disparities. In our paper, we theoretically justified a novel bi-level optimization perspective: controlling group sufficiency and accuracy. Simultaneously, other bi-level optimization and its relevant meta-learning algorithms could be further considered in the fair learning such as recurrent based gradient updating , layer-wise transformation or implicit gradient based approach .

Preliminaries

Intuitively, given a output score of the predictor $f(X)=\tau$, the conditional expectation of $Y$ is invariant across different subgroups. Namely, conditioning on the specific subgroup $A=a$ does not provide any additional information about the conditional expectation of $Y$. Then we could naturally define group sufficiency gap.

Specifically, $\textbf{Suf}_{f}$ measures the extent of group sufficiency violation, induced by the predictor $f$, which is taken by the expectation over $(X,A)$. Clearly, $\textbf{Suf}_{f}=0$ suggests that $f$ satisfies groups sufficiency and vice versa. For completeness, we also discuss other popular group fairness criteria: demographic parity and equalized odds.

Demographic Parity (DP), also known as statistical parity or independence rule, emphasizes that the expectation of the output score $f(X)$ is independent of $A$. further revealed that if $A\not\!\perp\!\!\!\perp Y$, group sufficiency and demographic parity could not be simultaneously achieved.

Equalized odds (EO) emphasizes the conditional expectation of output $f$ is invariant w.r.t. $A$, given the ground truth $Y$. reveal that if $\mathcal{D}(X,Y,A)>0$ and $A\not\!\perp\!\!\!\perp Y$, group sufficiency and equalized odds can not both hold.

The analysis reveals a general incompatibility between group sufficiency and DP/EO when $A\not\!\perp\!\!\!\perp Y$, which often occurs in practice. Besides, DP/EO based criteria generally suffers the well-known fair accuracy trade-off : enforcing the fair constraint degrades the prediction performance. This paper depicts that under the criteria of group sufficiency, these objectives could be both encouraged.

Upper bound of group sufficiency gap

To derive the theoretical results, we first introduce the group Bayes predictor.

The proof is inspired by . Specifically, Theorem 4.1 reveals that the upper bound of group sufficiency gap depends on the discrepancy between the predictor $f$ and $A$-group Bayes predictor $f_{A}^{\text{Bayes}}(X)$. Namely, given different subgroups $A=a$, the optimal predictor $f$ ought to be closed to all the group Bayes predictors $f_{A=a}^{\text{Bayes}}(X)$, $\forall a\in\mathcal{A}$.

Principled Approach

Based on the upper bound, we propose a principled approach to learn the predictor that achieves both small generalization error and group sufficiency gap.

The group sufficiency gap $\textbf{Suf}_{f}$ in randomized algorithm w.r.t. learned predictive-distribution $Q$ is upper bounded by:

Where KL is the Kullback–Leibler divergence. Corollary 5.1 further reveals that the upper bound is decomposed into two terms, showing in Fig.2.

Optimization term The optimization term is the average KL divergence between the learned distribution $Q$ and optimal predictive-distribution $Q^{\star}_{a}$ for each subgroup $A=a$. Minimizing the optimization term implies that the learned distribution $Q$ will be both fair and informative for the prediction, because it aims to minimize the upper bound of the group sufficiency gap $\textbf{Suf}_{f}$ and be close to the optimal predictive-distribution w.r.t. each $A=a$.

Approximation term The approximation term is the average KL divergence between the optimal distribution $Q^{\star}_{a}$ and the underlying data generation distribution. Given the distribution family $\mathcal{Q}$, it is a unknown constant. Besides, if the distribution family $\mathcal{Q}$ has a rich expressive power such as deep neural-network, the approximation term will be small . However, an extreme large distribution family $\mathcal{Q}$ could simultaneously yield a potential overfitting on finite samples. In this paper, the neural network is adopted and the approximation term is assumed to be a small constant. Thus, controlling $\textbf{Suf}_{f}$ implies minimizing the optimization term.

2 Challenge in learning limited samples

3 Q𝑄Q as an informative prior

We have demonstrated that $Q$ can achieve both fair and informative prediction. Therefore, we regard $Q$ as a prior information for minimizing the loss, yielding a bilevel objective.

Where $\lambda>0$ is the hyper-parameter. The proposed loss is a typical bilevel optimization. (1) In the lower-level, we aim to learn $\overline{Q}_{a}^{\star}$ for each $a\in\mathcal{A}$. Different from Eq. (1), the loss in lower-level adds a regularization term $\text{KL}(Q_{a}\|Q)$ as an informative prior in learning $\overline{Q}_{a}^{\star}$, given a fixed predictive-distribution $Q$. Moreover, Theorem 5.1 formally justified that optimizing the lower-level loss is to minimize the upper bound of the generalization error. (2) In the upper-level, $Q$ is updated through minimizing the average KL divergence between different $\overline{Q}_{a}^{\star}$, which controls the upper bound of $\textbf{Suf}_{f}$.

Supposing that datasets $\{S_{a}\}_{a=1}^{|\mathcal{A}|}$ with $S_{a}=\{(x^{a}_{i},y^{a}_{i})\}_{i=1}^{m}$ are i.i.d. sampled from $\mathcal{D}(x,y|A=a)$, the binary cross entropy (BCE) loss is upper bounded by $L$, $Q_{a}\in\mathcal{Q}$ is any learned distribution from dataset $S_{a}$ and $Q\in\mathcal{Q}$ is any distribution. Then with high probability $\geq 1-\delta$ with $\forall\delta\in(0,1)$, we have:

Discussions The proof is inspired by PAC-Bayes theorem such as . Secpficially, Theorem 5.1 reveals the generalization error in the lower-level is upper bounded by three terms. (a) Term (1) is the average empirical prediction error, which corresponds to the first term in the lower-level loss. (b) Term (2) indicates the average KL-divergence between the learned subgroup distribution $Q_{a}$ and the prior distribution $Q$, which corresponds to the second term in the lower-level loss. The combination of term (1-2) recovers the averaged lower-level loss w.r.t. $A$. In Theorem 5.1, the differences are in the square norm of KL divergence and setting the specific hyper-parameter: $\lambda=L\sqrt{|\mathcal{A}|/m}$. Thus optimizing the lower-level loss could control the generalization error. (c) When the confidence $\delta$ is fixed, term (3) will converge if $|\mathcal{A}|m\to+\infty$. Moreover, even if $m$ (the sample size in each subgroup) is quite small, a sufficient large number of subgroups $|\mathcal{A}|$ can also ensure the convergence of term (3).

For the sake of simplicity, we assumed the identical samples size $m$ in each subgroup $S_{a}$, while the theoretical result can be extended to subgroups with different samples $m_{a}$.

4 Practical Implementations

In this section, we develop a practical learning algorithm that can be applied to a wide range of differentiable and parametric models, including neural networks.

We choose the Isotropic Gaussian distribution (with diagonal covariance matrix) as the distribution family $\mathcal{Q}$, where the mean and covariance are set as $d$-dimensional parameter. Thus we need to learn the parameter $(\bm{\theta},\bm{\sigma})$ for fair and informative $Q\in\mathcal{Q}$. As for the subgroup $A=a$, we learn parameters $(\bm{\theta}_{a},\bm{\sigma}_{a})$ for $\overline{Q}^{\star}_{a}\in\mathcal{Q}$. It is worth mentioning that the Isotropic Gaussian distribution is selected for its computational efficiency in the optimization. We can use any distribution as long as the density function is differentiable with respect to the parameters.

Gradient Estimation

Based on the previous setting, we aim to optimize the bilevel objective to obtain the parameter of $Q$: $(\bm{\theta},\bm{\sigma})$. We use stochastic gradient descent (SGD) to optimize the parameters. In the lower-level, the loss in Sec. 5.3 is composed by the empirical prediction error and KL divergence term. The KL divergence has a closed form that can be differentiated efficiently. Specifically, since $Q$ and the subgroup specific $\overline{Q}^{\star}_{a}$ are factorized Gaussian, the KL divergence takes a simple closed form and the gradient can be easily calculated: $\text{KL}(\overline{Q}^{\star}_{a}\|Q)=\frac{1}{2}\sum_{i=1}^{d}\left\{\log\frac{\bm{\sigma}^{2}_{a}[i]}{\bm{\sigma}^{2}[i]}+\frac{\bm{\sigma}^{2}_{a}[i]+(\bm{\theta}_{a}[i]-\bm{\theta}[i])^{2}}{\bm{\sigma}^{2}[i]}-1\right\}$.

Re-parametrization trick

Proposed Algorithm

Based on the analysis, the algorithm is shown in Algorithm. 1 for solving the bilevel objective in Sec. 5.3. Specifically, we adopt the alternating optimization. Namely, in the lower-level, we fix $Q$ and optimize the subgroup specific predictor $\overline{Q}^{\star}_{a}$ through SGD. Then in the upper-level, we fix the learned $\overline{Q}^{\star}_{a}$ and update $Q$. Since we may face many subgroups, at each training epoch, we randomly sample a subset $\mathcal{A}^{{}^{\prime}}$ such that $|\mathcal{A}^{{}^{\prime}}|\ll|\mathcal{A}|$ for the memory saving.

Inference

Experiments

Dataset: Toxic Comments

Baselines

Since $f(X)$ is continuous, the group sufficiency gap is calculated by splitting the output of predictor into multiple intervals in $$ and computing the conditional expectation within each interval, as detailed in Appendix.

2 Experimental Results

We visualize the results in Fig. 3 for Amazon review product and Fig. 4 for toxic comment.

Accuracy and Fairness The accuracy and group sufficiency gap are depicted in Fig. 3(a) and Fig. 4(a). In Amazon review, the accuracy in the proposed approach has a slight decrease, compared with ERM. While the group sufficiency gap has improved by $3.0\%$, showing a significant improvement in the fairness. In toxic comments, the accuracy in proposed approach is nearly identical to the baseline, whereas group sufficiency gap has been significantly improved by $3.0$-$3.5\%$.

In Amazon review dataset, we visualize the top-9 users’ sufficiency gap in ERM, as shown in Fig. 3(b), where the gap of entire users is delegated to the Appendix. The proposed approach significantly reduces the group sufficiency gap of in most subgroups. The similar trend is also observed in Toxic dataset, as shown in Fig. 4(b), where the proposed approach has the nearly identical and small group sufficiency gap for each race. In contrast, the baselines exhibit significant group sufficiency bias on the Asian and Latino & other races.

Other sensitive attributes in Toxic comments Apart from adopting race as sensitive attribute, we also consider other possible sensitive attributes such as gender and religion, and the results are showed in the Appendix. The results in other sensitive attributes are similar to race, with improved fairness and no loss on accuracy.

Influence of $\lambda$. Fairness and accuracy can be simultaneously achieved. Theorem 5.1 suggests that there exists an optimal $\lambda$ in the generalization error bound. Then we changed the value of $\lambda$ in Amazon dataset, as shown in Fig. 5.

When $\lambda\to 0$, the subgroup specific parameters are simply learned from the limited samples within each subgroup. Then the fair predictor could not learn a proper prior from the subgroup specific predictor with a significant generalization error. Meanwhile, the group sufficiency gap is also large, which is consistent with : overfitting generally degrades the group sufficiency. When $\lambda$ is set between $[0.2,1]$, the generalization error is small (with a high accuracy) and group sufficiency gap is kept small, implying that both fairness and accuracy can be achieved. In contrast, if we set a large value for $\lambda\gg 0$, the predictor is unable to learn from the data but from the random prior $Q$. The prediction will be completely random (accuracy $=55\%$ when $\lambda=50$). When the predictor outputs a random guess, different from demographic parity (DP) or equalized odds (EO), the group sufficiency gap is also large. The analysis reveals that there exists an optimal $\lambda$ for simultaneously achieving accuracy and group sufficiency.

Conclusion

We conducted a novel analysis by simultaneously learning an informative and fair classifier for multiple or even many subgroups. We derived a novel principled algorithm. We further theoretically justified the generalization error and fair guarantees of the proposed framework. The empirical results in two real-world datasets demonstrated the effectiveness in both preserving the accuracy, as well as group sufficiency.

Discussion on Limitations

We proposed the analysis on learning group sufficiency and informative predictors, and developed a principled approach for it. Simultaneously there are several limitations to the proposed theory and algorithm. (1) In general, group sufficiency and DP/EO are incompatible. Controlling group sufficiency, for example, would cause DP/EO degradation. This would be problematic if DP/EO were preferred in practice. (2) We also assumed that the ground truth $A$-Bayes predictors would be similar across groups. However, this assumption could be violated, resulting in a highly non-trivial scenario. Thus, in order to evaluate the conditional distribution shift, we need to consider a new setting by collect sufficient data per subgroup.

Acknowledgments and Disclosure of Financial Support

We appreciate constructive feedback from anonymous reviewers and meta-reviewers. We also would like to thank Jun Xiao for the discussion and proof-reading the manuscript. C. Shui and C. Gagné acknowledge support from NSERC-Canada and the Canada CIFAR Chairs in AI. G. Xu, J. Li, C. Ling and B. Wang are supported by Natural Sciences and Engineering Research Council of Canada (NSERC), Discovery Grants program. T. Arbel is supported by International Progressive Multiple Sclerosis Alliance, the Canada Institute for Advanced Research (CIFAR) Artificial Intelligence Chairs program, the Natural Sciences and Engineering Research Council of Canada. Q. Chen is supported by China Scholarship Council.

References

Checklist

The checklist follows the references. Please read the checklist guidelines carefully for information on how to answer these questions. For each question, change the default [TODO] to [Yes] , [No] , or [N/A] . You are strongly encouraged to include a justification to your answer, either by referencing the appropriate section of your paper or providing a brief inline description. For example:

Did you include the license to the code and datasets? [Yes] See Section LABEL:gen_inst.

Did you include the license to the code and datasets? [No] The code and the data are proprietary.

Did you include the license to the code and datasets? [N/A]

Please do not modify the questions and only use the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.

Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope? [Yes]

Did you describe the limitations of your work? [Yes] The proposed theoretical analysis is restricted in the randomized algorithm.

Did you discuss any potential negative societal impacts of your work? [Yes] We study the algorithmic fairness. It is worth noting that ensuring group sufficiency may degrade other fair criteria such as DP/EO.

Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]

If you are including theoretical results…

Did you state the full set of assumptions of all theoretical results? [Yes] In the paper and Appendix.

Did you include complete proofs of all theoretical results? [Yes] In Appendix.

Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)? [Yes] We provided the code for the reproduction.

Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See the source code.

Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)? [Yes] We visualize the Boxplot of the results.

Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [N/A]

If you are using existing assets (e.g., code, data, models) or curating/releasing new assets…

If your work uses existing assets, did you cite the creators? [N/A]

Did you mention the license of the assets? [N/A]

Did you include any new assets either in the supplemental material or as a URL? [N/A]

Did you discuss whether and how consent was obtained from people whose data you’re using/curating? [N/A]

Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]

If you used crowdsourcing or conducted research with human subjects…

Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]

Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]

Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]

Appendix A Group sufficiency vs. demographic parity (DP) and equalized odds (EO)

For better understanding the properties of these three metrics, we consider the following scenario.

Consider the score predictor $f(X)$ uniformly outputs a value in $$for any$x\in\mathcal{X}$. i.e,$f(x)\sim\text{Unif}$. Then it is easy to verify the demographic party and equalized odds both satisfy. Since the output of predictor is completely independent with the data. Thus we have

The aforementioned counterexample suggests that group sufficiency shows different behaviors, where the pure random prediction could trivially achieve the EO/DP.

Moreover, based on , if $\mathcal{D}(X,Y,A)>0$ and $A\not\!\perp\!\!\!\perp Y$, the group sufficiency and demographic parity/equalized odds do not both hold. The aforementioned example also verifies this fact.

Appendix B Additional Facts of A𝐴A-Group Bayes predictor

For better understanding $A$-Group Bayes predictor, we could derive the following facts.

Proposition B.1 shows that given a subgroup $A=a$, $A$-group Bayes predictor simultaneously meets both fairness (group sufficiency) and informative (optimal predictor under binary cross-entropy loss). Unfortunately, $f^{\text{Bayes}}$ is impossible to estimate since it is related to the underlying data distribution $\mathcal{D}$, which is infeasible. Nevertheless, we can adopt $A$-group Bayes predictor $f^{\text{Bayes}}$ to derive an upper bound of group sufficiency gap $\textbf{Suf}_{f}$. The upper bound then holds for any predictor $f$ that can be learned from the observed data.

We first introduce the generalized Tower rule of the conditional expectation.

Let $(\Omega,\mathcal{F},P)$ be the probability space and two sub $\sigma$-algebras $\mathcal{G}_{1}\subseteq\mathcal{G}_{2}\subseteq\mathcal{F}$ are defined. Then we have

Based on the generalized tower rule, we have

Combining these two equations, we have the Fact 1:

B.2 Proof of Fact 2

Following , we can compute the optimal predictor of attribute $A=a$ under the binary cross-entropy by taking the functional derivative w.r.t. $f$:

Appendix C Upper bound of group sufficiency gap

Before deriving the theory, we need the following lemma.

Based on Lemma B.1, we can derive the main Theorem.

Thus the group sufficiency gap is upper bounded by:

Therefore, if $A$ takes only finite value ($|\mathcal{A}|<+\infty$) and follows uniform distribution with $\mathcal{D}(A=a)=1/|\mathcal{A}|$, then we have:

Appendix D Upper bound of group sufficiency gap in randomized algorithm

The third line is derived from Pinsker’s inequality. i.e, $\text{TV}(P\|Q)\leq\sqrt{\frac{1}{2}\text{KL}(P\|Q)}$.

Appendix E Generalization upper bound

We first demonstrate the following Lemma, which is based on .

Step 2

Then we could use the aforementioned Lemma to demonstrate the main theorem.

We adopt the lemma for the union of the whole training samples $S=\cup_{a\in\mathcal{A}}S_{a}$.

Through the decomposition property of KL divergence, we finally have:

Appendix F Computing group sufficiency gap from the data

In this paper, we need to compute the conditional expectation from the data. i.e,

where we have observed data $\{S_{a}\},a\in\mathcal{A}$. Since $f(x)$ is a continuous value, ranging from $$. Then we split$$ into sperate interevals:

We compute the expectation and conditional expectation within each interval. i.e:

Then for each group $A=a$, the group sufficiency gap is computed as:

We use the linear interpolation if the average values in each interval are not equal. Then the group sufficiency can be formulated as:

We assign the $\mathcal{D}(A=a)=\frac{1}{|\mathcal{A}|}$ as uniform distribution for ensuring fairness for each subgroup.

The demonstration is straightforward. By using the Bayes rule, we have

Thus iff $\mathcal{D}(A=a|f(X))=\frac{1}{|\mathcal{A}|}$, we have the equivalent form. Intuitively, $\mathcal{D}(A=a|f(X))$ refers the conditional probability of $A=a$, given the predicted score $f(X)$, which is related to the group membership inference such as . If $\mathcal{D}(A=a|f(X))$ is large, the subgroup index can be easily revealed via the algorithm output. If the algorithm can fully preserve the privacy, then $\mathcal{D}(A=a|f(X))=\frac{1}{|\mathcal{A}|}$.

Appendix G Experimental Details

In this part, we proposed a detailed description of the dataset and experiments settings.

The experiment is adapted from the protocol of . Specifically, we convert the original review score (ranging from 1-5) to the binary label: the positive review (score $\geq 4$) and negative review (score $\leq 3$). We sample and then fix 200 users from the original dataset, which contains the training (75-400 samples per user) , validation (75 samples per user), and test sets (75 samples per user).

The total training epoch is $100$. In each training epoch, we sample a small subset of users ($N_{\text{user}}=20$), then for each user we sample $50$ samples with replacement. The early stopping strategy is also adopted.

We adopt 4-layers fully connected neural network as the model, where the weights of the model follows the Gaussian distribution $Q_{a}$ or $Q$. The trade-off coefficient $\lambda$ ranges from $[0.01,50]$ and we fix $\lambda=0.4$ in the evaluation. We set all the Monte-Carlo samples as 5. More implementation details of experiments and parameter settings can be found in the code.

G.2 Toxic comments

We adopt the toxic comment dataset to predict whether the text comment is toxic or not, which has been observed the significant performance degradation on particular sub-populations.

Following , we first choose the race as sensitive attribute, which includes Black, White, Asian and Latino & others (4 subgroups).

The total training epoch is $100$. In each training epoch, we sample all the subgroups with the same sample size with replacement ($N=50$). The early stopping strategy is also adopted. The training ($N=33188$) validation ($N=3438$) and test set ($N=9744$) are following the protocol in .

We also consider the following sensitive attributes:

Religion. The religion includes Christian, Jewish, Muslin and others (such as Hindu, Buddhist, atheist).

Gender. The gender includes male, female and others (such as homosexual_gay_or_lesbian, bisexual, transgender, other_gender).

We adopt 4-layers fully connected neural network as the model, where the weights of the model follows the Gaussian distribution $Q_{a}$ or $Q$. The trade-off coefficient $\lambda$ ranges from $[0.01,50]$ and we fix $\lambda=0.6$ in the evaluation. We set all the Monte-Carlo samples as five $N=5$. More implementation details of experiments and parameter settings can be found in the code.

Appendix H Additional Results in Amazon review

Additional results of each subgroup’s fair performance and the probability calibration on the Amazon Review dataset are shown in Fig. 6 and Fig. 7.

Appendix I Additional Results in Toxic Comments

The additional results of the probability calibration on the Toxic Comments (Race) dataset is shown in Fig. 8.

I.2 Religion as sensitive attribute

The additional results of each subgroup’s fair performance and the probability calibration on the Toxic Comments (Religion) dataset are shown in Fig. 9 and Fig. 10.

I.3 Gender as sensitive attribute

The additional results of each subgroup’s fair performance and the probability calibration on the Toxic Comments (Gender) dataset are shown in Fig. 11 and Fig. 12.

It is worth noting that although the group sufficiency in three approaches is quite similar. However, the proposed approach shows a significant better probability calibration than baselines.

I.4 Results on different subgroup numbers

We visualize the results on different subgroup numbers, shown in Fig. 13. The results still suggest the consistently better results than baselines.

I.5 Additional Results on Adult dataset

The result suggests a consistently better group sufficiency with comparable accuracy.

I.6 Additional Results on vision dataset

We further consider CelebA dataset as a computer vision task . We follow the protocol of , which predicts the wavy hair $Y$ in the image $X$. We regard gender as sensitive attribute $A$. We further adopted Res18 as the backbone and three layers fully-connected (randomized) layers. We sub-sample 200 instances per subgroup and fine tune for maximum 20 epochs. All results are repeated 4 times and illustrated in Tab. 2.

The result also suggests a consistently better group sufficiency with comparable accuracy.

I.7 Evolution of Q during the training

We visualize the test accuracy and group sufficiency gap of fair predictor $Q$ during the training, shown in Tab. 3. The results are evaluated on the Toxic data with race as the sensitive attribute.