Detecting Cross-Geographic Biases in Toxicity Modeling on Social Media
Sayan Ghosh, Dylan Baker, David Jurgens, Vinodkumar Prabhakaran
Introduction
Online social media platforms have increasingly turned to NLP techniques to detect offensive and abusive content online at scale (e.g., the Perspective API), as a way to mitigate the harms it causes to people.www.perspectiveapi.com However, recent research has shown that these models often encode various societal biases against marginalized groups Waseem (2016); Caliskan et al. (2017); Park et al. (2018); van Aken et al. (2018); Dixon et al. (2018); Sap et al. (2019); Davidson et al. (2019); Hutchinson et al. (2020); Zhou et al. (2021) and can be adversarially deceived Hosseini et al. (2017); Gröndahl et al. (2018); Kurita et al. (2019a), potentially furthering disproportionate harm to those in the margins. These models are also challenged by several dimensions of noise, especially when compared with the corpora they are trained on. For instance, social media text across the globe may contain code switched language generated by multilingual speakers, neologisms, and other orthographic variations rarely seen in training data. This poses additional challenges to detect and mitigate fairness failures in these models.
In this paper, we propose a new weakly supervised method (outlined in Figure 1) to address shortcomings of current bias detection methods and robustly detect lexical biases in models (i.e. biases that associate the presence or lack of toxicity with certain words) when applied to noisy text (in particular, text generated by social media users from non-predominantly Anglophone nations). We draw upon the observation that social stereotypes and biases in data often reflect the public discourse at a specific point in time and place Fiske (2017); Garg et al. (2018). We use a cross-corpora analysis across seven different countries to reveal terms overrepresented in each country. We then cluster the model behaviour against term perturbations to qualify these biases further. Furthermore, we demonstrate the effectiveness of our method through human validation. Our method is task and genre agnostic; we expect it to work in any culture-laden tasks on genres of noisy text from platforms with global presence. Moreover, it can be applied in cases with and without ground truth labels. We conduct experiments on both cases and present a preliminary analysis on mitigation strategies to reduce harms created by these biases.
Background
Current research on detecting biases in NLP models primarily starts with a specific axis of injustice, and relies on annotated data Sap et al. (2019); Davidson et al. (2019) or lexicons that are salient to different subgroups. For example, Dixon et al. (2018) use a list of LGBTQ identity terms to identify varied outcomes in text classification and Hutchinson et al. (2020) use a list of terms that refer to persons with a disability. However, this reliance on lexicons and annotated data limit the focus to a handful of social injustices for which resources/lexicons are available, and largely ignores biases in non-Western contexts Sambasivan et al. (2021) and online environments containing creative or noisy language variation. Moreover, prior work shows that interpretation of hate varies significantly across different cultural contexts Salminen et al. (2018, 2019) and reveals that the performance state-of-the-art models such as BERT Devlin et al. (2019) on various tasks degrades significantly when used on noisy text Sarkar et al. (2020); Kumar et al. (2020).
Consequently, current approaches to identifying model biases miss 1) undesirable biases around words that are not captured in existing lexicons/datasets, and 2) desirable biases around offensive terms that are salient to different geocultural contexts that the models fail to capture. As an example, such biases are present even in a commonly-used toxicity detector like the Perspective API, as shown in Table 1. In the first two sets of rows, we see that the mentions of certain identity groups (e.g., muslims, Tamilian) causes the toxicity model to assign a higher toxicity score. The third and fourth set of rows demonstrate that the model does not recognize the toxicity association on the words presstitute and Madarchod,Presstitute is a portmanteau word blending the words press and prostitute, often used in Indian social media discourse; maadarchod is an abusive word in Hindi, commonly used in Indian social media discourse in English. whereas it does recognize the toxicity of words/phrases with the same/similar meanings. In this paper, we present a method to detect such undesirable lexical biases.
Methodology
We find undesirable lexical biases in two phases. The first phase aims to find words that are statistically overrepresented in tweets from specific countries that were deemed toxic by the model. Such overrepresentation could be due to different reasons: (1) the model has learned some of these words to be offensive and hence their presence causes it to assign a higher toxicity score, or (2) some of the words appear more often in offensive contexts (e.g., frequent victims of toxicity) but their presence is merely correlational (i.e., the model doesn’t have any toxicity association for these words). Another consideration is whether the association of a particular word towards toxicity is desirable or not. For some words, such as country-specific slur words, we want the model to be sensitive to their presence. On the other hand, we desire the models to not have biases towards culturally salient non-offensive words, such as religious concepts.
The second phase attempts to separate the overrepresented words along these two dimensions:
a descriptive axis that separates words representing correlational vs. causal associations
a prescriptive axis that separates words representing desirable vs. undesirable associations
Both phases are described in Figure 1 using tweets from India as an example. The causal-desirable associations (e.g., bastard) and correlational-undesirable associations (e.g., indian) capture the cases where the model is behaving as desired. On the other hand, the causal-undesirable associations (e.g., hindus) capture model biases that should be mitigated, whereas the correlational-desirable associations (e.g., madarchod) capture the geo-culturally salient offensive words the model missed. Our method aims to balance (i) achieving reasonably high precision and (ii) narrowing down the search space of words enough to be feasible in Phase 2 and any further human evaluation studies. We now discuss the methods we use for each phase.
First, for each country, we calculate the log-odds ratio with a Dirichlet informed prior Monroe et al. (2008) to find if term is statistically overrepresented in toxic tweets vs. non-toxic tweets in that country. Unlike the basic log-odds method, this method is robust against very rare and very common words as it accounts for a prior estimate of the expected frequency of each term. It also accounts for the variance in the word’s frequency by calculating the z-score. However, common profanities are likely overrepresented in toxic tweets globally, so this initial list must be further filtered to identify geographically-specific terms. Since we now have a multi-class setting, one could repeat the Monroe et al. (2008) method in a one-vs-rest manner. However, such an approach would miss the important interdependence between these tests.
By combining these two methods, we find culturally salient words that are overrepresented in toxic tweets in each country. We also conducted experiments using Monroe et al. (2008) method in a one-vs-rest manner (described above), as well as using Bamman et al. (2014) method for both steps; but they provided qualitatively worse results than the above approach.
2 Phase 2: Qualifying Associations
To determine the causal vs. correlational distinction, one could rely on counterfactual approaches such as perturbation sensitivity analysis (PSA) Prabhakaran et al. (2019). However, PSA (and similar approaches) aggregate the sensitivity to terms by calculating the mean difference in prediction scores across all sentences. Here, we investigate how the sensitivity varies across the range of toxicity scores, which gives important clues about the desirability of biases. Using naturally occurring sentences like Prabhakaran et al. (2019) would introduce numerous unknown correlational effects. Hence, we followed Hutchinson et al. (2020) and May et al. (2019) by building a set of template sentences (see Appendix A) that correspond to a range of toxicity scores. A few examples below:
For each term obtained in Phase 1, we measure the shift in toxicity in response to replacing the word person in the template sentence with that term.
Evaluation of Off-the-shelf Models
Our approach can notably be applied to cases where ground truth labels are not available. This is useful in deployment settings, where it may not be feasible and/or be expensive to annotate sufficient amounts of data regularly or where one does not have access to the training data but wants to understand biased outcomes. We provide an example of how our method could be used in this setting.
We use the Perspective API’s toxicity model for our analysis. We use 73 million tweets from across 7 countries with substantial English-speaking populations that are active online: India, Pakistan, Nigeria, Ghana, Jamaica, Mexico, and the Philippines. We focus on these countries as opposed to countries that are solely and primarily Anglophone as to better understand how models interact with “non-standard” English dialects, especially ones that are different from the data the model was likely trained on. Our data is collected from a 10% random sample of the tweets from 2018 and 2019, from the Decahose stream provided by Twitter. We pre-process tweets by removing URLs, hashtags, special characters and numbers, and applying uniform casing. We also lemmatize each resulting word to control for factors such as plurality using NLTK Loper and Bird (2002). The number of tweets from each country is shown in Table 2.
2 Results
Our Phase 1 analysis identified several hundred terms overrepresented in each country. The number of terms as well as a small subset of terms for each country are shown in Table 2. As evident, our method is effective at picking up words that are geography-specific, compared to simply using log-odds to identify overrepresentation in toxic versus non-toxic tweets; such an approach outputs more general, widely used profanities that (by nature) occur more commonly in offensive messages across all countries. Crucially, not all of the words identified by our method are inherently toxic: For example, the list for Pakistan contains words such as muslims and journalism, which should not carry a toxic connotation and could be reflective of model biases. Further, our method identifies emergent country-specific terms such as presstitutes (a slur used to refer to members of the press) and congi (a slur for supporters of the Congress Party, a large political party) in India.
Phase 2:
In Phase 2, we analyze the model behaviour in response to perturbations using each term from Phase 1, and cluster them based on model sensitivity. Here, we focus on nouns and named entities which represent of the terms from phase and are the most salient to analyze in terms of subgroups (although the method can be altered to use templates that allow for the use of other parts of speech). We use -means clustering; we selected based on manual inspection of ; this inspection showed that the majority of instances fall into four natural clusters, with lower merging dissimilar instances or higher producing clusters with very few instances. Figure 3 shows the model behaviour of each cluster we obtained for India, Mexico, and Nigeria; figures for the remaining countries are shown in Appendix B, with a sample of words in each cluster. We list a sample of words in each cluster for India below:
C0: muslim sexy indian hindu jihadi pakistani islam women feminist liberals italian arabs communist jews christian priest kafir australian
C1: morons bastards fools cock loser hypocrite coward rapists scum ignorant cursed retarded losers moronic arse sissy dolt scumbag cunts imbecile pervert slut boob pedophile prostitutes
C2: bloody shameless useless terrorist filthy killer pig disgrace arrogant corrupt bigot horrible selfish commie slave irritating filth troll murderer kutta liars sexual thug uneducated donkey
C3: fake sanghi blind journalist pappu mullah chaddi child appeaser wife country vala chankya tadipad mota dogs creature abuse animals bhakt jinahh pidi spineless shove
Qualitative inspection of the clusters reveals certain properties of the words within each. Cluster C0 (top left of Figure 3a) contains words such as feminist and muslim that increase the toxicity of the templates with the largest changes occurring in the middle, indicating undesirable biases in the model. In contrast, cluster C1 (top right) has words such as shithead that uniformly raise the toxicity of the templates to the very high range, regardless of the initial toxicity of the template.
Words in cluster C2 (bottom left) display a similar early trajectory to those in cluster C0, but converges towards the diagonal. Words in this cluster include presstitute and porki, both of which are country-specific slurs, suggesting that the model only has a weak signal about their toxicities. Similarly, cluster C3 (bottom right) consists of words that do not affect the templates much at all after replacement. Some words in this cluster are indeed not toxic, but we also see words such as congi that the model has not encountered before, but carry a negative connotation in Indian online discourse.
3 In-Community Validation
To quantitatively evaluate whether the word categories uncovered by our method correspond to real biases as viewed through members of a geographic community, we conducted a crowd-sourcing study via the Qualtrics platform. We note that even despite this step, there may be certain forms of bias that are not uncovered by our method. However, this form of validation attempts to judge whether the biases that our method does uncover have a correspondence with real-world judgments.
We recruited 25 raters per country, and each was asked to rate 60 terms (including 10 control words) to be (i) inoffensive, (ii) sometimes offensive, (iii) highly offensive, or (iv) don’t know this word. We obtained 5 ratings each for 250 words per country. We removed ratings from raters who got more than 3 out of 10 control words wrong. Full annotation details are provided in Appendix C. Despite recruiting a large pool of annotators for each instance, our annotators may not entirely capture the value systems of countries due to selection effects of who has access to the internet and is able to participate in Qualtrics work. For instance, raters from India in our human ratings potentially overrepresent views of the middle class, those of upper caste individuals, and views from a handful of states with tech hubs. As a result, their perception of what is or is-not offensive may not be representative of India at large.
We hypothesize that the words grouped into C0 would be judged identical to those in C3 (i.e., treated as false positive features), where C2 should be more offensive, potentially up to C1 (i.e., false negative). In aggregate, the annotator ratings largely matched our hypotheses across countries. Figure 4 shows the ratings for the clusters from India, Mexico and Nigeria (cf. Figure 3); here, we re-scale their categorical ratings into . Our results show that the words in C0 and C3 are often very similar in their perceived toxicity and, in the case of India, have no statistical difference in their toxicity ratings. Our analysis on Nigerian tweets does differ in this trend, however, where C0 is perceived to be more toxic than C3—and even slightly more than C2. Words in this cluster include feminist, junkie, africans, and maggot, which, though slightly negative in connotation, would not appear to be overly offensive and in-line with expectations for what C0 should contain. This difference for Nigeria in our case study exemplifies how in-community interpretations do differ from those of external readers and underscores the importance of including these viewpoints when assessing model errors.
The C2 clusters, which contains words like “presstitute” in India, were found offensive, matching our hypothesis of these words being false negatives. In all countries, the level of offensiveness was still less than the level for extremely toxic words in C1, but still larger than C3.Figures for other countries are shown in the Appendix B. The clusters found by our method do correspond to in-community judgments of toxicity and reflect meaningful groups of biases in the model, which can be used by experimenters for later bias testing and mitigation.
Towards Mitigating Undesirable Biases
Our pipeline provides an unsupervised approach to identifying potential latent biases in toxicity models. Ideally, practitioners aim not only to identify these biases but also to fix them before deploying their systems. However, our method identifies terms of different types: causal-desirable associations and correlational-undesirable associations, as well as causal-undesirable and correlational-desirable associations both of which require different approaches to handle. The clusters we find in Phase 2 approximate these different types of associations, and hence we expect them to behave differently for different mitigation approaches. Here, we test the impact of baseline mitigation techniques on reducing bias in these clusters.
Performing bias mitigation experiments involves re-training the model, and hence requires toxicity annotated training data. Since we do not have access to the training data used to train the Perspective API model, we employ the dataset provided by Jigsaw for the Kaggle competition on toxicity classification Jigsaw (2019) that contains public comments from the Civil Comments platform and annotated toxicity ratings (on a continuous scale in the range $>=0.52\times 10^{-5}$.The model was trained for three epochs with early stopping; ultimate model performance on a held-out 10% development set was highest after one epoch.
While some performance differences are inherent between this model and the Perspective API model, our method still identifies similar latent biases. Phase 1 of our pipeline identifies 150 candidate words with bias. These include words such as liberal, muslim, sanghi, and commie. In particular, we also note that the set of words flagged using the scores from our model is a strict subset of the words flagged using the scores from Perspective API. To quantify the effects of mitigation strategies, we further analyze 120 of these words that appear in at least 10 instances each for the toxic and non-toxic labels in the Jigsaw dataset.
2 Bias Measurement
As Borkan et al. (2019) note, biases in models can be nuanced and varied, and require multiple metrics that can provide insights on how to mitigate them. They compute a suite of five metrics based on subgroups of instances defined around the presence of different identity terms. We adopt a similar approach by defining 120 subgroups based on the presence of each of the 120 terms our bias evaluation pipeline identified.
The first three metrics are based on AUC. (1) Subgroup AUC reflects how separable are the toxic/non-toxic instances containing our target words. (2) Background Positive Subgroup Negative (BPSN) AUC measures how separable are non-toxic instances containing the target words from toxic instances without them; lower scores are typical of models with false positives. (3) Background Negative Subgroup Positive (BNSP) AUC measures how separable are non-toxic instances without the target words from toxic instances with them; lower scores are typical of models with many false negatives. As AUC, these scores also range $$ and higher scores are desirable.
The last two metrics compare the toxicity scoring distributions of the instances with the target words and those instances without. These metrics measure the Average Equality Gap (AEG) for toxic and non-toxic instances (separately) within , where scores close to 0 indicate no bias in how instances with the target word are treated. The AEG metrics capture cases where the scoring distributions of the target words are shifted but not sufficiently to cause re-orderings (by toxciity) that would be identified by the AUC metrics. We refer the reader to Borkan et al. (2019, §3.3) for a detailed discussion of what these metrics capture.
3 Mitigation Approaches
Mitigating undesirable biases in NLP models is an active research area Zhou et al. (2021), where counterfactual and data augmentation techniques tend to be the most common approach Dixon et al. (2018); Garg et al. (2019); Davani et al. (2020). In this section, we present experiments using three such , with the goal to demonstrate that different approaches are required to mitigate different types of biased associations our pipeline reveals.
Deletion: Here, we delete all training instances containing a biased word and re-do the fine-tuning of the DistillBERT model using the remaining instances. In doing this, we expect to remove undesirable associations around the target words learned from the training data, however it will remove a lot of useful information as a side effect.
Substitution: Here, we substitute all occurrences of the target words in the training data with a
Balance and Tune: Finally, we use an approach that further fine-tunes the model using a subset of the dataset where each target word has an equal number of toxic and non-toxic instances. Formally, for occurrences of the target word in instances with the toxic label and instances with the non-toxic, we select instances of each and fine-tune the model further only on these instances. We experiment with =10, 50, 100. For all three experiments, we use the same fine-tuning hyperparameters as in the baseline model.Separate hyperparameter tuning on the held-out development set showed these parameters to still be optimal.
4 Results
In aggregate, none of the mitigation strategies substantially reduced the bias, as shown in Table 3.Results for the Balance and Tune models were similar for all . We show results with other values in Appendix D. Three notable trends were observed. First, mitigation strategies varied in their net effect on recognizing toxicity. Substituting the biased word for
Second, the Substitution and Deletion models both increase bias on every metric except AEG+. Despite not training models on any instances containing the target terms, the resulting models still produced classification decisions biased against the target words. This result suggest that the pre-trained DistilBERT model itself is likely responsible for the latent bias, mirroring multiple results showing biases in these models Kurita et al. (2019b); Shwartz et al. (2020); Nadeem et al. (2020).
Third, the Balance and Tune strategy had negligible effect on both bias correction or toxicity performance. This result suggests that starting from a biased classifier based on a pre-trained language model, the bias cannot easily be corrected by simply rebalancing the class distribution in training data and further fine-tuning. Together, our results point to how simple but seemingly common-sense approaches fail to substantially reduce bias and thus motivate the need for (i) more sophisticated mitigation strategies or (ii) pre-trained language models free of such latent biases.
Do bias mitigation approaches change performance differently for each cluster of terms? Figure 5 shows cluster-level disaggregated bias metrics for the baseline model, as well as the increases/decreases obtained through the different mitigation approaches, showing that the bias metrics do vary across different clusters. The high AEG+/- of clusters C0 and C2 indicates that both clusters, on average, score higher than background examples, confirming the intuitive notions captured by the visual examination of the clusters (i.e. the terms in these clusters tend to be rated higher than their corresponding “neutral” templates). Cluster C1 also has higher AEG metrics, but these are mostly toxic words that the model already recognizes. On the other hand, C1 and C2 fares poorer than the other two on AUC bias metrics, especially in BPSN-AUC.
Our bias mitigation approaches also act differently on different clusters. Overall, the Balance and Tune technique has the most gains in bias metrics in C1, and moderate gains in C2, whereas it has negligible impact on C0 and C3. On the other hand, Deletion and Substitution approaches significantly worsened AEG metrics of C3, as well as AUC metrics, across the board except for BPSN-AUC of C1. These results demonstrate the need for different approaches to mitigate the different kinds of biased associations our pipeline reveals. However, more work needs to be done on finding which mitigation approaches are appropriate for different kinds of undesirable biases.
Discussion and Conclusion
Toxicity detection models commonly rely on training data that reflects discourse that is temporally and culturally situated. As such, models may contain biases by learning undesirable features that are not toxic or be blind to toxic language underrepresented in their training data. Here, we introduce a new weakly supervised approach for uncovering these biases.All code and data to replicate the results will be released at the url: https://github.com/sghosh73/Cross-Cultural-Bias-Detection Our method provides a key diagnostic tool for model creators and deployers to test for biases. From a fairness perspective, our analysis reveals biases present in the model along different dimensions, without having any labeled resources for those dimensions. For instance, C0 revealed biases around religion (muslim, hindu, jews), country/ethnicity (indian, pakistani, arabs), and ideology (feminist, liberals, communist) without looking for these axes a priori. This could be a valuable first step before deeper analysis of these biases.
At a practical level, our method reveals country-specific offensive terms that the model has not seen before (e.g., maadarchod). Such country-specific abusive language lexicons could aid NLP practitioners trying to make their models robust across geographies. Such lexicons could also aid human-rights organizations working on building region-specific hate-speech lexicons.https://www.peacetechlab.org/ Furthermore, our model also identifies portmanteau (blended) words that are used disproportionately in certain geo-cultural contexts (e.g., presstitute, and libtard in Indian social media discourse). Our method could help monitor emergent offensive language in different geographies and aid data augmentation efforts in keeping models up-to-date through surfacing emergent language in new geographies.
Ethical Considerations
The proposed method provides an efficient distantly-supervised method for practitioners to identify potential biases in their toxicity detection methods. Although intended strictly as beneficial, it could create the risk of overconfidence in a lack of bias by a particular model. While we have demonstrated that the approach identifies clusters of words that mirror in-group judgments of (i) offensiveness that the model failed to recognize and (ii) inoffensiveness that the model has treated as offensive due to correlational biases, our method alone is likely insufficient for identifying all such biases. Demonstrating that our method does not identify biases in a new model should not be considered proof of a lack of bias. Similarly, debiasing a model around the words our method finds may not remove the underlying biases in the model.
Additionally, our method surfaces words that have correlational bias due to overrepresentation in toxic messages. Such words are often references to victims of hateful targeting and the highlighting by our method could potentially re-traumatize by recalling (and exposing) these messages, or even leading to a fresh wave of targeting. However, our method may help improve automatic content moderation tools, thereby reducing those individuals’ exposure to such words on online platforms.
Finally, our method reveals country-specific abusive words. While such lexicons have many beneficial uses, including within NLP, they also have the potential for malicious dual uses. Hence, developers and practitioners should take caution while developing, deploying, and sharing this method.
Acknowledgements
We thank Ben Hutchinson for his insightful feedback on this paper, the crowdworkers from Qualtrics for rating the words in this study, and the anonymous reviewers for their helpful suggestions. The third author is supported in part by the Volkswagen Foundation.
References
Appendix A Templates
We present the list of templates as well as their corresponding toxicity scores according to the Perspective model.
Appendix B Further Results for Other Countries
We present results for the other countries we collected data for. Note that C3 and cluster C0 may not always be equivalent. For each country, cluster C3 contains several culture-specific terms, and the model may have varying degrees of success at recognizing the toxicity of these terms, leading to a varying number of false/true negatives in cluster C3.
Appendix C In-Community Analysis
To quantitatively evaluate whether the word categories uncovered by our method correspond to real biases as viewed through members of a geographic community, we conducted a crowd-sourcing study via the Qualtrics platform. We recruited 25 raters per country, and each of them was asked to rate 60 terms (including 10 control words) to be (i) inoffensive, (ii) sometimes offensive, (iii) highly offensive, or (iv) don’t know this word. Altogether, we obtained 5 ratings each for 250 words per country.
Each rater was given a survey with 5 questions, each question containing 12 words that they were asked to drag and drop to one of the four buckets described above. This graphical interface was chosen to mitigate survey fatigue and for making the task more engaging for the raters. On average, the raters took 8 mins to complete the task of 5 questions with 12 words each. The raters were paid between 12.50 depending on the country, for completing the task. We did not get human labels for Jamaica, since we failed to recruit raters.
Appendix D Additional Mitigation Results
Table 5 shows results for the Balance and Tune model with additional values of , along with the full results of all other models.