''Fifty Shades of Bias'': Normative Ratings of Gender Bias in GPT Generated English Text

Introduction

Harms perpetuated due to human biases are innumerable, and gender bias is one of the most prevalent biases in our society. Past work shows the harm due to gender-based bias ranges from under-representation in media stories Asr et al. (2021) to mis- or no representation in computational models Bolukbasi et al. (2016); Sun et al. (2019). Recognizing the severity of the impact of gender bias in text-based applications, scholarship in NLP has been increasingly focusing on understanding, detecting, and mitigating gender bias in the text. Past work on detecting gender bias has mostly focused on a lexica-based approach or used templatized sentences to create datasets that inform models for downstream tasks Bhaskaran and Bhallamudi (2019); Cho et al. (2019); Prates et al. (2020); Mohammad and Turney (2013a). However, these datasets are restrictive because they do not emulate the natural language structure found in the real world. This problem is further aggravated by real-world data with bias being sparse and difficult to mine Blodgett et al. (2021).

Further, annotation of gender bias is a challenging task. Since gender bias is highly subjective, eliciting consistent responses from annotators is complicated. Annotators’ perception of bias is heavily influenced by factors such as lived experience, education, community, personal sensitivity, and more Blodgett et al. (2020); Biester et al. (2022); Röttger et al. (2022). Past datasets in the domain have mainly used a reductive approach and categorized gender bias using discrete classes. However, Hada et al. (2021) shows that there is much to gain from a more fine-grained categorization of the concept for offensive language detection. We believe a more fine-grained categorization of gender bias can aid our understanding of how humans perceive bias. Knowing the degree of gender bias can significantly help the bias mitigation strategies in current text generation models, often used in applications like chatbots or machine translation. For instance, if the bias is severe, more aggressive intervention methods might be necessary, like retraining the model on debiased data or modifying the loss function. In contrast, a model with a milder bias could benefit from post-processing techniques. Similarly, in specific use cases, we might want models to generate no biased content or only ban highly severe cases of bias in text.

Since data collection and annotation is an expensive procedure, more so for tasks such as gender bias identification, which deal with data sparsity issues, there’s a growing interest in the community to leverage the fluent text generation and zero-shot learning capabilities of LLMs like GPT-3.5 and GPT-4 Wang et al. (2023); Eldan and Li (2023). These models are also being increasingly used in everyday applications. Therefore, it is imperative to understand the biases they can propagate. In our work, we prompt GPT-3.5-Turbo to generate graded gender-biased text. To ground the generation, we use a list of carefully curated seeds. This serves two purposes: (1) we can navigate data sparsity issues while still grounding GPT generations to real-world data, and (2) we can understand the biases GPT-3.5-Turbo can propagate via its generations. Studying (2) becomes increasingly relevant given that models have been shown to represent opinionation as a by-product of being trained on poorly representative data Santurkar et al. (2023) .

This paper introduces a novel dataset consisting of 1000 GPT-generated English text annotated for its degree of gender bias. The dataset includes fine-grained, real-valued scores ranging from 0 (least negatively biased) to 1 (most negatively biased) – normative gender bias ratings for the statements. Notably, this is the first time that comparative annotations have been utilized to identify gender bias. In its simplest form, comparative annotations involve presenting two instances to annotators simultaneously and asking them to determine which instance exhibits a greater extent of the targeted characteristic. This approach mitigates several biases commonly found in standard rating scales, such as scale-region bias Presser and Schuman (1996); Asaadi et al. (2019) and enhances the consistency of the annotations Kiritchenko and Mohammad (2017). However, this method necessitates the annotation of $N^{2}$ (where N = the number of items to be annotated) instance pairs, which can be prohibitive. Therefore, for the annotation of our dataset, we employ an efficient form of comparative annotation called Best—Worst Scaling (BWS) Louviere (1991); Louviere et al. (2015); Kiritchenko and Mohammad (2016, 2017).

Our annotation study provides valuable insights into how humans perceive bias and a nuanced understanding of the biases GPT models can generate. We conduct in-depth qualitative and quantitative analyses of our dataset to obtain insights into how different prompting strategies and source selection can affect the generation of statements. We analyze the different themes that humans consider to be more or less biased. We assess the performance of a few neural models used for related downstream tasks on our new dataset. Finally, we also investigate GPT-4’s reasoning capabilities to provide an appropriate reason for a given gender bias score. Dataset and Code available at: https://aka.ms/FiftyShadesofBias

Related Work

Existing studies on gender bias have relied on datasets that are either (a) templatized sentences or (b) sentences mined using lexical terms and rule-based patterns from web sources. The templatized sentences are structured as [Noun/Pronoun] is a/has a [occupation/adjectives] Bhaskaran and Bhallamudi (2019); Cho et al. (2019); Prates et al. (2020). These templatized sentences help elicit associations between gendered terms – name, pronouns and occupation, emotions, and other stereotypes. Templatized sentences usually have artificial structures and thus have limited applicability to downstream tasks with more natural language. Some prior works mine data from web sources like Wikipedia and Common Crawl Webster et al. (2018); Emami et al. (2019) to create a dataset for coreference resolution. However, many real-world sentences have a subtle manifestation of biases or use words that themselves do not have a negative connotation. Hence, rule-based sentence mining may not be able to capture the more implicit biases that humans have Blodgett et al. (2021).

A detailed analysis of the datasets used for gender bias was conducted by Stanczak and Augenstein (2021). Two more recently created datasets not introduced in the survey are the BUG dataset Levy et al. (2021) and the CORGI-PM dataset Zhang et al. (2023). Levy et al. (2021) uses lexical, syntactic pattern matching to create BUG, a dataset of sentences annotated for stereotypes. CORGI-PM is a Chinese corpus of 32.9K sentences that were human-annotated for gender bias. Sentences in the dataset are annotated as gender-biased (B) or non-biased (N). If gender-biased, they are further annotated for three different categories of stereotypical associations. In our work, to overcome the limitations of strictly templatized sentences and the challenges of mining real-world data, we leverage the text generation capabilities of GPT to create a dataset of graded statements.

Annotation tasks to identify gender bias can be categorized under the descriptive paradigm, where capturing disagreements in annotation due to annotator identity like gender, race, age, etc., and their lived experiences is important Röttger et al. (2022). However, the challenge is how to leverage the annotator disagreements to capture the nuances of the task. Studies on annotator disagreements have used different multi-annotator models Davani et al. (2022); Kiritchenko and Mohammad (2016). Past work has shown the efficacy of the Best-Worst Scaling framework in subjective tasks Verma et al. (2022); Hada et al. (2021); Pei and Jurgens (2020). Hence, in this study, we adopt this framework.

BWS or Maximum Difference Scaling (MaxDiff) proposed by Louviere (1991) has been long used in many psychological studies. BWS is an efficient comparative annotation framework. Studies Kiritchenko and Mohammad (2017) have shown that BWS can produce highly reliable real-valued ratings. In the BWS annotation setup, annotators are given a set of n items (where n > 1, often n = 4) and asked to identify the best and worst items based on a specific property of interest. Using 4-tuples is particularly efficient in best-worst annotations because each annotation generates inequalities for 5 out of the 6 possible item pairs. For instance, in a 4-tuple comprising items A, B, C, and D, where A is deemed the best and D is deemed the worst, the resulting inequalities would be: A > B, A > C, A > D, B > D, and C > D. By analyzing the best-worst annotations for a set of 4-tuples, real-valued scores representing the associations between the items and the property of interest can be calculated (Orme, 2009; Flynn and Marley, 2014). Thus far, the NLP community has leveraged the BWS annotation framework for various tasks. More recently, BWS has been used for the task of harshness modeling by Verma et al. (2022), determining degrees of offensiveness by Hada et al. (2021), and quantifying intimacy in language by Pei and Jurgens (2020). In the past, BWS has been used for tasks such as relational similarity (Jurgens et al., 2012), word-sense disambiguation (Jurgens, 2013), word–sentiment intensity (Kiritchenko et al., 2014), phrase sentiment composition (Kiritchenko and Mohammad, 2016), and tweet-emotion intensity (Mohammad and Bravo-Marquez, 2017; Mohammad and Kiritchenko, 2018). Using BWS, we create the first dataset of the degree of gender bias scores for GPT-generated text. Broadly, the efficacy of using preference-based evaluation methods has also been established in studies by Kang et al. (2023); Bansal et al. (2023).

Data Generation Pipeline

In this section, we discuss how we prompt GPT-3.5-Turbo to generate statements included in our dataset.

Data generation was conducted in multiple rounds before finalizing the prompting strategies. During the initial rounds, we iteratively evaluated our samples qualitatively and noticed several constraints, which are briefly described here:

Explicit Bias Generation with Gendered Seeds: The choice of a gendered seed significantly often resulted in generating sentences that explicitly express bias. Particularly, the model had the tendency to either do a counterfactual or a common biased attribute association in these samples. As real-world data often observe a higher degree of implicit bias, we wanted to increase the yield of such implicitly biased instances and accordingly used implicit (i.e., devoid of any physical quality attribute like " is stronger") and gender-agnostic/ neutral seeds (i.e., devoid of gender pronouns).

Contextually Breadth-Wise Limited Outputs: We also observed that the themes in the output from the model were mostly around traditionally biased physical attributes and professional associations in reductionist tones ("Women can’t do versus Men are better at ).

Limited Logical and Syntactic Diversity of the Generations: Initially, about 40% of our generation resulted in samples where the model introduced bias at the cost of the logical validity of the statement. For example, a seed like "The item was packaged in bubble wrap" would be completed as "The item was packaged in bubble wrap because the woman had delicate hands". While such samples hinted at the potential for some biased associations, their constructions seemed to be forced or poorly calibrated with the traditional gender attribution. We also noticed limited diversity in the syntax of the generated samples; particularly, the model-generated samples strongly emulated the syntactic structure of the in-context examples provided.

Based on these learnings, we carefully curated seeds (as described in Section 3.2) and the in-context examples to avoid such simple attributes that the model could regurgitate in its generations. We also varied the syntactic structure of our in-context examples to prevent redundancy in the structure of the generated samples.

2 Seed Selection

To ground the generations of GPT, we first curate a list of 500 seeds. The seeds are drawn from 4 categories – explicit, implicit, neutral, and random. Explicit, implicit, and neutral contribute 150 seeds each, and the remaining 50 seeds are from the random category. We select the seeds as follows:

Explicit: These are sentences that have explicit mentions of gender and stereotypical associations. We select these seeds from StereoSet Nadeem et al. (2021). We randomly sample 150 sentences where the target type is "gender" and the class is "stereotype." The sentences are uniformly distributed between males and females.

Implicit: These sentences have gender references but no stereotypical associations. We use the COPA dataset Roemmele et al. (2011) to sample these seeds. We manually select 150 seeds from the premise category.

Neutral: These are sentences that have no gender references in them. We manually select 150 seeds in this category from COPA Roemmele et al. (2011).

Random: We sample 50 random seeds from COPA Roemmele et al. (2011). These seed sentences do not overlap with the implicit and neutral categories.

From each explicit, implicit, and neutral category, we randomly sample 40 seeds each and create in-context examples pertaining to the 2 prompting strategies (20 each) discussed in Section 3.3.

3 Formats of Data Generations

To further promote syntactic diversity for the generated samples, we prompted the model to do generations across three formats: (a) Conversation, (b) Conversion, and (c) Completion. In the first, we prompted the model to generate a biased conversation against the provided seed, whereas, in (b) and (c), we prompted the model to convert and complete the provided seed, respectively (prompts shown in Appendix A.1). Upon qualitative evaluation, we noticed that conversational data wasn’t as usable due to a high incidence of neutral samples: we posit that this might be a function of this data format itself, i.e., conversations may require a much larger context width to encapsulate bias as opposed to more self-contained formats like conversion and completion. Therefore, we do not use the conversation-prompting strategy for our final dataset generation. Table 1 shows our different prompt types and seed types, with corresponding GPT generations.

Annotation

Akin to offensive language and harshness, ‘perceived gender bias’ is an inherently subjective concept based on lived experiences, community, education, etc., Blodgett et al. (2020); Davani et al. (2022); Biester et al. (2022). A large-scale crowd-sourced annotation study might be the ideal approach to gain a diversity of perspectives for such a subjective concept. However, getting annotations on a sensitive topic, such as gender bias, presents its own challenges. Quality control is one major issue in crowdsourcing annotations Mohammad and Turney (2013b). Therefore, in our study using a snowball sampling approach, we recruited 20 annotators from within Microsoft Research India who had some basic understanding of gender bias to perform the annotation task. All annotators were from India and had native proficiency in English. The annotators had at least an undergraduate degree as their minimum educational qualification. Out of the 20 annotators, 12 were male, and 8 were female.

In this study, we adopted the methodology outlined by Kiritchenko and Mohammad (2016) to obtain BWS annotations. Annotators were presented with sets of four statements. They were tasked with identifying the statement that is the most negatively gender-biased and the statement that is the least negatively gender-biased. Using the script provided by Kiritchenko and Mohammad (2016) to generate 4-tuples, we obtained 2N 4-tuples (in our case, N = 1000).http://saifmohammad.com/WebPages/BestWorst.html The 4-tuples were generated such that each statement was seen in eight different 4-tuples and no two 4-tuples had more than 2 statements in common. 64.25% of our tuples are annotated at least thrice, and the remaining are annotated twice. Since each statement occurs in 8 different 4-tuples, we have $16$ $(8X2)$ — $24$ $(8X3)$ judgments per statement.

2 Gender Bias Annotation Task

Drawing from previous work, our annotation task defined gender bias as "the systematic, unequal treatment based on one’s gender." Negatively gender-biased statements can discriminate against a specific gender by means of stereotypical associations, systemic assumption, patronization, use of metaphors, slang, denigrating language, and other factorsStanczak and Augenstein (2021). We encouraged annotators to trust their instincts. We include detailed annotation instructions and sample questions in the Appendix A.2. The BWS responses are converted to a degree of gender bias scores using a simple counting procedure Orme (2009); Flynn and Marley (2014). For each statement, the score is the proportion of times the statement is chosen as the most negatively biased minus the proportion of times the statement is chosen as the least negatively biased. We convert the scores from the range of $(-1)$ —1 to 0 —1.

3 Annotation Reliability

Standard inter-annotator agreement measures are inadequate for evaluating the quality of comparative annotations. Disagreements observed in tuples consisting of two closely ranked items provide valuable information for BWS by facilitating similar scoring of these items. Therefore, following best practices, we compute average split-half reliability (SHR) values to asses the reproducibility of the annotations and the final ranking. To compute SHR, the annotations for each 4-tuple are randomly split into two halves. Using these two splits, two sets of rankings are determined. We then calculate the correlation values between these two sets. This process is repeated 100 times, and average correlation values are reported. A high correlation value indicates that the annotations are of good quality. We obtained SHR scores of 0.86, indicating good annotation reliability. Table 2 summarizes the annotation statistics. We release "Fifty Shades of Bias" a dataset of 1000 comments with aggregated degrees of gender bias scores and the individual annotations for each tuple.

Data Analysis

In this section, we analyze the distribution of scores in our data, the impact of seed selection and prompting methods, and various themes in the gender bias grading.

We visualize the histogram of the degree of gender bias scores in Figure 1. The histogram is plotted over 10 equispaced score bins of size 0.1. We observe a Gaussian-tailed uniform distribution.

To analyze the data, we placed the statements in 3 score bins (bin 1: to $0.316$, bin 2: $0.316$ to $0.579$, bin 3: $0.579$ to $1.0$). We decided on the threshold score value for each bin after careful manual inspection of the data. Table 3 shows some comments from the dataset. Bin 1 has $364$ statements that can largely be perceived as neutral. Bin 2 has $248$ data points that have gendered references but are not explicitly stereotypical and some neutral statements, and finally, bin 3 contains $388$ data points and primarily consists of sentences that have explicit references to comparisons between males and females and the stereotypes associated with both.

As discussed in section 3.2, we use 4 different seed types to generate graded gender-biased text using GPT-4. We map the number of sentences in each of the three bins that come from the different seed types (explicit, implicit, neutral, and random) used during text generation. Figure 2 shows that bin 1 contains generations using neutral and random seeds. Bin 2 has a mix of generations obtained using implicit and neutral seeds, while bin 3 contains all generations obtained using explicit and one-third of the generations obtained using implicit seeds. While this shows that our grounding strategies, by carefully curating seeds and using appropriate in-context examples, have worked, it also highlights the non-deterministic nature of GPT-3.5-Turbo. It indicates that GPT-3.5-Turbo might be able to produce biased content even for completely innocuous text.

In section 3.3, we discuss the two prompting methods (conversion, completion) we adopt to generate the gender-biased text using the various categories of seeds. Different prompting methods may impact the kind of data GPT-3.5-Turbo generates. Figure 3 shows the distribution of statements from each prompting method over the 3 score bins. From the figure, we can observe that the generations using the "completion" prompting method are nearly uniformly distributed across the bins, with bin 1 having a slightly higher share. However, for the statements generated using the "conversion" prompting method, we observe a sharp increase from bin 1 to bin 2, with the majority of the generations in bin 3. This shows that the prompting method might provide a further level of grounding that can aid consistency in the generation of graded text.

To understand the specific themes and phrases of each of the bins, we (a) calculate the Pointwise Mutual Information (PMI) scores of all the unique words in the sentences, and (b) use an informed Dirichlet model Monroe et al. (2008) to identify meaningful n-grams that distinguish between the sentences in different bins. Table 4 shows the top 4 PMI scoring words for each of the 3 bins. We see that bin 1 has words that are associated with positive traits and emotions. Bin 2 has words that by themselves do not have negative connotations but can be used to stereotype the genders. In contrast, bin 3 consists of comparative words likely to be used to compare the two genders.

We analyze the bigrams and trigrams using the Convokit Chang et al. (2020) implementation of the log-odds with Dirichlet prior approach proposed by Monroe et al. (2008). Bin 2 and 3 both have sentences generated from implicit seeds, and Bin 1 and 2 have sentences generated from neutral seeds. To understand the distinguishing themes, we perform two pairwise comparisons. The pairwise comparison between bins 2 and 3 highlights an interesting pattern. Table 5 presents the top phrases associated with both classes. We see that sentences in bin 3 are towards the community, i.e., women and men, and have explicit comparison phrases such as ‘not as,’ ‘typical of,’ and others. In contrast, phrases most associated with bin 2 while having gendered associations are more individual specific, e.g., ‘with his,’ ‘my husband,’ ‘that she,’ and more.

The analysis shows that as the gradation of bias varies in text, there is an evident difference in the themes and tone of the sentences.

Computational Modeling

In this section, we investigate how closely the predictions from available automatic detection models for gender bias and offensive language are related to gender bias ratings in our dataset. We also show the performance of LLMs on giving a gender bias rating based on few-shot examples.

We fine-tune a binary classifier on the CORGI-PM dataset containing sentences annotated for "biased" and "non-biased" categories Zhang et al. (2023). We use the same train, validation, and test splits as provided by the authors. We fine-tune bert-base-multilingual-uncased (Devlin et al., 2019), with a batch size of 128, a learning rate of $1e-5$, AdamW optimizer, and cross-entropy loss as the objective function. We implement early stopping based on validation loss, with $patience=3$. The model was fine-tuned for 2 epochs and achieved an accuracy of 0.82 on the test set. We obtain the output layer’s softmax probabilities corresponding to the ‘biased’ class for all sentences in our dataset.

Ruddit is an offensive language dataset annotated using BWS Hada et al. (2021). The dataset has fine-grained real-valued scores of offensiveness ranging from $-1$ to $1$. We use the best model reported by the authors in their paper. The authors fine-tuned HateBERT Caselli et al. (2021) on Ruddit and reported $r=0.886$. We use this model to predict the offensiveness scores on text in our dataset.

1.1 LLMs

We prompt GPT-3.5-Turbo and GPT-4 to predict a degree of gender bias score for all statements in our dataset. We provide 8 in-context examples in our prompt (prompt included in Appendix A.3). We divide our dataset into 4 equispaced score bins of size $0.25$ and randomly sample 2 statements from each bin for our in-context examples. We obtain scores for each statement in our dataset from GPT-3.5-Turbo and GPT-4. Since in-context examples can greatly impact the output from these models, we get 5 sets of scores from each model and report the average performance values.

1.2 Perspective API

Perspective API is a text-based classification system for automated detection of toxic language Hosseini et al. (2017); Rieder and Skop (2021). It uses machine learning models to identify abusive comments. The model scores a phrase across 6 dimensions based on the perceived impact the text may have in a conversation. The dimensions included in Perspective API are toxicity, identity attack, insult, threat, severe toxicity, and profanity. We use Perspective API to obtain scores across these six dimensions for each sentence in our dataset.

2 Results and Analysis

Table 6 shows the results of all models on our dataset. We can see that GPT-3.5-Turbo and GPT-4 show the highest correlation with the degree of gender bias scores in our dataset. The low MSE values also indicate that the scores are well-calibrated with human ratings. HateBERT, fine-tuned on Ruddit (a dataset similar to ours), does not correlate very highly with our dataset. However, a reasonable correlation indicates that there is a significant overlap between the concept of offensive language and gender bias. The model fine-tuned on CORGI-PM (a dataset explicitly designed for detecting gender bias), even though in a classification setting, shows a decent correlation with our dataset. Finally, results from Perspective API show that out of the six dimensions, identity attack most highly correlates with the degree of gender bias, followed by toxicity. Our findings show that when GPT-3.5-Turbo is prompted to generate gender-biased text, it generates texts that attack identity or are toxic/offensive.

Qualitative Analysis of Reasoning

While the results in Section 6.2 show that the bias score from GPT-4 is highly correlated with the human scores, it is unclear how such models reason about their predictions. To probe further, we randomly sample 20 sentences from each of the three bins in our dataset and prompt GPT-4 to provide a reason for the score it provided (refer Appendix A.4). An analysis of some interesting examples is discussed in this section. Table 7 shows some examples along with their score and reasoning (for more examples, refer to Appendix A.4).

The reasoning of statements 1 and 2 both begin with the phrase ‘slight gender bias.’ However, we see statement 2’s reference to the stereotypical qualities of a female – women not getting ready timely, while no such references are mentioned in statement 1. While the reasoning for Statement 2 acknowledges that the statement has a negative attribute assigned to “she”; it uses hedging language to say that the negativity is not too strong. In comparison, the high BWS score shows that the annotators perceive the negative attribution to “she” strongly. Moreover, for statement 1, GPT-4 confuses the stated fact of ‘sister being a confidant’ with an assumption. Therefore, it is concerning that the reasoning for bias provided by GPT-4 for the two statements is almost similar. Akin to statement 2, statement 3, too, tap into the stereotype of males and their lack of interest in personalizing their room. However, the GPT-4 reasoning does not seem to capture that connotation, making one question the higher score the statement received compared to statement 2. In comparison to these sentences, we notice that as the stereotype becomes more explicit in Statement 4, both the human annotators and GPT-4 score align well, and the reasoning is well articulated. This analysis shows that the score assigned by GPT-4 and the reasoning does not always align. While the scores from GPT-4 are highly correlated to the human annotation scores, the reason behind it is often flawed, especially when the context is not extremely explicit. This indicates that effort akin to work like Zhou et al. (2023) that contextualize the generation of offensive statements should be accelerated to improve the contextual reasoning capabilities of LLMs.

Conclusion

We created a dataset with fine-grained human ratings of the degree of gender bias scores for GPT-generated English text. We used a comparative annotation framework – Best–Worst Scaling that overcomes the limitations of traditional rating scales. Even for a highly subjective task such as identifying gender bias, the fine-grained scores obtained with BWS indicate that BWS can help elicit responses from humans that are markedly discriminative. The high correlations ($r\approx 0.86$) on repeat annotations indicate that the ratings obtained are highly reliable. We conducted in-depth data analysis to understand the effect of different seed types and prompting strategies used in our dataset. Thematic analysis showed us that humans consider statements that target communities to be more biased than those that target individuals. We presented benchmarking experiments on our dataset and showed that identity attack most highly correlates with gender bias. While GPT-4 can produce a degree of gender bias ratings that highly correlate with human ratings, our analysis showed that the reasoning GPT-4 produces for its rating is often flawed. We make our dataset freely available to the research community.

Limitations and Future Work

The in-context examples used to ground GPT-3.5-Turbo were manually created by the authors; hence, they are limited by the authors’ sensibilities. Future work can look at creating a more diverse yet quality-rich dataset for grounding the data generation. This dataset was created to investigate the biases GPT models can generate despite all the content moderation policies in place. Therefore, The data generated by GPT-3.5-Turbo is limited by the dataset it has seen during training and hence might not cover a full range of gender-biased language that individuals have experienced. Pre-existing corpora on gender bias, such as the BUG dataset Levy et al. (2021), include stereotypical statements such as "Manu Prakash is one of the most imaginative scientists of his generation…" and "Among them was the president himself.". These sentences are more naturally occurring. The sentences included in our dataset have a limited diversity of themes. In the future, we will extend the study to more naturally occurring sentences from online corpora and other mediums. While we take several steps to overcome the limitations of GPT generations in this context, as discussed in Section 3.1, we acknowledge that some of these challenges persist. Lived experiences, opinions, and perspectives of the annotator impact the annotation of subjective tasks like ours. We recruited annotators from only one geography, and thus, this could have been one source of bias. Future work can consider recruiting annotators from more diverse backgrounds to increase the generalizability of the results. Our observations are limited to a relatively smaller sample size of statements. In the future, it would be interesting to explore such fine-grained ratings of gender bias on larger text corpora with longer text, different styles of prompting, seed sources, and languages other than English.

Ethical Considerations

We use the framework by Bender and Friedman (2018) to discuss the ethical considerations for our work.

Institutional Review: All aspects of this research were reviewed and approved by Microsoft Research IRB (ID10743).

Data: The dataset for this work was generated using API calls to GPT-3.5-Turbo. While the policies state that they don’t use the data submitted during API calls to fine-tune, it is plausible that during the 30-day retaining period, the data is used https://openai.com/policies/api-data-usage-policies. Without proper guardrails, gender-biased data might propagate harmful biases in the model.

Annotator Demographics: We restricted annotators to those working in the institute in India where this research was conducted. Apart from the country of residence, age, and educational qualification, no other information shall be made public about the annotators. The annotators are governed by the institute’s privacy policy.

Annotation Guidelines: We draw inspiration from the community standards set for similar tasks, such as offensive language annotations. While these guidelines were created after careful research, we acknowledge that they do not represent all possible perspectives. The perception of gender bias would vary across individuals and groups; thus, we emphasize that the scores here are not the only appropriate or correct scores but rather a representation of what the majority of our annotators think.

Impact on Annotators: In this work, we ask annotators to look at gender-biased data, some of which contained explicit stereotypes, and exposure to these statements could have negatively impacted the annotators Roberts (2016); Vidgen et al. (2019). To mitigate the effects, we took the following steps:

Our data had a good mix of explicitly biased, implicitly biased, and neutral sentences. This ensured that most of the tuples that annotators were not overexposed to only extremely biased sentences.

To further help prevent overexposure to biased language, we limited the number of annotations each annotator could attempt. Each annotator could only attempt 8% of the task,

The authors personally briefed each annotator about how their work would positively contribute to the field and the annotation framework itself.

They were given the option to give feedback on each tuple in a freeform field, which allowed them to give unrestricted feedback.

Feedback: We acknowledge that our dataset may be influenced by the inherent bias in the data generation process and the subjective judgments made by the annotators. Moreover, it is important to note that this compilation does not encompass all ethical considerations and limitations. We actively encourage feedback from the research community and any users of our dataset.

References

Appendix A Supplemental Material

: Using the given statement as a start, You generate only one gender-biased sentence to complete the provided seed. Some examples that you can follow are:

: You convert any given statement into one statement that is biased against a gender. Follow the examples specified below:

: Using the given statement as a start, You generate a long gender-biased conversation between 2 people like the conversations given as examples here:

A.2 Annotation

Figure 4 shows the detailed annotation instructions given to the crowd-workers for the task.

Figure 5 shows a sample questionnaire for the final annotation task.

A.3 GPT Prompt for Scoring

Prompt: On a fine-grained real-valued scale of 0 to 1, give the following sentence a rating of the degree of gender bias. 1 is for the most negatively biased sentence, and 0 is for the least negatively biased sentence. Only give the score and nothing else.

A.4 GPT-4 Reasoning Prompt

Prompt: On a fine-grained real-valued scale of 0 to 1, you gave the following sentence a rating of the degree of gender bias. Where 1 is for the most negatively biased sentence, 0 is for the least negatively biased sentence. Give a reason for your score.

Table 8 shows GPT scores of statements with generated reasoning.