Towards a Deep Multi-layered Dialectal Language Analysis: A Case Study of African-American English

Introduction

Over the years, social media users have leveraged online conversational platforms to perpetually express themselves online. For example, African American English (AAE)A dialectal continuum previously known as Northern Negro English, Black English Vernacular (BEV), Black English, African American Vernacular English (AAVE), African American Language (AAL), Ebonics, and Non-standard English (Labov, 1975; Bailey et al., 1998; Green, 2002, 2014; Baugh, 2008; Bland-Stewart, 2005; King, 2020). It is often referred to as African American Language (AAL) and African American English (AAE). In this work, we use the denotation AAE., an English language variety is often heavily used on Twitter Field et al. (2021); Blodgett et al. (2020). This dialect continuum is neither spoken by all African Americans or individuals who identify as BIPOC (Black, Indigenous, or People of Color), nor is it spoken only by African Americans or BIPOC individuals Field et al. (2021); Bland-Stewart (2005). In some cases, AAE, a low-resource language (LRL) may be the first (or dominant) language, rather than the second (or non-dominant) language of an English speaker.

Specifically, AAE is a regional dialect continuum that consists of a distinct set of lexical items, some of which have distinct semantic meanings, and may possess different syntactic structures/patterns than in Mainstream American English (MAE) (e.g., differentiating habitual be and non-habitual be usage) (Stewart, 2014; Dorn, 2019; Jones, 2015; Field et al., 2021; Bland-Stewart, 2005; Baugh, 2008; Blodgett et al., 2020; Labov, 1975). In particular, Green (2002) states that AAE possesses a morphologically invariant form of the verb that distinguishes between habitual action and currently occurring action, namely habitual be. For example, “the habitual be” experimenthttps://www.umass.edu/synergy/fall98/ebonics3.html by University of Massachusetts Amherst’s Janice Jackson.

However, AAE is perceived to be “bad english” despite numerous studies by socio/raciolinguists and dialectologists in their attempts to quantify AAE as a legitimized language Baugh (2008); Field et al. (2021); Bland-Stewart (2005); Labov (1975).

“[T]he common misconception [is] that language use has primarily to do with words and what they mean. It doesn’t. It has primarily to do with people and what they mean.” – Clark and Schober (1992)

Recently, online AAE has influenced the generation of resources for AAE-like text for natural language (NLP) and corpus linguistic tasks e.g., part-of-speech (POS) tagging Jørgensen et al. (2016); Blodgett et al. (2018), language generation Groenwold et al. (2020) and automatic speech recognition Dorn (2019); Tatman and Kasten (2017). POS tagging is a token-level text classification task where each token is assigned a corresponding word category label (see Table 1). It is an enabling tool for NLP applications such as a syntactic parsing, named entity recognition, corpus linguistics, etc. In this work, we incorporate a human-in-the-loop paradigm by directly involving affected (user) communities to understand context and word ambiguities in an attempt to study dialectal language inclusivity in NLP language technologies that are generally designed for dominant language varieties. Dacon and Liu (2021) state that,

“NLP systems aim to [learn] from natural language data, and mitigating social biases become a compelling matter not only for machine learning (ML) but for social justice as well.”

To address these issues, we aim to empirically study predictive bias (see Swinton (1981) for definition) i.e., if POS tagger models make predictions dependent on demographic language features, and attempt a dynamic approach in data-collection of non-standard spellings and lexical items. To examine the behaviors of AAE speakers and their language use, we first collect variable (morphological and phonological) rules of AAE language features from literature Labov (1975); Bailey et al. (1998); Green (2002); Bland-Stewart (2005); Stewart (2014); Blodgett et al. (2016); Elazar and Goldberg (2018); Baugh (2008); Green (2014) (see Appendix C). Then, we employ 5 trained sociolinguist Amazon Mechanical Turk (AMT) annotatorsA HIT approval rate $\geq$ 95% was used to select 5 bi-dialectal AMT annotators between the ages of 18 - 55, and completed $>$ 10,000 HITs and located within the United States. who identify as bi-dialectal dominant AAE speakers to address the issue of lexical, semantic and syntactic ambiguity of tweets (see Appendix B for annotation guidelines). Next, we incorporate a human-in-the-loop paradigm by recruiting 20 crowd-sourced diglossic annotators to evaluate AAE language variety (see Table 2). Finally, we conclude by expanding on the need for dialectal language inclusivity.

Related Work

Previous works regarding AAE linguistic features have analyzed tasks such as unsupervised domain adaptation for AAE-like language (Jørgensen et al., 2016), detecting AAE syntax(Stewart, 2014), language identification Blodgett and O’Connor (2017), voice recognition and transcription Dorn (2019), dependency parsing (Blodgett et al., 2018), dialogue systems (Liu et al., 2020), hate speech/toxic language detection and examining racial bias (Sap et al., 2019; Halevy et al., 2021; Xia et al., 2020; Davidson and Bhattacharya, 2020; Zhou et al., 2021; Mozafari et al., 2020; Xu et al., 2021; Koenecke et al., 2020), and language generation (Groenwold et al., 2020). These central works are conclusive for highlighting systematic biases of natural language processing (NLP) systems when employing AAE in common downstream tasks.

Although we mention popular works incorporating AAE, this dialectal continuum has been largely ignored and underrepresented by the NLP community in comparison to MAE. Such lack of language diversity cases constitutes technological inequality to minority groups, for example, by African Americans or BIPOC individuals, and may intensify feelings of disenfranchisement due to monolingualism. We refer to this pitfall as the inconvenient truth i.e.,

“[I]f the systems show discriminatory behaviors in the interactions, the user experience will be adversely affected.” — Liu et al. (2020)

Therefore, we define fairness as the model’s ability to correctly predict each tag while performing zero-shot transfer via dialectal language inclusivity.

Moreover, these aforementioned works do not discuss nor reflect on the “role of the speech and language technologies in sustaining language use” (Labov, 1975; Bird, 2020; Blodgett et al., 2020) as,

“… models are expected to make predictions with the semantic information rather than with the demographic group identity information” — Zhang et al. (2020).

Interactions with everyday items is increasingly mediated through language, yet systems have limited ability to process less-represented dialects such as AAE. For example, a common AAE phrase, “I had a long ass day” would receive a lower sentiment polarity score because of the word “ass”, a (noun) term typically classified as offensive; however, in AAE, this term is often used as an emphatic, cumulative adjective and perceived as non-offensive.

Motivation: We want to test our hypothesis that training each model on correctly tagged AAE language features will improve the model’s performance, interpretability, explainability, and usability to reduce predictive bias.

Dataset and Annotation

We collect 3000 demographically-aligned African American (AA) tweets possessing an average of 7 words per tweet from the publicly available TwitterAAE corpus by Blodgett et al. (2016). Each tweet is accompanied by inferred geolocation topic model probabilities from Twitter + Census demographics and word likelihoods to calculate demographic dialect proportions. We aim to minimize (linguistic) discrimination by sampling tweets that possess over 99% confidence to develop “fair” NLP tools that are originally designed for dominant language varieties by integrating non-standardized varieties. More information about the TwitterAAE dataset, including its statistical information, annotation process, and the link(s) to downloadable versions can be found in Appendix A.

2 Preprocessing

As it is common for most words on social media to be plausibly semantically equivalent, we denoise each tweet as tweets typically possess unusual spelling patterns, repeated letter, emoticons and emojisEmoticons are particular textual features made of punctuation such as exclamation marks, letters, and/or numbers to create pictorial icons to display an emotion or sentiment (e.g., “;)” $\Rightarrow$ winking smile), while emojis are small text-like pictographs of faces, objects, symbols, etc.. We replace sequences of multiple repeated letters with three repeated letters (e.g., Hmmmmmmmm $\rightarrow$ Hmmm), and remove all punctuation, “@” handles of users and emojis. Essentially, we aim to denoise each tweet only to capture non-standard spellings and lexical items more efficiently.

3 Annotation

First, we employ off-the-shelf taggers such as spacyhttps://spacy.io and TwitterNLPhttps://github.com/ianozsvald/ark-tweet-nlp-python; however, the Natural Language Toolkit (NLTK) (Loper and Bird, 2002) provides a more fine-grained Penn Treebank Tagset (PTB)https://www.guru99.com/pos-tagging-chunking-nltk.html along with evaluation metrics per tag such as F1 score. Next, we focus on aggregating the appropriate tags by collecting and manually-annotating tags from AAE/slang-specific dictionaries to assist the AMT annotators, and later we contrast these aggregated tags with inferred NLTK PTB inferred tags. In Figure 1, we display NLTK inferred and manually-annotated AAE tags from $k=300$ randomly sampled tweets.

The Online Slang Dictionary (American, English, and Urban slang)http://onlineslangdictionary.com - created in 1996, this is the oldest web dictionary of slang words, neologisms, idioms, aphorisms, jargon, informal speech, and figurative usages. This dictionary possesses more than 24,000 real definitions and tags for over 17,000 slang words and phrases, 600 categories of meaning, word use mapping and aids in addressing lexical ambiguity.

Word Typehttps://wordtype.org/ - an open source POS focused dictionary of words based on the Wiktionaryhttps://www.wiktionary.org project by Wikimediahttps://www.wikimedia.org. Researchers have parsed Wiktionary and other sources, including real definitions and categorical POS word use cases necessary to address the issue of lexical, semantic and syntactic ambiguity.

4 Human Evaluation

Methodology

In this section, we describe our approach to perform a preliminary study to validate the existence of predictive bias Elazar and Goldberg (2018); Shah et al. (2020) in POS models. We first introduce the POS tagging, and then propose two ML sequence models.

We consider POS tagging as it represents word syntactic categories and serves as a pre-annotation tool for numerous downstream tasks, especially for non-standardized English language varieties such as AAE (Zampieri et al., 2020). Common tags include prepositions, adjective, pronoun, noun, adverb, verb, interjection, etc., where multiple POS tags can be assigned to particular words due to syntactic structural patterns. This can also lead to misclassification of non-standardized words that do not exist in popular pre-trained NLP models.

2 Models

We propose to implement two well known sequence modeling algorithms, namely a Bidirectional Long Short Term Memory (Bi-LSTM) network, a deep neutral network (DNN) (Hochreiter and Schmidhuber, 1997; Graves and Schmidhuber, 2005) that has been used for POS tagging (Ling et al., 2015; Plank et al., 2016), and a Conditional Random Field (CRF) Lafferty et al. typically used to identify entities or patterns in texts by exploiting previously learned word data.

Taggers: First, we use NLTK (Loper and Bird, 2002) for automatic tagging; then, we pre-define a feature function for our CRF model where we optimized its L1 and L2 regularization parameters to 0.25 and 0.3, respectively. Later, we train our Bi-LSTM network for 40 epochs with an Adam optimizer, and a learning rate of 0.001. Note that each model would be accompanied by error analysis for a 70-30 split of the data with 5-fold cross-validation to obtain model classification reports, for metrics such as precision, recall and F1-score.

Operationalization of AAE as an English Language Variety

As (online) AAE can incorporate non-standardized spellings and lexical items, there is an active need for a human-in-the-loop paradigm as humans provide various forms of feedback in different stages of workflow. This can significantly improve the model’s performance, interpretability, explainability, and usability. Therefore, crowd-sourcing to develop language technologies that consider who created the data will lead to the inclusion of diverse training data, and thus, decrease feelings of marginalization. For example, CORAALhttps://oraal.uoregon.edu/coraal, is an online resource that features AAL text data, recorded speech data, etc., into new and existing NLP technologies, AAE speakers can extensively interact with current NLP language technologies.

Consequently, to quantitatively and qualitatively ensure fairness in NLP tools, artificial intelligence (AI) and NLP researchers need to go beyond evaluation measures, word definitions and word order to assess AAE on a token-level to better understand context, culture and word ambiguities. We encourage both AI and NLP practitioners to prioritize collecting a set of relevant labeled training data with several examples of informal phrases, expressions, idioms, and regional-specific varieties. Specifically, in models intended for broad use such as sentiment analysis by partnering with low-resource and dialectal communities to develop impactful speech and language technologies for dialect continua such as AAE to minimize further stigmatization of an already stigmatized minority group.

Conclusion

Throughout this work, we highlight the need to develop language technologies for such varieties, pushing back against potentially discriminatory practices (in many cases, discriminatory through oversight more than malice). Our work calls for NLP researchers to consider both social and racial hierarchies sustained or intensified by current computational linguistic research. By shifting towards a human-in-the-loop paradigm to conduct deep multi-layered dialectal language analysis of AAE to counter-attack erasure and several forms of biases such as selection bias, label bias, model overamplification, and semantic bias (see Shah et al. (2020) for definitions) in NLP.

We hope our dynamic approach can encourage practitioners, researchers and developers for AAE inclusive work, and that our contributions can pave the way for normalizing the use of a human-in-the-loop paradigm both to obtain new data and create NLP tools to better comprehend underrepresented dialect continua and English language varieties. In this way, NLP community can revolutionize the ways in which humans and technology cooperate by considering certain demographic attributes such as culture, background, race and gender when developing and deploying NLP models.

Limitations And Ethical Considerations

All authors must warrant that increased model performance for non-standard varieties such as underrepresented dialects, non-standard spellings or lexical items in NLP systems can potentially enable automated discrimination. In this work, we solely attempt to highlight the need for dialectal inclusivity for the development of impactful speech and language technologies in the future, and do not intend for increased feelings of marginalization of an already stigmatized community.

Acknowledgements

The authors would like to thank Shaylnn L.A. Crum-Dacon, Serena Lotreck, Brianna Brown and Kenia Segura Abá, Jyothi Kumar and Shin-Han Shiu for their support and the anonymous reviewers for their constructive comments.

References

Appendix A Dataset Details

Our collected dataset is demographically-aligned on AAE in correspondence on the dialectal tweet corpus by Blodgett et al. (2016). The TwitterAAE corpus is publicly available and can be downloaded from linkhttp://slanglab.cs.umass.edu/TwitterAAE/. Blodgett et al. (2016) uses a mixed-membership demographic language model which calculates demographic dialect proportions for a text accompanied by a race attribute—African America, Hispanic, Other, and White in that order. The race attribute is annotated by a jointly inferred probabilistic topic model based on the geolocation information of each user and tweet. Given that geolocation information (residence) is highly associated with the race of a user, the model can make accurate predictions. However, there a a low number messages that possess a posterior probabilities of NaN as these are messages that have no in-vocabulary words under the model.

Appendix B Annotator Annotation Guidelines

You will be given demographically-aligned African American tweets, in which we refer to these tweets as sequences. As a dominant AAE speaker, who identifies as bi-dialectal, your task is to correctly identify the context of each word in a given sequence in hopes to address the issues of lexical, semantic and syntactic ambiguity.

If you responded “yes” above, are you bi-dialectal?

If you responded “yes”, given a sequence, have you ever said, seen or used any of these words given the particular sequence?

Given a sequence, what are the SAE equivalents to the identified non-SAE terms?

For morphological and phonological (dialectal) purposes, are these particular words spelt how would you say or use them?

If you responded “no” above, can you provide a different spelling along with its SAE equivalent?

What is the context of each word given the particular sequence?

Given NLTK’s Penn Treebank Tagsethttps://www.guru99.com/pos-tagging-chunking-nltk.html, what is the most appropriate POS tag for each word in the given sequence?

B.2 Human evaluation of POS tags Protocol

Given the tagged sentence, are there any misclassified tags?

If you responded “yes” above, can you provide a different POS tag, and state why it is different?

Appendix C Variable Rules Examples

In this section we present a few examples of simple, deterministic phonological and morphological language features or current variable rules which highlight several regional varieties of AAE which typically attain misclassified POS tags. Please note that a more exhaustive list of these rules is still being constructed as this work is still ongoing. Below are a few variable cases (MAE $\rightarrow$ AAE), some of which may have been previously shown in Table 2:

Consonant (‘t’) deletion (Adverb case) : e.g. “just” $\rightarrow$ “jus”; “must” $\rightarrow$ “mus”

Contractive negative auxiliary verbs replacement: “doesn’t” $\rightarrow$ “don’t”

Contractive (’re) loss: e.g. “you’re” $\rightarrow$ “you”; “we’re” $\rightarrow$ “we”

Copula deletion: Deletion of the verb “be” and its variants, namely “is” and “are” e.g. “He is on his way” $\rightarrow$ “He on his way”; “You are right” $\rightarrow$ “You right”

Homophonic word replacement (Pronoun case): e.g. “you’re” $\rightarrow$ “your”

Indefinite pronoun replacement: e.g. “anyone” $\rightarrow$ “anybody”;

Interdental fricative loss (Coordinating Conjuction case): e.g. “this” $\rightarrow$ “dis”; ‘that’ $\rightarrow$ ‘dat”; “the” $\rightarrow$ “da”

Phrase reduction (present/ future tense) $\Rightarrow$ word (Adverb case): e.g. “what’s up” $\rightarrow$ “wassup”; “fixing to” $\rightarrow$ “finna”

Present tense possession replacement: e.g. “John has two apples” $\rightarrow$ “John got two apples”; “The neighbors have a bigger pool” $\rightarrow$ “The neighbors got a bigger pool”

Remote past “been” + completive (‘done’): “I’ve already done that” $\rightarrow$ “I been done that”

Remote past “been” + completive (‘did’): “She already did that” $\rightarrow$ “She been did that”

Remote past “been” + Present tense possession replacement: “I already have food” $\rightarrow$ “I been had food”; “You already have those shoes” $\rightarrow$ “You been got those shoes”

Term-fragment deletion: e.g. “brother” $\rightarrow$ “bro”; “sister” $\rightarrow$ “sis”; “your” $\rightarrow$ “ur”; “suppose” $\rightarrow$ “pose”; “more” $\rightarrow$ “mo”

Term-fragment replacement: “something” $\rightarrow$ “sumn”; “through” $\rightarrow$ “thru”; “for” $\rightarrow$ “fa”; “nothing” $\rightarrow$ “nun”