NaijaSenti: A Nigerian Twitter Sentiment Corpus for Multilingual Sentiment Analysis

Introduction

Sentiment analysis (SA) deals with the detection and classification of sentiment in texts [Pang and Lee (2007]. In recent years, SA has attracted considerable interest, which can be attributed to its many vital applications. However, most of the work on SA focuses on high-resource languages such as English [Yimam et al. (2020] while languages with a limited amount of data remain poorly represented [Nasim and Ghani (2020]. This problem is not unique to sentiment analysis, but affects NLP research as a whole [Joshi et al. (2020]. Recently, ?) and ?) examined how socio-cultural factors hinder NLP for low-resource languages, potentially resulting in economic inequities [Weidinger et al. (2021].

With more than 200 million people and 522 native languages, Nigeria is the most populous and linguistically diverse country in Africa, as well as the third most multilingual country in the world. https://www.ethnologue.com/guides/countries-most-languages However, due to the lack of training data for many NLP applications, these languages are underserved by digital technology. Therefore, a concerted effort is required to create resources for such languages [Adelani et al. (2021].

In this paper, we present NaijaSenti https://github.com/hausanlp/NaijaSenti—an open-source Twitter sentiment dataset for the four most spoken languages in Nigeria—Hausa, Igbo, Pidgin, and Yorùbá. This is the largest labelled sentiment dataset in these languages to date. As the Twitter API does not support these languages, we propose methods to enable the collection, filtering, and annotation of such low-resource language data. Overall, we annotated around 30,000 tweets in Hausa, Igbo, Yorùbá and Nigerian Pidgin (also known as Naija). The data highlight the challenges of sentiment analysis in these languages. For example, the absence of diacritics makes some tweets ambiguous in Yorùbá and Igbo. In addition, code-mixing is a common occurrence, with about 43% of Igbo tweets code-mixing between Igbo and English.

We conduct extensive experiments demonstrating that state-of-the-art pre-trained multilingual models achieve strong performance on sentiment classification on NaijaSenti. The best models have been explicitly trained on unlabelled data in African languages during pre-training such as AfriBERTa [Ogueji et al. (2021] or using language-adaptive fine-tuning [Pfeiffer et al. (2020].

The main contributions of this paper are:

We curate large-scale manually annotated code-mixed and monolingual sentiment datasets for Hausa, Igbo, Yorùbá and Pidgin languages.

We built a manually annotated sentiment lexicon in Hausa, Igbo, and Yorùbá. We also semi-automatically develop translated emotion and sentiment lexicons in these languages.

We curate the largest Twitter corpus in each language that can be useful for other NLP downstream tasks.

We present several benchmark experiments on sentiment analysis in Hausa, Igbo, Yorùbá, and Pidgin languages.

We make the datasets and code freely available to foster further research in the NLP community.

Related Work

Sentiment analysis for low-resource languages has recently gained popularity [Yimam et al. (2020, Xia et al. (2021, Jovanoski et al. (2021] due to the availability of relatively large amounts of tweets in such languages. Several studies have investigated using Twitter for sentiment analysis—by either automatically building a Twitter corpus or manually annotating one. Notable studies that automatically built Twitter corpora include ?), ?), and ?). More recently, ?) automatically built an Arabic Twitter sentiment analysis corpus using distant supervision and self-training. In contrast, other studies, such as ?), ?), ?), ?), and ?) employed native speakers or expert annotators to manually annotate the corpus. Our work is more closely related to ?) and ?), as it involves both the use of emoji as a distantly supervised approach for tweet extraction and the use of a translated sentiment lexicon to filter tweets before manual annotation [Nakov et al. (2019]

Despite advances in sentiment analysis for low-resource languages, indigenous Nigerian languages have received scant attention. This is mostly due to the absence of a freely accessible dataset in these languages. Nevertheless, there have been a significant number of studies on Nigerian code-mixed English [Nwofe (2017, Olaleye et al. (2018, Oyebode and Orji (2019, Kolajo et al. (2019, Rakhmanov (2020, Olagunju et al. (2020, Onyenwe et al. (2020, Honkanen and Müller (2021]. Most work on Nigerian languages has relied on automatically generated data, including the following:

Hausa

?) built a Twitter corpus and introduced combined Hausa and English features in a classifier.

Igbo

?) translated an English sentiment lexicon [Hu and Liu (2004] and manually added Igbo native words to create IgboSentiLex. ?) analysed Igbo emotion words using Interval Type-2 Fuzzy Logic.

Yorùbá

?) built a Yorùbá corpus from YouTube and applied a translated SentiWordNet for the sentiment analysis task. ?) created a multi-domain corpus (health, business, education, politics) and used different classic ML classifiers such as SVM to predict sentiment in text.

Pidgin

?) built a Pidgin tweet corpus and used a translated VADER English lexicon for sentiment analysis.

Table 1 summarizes the existing datasets for Nigerian languages; only two datasets are freely available, indicating that more work is needed to make indigenous datasets accessible and to stimulate research in these languages. To the best of our knowledge, ours is the first publicly available large-scale manually annotated dataset for sentiment analysis research in the following Nigerian languages: Hausa, Igbo, Yorùbá and Nigerian Pidgin (see Appendix 0 for the language description and characteristics.)

Data Collection and Cleaning

Twitter provides easy access to a large amount of domain-independent and topic-independent public opinionated user-generated data. We collected tweets using the Twitter Academic API https://developer.twitter.com/en/products/twitter-api/academic-research, which provides real-time and historical tweet data. The Twitter API supports retrieving tweets in 70 languages (including Amharic as the only African language) using language parameters. This makes it easy to extract a tweet in these languages. On the contrary, none of the languages considered in this work are supported by the API. Therefore, we consider different heuristic approaches to crawling tweets.

?) have shown that token-based filtering is a useful processing step for automatic language identification. Hence, we automatically built lists of common words (stopwords), which were verified by native speakers and used them to query the Twitter API to retrieve tweets in each language. ?) used emoticons and ?) used emojis as a distantly supervised approach to automatically classify subjective tweets as positive or negative. Using a similar approach, we used happy and sad emojis [Kralj Novak et al. (2015] in combination with stopwords to query the Twitter API to extract tweets that contain stopwords and emojis. In addition, we used the Google Language API to translate the Affin lexicon [Årup Nielsen (2011] into each of the languages (Hausa, Igbo, Yorùbá), except Pidgin. We then filtered the tweets using the translated Affin sentiment lexicon to improve the likelihood of annotating sentiment-bearing tweets [UzZaman et al. (2013].

Hashtags and Handles

We used Twitter hashtags to crawl tweets from trending topics (e.g., #Yorubaday) to collect sufficient tweets which are expected to be in the language under consideration. We also collect tweets from news handles (e.g., @bbchausa) which are expected to be factual and non-subjective. We selected the handles that tweet frequently in each language from the Indigenous Tweets http://indigenoustweets.com/ website.

One downside of this approach is that Twitter conversations with a popular Twitter handle may dominate the dataset and may introduce a bias towards certain topics. For example, a Hausa Twitter conversation that involves the handle @bbchausa and another conversation involving the handle @Rahmasadau make up 54% and 14% respectively of collected tweets associated with Hausa handles. Limiting the number of tweets per conversation mitigates this problem.

2. Language Detection and Data Cleaning

Stopwords overlap across indigenous languages in a multilingual society such as Nigeria [Caswell et al. (2020]. This results in tweets being collected in a language that differs from the query language. For example, using the stop word “nke” to crawl tweets in Igbo produces tweets in Hausa, such as “amin ya rabbi godiya nke”. To mitigate this, we collected tweets based on locations where a language is predominantly spoken, using the location, longitude, latitude and radius parameters (25 miles) to specify a circular geographic area.

We also used Google CLD3 https://github.com/google/cld3 and Natural Language API https://cloud.google.com/natural-language/docs to detect the language of the collected tweets. Pidgin is not supported by the API, so we used the stopword list to build an n-gram language detection tool to detect Pidgin. Before annotation, we cleaned the tweets. Retweets and duplicates were removed. We removed URLs and mentions, as well as trailing and redundant white spaces, converted all tweets to lowercase, and removed tweets with less than three words as they may contain insufficient information for sentiment analysis [Yang et al. (2018].

Annotation and the NaijaSenti Dataset

Our annotation guidelines focus on the classification of subjective tweets. A subjective tweet has a positive or negative emotion, opinion, or attitude [Refaee and Rieser (2014b]. We adapt a sentiment annotation guide from [Mohammad (2016] and define five classes: positive (POS), negative (NEG), neutral (NEU), mixed (MIX) and indeterminate (IND).

This occurs if a tweet implies positive sentiment, attitude and emotional state. For example, a tweet implies a positive opinion or sentiment (e.g., “I love iPhone"), positive emotional state (e.g., “we won the game last night"), expression of support (e.g., “I will vote for PDP"), thankfulness (e.g., “thank god she has not been kidnapped"), success (e.g., “I passed all my exams"), or positive attitude.

Negative (NEG) Sentiment:

This occurs if a tweet implies negative sentiment or emotion. For example, if a tweet implies negative sentiment (e.g., “ The iPhone camera is bad"), negative emotional states such as failure, anger, and disappointment.

Neutral (NEU):

This occurs if the user’s tweet does not imply any positive or negative language directly or indirectly. These are usually factual tweets, such as news.

Mixed (MIX):

This occurs if the user’s tweet implies both negativity and positivity directly or indirectly. For example, “I love an IPhone 10, but its camera is bad”.

Indeterminate (IND):

This occurs if the users’ tweet does not fall into either positive, negative, neutral, and mixed, or if the annotator can only guess the class of a tweet, especially in the case of proverbs or sarcasm without sufficient context. We additionally use this class to label tweets in a different language (not code-mixed).

2. Annotation Process

For each language, we recruited three native speakers as annotators. The Annotators are both males and females between the ages of 20 and 45. We also recruited a coordinator for each language to supervise and ensure the quality of the annotation task. Annotators and coordinators have backgrounds in either computer science or linguistics and were trained on the annotation task using the LightTag annotation tool [Perry (2021].

Data annotation is not a one-off process; it requires an agile approach with many iterations, collecting feedback from the annotators during the pilot stage, and refining the annotation guide to ensure that the annotators can achieve reasonable performance before moving to the next stage. We performed three iterations of the training and annotation practice of 100 tweets. For the first two iterations, the agreement among the annotators was poor. We asked the annotators for feedback and adapted a simplified sentiment questionnaire annotation guide [Mohammad (2016].

Tweets annotation:

The dataset was annotated in batches of 1,000 tweets by the annotators. For each batch, we adjudicated the cases in which the three annotators assigned a different label to a tweet. Annotators discuss these tweets, which allows them to address ambiguities, peculiar issues, and recommend ways to improve the annotation guidelines. We excluded these ambiguous tweets from the dataset. We iteratively update our annotation guide based on adjudication reports. Overall, the annotators annotated the following number of tweets: Hausa (35,000), Igbo (29,000), Pidgin (30,000) and Yorùbá (33,000).

Determining the gold label:

People often disagree on subjective concepts [Beddor (2019]. For example, person A, who has been using Apple products, says, “The Apple iPhone camera is better than the Samsung camera”, while person B says, “The Samsung camera is better”. This is an example of subjective disagreement in contrast to objective disagreement. Therefore, different from the simple majority vote approach [Davani et al. (2021], we introduced a new form of majority vote that involves an independent annotator who adjudicates subjective disagreement cases as follows: We determine a single gold label for sentiment analysis in accordance with prior work. Future work may alternatively leverage annotator disagreement [Fornaciari et al. (2021].

Three-way agreement: Similar to the majority vote approach, if all three annotators agree on a label, we consider the agreed sentiment class to be the gold standard.

Three-way disagreement: When all annotators disagree on a label, we discard the tweet.

Two-way partial disagreement: If two of the annotators agree on a label, and the third annotator has a partial disagreement. For example, if two annotators classify a tweet as POS (or NEG), and the other annotator classifies it as a non-contradicting class such as NEU, we consider the POS (or NEG) classification to be the gold standard.

Two-way disagreement: If two of the annotators agree on a label, and the third annotator has a total disagreement. For example, if two annotators identify a tweet as POS and another as NEG or vice versa, the majority vote is not the final class (in this case, POS). To resolve such subjective disagreement, independent annotators review the disagreement and assign a final label.

Sentiment lexicons

We created sentiment lexicons in three languages (Hausa, Igbo, and Yorùbá) based on NaijaSenti. We asked three annotators to tag words that convey negative or positive sentiment in a tweet. We used a simple majority vote. An independent annotator adjudicated cases where the annotators disagreed or only one person tagged a word as positive or negative. The distribution of the lexicon is presented in Table 6. We also created semi-automatically translated versions of the NRC emotion lexicon [Mohammad and Turney (2013] and the AFFIN sentiment lexicon [Årup Nielsen (2011] for Hausa, Igbo, and Yorùbá. We used the Google Translate API https://cloud.google.com/translate to translate the lexicon. Afterwards, professional human translators verified and corrected the translations and added missing diacritics.

3. Inter-Annotator Agreement

We used the Fleiss kappa ($\kappa$) reliability measure [Fleiss et al. (2013] to determine the inter-annotator agreement (IAA) between the three annotators. The IAA for the 5-class and adjusted 3-class agreements are shown in Table 3. The agreement between the five classes was not particularly high (e.g.,($\kappa$) = $0.35$) for Pidgin. However, according to the Fleiss classification [Fleiss et al. (2013], an agreement greater than $0.40$ is considered reasonable (moderate) and beyond chance.

We further computed the IAA ($\kappa$) (see Table 4) of each class with other classes to determine which classes the annotators find confusing or difficult and frequently disagree. Table 4 indicates that annotators generally have the lowest overall agreement in the MIXED class, which includes elements of both the positive and negative classes, and some annotators identify it as either negative or positive. This highlights the subtlety of annotating mixed sentiment on social media and is in contrast to reviews where the annotation of mixed sentiment is clearer [Potts et al. (2021]. To address this, we introduced an adjusted 3-class IAA agreement.

In the adjusted 3-class agreement, we considered only positive, negative, and neutral as valid sentiment classes. We selected only tweets that have at least two labels in the valid classes and discarded the rest. For the selected tweets, where the label between two annotators is valid and the third label is in the invalid sentiment (Indeterminate or Mixed), we changed the label to the agreed valid label. For instance, given three annotation labels of a tweet as POS, POS, MIX, the third label is changed to POS, whereas the annotation labels of POS, POS, NEU are left unchanged. Table 3 shows the final statistics of at least two agreed tweets of the various datasets after converting to the 3-class annotation, and their corresponding inter-annotator agreements (IAA) using the Fleiss’ Kappa ($\kappa$) metric.

To determine the performance of the IAA over time, Figure 1 shows the IAA in three languages over 30 batches. We hypothesised that as the annotators became more experienced with the task, their annotation quality would improve. However, the overall performance of the IAA deteriorates over time. Igbo has the lowest performance drop. This suggests that familiarity with the task does not necessarily improve IAA. Only Yorùbá annotators have some level of consistency that is not below $0.5$. Therefore, it is important to monitor the IAA as the annotation progresses and use some form of random quality check.

4. Human Evaluations

We assess human performance by re-annotating 200 random sample tweets by three different annotators [Warstadt et al. (2019, Nangia and Bowman (2019]. We take the majority vote as the final class. Human performance offers us an idea of the machine’s upper bound performance and the reproducibility of the first three annotators [Warstadt et al. (2019]. Table 5 shows the micro-F1 and Matthew’s correlation coefficient (MCC) [Jurman et al. (2012]. The human performance result validates the reliability of the corpus and is in line with previous literature [Rosenthal et al. (2017].

5. NaijaSenti Statistics

Table 3 shows the summary of our dataset with 5-class and adjusted 3-class. Other key statistical information, such as number of tokens, type of words, and type-token ration (TTR), which measure the lexical richness of a text are presented in Table 6. We also show the number of monolingual and code-mixed tweets in each dataset. The percentage of code-mixed tweets highlights the highly multilingual setting in Nigeria. Code-mixing is more prevalent in Igbo (43%) than in Hausa (23%) and Yorùbá (19%). Code-mixing between English and a native language is more common than between native languages, but it can also occur between more than two native languages.

Hausa does not have diacritics and therefore has an insignificant number of indeterminate cases (only 8), unlike Yorùbá and Igbo where the absence of diacritics may render a tweet incomprehensible and therefore lead to labelling it as indeterminate. Pidgin has the highest number of indeterminate cases. This is because some tweets appear to be pidgin, but they are Nigerian English and, therefore, we consider them indeterminate.

Tone in Yorùbá helps to give meaning to words in context, especially words that have the same orthographic representation. For instance, the sentence “Awon omo fo abo” does not have a meaning without diacritics, and the annotators classify it as indeterminate (IND). However, the same sentence with diacritics can have two opposite meanings: Àwọn ọmọ́ fọ abọ́ (The children washed the dishes) has a positive meaning, and Àwọn ọmọ́ fọ́ abọ́ (The children broke the dishes) is negative.

Similarly, tonality is heavily used in Igbo. Many Twitter users do not write Igbo with diacritics. One reason is the lack of an Igbo keyboard that accepts and shows diacritics. Even if such a keyboard exists, it is not used by many. Although it may be fairly easy to understand the sentiment of Igbo tweets in context on Twitter—either due to the presence of emojis or the context of the surrounding discourse, it is quite difficult and sometimes ambiguous to correctly annotate the tweets when they stand on their own. The example below highlights the impact of tone and punctuation marks on the same Igbo tweets but with different sentiment:

ò nwèkwàrà mgbe i naenwe sense ? – Will you ever be able to talk sensibly? – You’re a fool.

ò nwèkwàrà mgbe i naenwe sense – Sometimes you act with great maturity – I’m impressed.

Yes/No questions in Igbo are realized by a low tone on the subject pronoun, as in the first sentence above. So, with no tone and lacking punctuation, the author’s intended meaning is difficult to determine.

To create a benchmark dataset, we use only three sentiment classes: negative, neutral, and positive. We split tweets in each class by 70%, 10% and 20% ratios for the TRAIN, DEV and TEST splits as shown in Table 7.

Experimental Setup

Sentiment classification is a well-studied problem in NLP and many machine learning models have been developed for this task. State-of-the-art approaches on English data use pre-trained language models (PLMs) such as BERT [Devlin et al. (2019] and RoBERTa [Liu et al. (2019], which provide superior performance. Multilingual variants of PLMs provide an opportunity to quickly adapt to various languages, including languages not seen during training [Pfeiffer et al. (2020]. We compare several standard multilingual PLMs on the four languages. We fine-tune each model on the data of each language separately using the HuggingFace Transformer [Wolf et al. (2020]. Appendix B provides the details of the hyper-parameters used for training.

is a multilingual variant of BERT pre-trained on 104 languages, including one Nigerian language—Yorùbá. mBERT was pre-trained using masked language modeling (MLM) and next-sentence prediction task. We fine-tune the mBERT-base-cased model with 172M model parameters by adding a linear classification layer on top of the pre-trained transformer model.

XLM-R

Similar to mBERT, XLM-R [Conneau et al. (2020] is a multilingual variant of RoBERTa pre-trained on 100 languages, including Hausa as the only Nigerian language. Unlike mBERT, XLM-R only uses MLM during pre-training. We use XLR-base with 270M model parameters for fine-tuning on the NaijaSenti corpus.

RemBERT

scales up mBERT to a larger model size (559M) and decouples embeddings, which enables a larger output embedding size during pre-training, resulting in stronger pre-training and downstream performance [Chung et al. (2021]. RemBERT covers the three major Nigerian languages, except for Nigerian-Pidgin.

AfriBERTa

trains a RoBERTa-style model on 11 African languages [Ogueji et al. (2021] including all four Nigerian languages in NaijaSenti. The model was trained on less than 1GB of data (since most African languages are low-resourced). We use AfriBERTa-large with 126M parameters. AfriBERTa has been shown to perform competitively on an African NER dataset [Adelani et al. (2021] despite its small model size and limited pre-training data.

mDeBERTaV3

Unlike the other four models pre-trained on the MLM task, mDeBERTaV3 [He et al. (2021] makes use of ELECTRA-style [Clark et al. (2020] pre-training where a discriminator is trained to detect replaced tokens instead of predicting masked tokens. mDEBERTaV3 does not support any of the Nigerian languages. We use the mDEBERTaV3-base model with 276M model parameters similar to XLM-R-base.

2. Language Adaptive Fine-tuning

Many multilingual PLMs support only a few African languages. For example, mBERT only supports three African languages (Malagasy, Swahili, and Yorùbá). Language adaptive fine-tuning (LAFT) is an effective method of adapting PLMs to a new language by fine-tuning PLMs MLM on unlabeled texts in the new language [Pfeiffer et al. (2020]. The approach is similar to domain-adaptive fine-tuning [Howard and Ruder (2018, Gururangan et al. (2020]. LAFT has been shown to be very effective in improving NER performance in several African languages [Alabi et al. (2020, Muller et al. (2021, Adelani et al. (2021]. To further improve the LAFT performance, we perform vocabulary augmentation using 99 most frequent wordpieces inspired by [Chau et al. (2020, Pfeiffer et al. (2021] before further pre-training the PLM. We experimented on two collections of monolingual data: (1) Twitter domain (often very small; less than 50K tweets for Igbo and Yorùbá, and less than 600K tweets for Hausa and Nigerian-Pidgin), and (2) General domain (trained on mostly Common Crawl corpora, religious texts, and online news); for the latter, we use the checkpoints released by [Adelani et al. (2021].

3. Multi-task Sentiment Classification

In addition to fine-tuning separate models for each language, we trained a joint multi-task sentiment classification model on the four Nigerian languages by aggregating their training sets. The major advantage of this is that having a single model that can classify the sentiment in tweets in all major Nigerian languages facilitates deployment for practical applications. Knowledge from related languages may also be beneficial during transfer. This setting is possible because we are using multilingual PLMs that support multiple languages.

4. Cross-Lingual Transfer

Lastly, we evaluate the zero-shot performance of a sentiment classifier trained on English tweets from SemEval-2017 Task 4 [Rosenthal et al. (2017] on each of the four Nigerian languages. We also assess how many tweets from each of the Nigerian languages are needed to reach the zero-shot performance of a model transferred from English and to produce an accuracy score that is better than a majority classifier.

Experimental Results

Table 8 shows the performance of several sentiment classification models for three-way sentiment classification on four Nigerian languages. As the corpora do not have a balanced number of samples for each label, we also computed a majority classifier based on the dominant label in the corpus. hau, ibo and yor have more neutral tweets while pcm has more positive tweets. The performance of the majority classifier using the weighted F1-score is around $16-45\%$ for all languages and $33-56\%$ using Micro F1-score. On the other hand, PLMs have a minimum F1-score of $70\%$, demonstrating their usefulness for sentiment analysis.

are quite similar in most cases with about a $1-3\%$ performance difference. The performance may depend on the language being seen during pre-training. mBERT has a slightly better performance ($+0.7\%$) for yor than XLM-R likely because yor was seen during pre-training. Similarly, XLM-R performs better for hau. RemBERT achieves slightly better performance than mBERT and XLM-R-base, demonstrating that a model with more capacity can improve performance. Surprisingly, we found mDeBERTaV3 that has only seen hau gives better results ($77.8\%$) than other models except for AfriBERTa that has been exclusively trained on African languages. mDeBERTaV3 makes use of replaced token detection (RTD), which has been shown to give superior performance for English [Clark et al. (2020]. Overall, we found AfriBERTa to be the best baseline model for all languages because the model is more African language-centric. The main advantage of AfriBERTa is its smaller model size, which makes it easier to deploy especially on the African continent where most research labs cannot afford powerful GPUs.

Language adaptive fine-tuning (LAFT)

has been shown to improve over the baseline with additional pre-training on monolingual data in the domain or language. Table 8 shows some improvement over mBERT and XLM-R when we apply LAFT on the general domain, on average $2-3\%$ on hau, and yor, and $<1\%$ on ibo. For pcm, we only identified an improvement for mBERT ($+1.2\%$). Interestingly, applying LAFT on the Twitter domain did not improve performance. The main reason for this is the small size of the Twitter data. For example, hau was further pre-trained on CC100 [Conneau et al. (2020] corpus with over 318MB and 3 million sentences for the general domain, but for Twitter, we only have around 512K tweets (32MB), which are often short. In general, we found AfriBERTa to be competitive or better than LAFT for the Nigerian languages except for pcm.

Multi-task sentiment classification

We trained a single model on all languages with minimal drop in performance. In this setting, we only trained on the best two multilingual PLMs: AfriBERTa and mDeBERTaV3. We observe only a slight drop in performance with mDeBERTa ($-0.3\%$) while the AfriBERTa performance is the same. This indicates that we could easily deploy a single sentiment classification model for the four major Nigerian languages, instead of multiple monolingual models.

2. Zero-shot Cross-Lingual Transfer

Table 9shows the results of zero-shot transfer from English SemEval 2017 Task 4 tweets to the four Nigerian languages. The English SemEval corpus consists of 11,763 tweets in the training set. pcm has the best zero-shot performance across all models because of its linguistic similarity to English, its lexifier language. Similarly, we found an impressive zero-shot performance for hau with at least $50.0\%$ F1-score when we train on AfriBERTa, mDeBERTaV3 and RemBERT. For ibo, the performance is over $45.4\%$ on the three best PLMs while the zero-shot evaluation for yor is slightly lower ($36-43\%$). AfriBERTa gave the best overall result in the zero-shot transfer, and it is significantly better than the majority classifier (weighted average) for all languages: hau, ibo, pcm, and yor are better by $41.8\%$, $20.8\%$, $4\%$, and $19.1\%$ respectively.

3. Sample Efficiency in Transfer

Figure 2shows the result of training a sentiment classification model with different numbers of samples (10, 100, 500, 1K, 2.5K, 5K, 10K, and 15K). We fine-tune AfriBERTa on hau, ibo, and yor datasets of different sizes. We observe an F1 score of $38-40\%$ with only 10 examples, which already exceeds the majority voting performance in Table 8. Surprisingly, with only 100 sentences, we exceed the zero-shot transfer performance from English language, and with at least 1000 sentences, we already reach a decent performance of $70\%$ F1. This result shows that we can leverage a multitask sentiment classification model trained on Nigerian languages to quickly adapt to other African languages with as few as 100 or 1000 annotated samples. Overall, we identify headroom for model improvement particularly in the zero-shot and few-shot cross-lingual transfer settings.

Conclusions and Future Work

In this paper, we present NaijaSenti—the first publicly available large-scale and manually annotated Twitter sentiment dataset for the four main Nigerian languages (Hausa, Igbo, Nigerian-pidgin, and Yorùbá). We propose methods to enable the collection, filtering, and annotation of such low-resource language data. Additionally, we introduce a manually annotated sentiment lexicon in three languages (Hausa, Igbo, and Yorùbá). We present benchmark experiments on Twitter sentiment dataset using state-of-the-art pre-trained language models and transfer learning. The results indicate that language-specific models and language-adaptive fine-tuning perform the best on average. NaijaSenti has the potential to spark interest in sentiment analysis and other downstream NLP tasks in the languages involved. As future work, we plan to create benchmark experiments with our sentiment lexicon, and extend our dataset (NaijaSenti) to include other African languages (AfriSenti).

Acknowledgements

We thank Daan van Esch for feedback on a draft of this article. This work was carried out with support from Lacuna Fund, an initiative co-founded by The Rockefeller Foundation, Google.org, and Canada’s International Development Research Centre. The views expressed herein do not necessarily represent those of Lacuna Fund, its Steering Committee, its funders, or Meridian Institute. This work is also partially funded by the National Funds through the Portuguese funding agency, FCT - Fundação para a Ciência e a Tecnologia, within project LA/P/0063/2020. We thank Tal Perry for providing the LightTag annotation tool. Finally, David Adelani acknowledges the EU-funded Horizon 2020 projects: COMPRISE (http://www.compriseh2020.eu/) under grant agreement No. 3081705 and ROXANNE under grant number 833635.

Bibliographical References

References

Appendix

Appendix A Overview of the Languages

With over 522 native languages, Nigeria is the most multilingual country in Africa and the third most multilingual country in the world. https://www.ethnologue.com/guides/countries-most-languages Although linguistically very diverse, the majority of the population speaks either Hausa, Igbo, Yorùbá, or Nigerian-Pidgin. Therefore, our work focuses on these three indigenous Nigerian languages (Hausa, Yorùbá, and Igbo) and Nigerian-Pidgin.

Hausa is a Chadic (Afroasiatic) language that is spoken in 3 broad dialects https://www.mustgo.com/worldlanguages.com/hausa: Eastern, Western and Northern [Jaggar (2001]. Hausa is spoken by approximately 77 million people around the world, mostly in West Africa [Eberhard et al. (2022]. The language is written in two different scripts: Ajami and the more common Boko script. The Boko script uses the Latin characters without p, q, v and x as well as the following additional letters: consonants (\mb, \md, \mk, \my, kw, \mkw, gw, ky, \mky, gy, sh, ts) and vowels (the long a, i, o, u, e and two additional diphthongs ai and au). Hausa is a tonal language with two tones: low and high, represented by the grave (e.g. “è”) and acute (e.g. “é”) accents respectively, which are usually not marked in everyday writing. The sentence structure follows the Subject-Verb-Object (SVO) syntax.

Igbo (ibo):

Igbo belongs to the Benue-Congo group of the Niger-Congo language family and is spoken by over 27 million people [Eberhard et al. (2022]. It is the primary language of the Igbo people, an ethnic group of southeastern Nigeria, but is also spoken in some parts of Equatorial Guinea and Cameroon. There are approximately 30 Igbo dialects, some of which are not mutually intelligible. Igbo is written using the Ọnwụ orthography [Ohiri-Aniche (2007]. Ọnwụ consists of 28 consonants and 8 vowels. Standard Igbo consists of eight vowels, and thirty consonants. Igbo is a tonal language. Tone varies by dialect but in most dialects there are three main ones: high, low and downstep. A typical Igbo sentence follows subject-verb-object (SVO) order.

Yorùbá (yor):

Yorùbá belongs to the Yoruboid sub-branch of the Volta-Niger branch of the Niger-Congo language family. The language is spoken in the south-western parts of Nigeria stretching into some parts of Togo and Benin. The Yorùbá alphabet is based on the Latin script consisting of 18 consonants, 7 oral vowels, 5 nasal vowels and syllabic nasal consonants with additional characters likeẹ, ọ , ṣ, gb. The language uses tones: high, mid, and low tones. The Yoruba language is spoken by approximately 46 million people [Eberhard et al. (2022] , mostly in Nigeria, and Republic of Benin.

Nigerian-Pidgin (pcm):

Nigerian-Pidgin, also known as Naija, is an English-based creole language spoken as a lingua franca across regions in Nigeria. It is rooted in the Krio of the English-based creole language family with an estimate of about 40M and 80M first and second language speakers respectively. Nigerian Pidgin uses the Latin script but has no standardised orthographic representation. The phonology of the language displays no suprasegmental features such as tone as in other African languages and it makes heavy usage of loan words from African and European languages.

Appendix B Model Hyper-parameters for Reproducibility

For the pre-trained models, we fine-tune the models using HuggingFace transformer tool [Wolf et al. (2020] with the batch size of $32$, maximum sequence length of $128$, number of epochs of $20$, and default learning rate ($5e-5$) for all models except for XLM-R and RemBERT where we set learning rate to be $2e-5$ to ensure model convergence. All the experiments were performed Nvidia V100 and RTX 2080 GPUs.