Measuring Emotions in the COVID-19 Real World Worry Dataset
Bennett Kleinberg, Isabelle van der Vegt, Maximilian Mozes
Introduction
The outbreak of the SARS-CoV-2 virus in late 2019 and subsequent evolution of the COVID-19 disease has affected the world on an enormous scale. While hospitals are at the forefront of trying to mitigate the life-threatening consequences of the disease, practically all societal levels are dealing directly or indirectly with an unprecedented situation. Most countries are — at the time of writing this paper — in various stages of a lockdown. Schools and universities are closed or operate online-only, and merely essential shops are kept open.
At the same time, lockdown measures such as social distancing (e.g., keeping a distance of at least 1.5 meters from one another and only socializing with two people at most) might have a direct impact on people’s mental health. With an uncertain outlook on the development of the COVID-19 situation and its preventative measures, it is of vital importance to understand how governments, NGOs, and social organizations can help those who are most affected by the situation. That implies, at the first stage, understanding the emotions, worries, and concerns that people have and possible coping strategies. Since a majority of online communication is recorded in the form of text data, measuring the emotions around COVID-19 will be a central part of understanding and addressing the impacts of the COVID-19 situation on people. This is where computational linguistics can play a crucial role.
In this paper, we present and make publicly available a high quality, ground truth text dataset of emotional responses to COVID-19. We report initial findings on linguistic correlates of emotions, topic models, and prediction experiments.
Tasks like emotion detection Seyeditabari et al. (2018) and sentiment analysis Liu (2015) typically rely on labeled data in one of two forms. Either a corpus is annotated on a document-level, where individual documents are judged according to a predefined set of emotions Strapparava and Mihalcea (2007); Preoţiuc-Pietro et al. (2016) or individual -grams sourced from a dictionary are categorised or scored with respect to their emotional value Bradley et al. (1999); Strapparava and Valitutti (2004). These annotations are done (semi) automatically (e.g., exploiting hashtags such as #happy) Mohammad and Kiritchenko (2015); Abdul-Mageed and Ungar (2017) or manually through third persons Mohammad and Turney (2010). While these approaches are common practice and have accelerated the progress that was made in the field, they are limited in that they propagate a pseudo ground truth. This is problematic because, as we argue, the core aim of emotion detection is to make an inference about the author’s emotional state. The text as the product of an emotional state then functions as a proxy for the latter. For example, rather than wanting to know whether a Tweet is written in a pessimistic tone, we are interested in learning whether the author of the text actually felt pessimistic.
The limitation inherent to third-person annotation, then, is that they might not be adequate measurements of the emotional state of interest. The solution, albeit a costly one, lies in ground truth datasets. Whereas real ground truth would require - in its strictest sense - a random assignment of people to experimental conditions (e.g., one group that is given a positive product experience, and another group with a negative experience), variations that rely on self-reported emotions can also mitigate the problem. A dataset that relies on self-reports is the International Survey on Emotion Antecedents and Reactions (ISEAR)https://www.unige.ch/cisa/research/materials-and-online-research/research-material/, which asked participants to recall from memory situations that evoked a set of emotions. The COVID-19 situation is unique and calls for novel datasets that capture people’s affective responses to it while it is happening.
2 Current COVID-19 datasets
Several datasets mapping how the public responds to the pandemic have been made available. For example, tweets relating to the Coronavirus have been collected since March 11, 2020, yielding about 4.4 million tweets a day Banda et al. (2020). Tweets were collected through the Twitter stream API, using keywords such as ’coronavirus’ and ’COVID-19’. Another Twitter dataset of Coronavirus tweets has been collected since January 22, 2020, in several languages, including English, Spanish, and Indonesian Chen et al. (2020). Further efforts include the ongoing Pandemic Projecthttps://utpsyc.org/covid19/index.html which has people write about the effect of the coronavirus outbreak on their everyday lives.
3 The COVID-19 Real World Worry Dataset
This paper reports initial findings for the Real World Worry Dataset (RWWD) that captured the emotional responses of UK residents to COVID-19 at a point in time where the impact of the COVID-19 situation affected the lives of all individuals in the UK. The data were collected on the 6th and 7th of April 2020, a time at which the UK was under “lockdown” news (2020), and death tolls were increasing. On April 6, 5,373 people in the UK had died of the virus, and 51,608 tested positive Walker (now). On the day before data collection, the Queen addressed the nation via a television broadcast Guardian (2020). Furthermore, it was also announced that Prime Minister Boris Johnson was admitted to intensive care in a hospital for COVID-19 symptoms Lyons (2020).
The RWWD is a ground truth dataset that used a direct survey method and obtained written accounts of people alongside data of their felt emotions while writing. As such, the dataset does not rely on third-person annotation but can resort to direct self-reported emotions. We present two versions of RWWD, each consisting of 2,500 English texts representing the participants’ genuine emotional responses to Corona situation in the UK: the Long RWWD consists of texts that were open-ended in length and asked the participants to express their feelings as they wish. The Short RWWD asked the same people also to express their feelings in Tweet-sized texts. The latter was chosen to facilitate the use of this dataset for Twitter data research.
The dataset is publicly available.Data: https://github.com/ben-aaron188/covid19worry and https://osf.io/awy7r/.
Data
We collected the data of 2500 participants (94.46% native English speakers) via the crowdsourcing platform Prolifichttps://www.prolific.co/. Every participant provided consent in line with the local IRB. The sample requirements were that the participants were resident in the UK and a Twitter user. In the data collection task, all participants were asked to indicate how they felt about the current COVID-19 situation using 9-point scales (1 not at all, 5 moderately, 9 very much). Specifically, each participant rated how worried they were about the Corona/COVID-19 situation and how much anger, anxiety, desire, disgust, fear, happiness, relaxation, and sadness Harmon-Jones et al. (2016) they felt about their situation at this moment. They also had to choose which of the eight emotions (except worry) best represented their feeling at this moment.
All participants were then asked to write two texts. First, we instructed them to “write in a few sentences how you feel about the Corona situation at this very moment. This text should express your feelings at this moment” (min. 500 characters). The second part asked them to express their feelings in Tweet form (max. 240 characters) with otherwise identical instructions. Finally, the participants indicated on a 9-point scale how well they felt they could express their feelings (in general/in the long text/in the Tweet-length text) and how often they used Twitter (from 1never, 5every month, 9every day) and whether English was their native language. The overall corpus size of the dataset was 2500 long texts (320,372 tokens) and 2500 short texts (69,171 tokens). In long and short texts, only 6 and 17 emoticons (e.g. “:(“, “3”) were found, respectively. Because of the low frequency of emoticons, these were not focused on in our analysis.
Below are two excerpts from the dataset: Long text: I am 6 months pregnant, so I feel worried about the impact that getting the virus would have on me and the baby. My husband also has asthma so that is a concern too. I am worried about the impact that the lockdown will have on my ability to access the healthcare I will need when having the baby, and also about the exposure to the virus […] There is just so much uncertainty about the future and what the coming weeks and months will hold for me and the people I care about. Tweet-sized text: Proud of our NHS and keyworkers who are working on the frontline at the moment. I’m optimistic about the future, IF EVERYONE FOLLOWS THE RULES. We need to unite as a country, by social distancing and stay in.
2 Descriptive statistics
We excluded nine participants who padded the long text with punctuation or letter repetitions. The dominant feelings of participants were anxiety/worry, sadness, and fear (see Table 1)For correlations among the emotions, see the online supplement. For all emotions, the participants’ self-rating ranged across the whole spectrum (from “not at all” to “very much”). The final sample consisted to 65.15% of femalesFor an analysis of gender differences using this dataset, see van der Vegt and Kleinberg (2020). with an overall mean age of 33.84 years ().
The participants’ self-reported ability to express their feelings, in general, was (). When specified for both types of texts separately, we find that the ability to express themselves in the long text (, ) was higher than that for short texts (, ), Bayes factor .
The participants reported to use Twitter almost weekly (, ), tweeted themselves rarely to once per month (, ), and actively participated in conversations in a similar frequency (, ). Our participants were thus familiar with Twitter as a platform but not overly active in tweeting themselves.
Findings and experiments
We correlated the self-reported emotions to matching categories of the LIWC2015 lexicon Pennebaker et al. (2015). The overall matching rate was high (92.36% and 90.11% for short and long texts, respectively). Across all correlations, we see that the extent to which the linguistic variables explain variance in the emotion values (indicated by the ) is larger in long texts than in Tweet-sized short texts (see Table 2). There are significant positive correlations for all affective LIWC variables with their corresponding self-reported emotions (i.e., higher LIWC scores accompanied higher emotion scores, and vice versa). These correlations imply that the linguistic variables explain up to 10% and 3% of the variance in the emotion scores for long and short texts, respectively.
The LIWC also contains categories intended to capture areas that concern people (not necessarily in a negative sense), which we correlated to the self-reported worry score. Positive (negative) correlations would suggest that the higher (lower) the worry score of the participants, the larger their score on the respective LIWC category. We found no correlation between the categories “work”, “money” and “death” suggesting that the worry people reported was not associated with these categories. Significant positive correlations emerged for long texts for “family” and “friend”: the more people were worried, the more they spoke about family and — to a lesser degree — friends.
2 Topic models of people’s worries
We constructed topic models for both the long and short texts separately using the stm package in R Roberts et al. (2014a). The text data were lowercased, punctuation, stopwords and numbers were removed, and all words were stemmed. For the long texts, we chose a topic model with 20 topics as determined by semantic coherence and exclusivity values for the model Mimno et al. (2011); Roberts et al. (2014b, a). Table 3 shows the five most prevalent topics with ten associated frequent terms for each topic (see online supplement for all 20 topics). The most prevalent topic seems to relate to following the rules related to the lockdown. In contrast, the second most prevalent topic appears to relate to worries about employment and the economy. For the Tweet-sized texts, we selected a model with 15 topics. The most common topic bears a resemblance to the government slogan “Stay at home, protect the NHS, save lives.” The second most prevalent topic seems to relate to calls for others to adhere to social distancing rules.
3 Predicting emotions about COVID-19
It is worth noting that the current literature on automatic emotion detection mainly casts this problem as a classification task, where words or documents are classified into emotional categories Buechel and Hahn (2016); Demszky et al. (2020). Our fine-grained annotations allow for estimating emotional values on a continuous scale. Previous works on emotion regression utilise supervised models such as linear regression for this task Preoţiuc-Pietro et al. (2016), and more recent efforts employ neural network-based methods Wang et al. (2016); Zhu et al. (2019). However, the latter typically require larger amounts of annotated data, and are hence less applicable to our collected dataset.
We, therefore, use linear regression models to predict the reported emotional values (i.e., anxiety, fear, sadness, worry) based on text properties. Specifically, we applied regularised ridge regression modelsWe used the scikit-learn python library Pedregosa et al. (2011). using TFIDF and part-of-speech (POS) features extracted from long and short texts separately. TFIDF features were computed based on the 1000 most frequent words in the vocabularies of each corpus; POS features were extracted using a predefined scheme of 53 POS tags in spaCyhttps://spacy.io.
We process the resulting feature representations using principal component analysis and assess the performances using the mean absolute error (MAE) and the coefficient of determination . Each experiment is conducted using five-fold cross-validation, and the arithmetic means of all five folds are reported as the final performance results.
Table 4 shows the performance results in both long and short texts. We observe MAEs ranging between 1.26 (worry with TFIDF) and 1.88 (sadness with POS) for the long texts, and between 1.37 (worry with POS) and 1.91 (sadness with POS) for the short texts. We furthermore observe that the models perform best in predicting the worry scores for both long and short texts. The models explain up to 16% of the variance for the emotional response variables on the long texts, but only up to 1% on Tweet-sized texts.
Discussion
This paper introduced a ground truth dataset of emotional responses in the UK to the Corona pandemic. We reported initial findings on the linguistic correlates of emotional states, used topic modeling to understand what people in the UK are concerned about, and ran prediction experiments to infer emotional states from text using machine learning. These analyses provided several core findings: (1) Some emotional states correlated with word lists made to measure these constructs, (2) longer texts were more useful to identify patterns in language that relate to emotions than shorter texts, (3) Tweet-sized texts served as a means to call for solidarity during lockdown measures while longer texts gave insights to people’s worries, and (4) preliminary regression experiments indicate that we can infer from the texts the emotional responses with an absolute error of 1.26 on a 9-point scale (14%).
Emotional reactions to the Coronavirus were obtained through self-reported scores. When we used psycholinguistic word lists that measure these emotions, we found weak positive correlations. The lexicon-approach was best at measuring anger, anxiety, and worry and did so better for longer texts than for Tweet-sized texts. That difference is not surprising given that the LIWC was not constructed for micro-blogging and very short documents. In behavioral and cognitive research, small effects (here: a maximum of 10.63% of explained variance) are the rule rather than the exception Gelman (2017); Yarkoni and Westfall (2017). It is essential, however, to interpret them as such: if 10% of the variance in the anxiety score are explained through a linguistic measurement, 90% are not. An explanation for the imperfect correlations - aside from random measurement error - might lie in the inadequate expression of someone’s felt emotion in the form of written text. The latter is partly corroborated by even smaller effects for shorter texts, which may have been too short to allow for the expression of one’s emotion.
It is also important to look at the overlap in emotions. Correlational follow-up analysis (see online supplement) among the self-reported emotions showed high correlations of worry with fear () and anxiety () suggesting that these are not clearly separate constructs in our dataset. Other high correlations were evident between anger and disgust (), fear and anxiety (), and happiness and relaxation (). Although the chosen emotions (with our addition of ”worry”) were adopted from previous work Harmon-Jones et al. (2016), it merits attention in future work to disentangle the emotions and assess, for example, common ngrams per cluster of emotions (e.g. as in Demszky et al., 2020).
2 Topics of people’s worries
Prevalent topics in our corpus showed that people worry about their jobs and the economy, as well as their friends and family - the latter of which is also corroborated by the LIWC analysis. For example, people discussed the potential impact of the situation on their family, as well as their children missing school. Participants also discussed the lockdown and social distancing measures. In the Tweet-sized texts, in particular, people encouraged others to stay at home and adhere to lockdown rules in order to slow the spread of the virus, save lives and/or protect the NHS. Thus, people used the shorter texts as a means to call for solidarity, while longer texts offered insights into their actual worries (for recent work on gender differences, see van der Vegt and Kleinberg, 2020).
While there are various ways to select the ideal number of topics, we have relied on assessing the semantic coherence of topics and exclusivity of topic words. Since there does not seem to be a consensus on the best practice for selecting topic numbers, we encourage others to examine different approaches or models with varying numbers of topics.
3 Predicting emotional responses
4 Suggestions for future research
The current analysis leaves several research questions untouched. First, to mitigate the limitations of lexicon-approaches, future work on inferring emotions around COVID-19 could expand on the prediction approach (e.g., using different feature sets and models). Carefully validated models could help to provide the basis for large scale, real-time measurements of emotional responses. Of particular importance is a solution to the problem hinted at in the current paper: the shorter, Tweet-sized texts contained much less information, had a different function, and were less suitable for predictive modeling. However, it must be noted that the experimental setup of this study did not fully mimic a ‘natural’ Twitter experience. Whether the results are generalisable to actual Twitter data is an important empirical question for follow-up work. Nevertheless, with much of today’s stream of text data coming in the form of (very) short messages, it is important to understand the limitations of using that kind of data and worthwhile examining how we can better make inferences from that information.
Second, with a lot of research attention paid to readily available Twitter data, we hope that future studies also focus on non-Twitter data to capture emotional responses of those who are underrepresented (or non-represented) on social media but are at heightened risk.
Third, future research may focus on manually annotating topics to more precisely map out what people worry about with regards to COVID-19. Several raters could assess frequent terms for each topic, then assign a label. Then through discussion or majority votes, final topic labels can be assigned to obtain a model of COVID-19 real-world worries.
Fourth, future efforts may aim for sampling over a longer period to capture how emotional responses develop over time. Ideally, using high-frequency sampling (e.g., daily for several months), future work could account for the large number of events that may affect emotions.
Lastly, it is worthwhile to utilise other approaches to measuring psychological constructs in text. Although the rate of out-of-vocabulary terms for the LIWC in our data was low, other dictionaries may be able to capture other relevant constructs. For instance, the tool Empath Fast et al. (2016) could help measure emotions not available in the LIWC (e.g., nervousness and optimism). We hope that future work will use the current dataset (and extensions thereof) to go further so we can better understand emotional responses in the real world.
Conclusions
This paper introduced the first ground truth dataset of emotional responses to COVID-19 in text form. Our findings highlight the potential of inferring concerns and worries from text data but also show some of the pitfalls, in particular, when using concise texts as data. We encourage the research community to use the dataset so we can better understand the impact of the pandemic on people’s lives.
Acknowledgments
This research was supported by the Dawes Centre for Future Crime at UCL.