Do LLMs Understand Social Knowledge? Evaluating the Sociability of Large Language Models with SocKET Benchmark

Introduction

Interpersonal communication is more than just what is said. Understanding communication requires reasoning not only about the content of a message but also the social implications drawn from that message (Halliday, 1995). As NLP systems, particularly Large Language Models (LLMs), are increasingly used in interpersonal settings, these models’ abilities to understand social knowledge become critical. However, despite the recognized need for social knowledge (Hovy and Yang, 2021), the NLP field has limited abilities to test it. Here, we introduce SocKET, a new benchmark for evaluating social knowledge.

Evaluating NLP systems has remained a key component for benchmarking the field’s progress. Indeed, the rapid replacement of traditional models by LLM-based approaches was strongly motivated by substantial gains by LLMs on a variety of comprehensive Natural Language Understanding (NLU) benchmarks like SuperGLUE (Wang et al., 2019) and Natural Questions (Kwiatkowski et al., 2019). However, despite the fundamental social aspect of language, comprehensive benchmarks of social language remain absent. Instead, existing computational studies of social language have built individual datasets and models for specific types of information like empathy (Sharma et al., 2020), politeness (Danescu-Niculescu-Mizil et al., 2013), and humor (Van Hee et al., 2018). While beneficial, these semantic-level tasks omit broader social and narrative-level information (Li et al., 2021) and present only a narrow view of model performance.

We introduce SocKET (Social Knowledge Evaluation Tests), a theory-grounded, systematic collection of 58 social language tasks.The choice of the term “social knowledge” in framing stems from its use for a broad category in psychology (e.g., Turiel, 1983; Adolphs, 2009) that matched the capabilities we are interested in. SocKET covers five categories of social information: sentiment & emotion, trustworthiness, humor & sarcasm, offensiveness, and social factors, each motivated by specific theories. To examine models’ generalizability, SocKET includes four task formats: classification, regression, pairwise comparison, and span identification. This construction aims at assessing not only NLP models’ performances on individual tasks but their ability to perform multiple task types and to productively benefit from related tasks and task categories during learning.

Our study offers the following three contributions to the research community. (1) We motivate a theoretically-grounded organization of social tasks (§2) and subsequently introduce a new easy-to-use benchmark, SocKET, that systematically organizes 58 tasks (§3). (2) We benchmark multiple current LLM approaches to multitask NLU via standard supervised training and zero-shot LLMs (§4). Across all tests, our results show that baseline LLMs perform moderately, at best, but offer promising signs of being able to leverage task correlations. (3) We test the abilities of models to make use of cross-task transfer (§5) showing multi-task training on strongly correlated tasks can maintain or even improve performance in specific tasks, but doing so on weakly correlated tasks can hurt the overall performance of LLMs (§6). We release our framework code and prepackaged datasets at https://github.com/minjechoi/SOCKET and https://huggingface.co/datasets/Blablablab/SOCKET.

Social Information in Natural Language Processing

Language is inherently social, as meaning is constructed through social interactions (Wittgenstein, 1953). A substantial body of research in linguistic theory and communication studies have examined how social knowledge is communicated via language understanding. Theories of language grounded in interaction and communication systems such as Systemic Functional Linguistics (SFL) by Halliday et al. (1989) assert that the function and appropriacy of language in a given context is the key to our understanding of language and its use (Eggins, 2004; Allan, 2007; Halliday et al., 1989; Halliday, 2004). We use these insights to probe linguistic models for their ability to capture social information, which we define as information conveyed through text about broader metatextual function and contextual appropriacy of the utterances in conversation.

NLP Studies on Social Information Numerous studies have contributed to the development of datasets and models aimed toward identifying nuanced social information in language across diverse contexts. Computational linguists have modeled multiple forms of social information in language like sentiment (Buechel and Hahn, 2017), politeness (Fu et al., 2020), humor (Meaney et al., 2021), offensiveness (ElSherief et al., 2021), and intimacy (Pei and Jurgens, 2020), often achieving state-of-the-art results close to human performance in their respective settings. Studies such as Park et al. (2021) have also leveraged explicitly-given norms to train models to be more accurate in context-specific situations.

However, these plausible results may be achievable solely by focusing on the statistical and syntactical instead of the social aspects of language. Whether to make advances in language understanding in research or to ensure reliability and safety in deployment, it is of vital importance to study whether models are truly capable of gaining a generalizable understanding of social factors before employing them for tasks that require such knowledge (Hovy and Yang, 2021). The necessity for such understanding is exemplified by studies showing that, when measuring the same concept, the performance of a model can vary greatly when tested on a different dataset due to factors such as changes in dialect, speaker demographics, and dataset domain (Miller et al., 2020; Blodgett et al., 2016; Wang et al., 2022a).

Despite this importance, efforts towards aggregating and synthesizing various datasets into themes have been less practiced. One notable exception is the work of Kang and Hovy (2021), where the authors combine existing datasets on different linguistic styles to introduce a benchmark that enables them to study cross-style language understanding. Similarly, we present a benchmark curated from over fifty different tasks on different aspects of social information, which we group into five distinctive categories.

Examining the social knowledge of LLMs LLMs are ubiquitous in NLP and their success is attributed to the ability to capture language characteristics from the immense amount of text seen in pre-training and to effectively apply this information on downstream tasks, achieving state-of-the-art performances in many language understanding tasks (Chung et al., 2022a). LLMs have demonstrated less success when solving tasks directly related to social knowledge. For tasks that require social information such as detecting sarcasm (Farha et al., 2022) or patronizing language (Perez-Almendros et al., 2022), recent models exhibit only moderate performance. One major challenge is that compared to humans, LLMs have less capability to make predictions outside of the provided input and must perform reasoning only based on their innate social information (Sap et al., 2019b; Zhou et al., 2020). Yet, it is this very social knowledge that is crucial for human interactions and conversations and is a milestone that should be reached for LLMs to engage in meaningful communications with humans (Mahowald et al., 2023).

More recently, general-purpose LLMs trained with instruction-based prompts have been known to achieve strong performances, putting them to use in several practical domains such as summarization, question answering, and classification (Sanh et al., 2022). A newly emerging trend is to use curated prompts to identify the psychological capabilities of instruction-guided LLMs. Ruis et al. (2022) and Hu et al. (2022a) examine pragmatic understanding capabilities using prompts. Coupled with additional steps such as chain-of-thought (CoT) reasoning, this prompt-based approach has large potential for understanding whether LLMs can provide reasoning capabilities like humans.

The Inter-relatedness of Social Information Social language understanding requires accurately perceiving different dimensions and facets of communication that relate to one another. Interpersonal communication makes frequent use of humor (Schnurr, 2010), mitigation, also known as hedging, (Schneider, 2010), and swearing as a norm violation (Stapleton, 2003) in defining the contours of the social context for the speakers. Often, the pragmatics of these different dimensions of social language use are intertwined: communication with one dimension influences the interpretation of another, e.g., politeness and offensive speech (Culpeper, 2021), humor and politeness (Attardo, 2008), humor and offensiveness (Alberts, 1992), and mitigation and empathy (LI Hai-hui, 2019). Understanding one of these dimensions requires models to have the ability to recognize the related dimensions. While past computational work has largely focused on single dimensions, SocKET fills a key gap by testing whether models can accurately recognize multiple, interrelated social dimensions—and whether models can benefit in their understanding from cross-task transfer.

The SocKET Benchmark

Here, we describe the steps taken to curate SocKET as robust benchmark for identifying social information embedded in language in interpersonal communication contexts.

The task curation process began with a systematic review of literature on social from linguistics, communications, and psychology to identify likely categories of social knowledge. Then, possible datasets and tasks were identified through a systematic review of datasets published at ACL, EMNLP, NAACL, EACL, LREC, and SemEval since 2015. In this first pass, we selected more than 100 datasets and tasks to detect different types of social information in language (cf. Table LABEL:tab:all-candidate-datasets in Appendix B.9 for all candidate datasets and tasks). Tasks were selected based on membership in five categories of social language (described next) that are motivated as core aspects of social language understanding.

For each category, we include tasks of several distinct objectives: binary and multi-class classification, regression, pairwise similarity detection, and span identification.Other task types were initially considered (e.g., generation, paraphrasing) but such tasks were not feasible for all models and often were less standardized in their evaluation, complicating cross-task comparison if included. Where possible, we aim for diversity within categories and ensure one task for each objective. Candidate tasks were removed if it was found that training a bert-base-uncased model on the task achieved test performance over 0.95, which would provide little insight into progress at recognizing social information .

While this process identified many candidate tasks in multiple categories, the benchmark still defines only partial progress in social knowledge capabilities. Some abilities recognized by social sciences such as deceit have only one or two tasks proposed (Ott et al., 2011), providing limited data to measure progress. However, recognizing these as limitations (discussed in more detail in §8), SocKET provides a diverse set of tasks and capabilities, described next, for the field to begin to measure progress.

2 Task categories

Inspired by theories in interpersonal communication and interpersonal pragmatics, we provide a thematic organization of the tasks in SocKET into five related categories of social knowledge: Humor & Sarcasm, Offensiveness, Sentiment & Emotion, Social Factors, and Trustworthiness.

Humor & Sarcasm The practice of humor in conversations and interactions plays a key role in maintaining and forming positive social relations (Holmes, 2006; Brown et al., 1987; Ziv, 2010). We differ Humor & Sarcasm from Trustworthiness as a social information category because while both categories consider non-cooperative behaviors (Grice, 1975), humor is considered to be prosocial (Attardo, 2008). In instances where the humor is not considered to be prosocial and is instead of a derogatory nature, we consider it to be in the Offensiveness category. By nature, humor is a subjective concept that can differ depending on both demographic and contextual factors (Ruch, 2010), making humor detection a difficult task for LLMs. SocKET includes a number of tasks on humor that can occur in various contexts such as in social media (Meaney et al., 2021), short jokes (Meaney et al., 2021), and news headlines (Hossain et al., 2020). We also include tasks that require detecting relevant concepts of humor such as sarcasm (Khodak et al., 2018) and irony (Van Hee et al., 2018).

Offensiveness Detecting offensiveness using computational methods has gained large attraction in recent years due to the ubiquity of online communication and the necessity to implement automated content moderation to combat abusive behaviors (Spertus, 1997). However, most existing studies only focus on limited types of offensive languages (Jurgens et al., 2019). In this study, we consider offensiveness to be any explicit or implicit language directed towards individuals, entities, or groups (Waseem et al., 2017), and the tasks chosen are representative of this understanding. SocKET includes a list of offensiveness detection tasks covering different levels of harmful content and abusive language including both explicit and implicit hate (ElSherief et al., 2021), abuse (Vidgen et al., 2021), and humor-related offensiveness (Meaney et al., 2021). We also include forms of bias directed towards people and groups, as social bias enforces harmful stereotypes (Sap et al., 2020).

Sentiment & Emotion Emotion is a core element of interpersonal communication that can be communicated through human language in several aspects (Majid, 2012; Barrett et al., 2007). Social information is crucial in the ability to not only communicate, but also feel emotion. Theories of discretized emotion (Ekman, 1992) have been supported by empirical findings that humans use discrete labels learned through language to direct their emotional responses to stimuli (Lindquist and Barrett, 2008). Moreover, emotional responses have been shown to direct communication with peers (Lee et al., 2020), and expressing certain emotional responses—such as anger—have been shown to have social ramifications (Keltner et al., 1993). Interpreting emotions from text using computational tools has been a popular research topic across numerous areas in social sciences, enabling new discoveries at unprecedented scale (Jackson et al., 2022). In SocKET, we include a wide range of tasks from various domains such as daily dialogue (Li et al., 2017), written responses to news stories (Buechel and Hahn, 2017), and tweets using textual syntax (Mohammad et al., 2018), and also emojis (Barbieri et al., 2018).

Trustworthiness People can detect cues in language that determine the trustworthiness of a message (Newman et al., 2003), leading to studies that aim to quantify the level of trust in text using computational methods (Choi et al., 2020). In particular, this direction has gained attention from NLP communities following increased needs to combat and mitigate potential harms coming from the generation and dissemination of false information in online spaces (Wu et al., 2019). In SocKET we include tasks that require identifying perceived trust from several dimensions: impartiality (Pryzant et al., 2020), deception (Ott et al., 2011), propaganda (Martino et al., 2020), rumor (Ma et al., 2017) and bragging, as it is considered to be “unplain speaking" (Haiman, 1998; Jin et al., 2022).

Other Social Factors Finally, we include tasks of a more discursive and rhetorical type, that are understood to be more reliant on the contextual elements of social distance, power, and solidarity. In SocKET, the tasks included are empathy (Buechel et al., 2018), politeness (Hayati et al., 2021; Fu et al., 2020), intimacy (Pei and Jurgens, 2020) and complaints (Preoţiuc-Pietro et al., 2019). Politeness, like humor, is understood to be a non-cooperative prosocial behavior but unlike humor, is concerned with the act of “saving face” (Brown and Levinson, 1987). Empathy, shown to be closely related to politeness (Fukushima and Haugh, 2014), is heavily reliant on social positions in the context of the conversation (Macagno et al., 2022). Intimacy, however, has been shown to be more dependent on notions of time and space between people in dialogue (Márquez Reiter and Frohlich, 2020).

3 Dataset Summary

The final SocKET benchmark contains 58 tasks from 35 datasets, grouped into the five categories shown in Figure 1. We denote multiple tasks from the same dataset by adding the task name as a suffix following the dataset name and # symbol.

The collection of tasks chosen for SocKET makes it a comprehensive benchmark to measure language models’ abilities to capture underlying social information. Motivated by theories of systemic functional linguistics and interpersonal pragmatics, SocKET cuts across a number of dimensions of interpersonal communication, allowing it to also be a tool to better understand and interpret co-learning abilities and dependencies in sociolinguistic tasks. Having this ability allows researchers and users to more efficiently and effectively deploy NLP methods by providing empirical results on the limits and affordances of a variety of out-of-domain social language tasks.

In total, SocKET spans 2,616,342 items across all tasks, including 269,246 samples in the test set. However, experimenting with an evaluation set of size can be prohibitive due to model size, available resources, and considerations of the environment. Therefore, we also release a subset of our data as SocKETTe (SocKET but Tinier) that contains at most 1000 items per task in the test set, reducing the test set to 43,731 samples. In Appendix B.3, we show that performance on SocKETTe is highly correlated and we hope that this smaller subset enables more rapid progress.

Benchmarks on the Social Knowledge Capabilities of LLMs

We first train and evaluate several commonly used multitask LLMs on our datasets to obtain benchmark results, which provide a first glimpse of how good LLMs are at learning social knowledge tasks. Experiment details are described in Appendix §B.

BERT-based Finetuning We first apply the standard process of fine-tuning on pretrained LLMs. We select two of the most popular LLMs - BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019) - as well as two lightweight models known to achieve high performance on finetuning tasks - DeBERTa-V3 (He et al., 2021) and MiniLM (Wang et al., 2020).

Prompt-based finetuning Prompt-based finetuning has emerged as a flexible and effective means of adapting models to downstream tasks (Wei et al., 2021). As a benchmark, we include the performances of a T5 model (Raffel et al., 2020) trained on each task via finetuning. We manually design prompts for each task. For classification tasks, we use verbalizers to map the class to word labels and for regression tasks, we adopt a method similar to Gao et al. (2021) in that we use two anchor words “Yes” and “No” and consider the probability of predicting “Yes” as the final score. For span-based tasks, we train the model to directly generate the sequence outputs. A list of prompts can be found in Table 8 and Table 9 in the Appendix.

Zero-shot predictions We further apply our designed prompts to test the performances of LLMs in a zero-shot setting where no further finetuning is performed. Using the same prompts proposed in Table 8, we test SocKET on several widely used LLMs: GPT (Radford et al., 2018), GPT-J-6B Wang and Komatsuzaki (2021), OPT (Zhang et al., 2022), T5 (Raffel et al., 2020), LLaMA Touvron et al. (2023a), LLaMA-2 (Touvron et al., 2023b), BLOOM Workshop et al. (2023), BLOOMZ Muennighoff et al. (2022), FLAN-T5 Chung et al. (2022b), RedPajama Computer (2023), and Alpaca (Taori et al., 2023; Wang et al., 2022b). We also evaluate the performance of GPT-3.5 https://platform.openai.com/docs/models/gpt-3-5 using OpenAI’s API. Samples for which a model does not provide an appropriate label are automatically marked as incorrect. For each LLM variant, we test zero-shot results for different model sizes ranging between 110M and 13B parameters, which we report in Table 4 in the Appendix.

2 Results

We compare model performances across category type and task type as shown in Table 2. Each reported value is the average of the scores on every task within the specified group. The rationale behind using a unified average score is to provide a high-level comparison of the performances of zero-shot and fine-tuned models under various settings, including task type (regression/classification/pair/span) as well as dimension of social knowledge.

DeBERTa-V3 achieves the best overall performance after full training on each of the SocKET datasets, followed by other BERT-based models. The prompt-based finetuning of T5 performs worse than standard finetuning, especially on the pairwise classification and regression tasks. Meanwhile, most zero-shot models perform only slightly better than the baseline, indicating that prompting alone does not elicit correct social knowledge—though two models, google-flan-t5-xxl and GPT3.5, are much closer in performance to supervised models.

Social knowledge can be hard to infer Our benchmark results reveal that even our best-performing model leaves significant room for improvement, scoring just above 0.7 overall—compared with the models’ analogous performance on syntactic and discourse NLU tasks (He et al., 2021) which are often much higher. A comparison among categories of social knowledge reveals that humor & sarcasm is generally the easiest to detect, while trustworthiness is the hardest. This performance gap can be attributed to the level of understanding required for each dimension - while detecting humor or other social emotions can often be correlated with cues such as sentiment, detecting the level of trust within sentences requires more understanding of the context and may be harder to detect using computational models Choi et al. (2020). At a task level, we observe that models struggle most in span detection tasks. This is a complex task due to its open-ended nature, and thus BERT-based finetuning does not perform as well as in other types of tasks. We highlight that learning the various aspects of social knowledge is indeed a challenge for current LLMs, and thus call for the need for future models with improved social capabilities.

Supervised models significantly outperform zero-shot models Table 2 reveals that despite being much smaller in the number of parameters, finetuning supervised models such as MiniLM leads to much better performance than zero-shot models using state-of-the-art LLMs. All the zero-shot LLMs performed poorly, many on par with random baselines, apart from FLAN-T5. Figure 1 shows a detailed picture of how different LLM parameter sizes influence the ability to comprehend social knowledge tasks in a zero-shot setting. Surprisingly, we find that of the various training schemes FLAN-T5 is by far the most effective for inferring social knowledge, even with relatively small models. We speculate this performance is due to its initial pretraining on more than 1,000 tasks.

More parameters do not guarantee more social knowledge Another general trend we observe is a weak correlation between the number of parameters and overall performance within the same model architecture ($\rho=0.266$, $p=.08$). This is to some extent determined by the model’s ability to understand the task itself given an instruction prompt as well as a sample input, as larger models are capable of understanding a wider variety of tasks (cf. Appendix Table 6). Of course, it is also possible that larger LLMs could encode a greater amount of social knowledge through their greater parameter sizes. Interestingly, we observe that for some models, larger size does not always guarantee better performance. This is the case especially for BLOOM, T5 and GPT, where the largest model is not always the best performer within the group.

Models varied in the ability to follow instructions (Appendix Table 6). As expected, instruction-tuned models like FLAN-T5 and Alpaca are generally able to follow the prompt instructions, while other models may generate answers that are not provided in the options. For our social tasks, instruction-following was not significantly correlated with model size ($\rho$=0.08, p=0.60). Thus, lower model performance in Figure 1 is, in part, due to models being unable to answer questions relating to social knowledge.

When models are able to answer the question, are they right? Restricting only to instances in which a model outputs a valid answer reveals heterogeneity among different model groups (Figure 3), showing an interplay between model size, coverage, and performance. For architectures such as FLAN-T5 or BLOOMZ we observe a positive correlation between parameter size and performance, both in its ability to understand instructions and to make correct predictions. On the other hand, for certain architectures having larger parameters can actually make it worse at understanding instructions (e.g. LlaMA) or predicting correctly (e.g. OPT). Recognizing that measuring of instruction understanding and the accuracy of an LLM both depend on how strictly one chooses to map the predictions to an answer, overall, our results suggest that while LLMs do contain the potential for understanding social knowledge, additional steps such as finetuning or instruction tuning are likely needed for better social understanding.

Do we see Cross-task Transfer of Social Knowledge?

In this section, we examine the relations and dependencies between tasks using the predictions of LLMs trained on different tasks and test for dependencies between tasks that are predicted by theory.

Quantifying Task Dependency We quantify the dependency between two tasks as follows. We finetune a pretrained LLM on task $t_{A}$ to obtain a model $m_{A}$, which is used to make predictions on the test set of another task $t_{B}$. The correlation between the predicted values from model $m_{A}$ and the true labels of the test set of $t_{B}$ is considered as the task dependency that $t_{A}$ has on $t_{B}$. We report the absolute correlation value, as negatively correlated tasks are still informative. We describe how the correlations are obtained across different task types in the Appendix (§B.6). Span identification tasks are omitted from this analysis, resulting in $55\times 55$ scores. We also measure the pairwise correlation between models $m_{A}$ and $m_{B}$ as well as task dependency to gain an additional perspective of task similarity. Details for the model correlation can be found in Appendix §B.6 and Figure 7.

The task dependencies for all task pairs, shown in Figure 2, reveal salient block structures within the category,See Figure 6 for fully labeled version. especially for the Offensiveness, Sentiment & Emotion, and Social Factors categories, suggesting the existence of shared knowledge within our thematically grouped tasks. These correlations align with existing findings from interpersonal pragmatics on the relationships between social knowledge. For instance, increased self-disclosure or pain-related interactions are known to promote both intimacy (questionintimacy) and empathy (empathy) (Parks, 1981; Cano and Williams, 2010), two elements within the Social Factors category, while the usage of emojis (tweet_emoji) as effective symbols are indicative of emotional states such as valence (emobank#_valence) and arousal (emobank#_arousal) (Fischer and Herbert, 2021), which belong to the Sentiment & Emotion category.

The Offensiveness category shows mixed results in comparison with Arango et al. (2019), whose results show that hate speech datasets are often overfit and do not generalize well to other similar datasets . Figures 2 & 6, however, show that of the seven datasets included in SocKET, five of them included at least one task which showed comparable correlations when tested both within and out of domain. Indeed, PersonDirectedAbuse, a task labeled for offensive language specifically directed towards an individual, is actually predicted better by models fine-tuned on jigsaw# tasks than it was on its own.

Interestingly, correlations are scarce within the Humor & Sarcasm, and Trustworthiness categories. This is consistent with findings from (Hu et al., 2022b) which show that models without exposure to linguistic forms lack the requisite social information to perform well on non-literal pragmatic phenomena such as humor and deceit.

Another notable individual task is humor_rating from the Humor & Sarcasm dataset, which performs well as both the fine-tuning and predicted task alongside a number of tasks from the Emotion & Sentiment category—particularly discretized emotion tasks, as well as hateoffensive in the Offensiveness category—which labels comments as either “hateful," “offensive," or neither. While relationships between offensiveness and humor have been theorized as early as Freud (1960) and sentiment recognition has been shown to bolster offensive language detection (Liu, 2012), relatively little has been said regarding connections between the three categories and thus, this result presents an opportunity for further research.

We observe that politeness shows strong transfer with many of the offensive and hate speech detection tasks in the SocKET benchmark. In particular, those tasks with high correlation within the offensive category are highly correlated in predicting the politeness classification task. This finding is supported by literature showing that impoliteness can fall under the umbrella of offensive language (Bączkowska, 2021) and, although key differences exist in the pragmatics of the two, the constructs are closely related (Parvaresh, 2023; Culpeper, 2021).

Interestingly, regression tasks (from the hahackathon, emobank, and empathy datasets) in general have strong correlations with several other tasks. This trend suggests that tasks labeled with continuous variables may have more expressive power compared to ordinal or nominal categorization, and thus have a higher potential for stronger task dependencies. However, the magnitude of the correlation may be influenced by the relative value distributions of different correlation methods. This finding calls for a need for more datasets with continuous labels, which requires more effort but allows models to capture more fine-grained concepts of social knowledge.

Can Multi-task Training improve Social Knowledge?

Our findings reveal significant task transfer, both within and across task categories, which hints at shared knowledge among tasks. Linguistics studies of social language also note the interrelated perceptions of different dimensions such as humor and offensiveness (Culpeper, 2021; Attardo, 2008; Alberts, 1992; LI Hai-hui, 2019). We now examine whether LLMs can learn a more robust sense of social knowledge by training on multiple tasks.

Experimental Setup Recent studies have explored the possibility of multi-task training on LLMs, which is training a single model on several different tasks simultaneously, with effects of improving its performance on both seen and unseen tasks (Aghajanyan et al., 2021; Padmakumar et al., 2022). We apply multi-task training on SocKET, but make one clear distinction from prior work. Whereas previous studies have shown that multi-task training is especially effective when the grouped tasks are of similar types (Padmakumar et al., 2022), we introduce a new setting by grouping tasks instead by our defined categories of social knowledge. We expect that same-category tasks contain social knowledge that can be shared across tasks, resulting in LLMs that learn a more robust concept of the specific dimension than when trained on single tasks.

A popular method for multi-task training is pre-finetuning (Aghajanyan et al., 2021; Shi et al., 2022), which involves a first stage of finetuning on multiple tasks using task-specific heads on a shared encoder, then re-using the encoder for downstream tasks. We apply pre-finetuning in two different settings: (1) category-wise tasks, where we perform pre-finetuning on tasks grouped to the same category, and (2) all tasks, where all tasks of SocKET are included in the pre-finetuning stage. Consistent with prior work, we perform the second finetuning stage on individual tasks using the pre-finetuned model as initial weights (Aghajanyan et al., 2021). Other training details are identical to §4.

Results Multitask training had little to negative effect on task performance (Table 3). Although some tasks did benefit from being co-trained within category (Appendix Table 10)—particularly the Offensiveness category—when aggregated at the category level, the average performance is worse. In particular, the Humor & Sarcasm and Trustworthiness categories have the lowest levels of within-task and cross-task dependencies (§5). The performance drop is less strong in categories with high dependency, indicating that while multi-task training on similar tasks may not always improve performance, task-relatedness can help preserve performance when also learning task-specific new concepts. Together, our results suggest multi-task training on unrelated social tasks hurts overall performance—a result contrary to social science expectations of how social information is processed—and points to a need to further investigate cases when applying multi-task training as a practice to improve the social knowledge of LLMs.

Conclusion

People increasingly interact with LLMs in natural conversation. To what degree are these models able to pick up on the social cues? To help answer this question, we introduce SocKET, an NLP benchmark to evaluate how well models perform at learning and recognizing concepts of social knowledge. We provide benchmark results using several popular models and provide case studies of studying the inherent social capabilities of LLMs in a zero-shot setting. Surprisingly, LLMs perform moderately at best, with even large LLMs ($>$10b parameters) varying widely in their abilities. Additionally, we show that there exist significant task dependencies both within and across task categories, and that multi-task training on task categories can affect model performance. Our work contributes to the broader NLP community by fostering future efforts toward building and evaluating more socially responsible and coherent LLMs.

Limitations

Culture is an important aspects of understanding language, especially within the broader setting of multilingual NLP. In this study, however, we make a clear distinction between cultural knowledge and social knowledge, the latter of which is our focus for this study. Our work is grounded in social-psychological theory and the sociolinguistics of interpersonal communication, especially dyadic communication. Such studies are often aimed at phenomena that are widely shared across cultures while recognizing that cultural variation exists within how those phenomena are perceived. In contrast, work in anthropology or cultural studies provides a different perspective and grounding. Such work frequently focuses on cross-cultural perspectives and what is or is-not shared across cultures. For example, in language, the interpretation of whether something is polite can depend on gender norms (Mills, 2004) and cultural (Lorenzo-Dus and Bou-Franch, 2003), highlighting the potential context sensitivity. Similarly, the perception of toxicity can depend on the cultural identities of the reader Sap et al. (2019a); Ghosh et al. (2021). While highly valuable to study, cultural knowledge is a separate construct from social knowledge (though interrelated) and not the focus of this benchmark, though we hope that our work inspires other benchmarks to help assess such differences.

Regarding multilingual data, SocKET currently contains tasks based in English due to the limited availability of tasks in non-English. While there are a few datasets such as HAHA (Chiruzzo et al., 2020) in Spanish and DeTox (Demus et al., 2022) in German, we were not able to find sufficient numbers yet to provide a meaningful grouping. This highlights the importance of constructing datasets and frameworks capable of capturing social knowledge for a wide variety of languages, which we consider an important future step.

Interpersonal communication conveys a richness of different social information and despite our extensive literature review and data curation process, we fully acknowledge that other dimensions of social knowledge are not included in our current benchmark. In creating SocKET, our aim was to focus on diverse categories of social knowledge that have multiple tasks in order to get a more robust assessment of model capabilities, e.g., multiple tests of a model’s ability to recognize humor, in order to avoid the pitfalls of ascribing progress on the basis of a single task alone (Subramonian et al., 2023). Nevertheless, SocKET omits several notable dimensions or forms of social knowledge. Some social aspects of language such as pragmatic polysemy (Carston, 2021; Apresjan, 1974) and idioms (Strässler, 1982) either had too few similar datasets to form a theory-backed category, or there were no existing NLP datasets to test the construct. The latter is the case, especially in the case of linguistic techniques unique to recognize when a speaker is adopting community-specific dialects such as African-American English (Hyter et al., 2015; Rivers et al., 2012; Allan, 2007) and Queer Language (Barrett, 2006; Huebner, 2021; Harvey, 2000).

Social language understanding happens within a static, unspecified context for the current tasks in SocKET. However, the social context in which a message is said can dramatically alter its meaning. NLP is just beginning to incorporate the social context into language understanding (Hovy and Yang, 2021). While a handful of datasets have begun to explore modeling context explicitly, such as through the preceding conversation (Pavlopoulos et al., 2020; Menini et al., 2021), the identity of the speaker (Almagro et al., 2022), the social relationship between speakers (Jurgens et al., 2023), or explicit social norms (Park et al., 2021), there are currently too few of such tasks to compose a comprehensive benchmark with which to measure progress. Future datasets and benchmarks will be needed to study understanding social knowledge when controlling for context.

Thus, SocKET represents a starting point for modeling models’ abilities and provides room for improvement via the addition of new categories or constructs, as additional data becomes available. Further inclusion of other dimensions and corresponding tasks should be an ongoing goal.

SocKET fills a current gap for assessing the capabilities of LLMs on understanding social language. However, benchmarks as constructs have been rightly critiqued as markers of progress in NLP (e.g., Bowman and Dahl, 2021; Schlangen, 2021; Subramonian et al., 2023), due to aspects such as changing or narrowing the field’s definition of a task, overemphasizing or overselling progress in a particular area, or encouraging leaderboard chasing. In designing SocKET, we aimed to directly address the pitfalls of benchmark design by selecting a diverse set of social language understanding tasks that mirrored human capabilities recognized in social science studies; this selection helps ensure a broad measure of performance and that “progress” is not due to improved performance on one type of task. However, the benchmark itself does not capture all of social knowledge (nor do we claim as such) and we view it only as a starting point—a yardstick by which to measure current systems—with a need for new tasks and benchmarks as models advance in their social reasoning capabilities.

The use of a single metric to measure progress in an area or task can mask meaningful insight and fail to contextualize performance. While we follow common practice in NLP (e.g., Wang et al., 2018, 2019; Muennighoff et al., 2023) and report a single mean score in Table 2, the design of SocKET includes specific task categories and types designed to easily and meaningfully inspect what is ultimately contributing to the single score—e.g., are models performing well in classification but poorly in span recognition? Nevertheless, this design is a trade-off: A single score can and likely does promote leaderboard chasing by setting a clear goal to pursue, while completely disaggregated scores like those in Table 4 become unwieldy and make it hard to assess whether meaningful progress is being made when comparing two models. Here, we have opted to report both the overall average and averages for each category and type (10 scores total) in an attempt to balance these two tensions.

One major limitation of the current benchmark is we only tested LLMs that have up to 13B parameters. Recent studies show that the LLMs may start to show emergent abilities when they are scaled up above a certain threshold (Wei et al., 2022). Due to limited computational and financial resources, we are not able to test all very large language models, though we welcome future researchers to work on our benchmark and evaluate the sociability of new and larger LLMs.

Finally, our zero-shot model performance used curated prompts on pretrained models without any further finetuning. While it is widely known that instruction-based finetuning specific to downstream tasks can greatly improve performance, we deliberately chose not to do so. Finetuning LLMs with billions of parameters can leave a large carbon footprint, which we avoid for both financial and environmental reasons Hu et al. (2021); Liu et al. (2022); Lester et al. (2021).

Ethical Considerations

The interpretation of social information in communication is highly subjective in that it can largely vary depending on demographic and contextual factors. Nevertheless, several NLP datasets are created via crowdsourcing, which raises concerns on whether the dataset’s labels are truly representative of our society (Talat et al., 2022). Even within our benchmark, there is the possibility that for tasks such as offensiveness or humor the crowdsourced labels may undermine phrases that might disregard a specific demographic group, which may be inevitably picked up by LLMs that are trained and evaluated on these datasets. Improved versions of our benchmark should include datasets that are more inclusive in such contexts, which we call for future work.

There has been increasing concern over the amount of computing resources required for conducting deep learning research at scale, especially regarding LLMs where task performance is improved through larger datasets, increased model parameters, and longer training hours. The time and amount of computing resources required for training LLMs has become nontrivial (Bender et al., 2021), and it has been increasingly aware among machine learning practitioners to consider the carbon footprint of models and computing methods to minimize risks of global warming. This, combined with limited transparency of experiment results, may harm the very concept of open science. Keeping this in mind, we focused on conducting easily reproducible experiments that can be run on a single GPU within the time frame of hours or a couple of days at the longest. Some of our findings contribute towards this rightful direction, as can be seen in our investigation on multi-task training.

More importantly, we highlight the fact that the main contribution of our study is a thoroughly designed public framework of tasks for examining the social knowledge of LLMs. While it is indeed important to develop and improve LLMs that can perform better on several tasks, we believe that correctly evaluating the level of social knowledge engraved in these models is an equally important task. Without such scrutiny, the users of LLMs deployed in practical settings may be vulnerable to socially undesirable or unethical content. We sincerely hope that our efforts in producing SocKET can ease difficulties of conducting future studies that aim to examine and improve the social understanding of LLMs.

Acknowledgments

The authors thank reviewers for their timely and valuable feedback on the paper, with a special shout-out to R1 for their very detailed feedback which certainly made this paper better. We also thank the members of the Center for Social Media Responsibility, especially Paul Resnick and James Park for their support which enabled the initiation of this project. This work was supported by the National Science Foundation under Grant Nos. IIS-2007251, IIS-2143529, and 2137469. The third author was partially supported by grant SES-2200228 from the National Science Foundation.

References

Appendix A Details on dataset processing

The SocKET dataset consists of 58 tasks from 35 unique, public datasets. The datasets that make up the benchmark dataset are processed in a way that is meant to balance uniformity across datasets and tasks while minimizing deviations from the original dataset.

For all datasets, key changes from the original dataset are twofold:

Duplicates and unlabeled items are removed from all datasets. If duplicates occur across data splits, the splits are recombined, reshuffled, and split.

All datasets are split 80%/10%/10% between train/test/dev splits, respectively. Any datasets not split 80%/10%/10% are recombined, reshuffled, and split 80%/10%/10%.

All datasets were made compatible with the Hugging Face Datasets package.

Appendix B Experimental Details

All of our experiments were conducted on an Ubuntu 22.04.1 machine installed with NVIDIA RTX A5000 and A6000 GPUs. The Python packages used in our experiments include Pytorch 1.13, Transformers 4.21.3, and Pytorch Lightning 1.6.4.

B.2 Comparison of all models

Table 4 contains a detailed version of Table 2, where the scores of every single task are presented.

B.3 Details on the comparison between SocKET and SocKETTe

32 out of 58 tasks contained more than 1,000 test samples, resulting in a disparity between the sizes of the original SocKET and SocKETTe variants. To test that both datasets still offer comparable evaluations for testing models, we compare their scores for a supervised model and compare test set performances. For each task, we train a deberta-v3-base model, evaluate using the test sets of both versions, and compute the correlation between each setting using Pearson’s r score. We provide evaluation results of SocKETTe for our models in Table 5. Also, we show through Table 7 and Figure 5 that there exists a strong correlation between the evaluations of both versions, demonstrating that SocKETTe is indeed a representative sample of SocKET.

B.4 Details on language model finetuning (§4, §5, §6)

As our benchmark consists of four different task types: classification, regression, sentence pair detection, and span identification - we maintain a unified structure for each task where each sample is fed into the encoder of an LLM, and the output states are then fed into a task-specific head layer. For span detection tasks, we feed the last hidden layer into a bidirectional GRU (Chung et al., 2014), and then the output vectors of the GRU into a linear layer that transforms each vector into a dimension of 3, corresponding to the [B,I,O] labels for each token, following earlier work in span identification (Suman and Jain, 2021). For all other tasks, we feed the last hidden state of the encoder corresponding to the [CLS] token into a separate classifier/regression head consisting of two linear layers of hidden size 768 and a dropout probability of 0.1. We use the mean squared error loss for regression tasks and the cross-entropy loss for all other tasks.

B.4.2 Training strategies for language model finetuning (§4, §6)

When training models for the benchmark (§4) and the multi-task (§6) experiments, the learning rate was linearly increased for 6% of the training steps up to 1e-5 and linearly decreased afterward. All models were trained for a maximum of 10 epochs using three different seeds, with early stopping after validation performance did not increase for three consecutive epochs.

Our multi-task training in §6 requires two stages of training: (1) a pre-finetuning stage that simultaneously trains a model on multiple different tasks, and (2) a finetuning stage that loads the model trained from (1) and finetunes it to a single task. In the first stage, a single batch can include several different tasks and produce different types of losses. To obtain a unified loss that is differentiable, we aggregated the loss for each sample and sum them up, which we use for backpropagation. For both stages, we use the same aforementioned training steps and learning rate strategy.

For all settings, the training batch size was set to 32 with 16-bit precision enabled. Validation was made after each training epoch on the validation set using Pearson’s r correlation added by 1 and divided by 2 for regression tasks and macro F1 score for all other tasks. If there were multiple tasks considered due to multi-task training, the average of all task performances was used as the final validation score.

B.5 Details on prompt-based finetuning (§4, §5)

We use fix prompts fine-tuning for all the prompt-based models. The batch size was set as 32 for training. For every single task, we set 10 as the max epoch and do early stopping based on the validation loss. The learning rate is set as 5e-5.

For classification tasks, the model is fine-tuned to generate the target label. For regression tasks, we first normalized the scores into (0,1) and then split the labels into two groups. The model is fine-tuned to predict “yes” or “no” regarding the prompt question. During inference, the probability of the “yes” token is used as the prediction score. For span tasks, we directly train the model to generate the full answer.

B.6 Details on zero-shot predictions (§4, §5)

We use manually designed prompts for all the zero-shot prediction tasks and the prompts are shown in Table 8.

B.7 Computing correlation scores of task dependencies (§5)

Because our framework consists of several task types, it is challenging to obtain a unified metric of correlation across different task comparisons. We use the following rules to obtain correlation values:

Regression task & regression task: We compute the Pearson’s correlation coefficient of the two arrays.

Regression task & binary classification task: We compute the point biserial correlation coefficient of a continuous array and a binary array.

Regression task & multi-class classification task: We set up a linear regression task using the one-hot coded values of the multi-class array as independent variables and the continuous array as the dependent variable. We report the root of the R-squared value of the regression as correlation (Olsson et al., 1982).

Binary classification task & binary classification task: We compute the Matthews’ correlation coefficient (Matthews, 1975) from the two binary arrays.

Binary or multi-task classification task & multi-class classification task: We compute the Cramer’s V score (Cramér, 1999) from the two arrays of categorical variables.

B.8 Computing pairwise model similarities (§5)

We quantify the model similarity between two tasks as follows. We finetune a pretrained LLM on task $t_{A}$ to obtain a model $m_{A}$, and another LLM on task $t_{B}$ to obtain $m_{B}$. We obtain pairwise model similarities by inferring both models on a sufficiently large dataset—in this case the entire test set of all tasks—and computing the correlation of the two inferred arrays. We construct an undirected graph (Figure 7) where the thickness and color represent absolute correlation strength and polarity between the two models. The addition of polarity enables us to further discover strong negative correlations with task pairs such as politeness and offensiveness.

B.9 List of all potential tasks and datasets for SocKET