How would Stance Detection Techniques Evolve after the Launch of ChatGPT?
Bowen Zhang, Daijun Ding, Liwen Jing, Genan Dai, Nan Yin
Introduction
Personal stance towards an issue affects the decision making of an individual, while the stance holds by the public towards thousands of potential topics explains more. Stance detection is an important topic in research communities of both natural language processing (NLP) and social computing Küçük and Can (2020); AlDayel and Magdy (2021). Similar to all NLP tasks, early works on stance detection focused on rule-based approaches, and later made a transition into traditional machine learning based algorithms. Since 2014, deep learning models quickly become the mainstream techniques for stance detection. Later on, with the great success of Google’s bidirectional encoder representations from transformers (BERT) model, a new NLP research paradigm emerges which is utilizing large pre-trained language models (PLM) together with a fine tuning process. This pre-train and fine-tune paradigm provides exceptional performance for most NLP downstream tasks including stance detection, because the abundance of training data enables PLMs to learn enough general purpose features and knowledge for modeling different languages. Following BERT, more and more PLMs are proposed with different specialties and characteristics, including the ELMo series, the GPT series, the Turing series, varieties of BERT and many more.
ChatGPT is the most recent PLM optimized for dialogue and attracted over 1 million users within 5 days. Programmers use it to interpret code, artists use it to generate prompts for AIGC models, clerks use is to write and translate documents; writers challenge ChatGPT to write poems and film scripts and etc. To what extent will ChatGPT transform the society and people’s way of doing and thinking? For NLP experts, is ChatGPT just another pre-trained language model?
In this work we conduct experiments on ChatGPT for stance detection tasks by directly asking ChatGPT for the result. This approach can be considered as a zero-shot prompting strategy. Experimental results show that ChatGPT can achieve SOTA or similar performance for commonly used datasets including SemEval-2016 and P-Stance with a simple prompt. Since ChatGPT is trained for dialogues, it is surprisingly easy to know the reason of the model’s decision making by directly asking why. Furthermore, interacting with ChatGPT with a chain of inputs can potentially further improves the performance. This paper is structured as follows: after a brief overview of related work in Section 2, our proposed prompting methods and results are detailed in Section 3. Section 4 contains discussions and future work.
Related Work
Before getting into more detail, we first give a formal definition of stance detection. For an input in the form of a piece of text and a target pair, stance detection is a classification problem where the stance of the author of the text is sought in the form of a category label from this set: {Favor, Against, Neither}. Occasionally, the category label of Neutral is also added to the set of stance categories and the target may or may not be explicitly mentioned in the text. Researchers approach this task by converting it into a text classification task. Stance detection studies originally focused on parliamentary debates and gradually shifted to social media contents including Twitter, Facebook, Instagram, online blogs and etc. The techniques to approach these problems also evolve with time.
Early research works on stance detection from the 1950s mainly adopted rule-based techniques Anand et al. (2011); Walker et al. (2012). Since the 1990s, machine learning based models gradually replaced small scale rule-based methods. Traditional machine learning models build text classifiers for stance detection based on selected features. The effective algorithms for the classifiers are support vector machine (SVM) Addawood et al. (2017); Mohammad et al. (2017), logistic regression Ferreira and Vlachos (2016); Tsakalidis et al. (2018); Skeppstedt et al. (2017), naive bayes HaCohen-Kerner et al. (2017); Simaki et al. (2017), decision tree Wojatzki and Zesch (2016) and etc. With the fast advancement of deep learning in the 2010s, models based on deep neural networks (DNN) become mainstream in this field. These methods design neural networks with different structures and connections to obtain the desired stance classifier, which can be categorized as conventional DNN models, attention-based DNN models and graph convolutional network (GCN) models. Convolutional neural network (CNN) and long short-term memory (LSTM) models are most commonly used conventional DNN models Augenstein et al. (2016); Du et al. (2017); the attention-based methods mainly utilize target-specific information as the attention query, and deploy an attention mechanism for inferring the stance polarity Dey et al. (2018); Sun et al. (2018); and the GCN methods propose a graph convolutional network to model the relation between target and text Li et al. (2022); Zhang et al. (2020a); Conforti et al. (2021).
Inspired by the recent success of PLMs, fine-tuning methods have led to improvements in stance detection tasksLiu et al. (2021b). Fine-tuning models adapt PLMs by building a stance classification head on top of the “cls” token, and fine-tune the whole model. The PLMs are getting larger and larger because the performance and sample efficiency on downstream tasks are normally proportional to the scale of the model, and some abilities like the prompting strategies, popularized by GPT-3, are considered to be effective only when the model reaches a certain scale Wei et al. (2022). The main idea of prompt-based methods is mimicking PLMs to design a template suitable for classification tasks and then build a mapping (called verbalizer) from the predicted token to the classification labels to perform class prediction, which bridges a projection between the vocabulary and the label space. The prompting strategies provide further improvements for stance detection performanceShin et al. (2020). Models like LaMDA, GPT-3 and etc. also gain success on few-shot promptingWei et al. (2022), which alleviates the demand for large amount of training data and the tedious training process.
Generally speaking, stance detection techniques and NLP algorithms in general experienced four main paradigms: (1) rule-based models; (2) traditional machine learning based models; (3) deep neural network models and (4) PLM pre-train and fine-tune paradigm. Quite recently, the 5th paradigm "pre-train, prompt and predict" starts to draw wide attentionLiu et al. (2021a).
Methods and Results
Task definition: We use to denote the collection of data, where each denotes the input text and denotes the corresponding target. represents the number of instances. Stance detection aims to predict a stance label for the input sentence towards the given target by using the stance predictor.
In this Section, we reveal the performance of the ChatGPT method for stance detection. We utilize a special case of prompt to construct the stance predictor by creating a template of a direct question. Specifically, we directly ask the ChatGPT model the stance polarity of a certain tweet towards a specific target. Figure 1 shows an example. Given the input: “RT GunnJessica: Because i want young American women to be able to be proud of the 1st woman president #SemST”, the question for ChatGPT input is: “What is the attitude of the sentence : "RT GunnJessica: Because i want young American women to be able to be proud of the 1st woman president #SemST” to the target “Hillary Clinton" select from “favor, against’ or neutral’. For this particular example, ChatGPT returns a correct result.
Results: To compare the effectiveness of ChatGPT, we carried out experimental validations in the SemEval-2016 stance datasetMohammad et al. (2016) and P-Stance datasetLi et al. (2021). The SemEval-2016 is a dataset of 4870 tweets in English with manual annotation for stance towards 6 selected targets and ‘Hillary Clinton (HC)’, ‘Feminist Movement (FM)’ and ‘Legalization of Abortion (LA)’ are three commonly used ones. Similarly, P-Stance dataset contains 21574 English tweets with political contents with stance annotations towards three targets including “Donald Trump,” “Joe Biden,” and “Bernie Sanders.” (Note that, following Li et al. (2021), we only use favor and against labels for evaluation.) Since OpenAI has not provide API for using ChatGPT yet, experiments has only been conducted on these two benchmark datasets on social media texts. Following Zhang et al. (2020b); Liang et al. (2022) we use the F1-avg (the average of F1 on Favor and Against), and macro-F1 score (denoted as F1-m) for performance evaluation.
We constructed both zero-shot stance detection and in-domain stance detection setups for results comparison. The zero-shot setup means the model is directly tested without any adjustment with training data, which is a fair comparison with our proposed prompt method using ChatGPT. We also compared our zero-shot results of ChatGPT with other mainstream stance detection models in an in-domain setup, which means these models are optimized with 80% tweets as training data. The results are summerized in Table 1 to 3.
The results show that ChatGPT achieves SOTA results in zero-shot setup. For example, ChatGPT achieves a 16.6% improvement on average compared with the best competitor PT-HCL in zero-shot setup. Compared with in-domain setup, where these methods first learned from 80% training corpus, ChatGPT still yields better performance than all the baselines in most tasks.
Discussions and Future Work
Results in Section 3 demonstrate the emergent ability of ChatGPT on zero-shot prompting for stance detection tasks. By using a simple prompt of directly asking the dialogue model for the stance with no training, ChatGPT returns SOTA results in both zero-shot and in-domain setups. The launch of ChatGPT would potentially transform the whole research area. We would like to discuss three research directions which might further improve the performance of ChatGPT on stance detection tasks.
In this work, only one prompt template for stance detection has been tested with ChatGPT. Engineering the prompt template may further improve the zero-shot performance of using ChatGPT or unlock the use of ChatGPT to other NLP tasks. Futher studies can take the intuitive approach of manually selecting prompt templates or design an automated process for template selection.
ChatGPT is a language model trained for dialogues, thus it is a natural next step to ask the model why it provides certain answer. As shown in Figure 2 and 3, ChatGPT provides perfect explanations for why the given tweet is in favor of the target Hillary Clinton weather the stance is explicitly or implicitly expressed in the text. Such results indicate that ChatGPT carries out stance classification based on logic reasoning instead of pure probability calculation. These explanations opens up the possibility of building explanatory AI for stance detection.
(3)Can multi-round conversation help to improve the results?
ChatGPT has already shown exceptional results with zero-shot prompting, however, it is more powerful than a stance classifier. For some instances when ChatGPT cannot provide prediction results, it can still explain why it cannot produce a prediction, e.g. "the sentence does not mention or directly reference the target", as shown in 4 or even "instruct the speaker to express opinion with respect and empathy", as shown in 5. These explanations help us select the innately flawed data in the dataset, for which no model and even no human can accurately decide the stance only by the given information. For those flawed tweets, it is still possible to determine the stance of it by fixing the issue in the following conversation. In a multi-round conversation with ChatGPT, we can feed a variety of information to the model including background knowledge, missing part of the sentence, stance classification examples and etc. Future investigation on how to design a multi-round conversation may further improve the performance of ChatGPT model on more NLP tasks including stance detection.