A Unified Generative Framework for Aspect-Based Sentiment Analysis

Introduction

Aspect-based Sentiment Analysis (ABSA) is the fine-grained Sentiment Analysis (SA) task, which aims to identify the aspect term ($a$), its corresponding sentiment polarity ($s$), and the opinion term ($o$). For example, in the sentence “The drinks are always well made and wine selection is fairly priced”, the aspect terms are “drinks” and “wine selection”, and their sentiment polarities are both “positive”, and the opinion terms are “well made” and “fairly priced”.

Based on the combination of the $a$, $s$, $o$, there exist seven subtasks in ABSA. We summarize these subtasks in Figure 1. Specifically, their definitions are as follows:

$\bullet$ Aspect Term Extraction(AE): Extracting all the aspect terms from a sentence.

$\bullet$ Opinion Term Extraction (OE): Extracting all the opinion terms from a sentence.

$\bullet$ Aspect-level Sentiment Classification (ALSC): Predicting the sentiment polarities for every given aspect terms in a sentence.

$\bullet$ Aspect-oriented Opinion Extraction (AOE): Extracting the paired opinion terms for every given aspect terms in a sentence.

$\bullet$ Aspect Term Extraction and Sentiment Classification (AESC): Extracting the aspect terms as well as the corresponding sentiment polarities simultaneously.

$\bullet$ Pair Extraction (Pair): Extracting the aspect terms as well as the corresponding opinion terms simultaneously.

$\bullet$ Triplet Extraction (Triplet): Extracting all aspects terms with their corresponding opinion terms and sentiment polarity simultaneously.

Although these ABSA subtasks are strongly related, most of the existing work only focus 1$\sim$3 subtasks individually. The following divergences make it difficult to solve all subtasks in a unified framework.

Input: Some subtasks ( AE, OE, AESC, Pair and Triplet) only take the text sentence as input, while the remained subtasks ( ALSC and AOE) take the text and a given aspect term as input.

Output: Some tasks (AE, OE, ALSC, AOE) only output a certain type from $a$, $s$ or $o$, while the remained tasks (AESC, Pair and Triplet) return compound output as the combination of $a$, $s$ and $o$.

Task Type: There are two kinds of tasks: extraction task (extracting aspect and opinion) and classification task (predicting sentiment).

Because of the above divergences, a myriad of previous works only focus on the subset of these subtasks. However, the importance of solving the whole ABSA subtasks in a unified framework remains significant. Recently, several works make attempts on this track. Some methods(Peng et al., 2020; Mao et al., 2021) apply the pipeline model to output the $a$, $s$, $o$ from the inside sub-models separately. However, the pipeline process is not end-to-end. Another line follows the sequence tagging method by extending the tagging schema (Xu et al., 2020). However, the compositionality of candidate labels hinders the performance. In conclusion, the existing methods can hardly solve all the subtasks by a unified framework without relying on the sub-models or changing the model structure to adapt to all ABSA subtasks.

Motivated by the above observations, we propose a unified generative framework to address all the ABSA subtasks. We first formulate all these subtasks as a generative task, which could handle the obstacles on the input, output, and task type sides and adapt to all the subtasks without any model structure changes. Specifically, we model the extraction and classification tasks as the pointer indexes and class indexes generation, respectively. Based on the unified task formulation, we use the sequence-to-sequence pre-trained model BART (Lewis et al., 2020) as our backbone to generate the target sequence in an end-to-end process. To validate the effectiveness of our method, we conduct extensive experiments on public datasets. The comparison results demonstrate that our proposed framework outperforms most state-of-the-art (SOTA) models in every subtask.

In summary, our main contributions are as follows:

$\bullet$ We formulate both the extraction task and classification task of ABSA into a unified index generation problem. Unlike previous unified models, our method needs not to design specific decoders for different output types.

$\bullet$ With our re-formulation, all ABSA subtasks can be solved in sequence-to-sequence framework, which is easy-to-implement and can be built on the pre-trained models, such as BART.

$\bullet$ We conduct extensive experiments on four public datasets, and each dataset contains a subset of all ABSA subtasks. To the best of our knowledge, it is the first work to evaluate a model on all ABSA tasks.

$\bullet$ The experimental results show that our proposed framework significantly outperforms recent SOTA methods.

Background

In this section, we first review the existing studies on single output subtasks, and then turn to studies focusing on the compound output subtasks.

Some researches mainly focus on the single output subtasks. The AE, OE, ALSC and AOE subtasks only output one certain type from $a$, $s$ or $o$.

AE Most studies treat AE subtask as a sequence tagging problem (Li and Lam, 2017; Xu et al., 2018; Li et al., 2018b). Recent works explore sequence-to-sequence learning on AE subtask, which obtain promissing results especially with the pre-training language models (Ma et al., 2019; Li et al., 2020).

OE Most studies treat OE subtask as an auxiliary task (Wang et al., 2016a, 2017; Wang and Pan, 2018; Chen and Qian, 2020; He et al., 2019). Most works can only extract the unpaired aspect and opinion termsIt is also referred to as the AE-OE co-Extraction.. In this case, opinion terms are independent of aspect terms.

ALSC Tang et al. (2016a) use the long short term memory (LSTM) network to enhance the interactions between aspects and context words. Wang et al. (2016b); Liu and Zhang (2017); Ma et al. (2017); Tay et al. (2018) incorporate the attention mechanism into the LSTM-based neural network models to model relations of aspects and their contextual words. Other model structures such as convolutional neural network (CNN) (Li et al., 2018a; Xue and Li, 2018), gated neural network (Zhang et al., 2016; Xue and Li, 2018), memory neural network (Tang et al., 2016b; Chen et al., 2017) have also been applied.

AOE This subtask is first introduced by Fan et al. (2019) and they propose the datasets for this subtask. Most studies apply sequence tagging method for this subtask (Wu et al., 2020; Pouran Ben Veyseh et al., 2020).

1.2 Compound Output Subtasks

Some researchers pay more attention and efforts to the subtasks with compound output. We review them as follows:

AESC. One line follows pipeline method to solve this problem. Other works utilize unified tagging schema (Mitchell et al., 2013; Zhang et al., 2015; Li et al., 2019) or multi-task learning (He et al., 2019; Chen and Qian, 2020) to avoid the error-propagation problem (Ma et al., 2018). Span-based AESC works are also proposed recently (Hu et al., 2019), which can tackle the sentiment inconsistency problem in the unified tagging schema.

Pairs Zhao et al. (2020) propose to extract all ($a$, $o$) pair-wise relations from scratch. They propose a multi-task learning framework based on the span-based extraction method to handle this subtask.

Triplet This subtask is proposed by Peng et al. (2020) and gains increasing interests recently. Xu et al. (2020) design the position-aware tagging schema and apply model based on CRF (Lafferty et al., 2001) and Semi-Markov CRF (Sarawagi and Cohen, 2004). However, the time complexity limits the model to detect the aspect term with long-distance opinion terms. Mao et al. (2021) formulate Triplet as a two-step MRC problem, which applies the pipeline method.

2 Sequence-to-Sequence Models

The sequence-to-sequence framework has been long studied in the NLP field to tackle various tasks Sutskever et al. (2014); Cho et al. (2014); Vinyals et al. (2015); Luong et al. (2015). Inspired by the success of PTMs (pre-trained models) Qiu et al. (2020); Peters et al. (2018); Devlin et al. (2019); Brown et al. (2020), Song et al. (2019); Raffel et al. (2020); Lewis et al. (2020) try to pre-train sequence-to-sequence models. Among them, we use the BART Lewis et al. (2020) as our backbone, while the other sequence-to-sequence pre-training models can also be applied in our architecture to use the pointer mechanism Vinyals et al. (2015), such as MASS Song et al. (2019).

BART is a strong sequence-to-sequence pre-trained model for Natural Language Generation (NLG). BART is a denoising autoencoder composed of several transformer Vaswani et al. (2017) encoder and decoder layers. It is worth noting that the BART-Base model contains a 6-layer encoder and 6-layer decoder, which makes it similar number of parametersBecause of the cross-attention between encoder and decoder, the number of parameters of BART is about 10% larger than its counterpart of BERT Lewis et al. (2020). with the BERT-Base model. BART is pretrained on denoising tasks where the input sentence is noised by some methods, such as masking and permutation. The encoder takes the noised sentence as input, and the decoder will restore the original sentence in an autoregressive manner.

Methodology

Although there are two types of tasks among the seven ABSA subtasks, they can be formulated under a generative framework. In this part, we first introduce our sequential representation for each ABSA subtask. Then we detail our method, which utilizes BART to generate these sequential representations.

$\bullet$ AE : $Y=[a_{1}^{s},a_{1}^{e},...,a_{i}^{s},a_{i}^{e},...]$,

$\bullet$ OE : $Y=[o_{1}^{s},o_{1}^{e},...,o_{i}^{s},o_{i}^{e},...]$,

$\bullet$ AESC : $Y=[a_{1}^{s},a_{1}^{e},s^{p}_{1},...,a_{i}^{s},a_{i}^{e},s^{p}_{i},...]$,

$\bullet$ Pair: $Y=[a_{1}^{s},a_{1}^{e},o_{1}^{s},o_{1}^{e},...,a_{i}^{s},a_{i}^{e},o_{i}^{s},o_{i}^{e},$$...]$,

$\bullet$ Triplet : $Y=[a_{1}^{s},a_{1}^{e},o_{1}^{s},o_{1}^{e},s^{p}_{1},...,a_{i}^{s},a_{i}^{e},o_{i}^{s},$ $o_{i}^{e},s^{p}_{i},...]$,

The above subtasks only rely on the input sentence, while for the ALSC and AOE subtasks, they also depend on a specific aspect term $a$. Instead of putting the aspect term on the input side, we put them on the target side so that the target sequences are as follows:

$\bullet$ ALSC : $Y=[\underline{a^{s}},\underline{a^{e}},s^{p}]$,

$\bullet$ AOE : $Y=[\underline{a^{s}},\underline{a^{e}},o_{1}^{s},o_{1}^{e},...,o_{i}^{s},o_{i}^{e},...]$, where the underlined tokens are given during inference. Detailed target sequence examples for each subtask are presented in Figure 3.

2 Our Model

As our discussion in the last section, all subtasks can be formulated as taking the $X=[x_{1},...,x_{n}]$ as input and outputting a target sequence $Y=[y_{1},...,y_{m}]$, where $y_{0}$ is the start-of-the-sentence token. Therefore, different ABSA subtasks can be formulated as:

To get the index probability distribution $P_{t}=P(y_{t}|X,Y_{<t})$ for each step, we use a model composed of two components: (1) Encoder; (2) Decoder.

Encoder The encoder part is to encode $X$ into vectors $\mathbf{H}^{e}$. We use the BART model, therefore, the start of sentence ($<$s$>$) and the end of sentence ($<$/s$>$) tokens will be added to the start and end of $X$, respectively. We ignore the $<$s$>$ token in our equations for simplicity. The encoder part is as follows:

where $C=[c_{1},...,c_{l}]$ is the class token listIn our implement, $y_{t}\in[1,n+l]$. The $x_{1}$ has the pointer index 1. .

After that, we use the BART decoder to get the last hidden state

During the training phase, we use the teacher forcing to train our model and the negative log-likelihood to optimize the model. Moreover, during the inference, we use the beam search to get the target sequence $Y$ in an autoregressive manner. After that, we need to use the decoding algorithm to convert this sequence into the term spans and sentiment polarity. We use the Triplet task as an example and present the decoding algorithm in Algorithm 1, the decoding algorithm for other tasks are much depicted in the Supplementary Material.

Experiments

We evaluate our method on four ABSA datasets. All of them are originated from the Semeval Challenges (Pontiki et al., 2014a, b, c), where only the aspect terms and their sentiment polarities are labeled.

The first dataset($\mathcal{D}_{17}$Each dataset only contains a subset of all ABSA subtasks. We use the published year of the dataset to distinguish them.) is annotated by Wang et al. (2017), where the unpaire opinion terms are labeled. The second dataset($\mathcal{D}_{19}$) is annotated by Fan et al. (2019), where they pair opinion terms with corresponding aspects. The third dataset($\mathcal{D}_{20a}$) is from Peng et al. (2020). They refine the data in <$a$, $o$, $s$> triplet form. The fourth dataset($\mathcal{D}_{20b}$) from Xu et al. (2020) is the revised variant of Peng et al. (2020), where the missing triplets with overlapping opinions are corrected. We present the statistics for these four datasets in Table 1.

2 Baselines

To have a fair comparison, we summarize top-performing baselines of all ABSA subtasks. Given different ABSA subtasks, datasets, and experimental setups, existing baselines can be separated into three groups roughly as shown in Table 2.

The baselines in the first group are conducted on $\mathcal{D}_{17}$ dataset, covering the AE, OE, ALSC, and AESC subtasks. Span-based method SPAN-BERT (Hu et al., 2019) and sequence tagging method, IMN-BERT (He et al., 2019) and RACL-BERT (Chen and Qian, 2020), are selected. Specifically, the IMN-BERT model is reproduced by Chen and Qian (2020). All these baselines are implemented on BERT-Large.

The baselines of the second group are conducted on $\mathcal{D}_{19}$ dataset, mainly focusing on AOE subtask. Interestingly, we find that sequence tagging method is the main solution for this subtask (Fan et al., 2019; Wu et al., 2020; Pouran Ben Veyseh et al., 2020).

The baselines of the third group are mainly conducted on $\mathcal{D}_{20a}$ and $\mathcal{D}_{20b}$ datasets, which could cover almost all the ABSA subtasks except for one certain subtask depending on the baseline structures. For the following baselines: RINANTE (Dai and Song, 2019), CMLA (Wang et al., 2017), Li-unified (Li et al., 2019), the suffix “+” in Table 2 denotes the corresponding model variant modified by Peng et al. (2020) for being capable of AESC, Pair and Triplet.

3 Implement Details

Following previous studies, we use different metrics according to different subtasks and datasets. Specifically, for the single output subtasks AE, OE, and AOE, the prediction span would be considered as correct only if it exactly matches the start and the end boundaries. For the ALSC subtask, we require the generated sentiment polarity of the given aspect should be the same as the ground truth. As for compound output subtasks, AESC, Pair and Triplet, a prediction result is correct only when all the span boundaries and the generated sentiment polarity are accurately identified. We report the precision (P), recall (R), and F1 scores for all experimentsDue to the limited space, we would present detailed experiments for each dataset in the Supplementary Material..

4 Main Results

On $\mathcal{D}_{17}$ dataset (Wang et al., 2017), we compare our method for AE, OE, ALSC, and AESC. The comparison results are shown in Table 3. Most of our results achieve better or comparable results to baselines. However, these baselines yield competitive results based on the BERT-Large pre-trained models. While our results are achieved on the BART-Base model with almost half parameters. This shows that our framework is more suitable for these ABSA subtasks.

On $\mathcal{D}_{19}$ dataset (Fan et al., 2019), we compare our method for AOE. The comparison results are shown in Table 4. We can observe that our method achieves significant P/R/F1 improvements on 14res, 15res, and 16res. Additionally, we notice that our F1 score on 14lap is close to the previous SOTA result. This is probably caused by the dataset domain difference as the 14lap is the laptop comments while the others are restaurant comments.

On $\mathcal{D}_{20a}$ dataset (Peng et al., 2020), we compare our method for AESC, Pair, and Triplet. The comparison results are shown in Table 5. We can observe that our proposed method is able to outperform other baselines on all datasets. Specifically, we achieve the better results for Triplet, which demonstrates the effectiveness of our method on capturing interactions among aspect terms, opinion terms, and sentiment polarities. We also observe that the Span-based methods show superior performance to sequence tagging methods. This may be caused by the higher compositionality of candidate labels in sequence tagging methods (Hu et al., 2019). As the previous SOTA method, the Dual-MRC shows competitive performance by utilizing the span-based extraction method and the MRC mechanism. However, their inference process is not an end-to-end process.

On $\mathcal{D}_{20b}$ dataset (Xu et al., 2020), we compare our method for Triplet. The comparison results can be found in Table 6. Our method achieves the best results with nearly 7 F1 points improvements on 14res, 15res, and 16res. Our method achieves nearly 13, 9, 7, 12 points improvements on each dataset for the recall scores compared with other baselines. This also explains the drop performance of the precision score. Since $\mathcal{D}_{20b}$ is refined from $\mathcal{D}_{20a}$, we specifically compare the Triplet results of the corresponding dataset in $\mathcal{D}_{20a}$ and $\mathcal{D}_{20b}$. Interestingly, we discover that all baselines have a much bigger performance change on 15res. We conjecture the distribution differences may be the cause reason. In conclusion, all the experiment results confirm that our proposed method, which unifies the training and the inference to an end-to-end generative framework, provides a new SOTA solution for the whole ABSA task.

Framework Analysis

To better understand our proposed framework, we conduct analysis experiments on the $\mathcal{D}_{20b}$ dataset (Xu et al., 2020).

To validate whether our proposed framework could adapt to the generative ABSA task, we metric the invalid predictions for the Triplet. Specifically, since the Triplet requires the prediction format like $[a^{s},a^{e},o^{s},o^{e},s^{p}]$, it is mandatory that one valid triplet prediction should be in length 5, noted as “5-len”, and obviously all end index should be larger than the corresponding start index, noted as “ordered prediction”. We calculate $\frac{number\ of\ non-5-len}{total\ prediction}$, referred to as the “Invalid size”, and the $\frac{number\ of\ non-ordered\ prediction}{total\ 5-len\ prediction}$, referred to as the “Invalid order”. The “Invalid token” means the $a^{s}$ is not the start of a token, instead, it is the index of an inside subword. From Table 7, we can observe that BART could learn this task form easily as the low rate for all the three metrics, which demonstrate that the generative framework for ABSA is not only a theoretically unified task form but also a realizable framework in practical. We remove these invalid predictions in our implementation of experiments.

As shown in Table 4, we give some analysis on the impact of the beam size, as we are a generation method. However, the beam size seems to have little impact on the F1 scores.

Conclusion

This paper summarizes the seven ABSA subtasks and previous studies, which shows that there exist divergences on all the input, output, and task type sides. Previous studies have limitations on handling all these divergences in a unified framework. We propose to convert all the ABSA subtasks to a unified generative task. We implement the BART to generate the target sequence in an end-to-end process based on the unified task formulation. We conduct massive experiments on public datasets for seven ABSA subtasks and achieve significant improvements on most datasets. The experimental results demonstrate the effectiveness of our method. Our work leads to several promising directions, such as sequence-to-sequence framework on other tasks, and data augmentation.

Acknowledgements

We would like to thank the anonymous reviewers for their insightful comments. The discussion with colleagues in AWS Shanghai AI Lab was quite fruitful. We also thank the developers of fastNLPhttps://github.com/fastnlp/fastNLP. FastNLP is a natural language processing python package. and fitloghttps://github.com/fastnlp/fitlog. Fitlog is an experiment tracking package. This work was supported by the National Key Research and Development Program of China (No. 2020AAA0106700) and National Natural Science Foundation of China (No. 62022027).

Ethical Considerations

For the consideration of ethical concerns, we would make detailed description as follows:

(1) All the experiments are conducted on existing datasets, which are derived from public scientific papers.

(2) We describe the characteristics of the datasets in a specific section. Our analysis is consistent with the results.

(3) Our work does not contain identity characteristics. It does not harm anyone.

(4) Our experiments do not need a lot of computer resources compared to pre-trained models.

References

Appendix A Supplemental Material

We use the triangular learning rate warmup. All experiments are conducted in the Nvidia Ge-Force RTX-3090 Graphical Card with 24G graphical memory.

The averages running time for experiments on each dataset is less than 15 minutes. The number of parameters is as follows:

$\bullet$ BART-Base model: 12 layers, 768 hidden dimensions and 16 heads with the total number of parameters, 139M;

$\bullet$ BERT-Base model: 12 layers, 768 hidden dimensions and 12 heads with the total number of parameters, 110M.

A.2 Decoding Algorithm for Different Datasets

In this part, we introduce the decoding algorithm we used to convert the predicted target sequence $Y$ into the target span set $L$. These algorithm can be found in Algorithm 2, 3, 4.

A.3 Detailed Experimental Setup

As the different subtasks are conducted on different datasets, specifically, we conduct the following experiments on each dataset:

$\bullet$ On the $\mathcal{D}_{17}$ dataset, we conduct the AESC and the OE in multi-task learning method. To that end, we feed the pre-defined task tags “$<$AESC$>$” and “$<$OE$>$” to the decoder first. For example, for the input “The drinks are always well made and wine selection is fairly priced” from $\mathcal{D}_{17}$ dataset, we define the AESC sequence and the OE target sequence as “$<$AESC$>$, 1, 1, POS, 7, 8, POS, $<$/s$>$” and “$<$OE$>$, 4, 5, 10, 11, $<$/s$>$”.

$\bullet$ On the $\mathcal{D}_{19}$ dataset, we conduct the AOE. As the AOE subtask requires to detect the opinion terms given aspect terms in advance, the aspect terms need to be fed to our decoder first. For the aforementioned example sentence from $\mathcal{D}_{19}$ dataset, we define the AOE target sequence as “ 1, 1, 4, 5, $<$/s$>$” and the “ 7, 8, 10, 11, $<$/s$>$”.

$\bullet$ On the $\mathcal{D}_{20a}$ and $\mathcal{D}_{20b}$ datasets, we conduct the Triplet Extraction. For the aforementioned example sentence from $\mathcal{D}_{20a}$ and $\mathcal{D}_{20b}$ dataset, we define the Triplet target sequence as “1, 1, 4, 5, POS, 7, 8, 10, 11, POS, $<$/s$>$”. Specific Subtask Metrics

$\bullet$ On the $\mathcal{D}_{17}$ dataset, we get the AESC and OE results directly. Following previous work, we only calculate the metrics for AESC and ALSC from those true positive AE predictions. Specifically, the F1

$\bullet$ On the $\mathcal{D}_{19}$ dataset, we get the AOE results directly. The metrics for AOE are standard Precision, Recall and the F1 score.

$\bullet$ On the $\mathcal{D}_{20a}$ and $\mathcal{D}_{20b}$ datasets, we get the Triplet results directly. We preserve the $<$AT,OT$>$ for Pair metric and $<$AT, SP$>$ for AESC metric. The metrics for them are standard Precision, Recall and the F1 score.