POINTER: Constrained Progressive Text Generation via Insertion-based Generative Pre-training

Introduction

Real-world editorial assistant applications must often generate text under specified lexical constraints, for example, convert a meeting note with key phrases into a concrete meeting summary, recast a user-input search query as a fluent sentence, generate a conversational response using grounding facts Mou et al. (2016), or create a story using a pre-specified set of keywords Fan et al. (2018); Yao et al. (2019); Donahue et al. (2020).

Generating text under specific lexical constraints is challenging. Constrained text generation broadly falls into two categories, depending on whether inclusion of specified keywords in the output is mandatory. In soft-constrained generation Qin et al. (2019); Tang et al. (2019), keyword-text pairs are typically first constructed (sometimes along with other conditioning information), and a conditional text generation model is trained to capture their co-occurrence, so that the model learns to incorporate the constrained keywords into the generated text. While soft-constrained models are easy to design, even remedied by soft enforcing algorithms such as attention and copy mechanisms Bahdanau et al. (2015); Gu et al. (2016); Chen et al. (2019), keywords are still apt to be lost during generation, especially when multiple weakly correlated keywords must be included.

Hard-constrained generation Hokamp and Liu (2017); Post and Vilar (2018); Hu et al. (2019); Miao et al. (2019); Welleck et al. (2019), on the other hand, requires that all the lexical constraints be present in the output sentence. This approach typically involves sophisticated design of network architectures. Hokamp and Liu (2017) construct a lexical-constrained grid beam search decoding algorithm to incorporate constraints. However, Hu et al. (2019) observe that a naive implementation of this algorithm has a high running time complexity. Miao et al. (2019) introduces a sampling-based conditional generation method, where the constraints are first placed in a template, then words in a random position are either inserted, deleted or updated under a Metropolis-Hastings-like scheme. However, individually sampling each token results in slow convergence, as the joint distribution of all the tokens in a sentence is highly correlated. Welleck et al. (2019) propose a tree-based text generation scheme, where a token is first generated in an arbitrary position, and then the model recursively generates words to its left and right, yielding a binary tree. However, the constructed tree may not reflect the progressive hierarchy/granularity from high-level concepts to low-level details. Further, the time complexity of generating a sentence is $\mathcal{O}(n)$, like standard auto-regressive methods.

Motivated by the above, we propose a novel non-autoregressive model for hard-constrained text generation, called Pointer (PrOgressive INsertion-based TransformER). As illustrated in Table 1, generation of words in Pointer is progressive, and iterative. Given lexical constraints, Pointer first generates high-level words (e.g., nouns, verbs and adjectives) that bridge the keyword constraints, then these words are used as pivoting points at which to insert details of finer granularity. This process iterates until a sentence is finally completed by adding the least informative words (typically pronouns and prepositions).

Due to the resemblance to the masked language modeling (MLM) objective, BERTDevlin et al. (2019) can be naturally utilized for initialization. Further, we perform large-scale pre-training on a large Wikipedia corpus to obtain a pre-trained Pointer model that which can be readily fine-tuned on specific downstream tasks.

The main contributions of this paper are summarized as follows. ($i$) We present Pointer, a novel insertion-based Transformer model for hard-constrained text generation. Compared with previous work, Pointer allows long-term control over generation due to the top-down progressive structure, and enjoys a significant reduction over emperical time complexity from $\mathcal{O}(n)$ to $\mathcal{O}(\log n)$ at best. ($ii$) Large-scale pre-training and novel beam search algorithms are proposed to further boost performance. ($iii$) We develop a novel beam search algorithm customized to our approach, further improving the generation quality. ($iv$) Experiments on several datasets across different domains (including News and Yelp) demonstrates the superiority of Pointer over strong baselines. Our approach is simple to understand and implement, yet powerful, and can be leveraged as a building block for future research.

Related Work

Large-scale pre-trained language models, such as BERT Devlin et al. (2019), RoBERTa Liu et al. (2019), XLNet Yang et al. (2019), Text-to-text Transformer Raffel et al. (2019) and ELECTRA Clark et al. (2020), have achieved great success on natural language understanding benchmarks. GPT-2 Radford et al. (2018) first demonstrates great potential for leveraging Transformer models in generating realistic text. MASS Song et al. (2019) and BART Lewis et al. (2019) propose methods for sequence-to-sequence pre-training. UniLM Dong et al. (2019) unifies the generation and understanding tasks within a single pre-training scheme. DialoGPT Zhang et al. (2020) and MEENA Adiwardana et al. (2020) focus on open-domain conversations. CTRL Keskar et al. (2019) and Grover Zellers et al. (2019) guide text generation with pre-defined control codes. To the best of our knowledge, ours is the first large-scale pre-training work for hard-constrained text generation.

Non-autoregressive Generation Many attempts have been made to use non-autoregressive models for text generation tasks. For neural machine translation, the promise of such methods mostly lies in their decoding efficiency. For example, Gu et al. (2018) employs a non-autoregressive decoder that generates all the tokens simultaneously. Generation can be further refined with a post-processing step to remedy the conditional independence of the parallel decoding process Lee et al. (2018); Ghazvininejad et al. (2019); Ma et al. (2019); Sun et al. (2019); Kasai et al. (2020). Deconvolutional decoders Zhang et al. (2017); Wu et al. (2019) have also been studied for title generation and machine translation. The Insertion Transformer Stern et al. (2019); Gu et al. (2019); Chan et al. (2019) is a partially autoregressive model that predicts both insertion positions and tokens, and is trained to maximize the entropy over all valid insertions, providing fast inference while maintaining good performance. Our Pointer model hybridizes the BERT and Insertion Transformer models, inheriting the advantages of both, and generates text in a progressive coarse-to-fine manner.

Method

Let $X=\{x_{0},x_{1},\cdots,x_{T}\}$ denote a sequence of discrete tokens, where each token $x_{t}\in V$, and $V$ is a finite vocabulary set. For the hard-constrained text generation task, the goal is to generate a complete text sequence $X$, given a set of key words $\hat{X}$ as constraints, where the key words have to be exactly included in the final generated sequence with the same order.

Let us denote the lexical constraints as $X^{0}=\hat{X}$. The generation procedure of our method can be formulated as a (progressive) sequence of $K$ stages: $S=\{X^{0},X^{1},\cdots,X^{K-1},X^{K}\}$, such that for each $k\in\{1,\ldots,K\}$, $X^{k-1}$ is a sub-sequence of $X^{k}$. The following stage can be perceived as a finer-resolution text sequence compared to the preceding stage. $X^{K}$ is the final generation, under the condition that the iterative procedure is converged (i.e., $X^{K-1}=X^{K}$).

Table 1 shows an example of our progressive text generation process. Starting from the lexical constraints ($X_{0}$), at each stage, the algorithm inserts tokens progressively to formulate the target sequence. At each step, at most one new token can be generated between two existing tokens. Formally, we propose to factorize the distribution according to the importance (defined later) of each token:

where $p(X^{k}|X^{k-1})=\prod_{x\in X^{k}-X^{k-1}}p(x|X^{k-1})$. The more important tokens that form the skeleton of the sentence, such as nouns and verbs, appear in earlier stages, and the auxiliary tokens, such as articles and prepositions, are generated at the later stages. In contrast, the autoregressive model factorizes the joint distribution of $X$ in a standard left-to-right manner, i.e., $p(X)=p(x_{0})\prod_{t=1}^{T}p(x_{t}|x_{<t})$, ignoring the word importance. Though the Insertion Transformer Stern et al. (2019) attempts to implement the progressive generation agenda in (1), it does not directly address how to train the model to generate important tokens first.

2 Data Preparation

Designing a loss function so that $(i)$ generating an important token first and $(ii)$ generating more tokens at each stage that would yield a lower loss would be complicated. Instead, we prepare data in a form that eases model training.

The construction of data-instance pairs reverses the generation process. We construct pairs of text sequences at adjacent stages, i.e., $(X^{k-1},X^{k})$, as the model input. Therefore, each training instance $X$ is broken into a consecutive series of pairs: $(X^{0},X^{1}),\cdots,(X^{K-1},X^{K})$, where $K$ is the number of such pairs. At each iteration, the algorithm masks out a proportion of existing tokens $X^{k}$ to yield a sub-sequence $X^{k-1}$, creating a training instance pair $(X^{k-1},X^{k})$. This procedure is iterated until only less than $c$ ($c$ is small) tokens are left.

Two properties are desired when constructing data instances: ($i$) important tokens should appear in an earlier stage , so that the generation follows a progressive manner; ($ii$) the number of stages $K$ is small, thus the generation is fast at inference time.

Token Importance Scoring We consider three different schemes to assess the importance score of a token: term frequency-inverse document frequency (TF-IDF), part-of-speech (POS) tagging, and Yet-Another-Keyword-Extractor (YAKE) Campos et al. (2018, 2020). The TF-IDF score provides the uniqueness and local enrichment evaluation of a token at a corpus level. POS tagging indicates the role of a token at a sequence level. We explicitly assign noun or verb tokens a higher POS tagging score than tokens from other categories. YAKE is a commonly used unsupervised automatic keyword extraction method that relies on statistical features extracted from single documents to select the most important keywords Campos et al. (2020). YAKE is good at extracting common key words, but relatively weak at extracting special nouns (e.g., names), and does not provide any importance level for non-keyword tokens. Therefore, we combine the above three metrics for token importance scoring. Specifically, the overall score $\alpha_{t}$ of a token $x_{t}$ is defined as $\alpha_{t}=\alpha^{\text{TF-IDF}}_{t}+\alpha^{\text{POS}}_{t}+\alpha^{\text{YAKE}}_{t}$, where $\alpha^{\text{TF-IDF}}_{t}$, $\alpha^{\text{POS}}_{t}$ and $\alpha^{\text{YAKE}}_{t}$ represent the TF-IDF, POS tagging and YAKE scores (each is rescaled to $$), respectively.

Additionally, stop words are manually assigned a low importance score. If a token appears several times in a sequence, the latter occurrences are assigned a decayed importance score to prevent the model from generating the same token multiple times in one step at inference time. We note that our choice of components of the importance score is heuristic. It would be better to obtain an unbiased/oracle assessment of importance, which we leave for future work.

DP-based Data Pair Construction Since we leverage the Insertion-based Transformer, which allows at most one new token to be generated between each two existing tokens, sentence length at most doubles at each iteration. Consequently, the optimal number of iterations $K$ is $\log(T)$, where $T$ is the length of the sequence. Therefore, generation efficiency can be optimized by encouraging more tokens to be discarded during each masking step when preparing the data. However, masking positional interleaving tokens ignores token importance, and thus loses the property of progressive planning from high-level concepts to low-level details at inference time. In practice, sequences generated by such an approach can be less semantically consistent as less important tokens occasionally steer generation towards random content.

We design an approach to mask the sequence by considering both token importance and efficiency using dynamic programming (DP). To accommodate the nature of insertion-based generation, the masking procedure is under the constraint that no consecutive tokens can be masked at the same stage. Under such a condition, we score each token and select a subset of tokens that add up to the highest score (all scores are positive). This allows the algorithm to adaptively choose as many high scored tokens as possible to mask.

Formally, as an integer linear programming problem Richards and How (2002), the objective is to find an optimal masking pattern $\Phi=\{\phi_{1},\cdots,\phi_{T}\}$, where $\phi_{t}\in\{0,1\}$, and $\phi_{t}=1$ represents discarding the corresponding token $x_{t}$, and $\phi_{t}=0$ indicates $x_{t}$ remains. For a sequence $X^{\prime}$, the objective can be formulated as:

where $\alpha_{\max}=\max_{t}\{\alpha_{t}\}$. Though solving Eq. (3.2) is computationally expensive, one can resort to an analogous problem for a solution, the so-called House Robbery Problem, a variant of Maximum Subarray Problem Bentley (1984), where a professional burglar plans to rob houses along a street and tries to maximize the outcome, but cannot break into two adjacent houses without triggering an alarm. This can be solved using dynamic programming Bellman (1954) (also known as Kadane’s algorithm Gries (1982)) as shown in Algorithm 1.

3 Model Training

With all the data-instance pairs $(X^{k-1},X^{k})$ created as described above as the model input, we optimize the following objective:

where $X^{+}\triangleq X^{k}-X^{k-1}$, and $\Phi^{k-1}$ denotes an indicator vector in the $k$-th stage, representing whether an insertion operation is applied in a slot.

As illustrated in Figure 1, while the MLM objective in BERT only predicts the token of a masked placeholder, our objective comprises both ($i$) likelihood of an insertion indicator for each slot (between two existing tokens), and ($ii$) the likelihood of each new token conditioning on the activated slot. To handle this case, we expand the vocabulary with a special no-insertion token $\mathtt{[NOI]}$. During inference time, the model can predict either a token from the vocabulary to insert, or an $\mathtt{[NOI]}$ token indicating no new token will be inserted at a certain slot at the current stage. By utilizing this special token, the two objectives are merged. Note that the same insertion transformer module is re-used at different stages. We empirically observed that the model can learn to insert different words at different stages; it presumably learns from the “completion level” (how discontinuous the context is) of the current context sequence to roughly estimate the progress up to that point.

During inference time, once in a stage ($X^{k}$), all the slots predict $\mathtt{[NOI]}$ for the next stage, the generation procedure is converged and $X^{k}$ is the final output sequence. Note that to account for this final stage $X^{k}$, during data preparation we incorporate an $(X,N)$ pair for each sentence in the training data, where $N$ denotes a sequence of $\mathtt{[NOI]}$ with the same length of $X$. To enable the model to insert at the beginning and end of the sequence, an $\mathtt{[SOS]}$ token and an $\mathtt{[EOS]}$ token are added in the beginning and at the end of each sentence, respectively.

In light of the similarity with the MLM objective, we use BERT model to initialize the Insertion Transformer module.

Large-scale Pre-training In order to provide a general large-scale pretrained model that can benefit various downstream tasks with fine-tuning, we train a model on the massive publicly available English Wiki dataset, which covers a wide range of topics. The Wiki dataset is first preprocessed according to Sec. 3.2. We then initialize the model with BERT, and perform model training on the processed data using our training objective (3). After pre-training, the model can be used to generate an appropriate sentence with open-domain keyword constraints, in a tone that represents the Wiki style. In order to adapt the pre-trained model to a new domain (e.g., News and Yelp reviews), the pre-trained model is further fine-tuned on new datasets, which empirically demonstrates better performance than training the model on the target domain alone.

4 Inference

During inference time, starting from the given lexical constraint $X^{0}$, the proposed model generates text stage-by-stage using greedy search or top-K sampling Fan et al. (2018), by applying the Insertion Transformer module repeatedly until no additional token is generated. If a $\mathtt{[NOI]}$ token is generated, it is deleted at the next round.

Inner-Layer Beam Search According to (3), all new tokens are simultaneously generated based on the existing tokens at the previous stage. Despite of being fully parallel, like BERT Yang et al. (2019) and NAT Ghazvininejad et al. (2019); Kasai et al. (2020) this approach suffers from a conditional independence problem in which the predicted tokens are conditional-independently generated and are agnostic of each other. This can result in generating repeating or inconsistent new tokens at each generation round.For example, from an existing token “and”, the model generates “clean and clean”.

To address this weak-dependency issue, we perform a modified beam search algorithm for decoding. Specifically, at stage $k$, suppose the existing tokens from last stage are $X^{k-1}=\{x^{k-1}_{1},\cdots,x^{k-1}_{T_{k-1}}\}$ , where $T_{k-1}$ is the length of $X^{k-1}$. For predicting next stage $X^{k}$, there will be $T_{k-1}$ available slots. A naive approach to perform beam search would be to maintain a priority queue of top $B$ candidate token series predictions when moving from the leftmost slot to the rightmost slot. At the $t$-th move, the priority queue contains top $B$ sequences for existing predicted tokens: $(s_{1}^{(b)},\cdots,s_{t-1}^{(b)})$, where $s_{i}^{(b)}$ denotes the predicted token for the $i$-th slot in the $b$-th ($b\in\{1,\cdots,B\}$) sequence. The model then evaluates the likelihood of each item (including $\mathtt{[NOI]}$) in the vocabulary for the slot $s_{t}$, by computing the likelihood of $(s_{1}^{(b)},x^{k-1}_{1},\cdots,s_{t-1}^{(b)},x^{k-1}_{t-1},s_{t},x^{k-1}_{t},\mathtt{[NOI]},\cdots$, $\mathtt{[NOI]}$,$x^{k-1}_{T_{k-1}})$. This is followed by a ranking step to select the top $B$ most likely series among the $VB$ series to grow. However, such a naive approach is expensive, as the runtime complexity takes $\mathcal{O}(TBV)$ evaluations.

Instead, we approximate the search by constraining it in a narrow band. We design a customized beam search algorithm for our model, called inner-layer beam search (ILBS). This method applies an approximate local beam search at each iteration to find the optimal stage-wise decoding. At the $t$-th slot, ILBS first generates top $B$ token candidates by applying one evaluation step based on existing generation. Prediction is limited to these top $B$ token candidates, and thus the beam search procedure as described above is applied on the narrow band of $B$ instead of the full vocabulary $V$. This reduces the computation to $\mathcal{O}(TB^{2})$.

Experiments

We evaluate the Pointer model on constrained text generation over News and Yelp datasets. Details of the datasets and experimental results are provided in the following sub-sections. The pre-trained models and the source code are available at Github https://github.com/dreasysnail/POINTER.

We evaluate our model on two datasets. The EMNLP2017 WMT News datasethttp://www.statmt.org/wmt17/ contains 268,586 sentences, and we randomly pick 10k sentences as the validation set, and 1k sentences as the test set. The Yelp English review dataset is from Cho et al. (2018), which contains 160k training examples, 10k validation examples and 1k test examples. These two datasets vary in sentence length and domain, enabling the assessment of our model in different scenarios.

The English Wikipedia dataset we used for pre-training is first pre-processed into a set of natural sentences, with maximum sequence length of 64 tokens, which results in 1.99 million sentences for model training in total (12.6 GB raw text). On average, each sentence contains 27.4 tokens.

For inference, we extract the testing lexical constraints for all the compared methods using the 3rd party extracting tool YAKEhttps://github.com/LIAAD/yake. The maximum length of the lexical constraints we used for News and Yelp is set to 4 and 7, respectively, to account the average length for News ($27.9\approx 4\times 2^{3}$) and Yelp ($50.3\approx 7\times 2^{3}$), as we would hope the generation can be done within 4 stages.

Baselines We compare our model with two state-of-the-art methods for hard-constrained text generation: ($i$) Non-Monotonic Sequential Text Generation (NMSTG) Welleck et al. (2019), and ($ii$) Constrained Sentence Generation by Metropolis-Hastings Sampling (CGMH) Miao et al. (2019). We also compared with an autoregressive soft-constraint baselineGao et al. (2020). Note that the Insertion Transformer Stern et al. (2019) focuses on machine translation rather than hard-constrained generation task, and therefore is not considered for comparison. Other methods based on grid beam search typically have long inference time, and they only operate on the inference stage; these are also excluded from comparison. For all compared system, we use the default settings suggested by the authors, the models are trained until the evaluation loss does not decrease. More details are provided in the Appendix.

Experiment Setups We employ the tokenizer and model architecture from BERT-base and BERT-large models for all the tasks. BERT models are used as our model initialization. Each model is trained until the validation loss is no longer decreasing. We use a learning rate of 3e-5 without any warming-up schedule for all the training procedures. The optimization algorithm is Adam Kingma and Ba (2015). We pre-train our model on the Wiki dataset for 2-4 epochs, and fine-tune on the News and Yelp datasets for around 10 epochs.

Evaluation Metrics Following Zhang et al. (2020), we perform automatic evaluation using commonly adopted text generation metrics, including BLEU Papineni et al. (2002), METEOR Lavie and Agarwal (2007), and NIST Doddington (2002). Following Kann et al. (2018), to assess the coherence of generated sentences, we also report the perplexity over the test set using pre-trained GPT-2 medium (large) modelhttps://github.com/openai/gpt-2. We use Entropy Zhang et al. (2018) and Dist-n Li et al. (2016) to evaluate lexical diversity.

2 Experimental Results

News Generation We first conduct experiments on the News dataset to generate sentences from 4 lexical constraints. Quantitative results are summarized in Table 2 (upper). Some qualitative examples including the progressive generations at each stage are provided in Table 3 and Appendix B. Pointer is able to take full advantage of BERT initialization and Wiki pre-training to improve relevance scores (NIST, BLEU and METEOR). Leveraging the ILBS or using a larger model further improves most automatic metrics we evaluated The ILBS for larger models performs similarly to greedy decoding, and thus is omitted from comparison. For diversity scores, as CGMH is a sampling-based method in nature, it achieves the highest Dist-n scores (even surpasses human score). We observed that the length of generated sentences, the diversity scores and the GPT-2 perplexity from Pointer are close to human oracle.

Yelp Generation We further evaluate our method on the Yelp dataset, where the goal is to generate a long-form text from more constraints. Generating a longer piece of text with more lexical constraints is generally more challenging, since the model needs to capture the long-term dependency structure from the text, and effectively conjure up with a plan to realize the generation. Results of automatic evaluation are provided in Table 2 (lower). Generated examples are shown in Table 4 and Appendix C. Generally, the generation from our model effectively considers all the lexical constraints, and is semantically more coherent and grammatically more fluent, compared with the baseline methods. The automatic evaluation results is generally consistent with the observations from News dataset, with an exception that Dist-n scores is much lower than the human Dist-n scores. Compared with greedy approach, at a cost of efficiency, ILBS is typically more concise and contains less repeated information, a defect the greedy approach occasionally suffers (e.g., Table 4, “delicious and delicious”).

For both datasets, most of the generations converges with in 4 stages. We perform additional experiments on zero-shot generation from the pre-trained model on both datasets, to test the versatility of pre-training. The generated sentences, albeit Wiki-like, are relatively fluent and coherent (see examples in Appendix B and C), and yield relatively high relevance scores (see Appendix E for details). Interestingly, less informative constraints are able to be expanded to coherent sentences. Given the constraint “is to from”, our model generates “it is oriented to its east, but from the west”.

The autoregressive soft-constraint baselineGao et al. (2020) has no guarantee that it will cover all keywords in the given order, thus we omit it in the Table 2. For this baseline, the percentage of keywords that appear in the outputs are 57% and 43% for News and Yelp datasets, respectively. With the similar model size (117M), this baseline’s performance is worse than ours approach in automatic metrics for News dataset (BLEU4: $2.99\to 1.74$; NIST4: $3.22\to 1.10$; METEOR: $16\%\to 9\%$; DIST2: $61\%\to 58\%$; PPL: $66\to 84$). The performance gap in Yelp dataset is even larger due to more lexical constraints.

Human Evaluation Using a public crowd-sourcing platform (UHRS), we conducted a human evaluation of 400 randomly sampled outputs (out of 1k test set) of CGMH, NMSTG and our base and large models with greedy decoding. Systems were paired and each pair of system outputs was randomly presented (in random order) to 5 crowd-sourced judges , who ranked the outputs pairwise for coherence, informativeness and fluency using a 5-point Likert-like scale. The human evaluation template is provided in Appendix G. The overall judge preferences for fluency, informativeness and semantic coherence are presented as percentages of the total ”vote” in Table 5. P-values are all p¡0.00001 (line 721), computed using 10000 bootstrap replications. For inter-annotator agreement, Krippendorff’s alpha is 0.23 on the News dataset and 0.18 on the Yelp dataset. Despite the noise, the judgments show a strong across-the-board preference for Pointer(base) over the two baseline systems on all categories. A clear preference for the human ground truth over our method is also observed. The base and large models show comparable human judge preferences on the News dataset, while human judges clearly prefer the large model on Yelp data (see Appendix D for more details).

Running-time Comparison One of the motivations of this work is that at each stage the generation can be parallel, leading to a significant reduction in training and inference. We compare the model training time and the inference decoding time of all the methods on the Yelp dataset, and summarize the results in Table 6. The evaluation is based on a single Nvidia V100 GPU. Training time for CGMH and Pointer is relatively fast, while NMSTG processes fewer tokens per second since it needs to generate a tree-like structure for each sentence. With respect to inference time, CGMH is slow, as it typically needs hundreds of sampling iterations to decode one sentence.

We note there is no theoretical guarantee of $\mathcal{O}(\log N)$ time complexity for our method. However, our approach encourages filling as many slots as possible at each stage, which permits enables the model to achieve an empirical $\mathcal{O}(\log N)$ speed. In our experiment 98% of generations end within 4 stages.

Note that our method in Table 6 uses greedy decoding.

ILBS is around 20 times slower than greedy. The large model is around 3 times slower than the base model.

Conclusion

We have presented Pointer, a simple yet powerful approach to generating text from a given set of lexical constraints in a non-autoregressive manner. The proposed method leverages a large-scale pre-trained model (such as BERT initialization and our insertion-based pre-training on Wikipedia) to generate text in a progressive manner using an insertion-based Transformer. Both automatic and human evaluation demonstrate the effectiveness of Pointer. In future work, we hope to leverage sentence structure, such as the use of constituency parsing, to further enhance the design of the progressive hierarchy. Our model can be also extended to allow inflected/variant forms and arbitrary ordering of given lexical constraints.

References

Appendix

Appendix A Baseline and Experimental Details

For NMSTG, we first convert the lexical constraints into a prefix sub-tree, and then sample a sentence to complete the sub-tree. We use the default settings suggested by the authors, and use an LSTM with hidden size of 1024 as the text generator, and select the best performed variants (annealed) as our baseline. For CGMH, we use their default setting, which uses an LSTM with hidden size of 300, and set the vocabulary size as 50k. Both models are trained until the evaluation loss does not decrease. During inference, we run CGMH for 500 iterations with default hyperparameters.

For experiment setup, we employ the tokenizer from BERT, and use WordPiece Embeddings Wu et al. (2016) with a 30k token vocabulary for all the tasks. A special no-insertion token $\mathtt{[NOI]}$ is added to the vocabulary. We utilize the BERT-base and BERT-large models with 12 self-attention layers and 768 hidden dimensions as our model initialization. Each model is trained until there is no progress on the validation loss. We use a learning rate of 3e-5 without any warming-up schedule for all the training procedures. The optimization algorithm is Adam Kingma and Ba (2015). We pre-train our model on the Wiki dataset for 2 epochs, and fine-tune on the News and Yelp datasets for around 10 epochs.

Appendix B Additional Generated Examples for News Dataset

We provide two examples on News dataset for how the model progressively generates the sentences in Table 7. All the generations are from the Pointer large model using greedy decoding.

In this section, we also provide some additional examples from the 1k news test data.

Appendix C Additional Generated Examples for Yelp Dataset

We provide two examples on Yelp dataset for how the model progressively generates the sentences in Table 8. All the generations are from the Pointer large model using greedy decoding.

We also provide some additional examples from the Yelp test set. The results includes keywords, human oracle, CGMH, NMSTG and our models. For our models, we include Pointer base and large models with greedy decoding and base model with ILBS. The large model with ILBS is time consuming so we omit them from the comparison.

Appendix D Additional Human Evaluation information and Results

There were 145 judges in all: 5 judges evaluated each pair of outputs to be reasonably robust against spamming. P-values are all p¡0.00001 (line 721), computed using 10000 bootstrap replications. Judges were lightly screened by our organization for multiple screening tasks.

We present the additional human evaluation results on Pointer large model vs base model in table 11. In general, for the news dataset the results are mixed. For the yelp dataset, the large model wins with a large margin. All results are still far away from the human oracle in all three aspects.

Appendix E Additional Automatic Evaluation Results

We provide the full evaluation result data including Wikipedia zero-shot learning results in Table 9 and Table 10. Note that zero-shot generations from Wikipedia pre-trained model yield the lowest perplexity, presumably because the Wikipedia dataset is large enough so that the model trained on it can learn language variability, thus delivering fluent generated results.

Appendix F Inference Details

During inference time, we use a decaying schedule to discourage the model from generating non-interesting tokens, including $\mathtt{[NOI]}$ and some other special tokens, punctuation and stop words. To do this, we use a decay multiplier $\eta$ on the logits of these tokens before computing the softmax. The $\eta$ is set to be $\eta=\min(0.5+\lambda*s)$, where $s$ is the current stage and $\lambda$ is an annealing hyper-parameter. In most of the experiments, $\lambda$ is set at $0.5$

Appendix G Human Evaluation Template

See Figure 2 for human evaluation template