Prompt Waywardness: The Curious Case of Discretized Interpretation of Continuous Prompts

Introduction

Recent work has shown the surprising power of continuous prompts to language models (LMs) for controlled generation and for solving a wide range of tasks Li and Liang (2021); Lester et al. (2021); Min et al. (2022). Despite these successes, the resulting continuous prompts are not easy to interpret Shin et al. (2020). Is it possible to come up with meaningful discrete (textual) interpretations of continuous prompts, especially ones that provide a faithful explanation of the prompt’s behavior?

Towards addressing this question, we propose and investigate the Prompt Waywardness hypothesis (§3.2), a surprising disconnect between the intended behavior of continuous prompts and their nearest-neighbor discrete (language) representations. Nearest-neighbor projection via dot product has been previously used to study properties of continuous word embeddings Mikolov et al. (2013); Hashimoto et al. (2016) and is commonly performed in the final layer of modern generative LMs Radford et al. (2019); Raffel et al. (2020). In particular, we show that one can find continuous prompts that perform a desired task while, at the same time, project to any given target text. This indicates that there is little correspondence between continuous prompts and their discrete interpretation. For instance, a continuous prompt that effectively solves the sentiment classification task in Fig.1, when projected onto discrete space, might appear as the definition of a different task (“flip the sentiment”). Intuitively, continuous prompt optimization is a highly non-convex problem with numerous local minima. More surprisingly, many of these prompts (e.g., those near embedded text) are very effective at solving a desired task.

We conduct extensive analyses showing Waywardness on five classification datasets (§4). Empirically, we find the existence of wayward prompts — prompts that solve each of these tasks while projecting to arbitrary natural language text. To study a variety of projected text, we experiment with 60+ sentences, either discrete prompts from other tasks (from Mishra et al. 2022b) or random sentences from a large text corpus. We observe that it is possible to find prompts that project to a given discrete prompt (token overlap 94% F1) while scoring within 2% accuracy of the best continuous prompts-based solution for the task. Further analysis shows that the effect of Waywardness gets worse for larger models and longer prompts. We explain this surprising behavior by relating it to several structural properties of large language models (§5).

We discuss several social and research implications of prompt waywardness, to help guide future research on prompt based models (§6). First and foremost, despite many promising attributes of continuous prompts, interpreting them is non-trivial and will require further research. In fact, careless interpretation of continuous prompts can result in vulnerabilities against malicious attacks concealed under the guise of benign discrete representation. Further, the loose correspondence between continuous and discrete prompts poses a challenge for future research in differentiable interpretable-prompt optimization – optimization in search of human readable discrete prompts through the continuous space. Our work shows that continuous and discrete prompts, despite their seeming similarity, are quite different and the results from one may not always transfer to the other. We hope these findings will motivate further innovations in the prompting literature for NLP models.

Related Work

There is a line of work focused on tuning continuous prompts Li and Liang (2021); Lester et al. (2021); Zhong et al. (2021); Qin and Eisner (2021); Zhou et al. (2021); Zhong et al. (2021). These works present different approaches to discovering a continuous prompt (which is an array of real numbers) for addressing an end task, though the interpretability of the resulting prompts remains an open question. This paper investigates the feasibility of interpreting a learned continuous prompt and its connection to discrete prompts.

Discrete prompts.

The release of GPT-3 Brown et al. (2020) initiated a body of work on the emergent ability of LMs to follow discrete natural language prompts. Consequently, several follow-up studies have used manually-designed discrete prompts for probing LMs Petroni et al. (2019); Jiang et al. (2020), improving LMs’ few-shot ability Schick and Schütze (2021); Gao et al. (2021); Le Scao and Rush (2021), and their zero-shot ability as well as transferability Mishra et al. (2022a); Reynolds and McDonell (2021). Most importantly, discrete prompts have the advantages of being human-readable and thus easily interpretable though we do not have efficient and algorithmic ways of reconstructing them. For example, Shin et al. (2020)’s algorithm discovers discrete prompts, yet the results are not human readable. Prior work also finds that model performance is highly sensitive to small changes in wordings (Mishra et al., 2022a) and that optimization over the discrete prompt space is non-trivial and often highly unstable. Our findings here about the disconnect between continuous prompts and their discrete interpretation provides another perspective on the difficulty of discovering discrete prompts via continuous optimization algorithms that (directly or indirectly) leverage the continuous space (more discussion in §6).

Prompt Waywardness

The $d$-projection maps the continuous inputs to nearest neighbor discrete elements, where for each position $l$ ($1\leq l\leq L$), one of the possible (and perhaps most straightforward) methods for interpreting a continuous prompt is defined as a projection onto nearest neighbor representations Mikolov et al. (2013); Hashimoto et al. (2016):

where $\delta_{l}$ is a one-hot vector corresponding to the word with the closest (highest dot product) embedding to the $l$-th position of continuous prompt $p_{c}$.

These projections are used in the first and last layer of virtually all modern LMs, such as GPT2.

Solving tasks with continuous prompts.

Consider any machine learning model $M$ (typically a pre-trained model) that takes textual input $x$ and produces output $y$. Normally, the parameters of $M$ are learned so as to optimize behavior on a task with a dataset $\mathcal{D}=\{(x,y)\}$ of input/output pairs. In prompt tuning (Lester et al., 2021), one freezes the parameters of $M$ and instead optimizes for a prompt $p$ that, when fed in conjunction with $x$, makes $M$ produce the desired output $y$. Thus, $p$ represents the only learnable parameters in this method. When $p$ is a discrete prompt with $k$ tokens, it can be simply concatenated with $x$, denoted $p+x$. In our study, $p$ will be a continuous prompt (of length equal to the embedding of $k$ tokens). We will concatenate it with the embedding of the input $x$. For simplicity and with some abuse of notation, we use $p+x$ to denote concatenation in this continuous case as well.

One can quantify the amount of loss incurred when using a continuous prompt $p$ as follows:

Minimizing this loss function (empirical risk minimization) over $p$ recovers a minimum risk continuous prompt for this dataset:

2 The Waywardness Hypothesis

It is worth emphasizing that the hypothesis is stated for any task and any set of discrete prompts, even if they are irrelevant or contradictory.While our focus is on the use of continuous prompts for solving datasets (one prompt shared among many instances), one can imagine applications of the same conjecture to special use cases such as controlled generation Dathathri et al. (2019) with one prompt per instance.

3 Finding Wayward Prompts

where $p_{c}$ is the only learnable parameter, and $\gamma$ is a hyperparameter.

There are at least two natural ways to define the distance measure $\mathsf{dist}(p_{c},p_{d})$ between a continuous prompt $p_{c}$ and a discrete target prompt $p_{d}$, by converting one so that both are in the same space:

The first of these places both $p_{c}$ and $p_{d}$ in the continuous space and computes the squared-L2 norm, normalized by the prompt length. This is used in our training loss (Eq.5) implementation. The second places both in discrete space (text) and computes the standard word-level token overlap F1 score.Ignoring punctuation marks and articles, and applying lemmatization. This is used during our evaluation.

Empirical Support of Waywardness

We empirically investigate the Prompt Waywardness hypothesis (§3.2) using our modification (§3.3) of the prompt tuning method from Lester et al. (2021). We show that given an arbitrary and irrelevant discrete prompt $p_{d}$, it is possible to learn a continuous prompt that is mapped to $p_{d}$ while retaining its accuracy on a given dataset. Scripts needed to reproduce our results: https://github.com/Alrope123/prompt-waywardness

Following the setup of Min et al. (2022), we select a diverse set of 5 classification datasets: SST-2 (Socher et al., 2013), SST-5 (Socher et al., 2013), AGNews (Zhang et al., 2015), Subj (Pang and Lee, 2004) and TREC (Voorhees and Tice, 2000). Statistics and the unconstrained accuracy of each dataset are provided in Table 1.

Discrete Target Projections.

We compile two sets of discrete target prompts: (1) 32 target prompts for solving tasks from Natural-Instructionshttps://instructions.apps.allenai.org dataset Mishra et al. (2022b) that are distinct from and intentionally orthogonal to the end tasks considered here. These were chosen by excluding discrete target prompts that have high lexical overlap with other discrete prompts; this is because we found lexically similar prompts are often semantically similar even when written for different subtasks. (2) 30 random sentences from PILE,https://pile.eleuther.ai a large-scale, diverse text corpus used to pretrain GPT-J, the largest public causal language model (Wang and Komatsuzaki, 2021). The sampled sentences were drawn from a Poisson distribution with $\lambda=14$, which makes the average length of the sentence to be consistent to those in Natural-Instructions. These sentences are selected to have little or no token overlap with the true definition of the target tasks. See Table 3 for a few examples.

Evaluation metrics.

For all experiments, we report two metrics: (1) the task accuracy We did not consider alternatives like Macro-F1 because all datasets are roughly balanced across different classes. as well as (2) prompt F1, the word-level token overlap F1 score computed as in Eq.8, since it easy to interpret and is commonly used for evaluating the textual output of models Rajpurkar et al. (2016).

Models.

For evaluation, we use GPT2 Radford et al. (2019) an auto-regressive LM which has extensively been used in many NLP applications. Unless otherwise specified, we use a ‘large’ variant consisting of 774M parameters.

Implementation details.

2 Main Results

Table 2 summarizes the results. Across all datasets, we find that it is possible to learn a continuous prompt $p_{c}$ whose discrete projection is very close to $p_{d}$ and mostly retains the task accuracy. There is a trade-off between the task accuracy and prompt F1, which can be controlled by the choice of $\gamma$ (more extensive ablations in the forthcoming paragraphs (§4.3)). Overall, with $\gamma=0.01$, it is possible to achieve $\geq 94\%$ prompt F1 with under $2\%$ relative drop in task accuracy. The only outlier is the TREC dataset where we achieved a prompt F1 score of $86\%$ for a $\hat{\Delta}=2.3\%$ relative drop in accuracy. This might be due to the difficulty of learning effective prompts on TREC (also discussed by Min et al. (2022)).

Example prompts with varying values of prompt F1 scores are shown in Table 3. A prompt F1 $\geq 94\%$ generally indicates one word mismatch with almost no semantically meaningful difference.

3 Further Analysis

Fig. 3 shows the trade-off between task accuracy and the prompt F1 when varying $\gamma$ from 0 to 0.03. As $\gamma$ increases, the task accuracy goes down while the prompt F1 increases. The drop in task accuracy is relatively minor—it is possible to learn a continuous prompt for which prompt F1 is near 1.00 and the accuracy drop relative to the unconstrained accuracy is less than 1%.

Effect of Prompt Length (L𝐿L).

We randomly sample sentences from The PILE with a constraint that its length must be $L$ (chosen from $\{4,7,14,28,56\}$). The left and the middle parts of Fig. 4 illustrate the results. We find that when $L$ is very small (e.g., 4) it is relatively difficult to learn a continuous prompt $p_{c}$ that is close to $p_{d}$ (F1<60%) while retaining the task accuracy. This is likely because the prompt being too short significantly hurts the expressivity of the prompt. Nonetheless, when $L$ is reasonably larger, e.g., $14$ (the average length of in Natural Instructions) or longer, all cases lead to a continuous prompt with near 1.0 prompt F1 and little accuracy drop.

Effect of Model Size.

We vary the size of the GPT2 models—small, medium, large, and XL—with 124M, 355M, 774M, and 1.5B parameters, respectively. Figure 5 (right) reports the result on SST-2. We find that (1) across different sizes of the LM, our findings in learning continuous prompts with the prompt F1 of near 1.0 and little drop in the accuracy generally hold, and (2) in particular, the drop in accuracy is more negligible with larger LMs (0.2% with XL, 0.5–0.7% with medium and large, 1.2% with small).

Projection onto true task definitions.

More interestingly, as shown in Table 5, continuous prompts that project to “true” target prompts are no more effective at solving the task than continuous prompts that project to the 62 irrelevant target prompts considered earlier (Table 2). Specifically, the average performance gap $\Delta$ (relative to unconstrained prompts of the same length) is about the same ($\approx 1.5\%$) for continuous prompts that map to true task definitions compared to prompts that map to irrelevant text. This further bolsters the waywardness hypothesis—continuous prompts don’t relate to the task being solved.

Explaining Waywardness

Here we provide intuitions behind the factors that enable Prompt Waywardness.

This intuitively suggests that there is a whole region of continuous prompts that corresponds to a fixed discrete representation (Fig.6). The remaining question is, how is this region able to have a diverse set of prompts that can solve a variety of tasks? This is addressed next.

Deep models give immense expressive power to earlier layers.

The deeper a network is, the more expressivity it has with respect to its inputs Telgarsky (2016); Raghu et al. (2017). Since continuous prompts reside just before the first layer, they enjoy a lot of expressivity. Therefore, no matter how narrow the regions corresponding to individual tokens are (Fig.6), they are extremely powerful in solving a variety of tasks. Previously in §4.2 we provide an empirical analysis showing evidence that the effect of Waywardness is stronger in deeper models.

Implications of Prompt Waywardness

We discuss the implications of these findings on several inter-related lines of research. Note that all the following statements are valid within the boundaries of the existing architectures. Moving beyond these barriers likely requires major innovations in terms of LM architectures or how continuous prompts are optimized.

Given the intuitions behind and empirical support for the Waywardness hypothesis (§5), faithful discrete interpretations of continuous prompts via common discrete projections (like nearest-neighbor projection) are unlikely to be robust based on current approaches. It is an open question whether there is a better way of interpreting continuous prompts with human language, or whether explaining and interpreting continuous prompts via human language is inherently impossible because they lie in completely different spaces. Future work may investigate more on this topic in order to improve the interpretability of prompt-based language models.

Risk of interpreting continuous prompts: concealed adversarial attacks.

It is not difficult to imagine a future where proprietary model development is driven by fine-tuned continuous prompts. In such a world, not addressing the challenges involved in discrete interpretation of continuous prompts can lead to harmful (and potentially, adversarial) consequences Slack et al. (2020); Wallace et al. (2021), as discussed below.

We consider the following scenario: a model designer comes up with a set of continuous prompts that solve a target task (e.g., ranking resumes according to each applicant’s qualifications and merits). Whether intentionally or not, such prompts may maliciously target, for example, a minority group. To assure their customers, the model designer uses the projection of the prompt that expresses a benign definition for the task, which does not reveal the true nature of the egregious behavior. The customers might even evaluate the prompt on a few instances but not notice this harmful behavior, e.g., when it effects a minority group not in the evaluation set. In a way, the benign discrete projections may provide a false sense of security.

Optimizing discrete prompts through continuous prompts can be degenerate.

This a joint optimization towards a utility objective (the extent to which it can solve dataset $\mathcal{D}$) and a human readability objective. According to the Waywardness hypothesis, there are $p_{c}$’s that assign high mass to the utility term while also mapping to human interpretable text that is irrelevant (or even contradictory) to the task solved by the prompt – hence, degenerate solutions.

In summary, Waywardness presents a challenge for searching effective discrete prompts via continuous optimization. The recent works have used additional signals such as domain-specific constraints Qin et al. (2020); Khot et al. (2021); Qin et al. (2022) to alleviate these challenges. We hope to see more design innovations in this direction.

Gradients alone are insufficient to reverse engineer a model.

Suppose we are given a fixed (fine-tuned or otherwise) model $M$ (e.g., an open question-answering model) and an expected output $y$ from this model (e.g., $y=$“Joe Biden”). Can we use gradients with respect to an LM’s output to find a semantically meaningful input that makes a frozen model $M$ generate a particular output?

Our findings and the earlier argument about continuous differentiable optimization suggests this may not be feasible with current methods. To see the correspondence to Prompt Waywardness, we can replace $\mathcal{D}$ in Eq.9 with the desired outcome $y$ and run the optimization over word distributions (cf. Footnote 11). While gradients can guide towards some input that makes $M$ produce $y$, their interpretation is likely unfaithful to the task being solved by $M$. In the context of the above example ($M$ being a QA system), gradients might lead to inputs maximize the probability assigned to “Joe Biden”, although this input will likely be neither fluent nor semantically descriptive of “Joe Biden”.

Nevertheless, as noted earlier, gradients are still useful when they are applied using domain-specific constraints. For example, one can find local (word-level) perturbations that lead to a certain adversarial outcome, if the perturbations are restricted to well-defined semantic categories (e.g., “blue” can be perturbed to any other color name) Sha (2020); Guo et al. (2021); Yuan et al. (2021).

Continuous prompt tuning does not necessitate task-specific initialization.

Recent works on continuous prompt-tuning have shown the effectiveness of initialization from embeddings of random common words Lester et al. (2021); Min et al. (2022), despite these words being irrelevant to the task solved by these prompts. This, however, makes sense given the observations made in this work regarding the existence of effective prompts around word embeddings.

Conclusion

The prompting literature has seen many parallel developments around continuous and discrete prompts, as efficient alternatives to fine-tuning models with tens of millions of parameters. Our work introduced the Prompt Waywardness hypothesis, which expresses a surprising disconnect between continuous and discrete prompts: given a downstream task, for any discrete target prompt $p_{d}$, there exists a continuous prompt that projects to $p_{d}$ while achieving strong performance on the task. We provided empirical evidence for this hypothesis, studied various parameters around it, and ended with several implications of this hypothesis.

While our experiments are done on the GPT family, we expect our findings to apply to a broader set of architectures that, in one way or another, use similar mechanisms for mapping discrete elements to continuous representations and vice versa. Similarly, while our projection to the discrete space (Eq.2) is a popular operator in the field (cf. Footnote 1), the intuition explained in Propositions B and B of the Appendix suggests similar behavior for a broad class of projection operators.

Prompt Waywardness identifies challenges for future progress on algorithmic methods for the discovery of human readable prompts that are faithful to the task they solve. We hope the observations made in this work motivate architectural innovations that overcome such challenges and guide future steps in the prompting literature.

Acknowledgment

The authors are thankful to Lisa Li, Nicholas Lourie, and Vered Shwartz for helpful discussions, and the Beaker team at AI2 for their support with the experiments. This work was supported in part by DARPA MCS program through NIWC Pacific (N66001-19-2-4031), DARPA SemaFor program, and Google Cloud Compute.

References

Appendix A Additional Experimental Details

Here we include several experimental details (§4) that did not fit in the main text. For the experiments we used A100 GPUs with 40G memory. In terms of the time GPU time of the experiments, each round of training and inference for each seed took about around 6 min. Therefore, the total GPU hours for our main experiment (Table 2) adds up to 93 hours (6 mins $\times$ 3 seeds $\times$ 5 datasets $\times$ 62 prompts = 5580 mins).

Appendix B The mapping between continuous and discrete space is not one-to-one

As argued in §5, the mapping between the space of discrete input and that of word embeddings (Fig.2) is not a bijection. While a discrete target prompt is mapped to exactly one continuous prompt (via its embedding, Eq.1), the reverse is not true: except for some unnatural or rare cases (as formalized in the following propositions) there are infinitely many continuous prompts that project back to a fixed discrete target prompt (via Eq.2).

The infinite-to-one mapping aspect is not limited to the class of nearest-neighbor projection operators. It is rather an inherent property of the interaction between continuous and discrete spaces, and holds for a broader family consisting of all but a negligible portion of possible projection operators:

Proof of Prop. B: