Relying on the Unreliable: The Impact of Language Models' Reluctance to Express Uncertainty

Introduction

Natural language is becoming the default interface for humans to engage with artificial intelligence systems whether it be information seeking, summarization, or image captioning Bommasani et al. (2021); Brown et al. (2020); Ouyang et al. (2022). As input, natural language serves as a rich and versatile medium, enabling users to articulate intricate tasks and inquiries effectively. As for output, natural language provides an opportunity for language models (LMs) to generate not only informative, but nuanced responses that better support the collaboration between humans and AI.

A pivotal aspect of fostering reliable human-AI interactions lies in the apt communication of model uncertainties Cai et al. (2019); Kizilcec (2016); De-Arteaga et al. (2020), typically defined as the probability assigned to a model’s prediction. Recent work in language generation reflects a shift towards using natural language as a means to convey model confidences (e.g., “I’m fairly confident it’s”, "According to Wikipedia it’s" Mielke et al., 2022; Lin et al., 2022; Zhou et al., 2023). Such features are known in the linguistics literature as epistemic markers, which serve to convey a speaker’s stance and commitment, which support human communication and decision making Babrow et al. (1998); Brashers et al. (2000); Tseng and Zhang (2023). Since epistemic markers play an important role in human-human communication and decision making Budescu et al. (1988); Windschitl and Wells (1996); Druzdzel (1989), we hypothesize that the use of these markers by LMs will also have an impact on human-AI interactions.

Our work begins with an examination of how LMs communicate uncertainties to end-users in realistic information seeking scenarios (§3). Specifically, we elicit responses from popular, publicly-deployed models including GPT, LLaMA-2, and Claude by prompting them to provide epistemic markers when answering multiple choice questions Brown et al. (2020); OpenAI (2023); Touvron et al. (2023); Anthropic (2022). Our analysis reveals that LMs are reluctant to share model uncertainties, despite errors in their generations. LMs can be explicitly prompt to use epistemic markers, but are more likely to generate expressions of certainty than uncertainty, despite an average 47% error rate among high confidence responses, (e.g., I’m confident the capital of Tanzania is Dar es Salaam. [Incorrect]).

We then investigate the behavioral responses of end-users to model-generated epistemic markers (§4). While linguists and psychologists have long focused on the interpretation of epistemic markers by humans Budescu et al. (1988); Windschitl and Wells (1996); Wallsten et al. (1986), the pragmatic implications of speakers of epistemic markers being AI systems, combined with dangers of over-reliance on AI Bussone et al. (2015); Jacobs et al. (2021); Bansal et al. (2021b); Buçinca et al. (2021) could drastically change their interpretation compared to human-spoken ones. We address this gap by conducting several user studies to measure how individuals interpret and respond to uncertainties as articulated by LMs in calibrated and miscalibrated settings (Figure 1). Our findings indicate that users are heavily reliant on LM generated expressions of high confidence (e.g., "I’m sure it’s…"), but are also surprisingly reliant on plain statements (e.g., "The answer is…"). Additionally, we find that even minor miscalibrations in how a model uses epistemic markers can lead to long-term harms in human performance.

Lastly, given our findings on model overconfidence and human reliability of LM generations, we pinpoint the origins of model overconfidence (§5). We investigate model artifacts such as base models, instruction-tuned models, reward models, and human feedback datasets to isolate the origins of model confidence. Our investigation identifies the process of reinforcement learning with human feedback (RLHF) as a key contributing factor and we uncover that human annotators are biased against expressions of uncertainty.

Together, our findings expose the shortcomings of how LMs currently use epistemic markers, outlines the risks that they pose on downstream users, and put forward mitigating solutions.

Epistemic Markers in Language Models

Our work focuses on the alignment between LM accuracy and LM-articulated epistemic markers as perceived by users. This is referred to as linguistic calibration Mielke et al. (2022) which builds off of work in both linguistics and machine learning.

Linguists has extensively studied epistemic markers as ubiquitous linguistic features that signal the speaker’s commitment and stance. These markers broadly fall into two categories: weakeners—expressions of uncertainty, and strengtheners—expressions of certainty. Numerous other categories include: hedges Lakoff (1975), boosters Hyland (2005, 2014), evidentials Aikhenvald (2004), approximators Prince et al. (1982), and factive verbs Kiparsky and Kiparsky (1970).

In machine learning, work has focused on improving model calibration Jiang et al. (2021); Desai and Durrett (2020); Jagannatha and Yu (2020); Kamath et al. (2020); Kong et al. (2020) by calibrating the confidence value assigned by a model and model accuracy through a measure called ECE Naeini et al. (2015). Recent work has focused explicitly on how pretraining Hendrycks et al. (2019) and scaling Srivastava et al. (2022); Chen et al. (2022) impacts the calibration of language models. Most relevant to our work is Dhuliawala et al. (2023)’s studies on how humans interpret numerical confidences in calibrated and miscalibrated settings. A key challenge remains, as numeric confidence values are known to be challenging for users to interpret Miller (2019). Our work aims to fill a gap by providing a more comprehensive understanding of how humans interpret LM-generated epistemic markers.

We begin by eliciting open-ended generations, a departure from prior methods which prompts models to produce a predefined set of confidence expressions either numerically Kadavath et al. (2022); Tian et al. (2023); Liu et al. (2023); Xiong et al. (2023); Tanneru et al. (2023) or verbally (based on an ordinal scale) Lin et al. (2022); Mielke et al. (2022). This strategy enables a bottom-up approach towards understanding how LMs generate epistemic markers and more closely mimics how real-world deployed users might engage with chat models.

In addition to examining what LMs generate, we also reveal how users behave in response to these generations in both static and interactive settings. By focusing on how humans interpret these markers, we provide a framework for how to measure the harms of linguistic miscalibration from the lens of downstream users.

How do LMs use Epistemic Markers?

To answer our first motivating question, we investigate how LMs such as GPT, LLaMA-2, and Claude express uncertainties within a broad and challenging, question-answering context. We find that LMs prefer to respond with answers free of epistemic markers and when LMs do use epistemic modifiers, they rely too much on strengtheners, leading to overconfident but incorrect generations.

Our objective is to assess the potential harms and safety risks associated with widely used publicly deployed models like GPT, LLaMA-2, and Claude.Models were accessed June - November 2023. We pose a diverse set of questions from the Massive Multitask Language Understanding benchmark (MMLU Hendrycks et al., 2021), a dataset comprised of multiple-choice questions with four options each. These questions span 57 subjects that assess both language model knowledge and problem-solving skills. In this section, we design confidence eliciting prompts and measure if there are systematic trends in how LMs use strengtheners and weakeners.

We systematically prompt LMs for epistemic markers by designing three types of open-ended prompt instructions. We modify the base template from the original MMLU paper by appending additional instructions crafted to elicit: 1) epistemic markers “Please answer the question and provide your certainty level” (Epi-M), 2) chain-of-thought reasoning (CoT), “Explain your thought process step by step” or 3) a combination of both “Using expressions of uncertainty, explain your thought process step by step” (Epi-M+CoT) (Figure 2). Previous studies have shown that chain-of-thought prompts can enhance model behavior through step-by-step reasoning, and we hypothesized that the process of articulating reasoning might also generate epistemic markers Wei et al. (2022); Suzgun et al. (2022); Wang et al. (2022).

To ensure the generalizability of our results, we employ snowball sampling to generate a list of diverse prompts, gathering additional paraphrases of prompts from Amazon Mechanical Turk Workers and GPT-3.5 (details in A).

We prompted nine models (text-davinci-003, GPT-3.5-Turbo, GPT-4, LLaMA-2 7B, LLaMA-2 13B, LLaMA-2 70B, Claude-1, Claude-2, Claude- Instant-1) using 49 prompts on 284 questions, resulting in a total of 125,244 queries.Duplicated question in the development set of MMLU benchmark, resulting in 284 instead of 285 questions. Using a zero-shot prompting approach, we aim to simulate the interactions of end-users. We set the temperature to 0.3 OpenAI (2023); Anil et al. (2023), with no stop tokens, and limit the token generation length to a maximum of 400 tokens (Details in C).

Eliciting and Classifying Epistemic Markers

The authors then qualitatively code for epistemic markers in generated responses, iteratively identifying the markers within the responses and categorizing them as strengtheners and weakeners. Using Regex pattern matching, we automatically identify epistemic markers in response generations.A future iteration could include training a classifier to identify codes at the trade-off of lowered transparency and interpretability. In contrast to previous analyses that primarily examined single-word lexical forms of epistemic markers like “anyway”, “should,” and “obviously” Islam et al. (2020), our analysis specifically focuses on phrase-levels of epistemic markers. Our qualitative coding yielded a total of 76 strengtheners and 105 weakeners. See tables 8 and 9 for the most commonly generated expressions.

2 Findings

LMs fail to incorporate epistemic markers in their responses when prompted with the base template. Only 5% of the generated answers include any type of epistemic markers (Table 1), indicating that the majority of responses seen by end-users lack information regarding model uncertainties. We refer to responses that don’t contain any epistemic markers as plain statements (e.g., “(A)” or “The answer is (A)”).

When using chain-of-thought instructions or explicit instructions to elicit epistemic markers, LMs can be encouraged to produce more epistemic markers. Resulting in 16% and 71% of generations incorporating epistemic markers respectively.

Models are biased towards using strengtheners.

Six out of nine models have a preference to generate significantly more strengtheners than weakeners. Our results indicate that an average of 20% of generations had strengtheners meanwhile only 14% had weakeners, and this bias is true among prompts eliciting for certainty, with or without CoT (Table 1). We see this trend emerge strongly among GPT and LLaMA-2 chat models, with the Claude-2 being more balanced in its generation of strengtheners and weakeners (Figure 3). Interestingly, we find that smaller models (LLaMA-2-7B and Claude-Instant-V1Announced as a ”lighter” version of Claude. See: https://www.anthropic.com/index/introducing-claude) have a higher use of weakeners over strengtheners, contrasting with larger models.

Overconfidence results in confident but inaccurate generations.

We find that across all generations, only 53% of generations with expressions of certainty are correct (random guessing accuracy being 25%). Although this accuracy rate is higher than accuracies among weakeners (32%), this is an alarmingly high rate of errors among strengtheners. Furthermore due to the high rate of generations of strengtheners, 17% of all incorrect answers include strengtheners.

3 Discussion

Our findings illustrate that models struggle to appropriately use epistemic markers. First, models are reluctant to produce uncertainties, even when asked with CoT prompts, presenting a veil of tacit certainty. When explicitly prompted to verbalize model confidences, LMs are prone to overuse expressions of certainty even when the output is incorrect, creating potential downstream harms (see §4). The overuse of expressions of certainty is likely to contribute to the existing problem of human over-reliance on AI predictions Jacobs et al. (2021); Bussone et al. (2015) and explanations Bansal et al. (2021b); Poursabzi-Sangdeh et al. (2021); Wang and Yin (2021); Ehsan et al. (2021).

In the space of information seeking, the need to express uncertainties and knowledge limitations will likely continue to persist. The need for linguistic calibration is emerging as a new harm as LMs play a bigger role in human-LM collaborations and will likely persist even as model performance improves. Moving forward, we must examine how to mitigate the harms of model overconfidence and how to best build cognitive forcing designs, such as verbalized uncertainties, to discourage human overreliance of AI systems Buçinca et al. (2021).

Human Interpretations of Uncertainty

With more robust understanding of how LMs use epistemic markers, we shift our focus to the second inquiry: how do humans interpret LM-generated epistemic markers? Using a subset of expressions generated by LMs, we set up a task to evaluate the effect of these markers on user reliance on AI. We find that users by default are highly reliant on LM-generated responses and that even minor miscalibrations in systems can have long-term consequences in human-LM collaborations.

We create a self-incentivized task where users must accrue points by correctly answering challenging trivia questions, deciding whether to rely on an AI agent’s response for help. We situate users in an imagined game scenario where they are asked to interact with AI agent named Marvin, a set up adopted from an user-AI interaction study from Bansal et al. (2019a).The original task involves users classifying shapes. We adopt the decision making setup from this work. In the game, the user shown a question (e.g., "What is the capital of Palau?") and a response generated by Marvin that includes epistemic markers (e.g., "I’m certain its Ngerulmud"). The user must decide whether to rely on Marvin’s answer or whether to indicate that they’ll look it up themselves later.

Trivia Question Selection

We control for uniformity in scenario content by limiting questions to country capital trivia. Our task should ensure that the users’ assessment of AI reliability stems from the model’s use of epistemic cues rather than users’ own prior knowledge. Hence, we select the most challenging trivia questions as ranked from Sporcle, an online trivia platform.https://www.sporcle.com/games/g/worldcapitals/results By selecting countries where participants are unlikely to know the answer, we encourage users to primarily use LM-generated epistemic markers rather than their own knowledge to make decisions.

Recruitment Process

We launch the task using Prolific and Qualtrics and inform the participants of the nature and the risks of the task through a consent form. The task is compliant with internal review board (IRB) protocols.

Template Selection

We select the most frequently occurring expressions of certainty and uncertainty from §3 and transform them into prefixes in the context of question answering (e.g., "I think it’s", "Perhaps it’s"). We filter for naturalistic expressions, avoiding any template duplication.

For details on recruitment process and template selection see Appendix B.

2 Experimental Settings

We first use our task to evaluate how participants rely on LM generated epistemic markers (from §3). Participants are shown a set of trivia questions and the beginning of a response (e.g., "I think it’s…" Figure 4). Users are asked whether or not they’d like to rely on Marvin’s answer, and since the users do not see any answers, they are simply expressing their reliance of epistemic markers as generated by LMs. Participants are presented with strengtheners, weakeners, and plain statements, which are expressions free of epistemic markers like, “The answer is (A)”. We recruited 25 participants and each were shown 106 questions.

Setting 2: Interactive Settings

The next three settings are interactive settings. Participants engage in 50 rounds of question-answering where in each round they are 1) shown a question, 2) shown Marvin’s predicted response with epistemic markers, 3) asked to make a decision, and 4) given feedback on their decision. Providing users with feedback gives users the opportunity to build a mental model of how Marvin performs Bansal et al. (2019a) and allows us to measure the harms that may arise from long-term interaction Lee et al. (2022). We recruited 25 new participants for each setting.

In these experimental settings, we introduce a scoring system, also modified from Bansal et al. 2019a. The scoring set-up is designed such that the only way to have a positive score is to rely on Marvin correctly based on reading the epistemic markers (see Table 2).To avoid spammers, we filtered the bottom 20% participants based on their performance on this task.

Setting 2A: Calibrated Setting

In the calibrated setting, Marvin’s responses are calibrated with the expected human interpretations of epistemic markers from the control setting (i.e., strengtheners appear with correct answers and weakeners appear with incorrect answers).

Setting 2B: Overconfident Setting

In the overconfident setting, Marvin will use strengtheners when generating the incorrect answer (e.g., "I’m sure the capital of Vanuatu is Luganville" [Incorrect]). This setting has five overconfident responses which will appear in the first 20 questions; the remaining 30 questions are calibrated.To create consistency in how the answers are incorrect, the largest non-capital city is used instead of the capital city.

Setting 2C: Underconfident Setting

In the underconfident setting, Marvin will use weakeners when generating the correct answer (e.g., "I’m not certain, but maybe the capital of Vanuatu is Port Vila" [Correct]). Again, five underconfident responses will appear in the first twenty questions.

3 Findings

When users are presented with weakeners, approximately 90% of users will choose to look up the answer themselves. When users are presented with expressions of strengtheners, nearly 90%, users will rely on Marvin’s response (See tables 6 and 7 for details). However, surprisingly, plain statements, which are expressions free of epistemic markers like, “The answer is (A)” or “(A)”, are relied on by users nearly 90% of the time. In other words, without communicating any epistemic markers, humans interpret this as a sign of model certainty.

Users Effectively Leverage Calibrated LM-Generated Epistemic Markers

In the calibrated setting, our results illustrate that users are able to learn mental models of epistemic markers after approximately 20 rounds. In the early stages, users rely on strengtheners 94% of the time and on weakeners 7% of the time. After 20 rounds, most participants are able to nearly perfectly leverage Marvin’s epistemic markers with users learning to rely on strengtheners 99% of the time and weakeners nearly 1% of the time, averaging an accuracy of 97% across all rounds. The high performance in this calibrated setting also validates that participants are primarily relying on epistemic markers, to answer these questions. If they had been only relying on their own knowledge, it would be unlikely to see participants perform nearly perfectly on such challenging trivia questions.

Users are Overreliant on Overconfident Responses

Participants in the overconfident setting are too reliant of strengtheners. In the first 20 rounds, users relied on 81% of strengtheners, when only 66% of the generations with strengtheners were correct, accruing on greater penalties than necessary. Because participants are unlikely to know the answer to the question, we see that participants mistakenly rely on incorrect, confident generations an average 73% of the time. The consequences of system miscalibration continued to negatively impact human performance, even after several rounds of calibrated model answers. Participants averaged 76% in the miscalibrated rounds and 86% on the calibrated rounds (Figure 5).

Users in the Overconfident Setting Also Incorrectly Relied on Weakeners

An unexpected effect of the overconfident responses was that participants started to interpret weakeners differently. In the overconfident setting, strengtheners were used incorrectly but weakeners were used correctly (only occurring when the answer is wrong). However, we see participants respond to weakeners differently, with users relying on answers with weakeners nearly 10%, when none of the answers with weakeners were correct.

Users are Underreliant on Weakeners in Underconfident LMs

In the underconfident settings, users are not reliant enough of weakeners — 33% of generations with weakeners were correct but users only relied on those generations 17% of the time. Participants in the underconfident setting averaged 66% in the miscalibrated rounds but 98% on the calibrated rounds, matching the performance in the calibrated setting. This is in contrast to the overconfident setting where users’ mental models formed correctly, even after the same number of calibrated rounds.

4 Discussion

The perception that plain statements indicate confidence for humans illustrates the harms in the status quo of how LMs’ reluctance to use epistemic markers. The lack of epistemic markers is perceived not as neutral language but as confident language. Although currently models struggle to correctly use epistemic markers in calibrated ways (§3), efforts to linguistically calibrate models is necessary as the absence of these calibration efforts presents significant harms for human-AI collaboration. For example, LM hallucinations are not only factually incorrect Ji et al. (2023); Maynez et al. (2020); Zhang et al. (2023), but are also interperted as high certainty due to the lack of epistemic markers. One potential design recommendation is to generate weakeners without explicit elicitation and use plain statements only when the model is confident.

Miscalibrations in Strengtheners Impact Interpretation of Weakeners

Miscalibration in the use of strengtheners resulted in users interpreting weakeners incorrectly as well. Our findings signal that miscalibration in one dimension (e.g., the incorrect use of strengtheners) leads users to distrust other areas of epistemic markers. As a recommendation, researchers measuring the harms of miscalibration must also consider how miscalibration impacts the users’ mental models of the whole system, rather than the perception of just an individual component. This work ties into existing literature on how mental models of AI systems are affected by numerous complexities such as accuracy Bansal et al. (2021a), warmth McKee et al. (2022), and system updates Bansal et al. (2019b).

Long-Term Effects of Overconfidence

Our findings signal that mental models of language models are developed early in LM-interactions, potentially resulting in long-term harms, even after models become calibrated later on. This is consistent with work from Dhuliawala et al. (2023) who conducted human interpretations of numerical uncertainties from LMs. Similarly, we find that users can correct a mental model developed from an underconfident model but struggle to do so with overconfident models. Unfortunately, publicly deployed models are overconfident, creating not only reliability harm now, but also potentially creating long-term algorithmic aversion Dietvorst et al. (2015) to future models.

Origin of Model Overconfidence

Our work illustrates that LMs are overconfident and that humans are highly reliant on plain statements and statements with strengtheners; all of which is exacerbated by the fact that models are often incorrect when expressing certainty. Here, we turn to our last motivating question: What is the origin of model confidence and what are potential mitigation solutions moving forward? In this section, we pinpoint LM overconfidence to an artifact of the RLHF process, specifically a bias from human annotators against uncertainty.

Current state-of-the-art models are trained using a number of techniques to support human-AI collaboration through natural language. Starting with a pretrained base model, one of the most popular techniques is to use supervised fine-tune (SFT) using human instructions. The model is then trained with reinforcement learning with human feedback (RLHF), where a reward model is learned through human preferences of pairwise comparisons of texts.

2 Methods

We perform analysis on the models from the GPT and LLaMA-2 family to identify the origin of model overconfidence. Using the same prompting strategy as §3, we measure how base models and supervised fine-tuned models compare to their RLHF counterparts when it comes to generating expressions of certainty.

We compare three models from the GPT3 family, davinci, text-davinci-002, and text-davinci-003 which are base, supervised fine-tuned, and RLHF models respectively.Models were accessed June - November 2023. We then compare LLaMA-2 models base models with their SFT+RLHF counterparts.

Reward Modeling

We then directly probe a open-sourced reward model trained on human feedback datasets and assess their scoring. We prompt the model with a question-response pair where the question is "What is the capital of X?" and the response is an epistemic marker like "I think it’s". We test the reward model on 183 question answer pairs across a subset of 30 commonly occurring templates generated by LMs in §3 and compare the model scores with human judgements from §4.

Human Annotated Datasets

Lastly, we examine the datasets that were used to train the open-sourced reward model. Specifically, we examine the datasets: OpenAI’s “WebGPT comparison” and “Summarize with Feedback”, Dahoa’s “Synthetic Instruct GPT Pairwise” dataset, and Anthropic’s “Helpful and Harmless” dataset.Reward model and link to datasets: https://huggingface.co/OpenAssistant/reward-model-deberta-v3-large-v2 We then measure how often strengtheners and weakeners are preferred by human annotators in these datasets.

3 Findings

We quantitatively observe that RLHF-ed models emit more strengtheners than weakeners, which contrasts to the base and instruction-tuned variants where the pattern is the opposite (Figure 6). This suggests that this preference for strengtheners is introduced during the RLHF process.

Reward Modeling Is Biased Towards Certainty

Reward modeling prefers plain statements with an average score of 4.03, followed by strengtheners with a score of 0.82 (Table 3). However, there is a strong penalty applied to weakeners, with the average rewards score of -1.86. Lower scores in reward modeling would in turn influence how LMs generate natural language, leading to a bias in avoiding the language of uncertainty at generation time.

Human Raters are Biased Against Uncertainty

We annotate texts in each datasets as containing strengtheners and weakeners and measure how these rates vary across the chosen and rejected texts as annotated by human judges. Given the prevalence of model overconfidence, we hypothesized that there may be a preference for strengtheners in chosen texts. We find that this is not the case. In fact, there is a slight, but significant preference for strengtheners in rejected texts (2.95% vs 2.72%).95% CI calculated using bootstrap sampling. In a pairwise comparison, we see that plain text is actually slightly preferred over strengthened texts (chosen 9% more often). This shows that there is not a human bias for strengtheners. However, we do find that weakeners appears significantly more often among rejected texts (5.02%) compared to chosen texts (4.47%). In a pairwise comparison, weakeners are chosen 8% less often than strengtheners and 9% less often than plain texts. This highlights that annotators don’t have a bias for certainty, rather there is a bias against weakeners, matching the results seen from the reward model experiments.

4 Discussion

Our investigation into RLHF datasets also reveals that humans have implicit biases towards other dimensions of language which may not be known in the annotation phase. The artifact of humans having a bias against uncertainty adds to the list of implicit biases in annotations Gururangan et al. (2018) (e.g., text length Saito et al. (2023), toxicity detection Sap et al. (2022), political leanings Santurkar et al. (2023)). However, the human bias against uncertain language is particularly harmful as it causes RLHF models to be reluctant in their generation uncertainty, negatively impacting human over-reliance on LMs. Simple interventions such as swapping labels on annotated datasets could potentially mitigate some overconfidence harms but could also risk introducing new, yet to be known biases. However, RLHF processes remain significantly understudied due to closed processes and lack of access to datasets and documentation. Investigations are needed in RLHF datasets and that annotation process to fully understand the potential implicit biases that are introduced through human preference annotations.

Beyond Mimicking Human Language

As LMs evolve towards generating more nuanced language, such as uncertainties, a shift in design is prudent. The work of Hollan and Stornetta (1992) discusses the need to design technology that goes beyond simply mimicking that of the real world. We could apply this design thinking towards designing natural language as an interface: when it comes to expressions of uncertainty, mimicking humans (or mimicking what humans prefer) might not be the end goal. Instead, we could design LMs to verbalize uncertainty in ways that would increase cognitive engagement and lower human overreliance on imperfect models Buçinca et al. (2021). Although humans may have biases in their interpretations and preferences for expressions of uncertainty, we can design LM interfaces to account for or counteract these potential harms.

Conclusion

Our work set out to explore how users interpret epistemic markers as generated by LMs in an effort to better understand the shortcomings of human-LM communications. We find that LMs are overconfident in their generations and that users are highly reliant on LM responses whether there is implicit or explicit confidence. We trace the origin of model overconfidence to the RLHF process and find that annotators have a bias against uncertainty in text.

Limitations

Our study focused exclusively on how language models generate epistemic markers in the English language Bender (2019). Humans greatly differ in their use of hedges and strengtheners across languages, contexts, and cultures Itani (1995); Lauwereyns (2002); Yagız and Demir (2014); Nguyen Thi Thuy (2018); Mur-Duenas (2021), future studies could consider how non-English language models differ in their use of strengtheners and weakeners.

Our human studies recruited U.S. based participants exclusively and their willingness to rely on epistemic markers is shaped by their experiences and cultural context. Our results thus illustrate a narrow and U.S.-centric view of how humans might interpret epistemic markers Henrich et al. (2010); Atari et al. (2023). Participants from other cultural backgrounds might reveal different findings on how humans rely on LM-generated epistemic markers.

The Ambiguity of Weakened Strengtheners

LMs articulated epistemic markers that fell in between strengtheners and weakeners, which we labeled as labeled as weakened-strengtheners. This schema closely follows Sanders and Spooren (1996)’s schema of certainty, uncertainty, and semi-certainty markers. In our human experiments, we found that participants displayed great variance in their reliance of weakened strengtheners as some humans appear to rely on weakened strengtheners meanwhile others do not (see Table 6). This ambiguous interpretation of weakened strengtheners highlights a potential risk for miscommunication; the prolific use of them could lead to more confusion and misinterpretation than clarity. Further work on the more nuanced features of epistemic markers is needed.

Gap Between Human Experiments and Real Self-Incentivized Users

A gap still exists between self-incentivized users and the participants who we recruited for our experiments. Despite our best efforts to situate users in a real-life scenario, the harms we uncover here will likely differ from that of users in a real deployed setting. The contexts in which users might engage with these chat models would like also influence their interpretability of epistemic markers Grice (1975); Goodman and Frank (2016). Changes such as the metaphors associated with the agent itself could have significant impacts on user reliance behaviors Khadpe et al. (2020). Further investigations and in-depth user studies and interviews may be needed to comprehensively study the harms of LM overconfidence.

Acknowledgements

Thank you so much to Myra Cheng, Kristina Gligorić, Dan Jurafsky, Abhilasha Ravichander, Tal August, Luca Soldaini and the AI2 Mosaic team for their helpful feedback and advice! Kaitlyn Zhou is supported by the Stanford Graduate Fellowship.

References

Appendix A Details on Prompt Paraphrases

Initially, the authors generated a list of prompts which was paraphrased by Amazon Mechanical Turk workers (details in Figure 7). The resulting paraphrased prompts then served as seed prompts for GPT-3.5 to generate additional variations. We take measure to maintain the neutrality of prompts; with keywords of certainty and uncertainty appearing together in random order (details in Table 4, 5).

Appendix B Details on Experiments from Section 4

We aimed to pay participants an average of $15 USD an hour (average actual payment was$17.77 USD/hour). Participants were filtered out to be English speaking, U.S. based, with an approval rating of at least 97% and had completed 100 or more tasks on Prolific. Each experiment had 25 participants. Human experiments were run throughout the months of September - November 2023.

Our research teach sought and received exemption from our internal review board (IRB). We do not collect sensitive or demographic information. The exemption does not require a consent form but we used a consent form an collected informed consent from all our participants.

Most Frequently Occurring Expressions

Expressions were filtered out if they were nearly identical to each other to avoid duplicate templates (e.g.,"I do not know" vs "I don’t know") as well as expressions which were would be primarily numeric or ordinal (e.g., "Confidence: 90%" or "Certainty: High" as it would break form from the other naturalistic expressions of certainty/uncertainty).

Scoring Details

Specifically, participants receive a point if they rely on Marvin’s response and Marvin is correct but lose a point if Marvin is incorrect. If they choose to look up the answer, they will neither gain nor lose a point. Half the answers are wrong so choosing to always to rely on Marvin or always looking up the answer will yield a total score of 0. The user’s score is updated and shown to the user after every question and users are explicitly informed that their performance on this task is independent to their compensation.

Appendix C Insensitivity to Increases in Temperature

A recognized mitigation strategy to address model miscalibration (i.e., calibrating model accuracy with model confidence) among RLHF models is to increase the temperature Kadavath et al. (2022). The intuition is that reward modeling encourages the model to concentrate their predictions towards those which would score highest in reward modeling and an increase in temperature could help resolve these issues Kadavath et al. (2022). We experiment with using maximum temperatures, only to observe the persistent effect of LMs preferring generating strengtheners over weakeners. This mitigation strategy also illustrates that the issue of model calibration is separate from linguistic calibration of LMs.