OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation

Introduction

Recent advancements in multi-modal large language models (MLLMs) has greatly elevated general-purpose foundation models to unprecedented levels. These models enable users to interact using images as input, facilitating free-flowing communication based on the content of these images. The impressive abilities of MLLM allows it to be adept at a variety of vision tasks , meanwhile easily handling some complex content comprehension or generation .

Notwithstanding their remarkable versatility, MLLMs also grapple with a significant challenge known as the “hallucination” problem. Specifically, MLLMs often hallucinate incorrect statements to the user-provided image and prompts, e.g., producing irrelevant or nonsensical responses, indentifying inaccurate objects in terms of colors, quantities and locations that do not exist in the image. This flaw poses substantial risks for practical applications of MLLMs to become a trustworthy assistant. For instance, in model-assisted autonomous driving scenarios, such misinterpretations of road scene images may lead to wrong judgments of system and serious traffic accidents.

Various approaches have been proposed to reduce hallucinations in MLLMs. While these method incur substantial additional costs, including the annotation budget for extra instruction data for training , the integration of external knowledge or models, etc.

In this paper, we delve into the challenge of mitigating hallucination with the of the MLLMs during inference, without introducing additional data, models, or knowledge. Our investigation commences with a noteworthy ‘partial over-trust’ observation found while visualizing self-attention maps for decoded sequences. As illustrated in Figure 2, we discern a recurring pattern where the inception of many hallucinated contents aligns with the subsequent tokens generated after a columnar attention pattern. Notably, these columnar attention patterns often manifest on tokens that lack substantial informativeness, e.g., full stop or quotation marks. Intuitively, this peculiarity reveals a weird fact that, a token exhibiting a columnar attention pattern typically possesses limited information, yet exerts a pronounced influence on the prediction of all subsequent tokens, and as we calculated in Figure 3, most of the subsequent contents contain reasoning or hallucinations. With the above observation, we hypothesize that such tokens serve as a summary token, i.e., aggregating the crucial knowledge from previous tokens in the sequence and guiding the subsequent tokens generation.

However, the MLLMs are expected to focus on the image and provide an precise understanding, which is conflicts with its partial over-trust tendence. In detail, the subsequent tokens may ignore the forehead image tokens (usually the first several tokens in the sequence) and over-trust the summary tokens via their strong attention attended, leading to hallucinations raised by the model bias, e.g., hallucinating “cars” based on the “road” mentioned in the previous sentence.

To alleviate the partial over-trust issue, we present OPERA, a novel MLLM decoding approach grounded in an Over-trust Penalty and a Retrospection-Allocation strategy. The over-trust penalty introduces a weighted score for the candidate selection step in the Beam Search , so that the candidate with an over-trust pattern will have lower priority to be selected. Specifically, for each decoding token, we investigate the local window segmented on the self-attention map of the decoded sequence, and devise a column-wise metric to calculate the intensity of knowledge aggregation patterns. This metric produces a value that indicates the over-trust degree between in-window tokens and the summary tokens. It is naturally incorporated with the model logits predicted for the next token in the Beam Search and penalizes the appearance of over-trust patterns. Further, considering the hysteresis of the appearance of the knowledge aggregation pattern, the hallucination may exist in all the candidates when it can be observed. We propose a retrospection-reallocation strategy to help the decoding process roll back to the position of the summary token and re-select better candidates that can avoid such a pattern. Such retrospection is triggered when the location overlap of the maximum of in-window penalty scores reaches a threshold.

With extensive experiments on various benchmarks and hallucination metrics, along with comprehensive GPT-4/GPT-4v assessments, our OPERA demonstrates the generalized hallucinations-reducing performance on several MLLM models. Our contributions can be summarized as follows:

To the best of our knowledge, our OPERA is the first to alleviate the MLLMs’ hallucination issue without introducing any data, knowledge, or training.

We reveals the appearance of hallucinations and over-trust patterns, and propose a penalty-based decoding method equipped with retrospection-reallocation strategy.

Extensive evaluation including GPT assessments prove the superior performance of OPERA, which serves as a nearly free-lunch to mitigate hallucinations.

Related Work

Recent progresses of computational resources has greatly facilitated the research into large-scale foundational models incorporated with multi-modal learning. Powered by open-sourcing large language models such as LLaMA and Vicuna , MLLMs understand and generate diverse content in a more comprehensive way by integrating information from different modalities, such as text, images, and audio. The series of CLIP and BLIP well aligns the text features and image features. LLaVA , InstructBLIP and MiniGPT-4 take a step forward in this field, allowing users to interact with these intelligence with images and texts as prompts. All of them share the same two training phases, i.e., pre-trained feature alignment and instruction fine-tuning, to help the model to comprehend the format of instruction input. Shikra incorporates grounding data and teaches the model to understand the grounding knowledge in the given images. All of aforementioned MLLM models suffer from severe hallucination problems. Consequently, we mainly conduct the experiments on these four models in our paper.

2 Hallucination in Large Foundation Models

The hallucination refers to the generation of text that is either irrelevant, factually incorrect, or nonsensical in the given context, which is quite severe in current large foundation models. This issue can arise due to overfitting to specific patterns in the training data, lack of understanding of real-world facts, or an inability to effectively contextualize the given input. The primary concern regarding hallucination in LLMs is the factual accuracy of generated content, i.e., conflicting with world knowledge or common sense. In MLLMs, the primary worry centers around faithfulness, i.e., assessing whether the generated answers conflict with user-provided images. Researches on mitigating current LLMs’ hallucination issues often focuses on several aspects, including refining the training process, using larger and more diverse datasets , or implementing post-training evaluation and correction mechanisms . While for MLLMs, relevant researches are still quite few . However, most of these countermeasures have a large drawback that, they either introduce large quantities of extra data, or resort to more powerful external models or knowledge. Compared with them, our OPERA serves as nearly free lunch for alleviating the hallucination issue, which does not incur extra training, data, or knowledge.

3 Decoding Strategy in Language Models

Decoding strategies in language models are crucial for determining how these models generate text. They play a pivotal role in shaping the output’s quality, relevance, and coherence. Greedy Decoding simply selects the most likely next word at each step. While fast and computationally efficient, greedy decoding often leads to repetitive and less varied text. Beam Search is a more sophisticated approach, beam search keeps track of a predefined number of hypotheses at each step, expanding on them to find a more optimal sequence. Top-k Sampling adds randomness to the generation process by randomly selecting from the top-k likely next words, introducing diversity in the output but can sometimes produce less coherent results. Top-p (Nucleus) Sampling is an evolution of Top-k, Nucleus sampling considers a dynamic number of words that cumulatively reach the probability $p$. This method provides a balance between randomness and relevance, often leading to more coherent and interesting outputs than Top-k sampling. DoLa decoding is a recently proposed decoding method that aims to mitigate the hallucinations in MLLMs, which contrasts the logits of mature layer and pre-mature layers and rescale the increments as the output. In this paper, we compare our proposed OPERA with these common decoding strategies, focusing on the performance on the hallucination issues of MLLMs.

Method

In the following, we first formulate the generation procedure of the MLLMs for the easy understanding of our OPERA, then introduce the calculation of the proposed Over-Trust Logit Penalty and Retrospection-Allocation Strategy respectively.

The generation procedure of LLMs could be parsed into three components: input formulation, model forward, decoding.

Input Formulation. The input of MLLMs contains both image and text. Putting aside the specific architecture difference, the MLLMs commonly use a vision encoder to extract visual tokens from the raw images, and map them into the LLMs’ input space with a cross-modality mapping module. The mapped visual tokens are used as part of the LLM input, along with the text input. We denote the visual tokens as $\mathbf{x}^{v}=\{x_{0},x_{1},\ldots,x_{N-1}\}$. Here $N$ is the length of the visual tokens and it is a fixed number in most cases. Correspondingly, the input text is tokenized with the tokenizer and we denote it as $\mathbf{x}^{p}=\{x_{N},x_{N+1},\ldots,x_{M+N-1}\}$. The image and text tokens are concatenated as the final input sequence and we denote it as $\{x_{i}\}_{t=0}^{T-1}$ that $T=N+M$.

Model Forward. The MLLM is trained in an auto-regressive manner with a causal attention mask, each token predicts its next token based on the previous tokens, formally:

where $\mathbf{h}$ is the output hidden states of the last layer of the MLLM.

Next, MLLMs use a vocabulary head $\mathcal{H}$ to project the hidden states $\mathbf{h}$ and get the logits (or probabilities) for the next token prediction, formally:

where we use $x_{<t}$ to simplify the sequence $\{x_{i}\}_{i=0}^{t-1}$ and $\mathcal{X}$ means the whole vocabulary set.

Decoding. Based on the logits $p(x_{t}|x_{<t})$, there are several decoding strategy developed, including Greedy Decoding, Beam Search, DoLa, etc. The decoded token is concatenated to the last of the original input text for the next-round generation, until the generation is ended.

Our OPERA is based on the Beam Search , which is a accumulated-score-based decoding strategy. Briefly, With a given beam size $N_{beam}$, the Beam Search keeps $N_{beam}$ candidate sequences, where each candidate is a decoded sequence $\mathbf{x}^{N_{beam}}$ with a beam score. When decoding token $x_{t}$, each candidate hypothesis will select $N_{beam}$ candidate tokens based on the Top-$N_{beam}$ probabilities in the logits. And finally, the decoding procedure will output the hypothesis wins the best beam score.

2 Over-Trust Logit Penalty

As we analyzed in Sec.1, there exists a high-probability co-currence between the hallucination and the knowledge aggregation patterns. However, such pattern has a significant hysteresis, i.e., the patterns can not be immediately observed when the corresponding token is decoded, but after several subsequent tokens been decoded, and the hallucination may already occurred.

In response to the hysteresis, we propose ‘Over-Trust Logit Penalty’, an accumulative penalty weighted in the beam score, which influences the selection of both the current token and the candidate sequence. A candidate sequence accumulated with a large penalty will have a lower priority to be selected so that the output with hallucinations will be possibly omitted.

In practice, we investigate a local window on the self-attention weights and leverage column-wise product to calculate the metric values. Denote the current generated sequence as $\{x_{i}\}_{i=0}^{t-1}$ and their casual self-attention weights $\{\omega_{t-1,j}\}_{j=0}^{t-1}$ paid on the next token prediction, in which the weights can be depicted by softmax result as $\omega=\text{SoftMax}(\frac{QK^{\top}}{\sqrt{D}})$ and $Q$, $K$, $D$ denote query feature, key feature, feature dimension respectively. We consider to gather all of previous self-attention weights in a local window for characterizing the knowledge pattern, i.e., the local window attention is defined as

where $k$ denotes the size of local window we cropped on the attention map, $\omega_{i,j}$ means the attention weight assigned by the $j^{th}$ token to the $i^{th}$ token. There are two points should be clarified: 1) our window does not involve the attention weights of image tokens or prompt tokens because we only concentrate on the knowledge aggregation patterns on generated tokens, i.e., $t-k\geq N+M$. 2) we select the maximum weight in multi-head attentions and re-normalize the values since it usually indicates the strong confidence of models.

With the local window attention weights $\mathbf{W}_{t-1}^{k}$, we can calculate upon a simple metric to describe the size of the knowledge aggregation pattern. Specifically, we first do some preprocess on $\mathbf{W}_{t-1}^{k}$, including filling the upper triangle of the matrix with zeros and scaling up the attention values as the values are usually too small, i.e.,

where $\{\omega_{i,j}\}_{j=i+1}^{t-1}$ are zeros and $\sigma$ is a configurable scaling factor.

As illustrated in Figure 4, we then conduct the column-wise multiplication on the lower triangle of the attention matrix and obtain a vector of column-wise scores. Intuitively, the larger score indicates the stronger pattern that exists at the corresponding location. Thus, we select the maximum value of the column-wise score vector as the characteristic of knowledge aggregation patterns. Formally,

Until now, we have an salient metric to detect the occurring of knowledge aggregation patterns within the local window. With the concern of calculation efficiency and the penalty should not bias the model to unreasonable output, we choose the top-$N_{can}$ in the logit of each beam to consist a candidate set $\mathcal{Y}$, where $|\mathcal{Y}|=N_{can}*N_{beam}$ and $N_{beam}$ is the number of beams. In this way, we limit the prediction within the candidate set and incorporate $\phi(w_{\leq t})$ with the model logits to predict the next token, i.e.,

where $w_{\leq t}$ simplifies all of attention weights obtained by feeding forward the sequence $\{x_{0},x_{1},\ldots,x_{t}\}$.

3 Retrospection-Allocation Strategy

With the over-trust logit penalty, we can successfully detect the occurrence of patterns after several subsequent tokens are generated. Normally, the penalty term is able to penalize the candidates which have knowledge aggregation patterns, and encourage other candidates to be predicted. While there still exists a few cases that all of the candidates get penalized and the hallucination already occurred

This case motivates us to rethink the origin of such aggregation patterns: it is caused by the first few subsequent tokens over-trusting the summary token, and the penalty failed to correct them. So an intuitive while aggressive idea is that the pattern will be greatly weakened if we could exclude the tokens that lead to hallucination and re-choose the proper first few tokens after the summary token.

To this end, we propose the Retrospection-Allocation strategy. Specifically, when the decoding procedure encounters the knowledge aggregation pattern and the hallucination is inevitable, it rolls back to the summary token and selects other candidates for the next token prediction except for the candidates selected before. Empirically, the condition of decoding retrospection is designed as the location overlap of the maximum value in column-wise scores that corresponds to several consecutive tokens, where we manually set the threshold counts as $r$. Rather than the maximum value that varies between different models, location counting is a much more robust and general metric for the decision.

The whole retrospection process is illustrated in Figure 5. Based on Sec. 3.2, we can easily derive the location coordinate $c$ of the maximum score via Eq. (5). Consequently, we can obtain the location coordinate set of several recently decoded tokens $x_{t-l},\ldots,x_{t-1}$, i.e.,

where $l>r$ should be specified. We set $l=k$ by default.

Given a sequence $\{x_{0},x_{1},\ldots,x_{t-1}\}$ and its recent location coordinate set $\mathcal{C}$, we can easily check whether the coordinates are consistent. Formally, the overlap times can be calculated by

If $N_{overlap}\geq r$, we consider to implement retrospection, regarding $s=\text{Mode}(\mathcal{C})$ as the location of the summary token. Suppose the sequence $\{x_{0},x_{1},\ldots,x_{s},\ldots,x_{t-1}\}$ that has presented knowledge aggregation pattern at the summary token $x_{s}$, we intend to roll the decoding procedure back to the sequence $\{x_{0},x_{1},\ldots,x_{s}\}$ and select the new next token in the complementary set $\mathcal{Y}/\{x_{s+1}\}$. Since the subsequent rollback will be further forward than previous ones, we manually specify that the rollback location $s$ must be monotonically not decreasing. Additionally, we configure a maximum time $\beta$ for rollback and consider to roll back to $\{x_{0},x_{1},\ldots,x_{s-1}\}$ if $x_{s}$ has already reached the maximum rollback times.

Experiment

Models. We select four of the most representative MLLM models for evaluation, including InstructBLIP , MiniGPT-4 , LLaVA-1.5 and Shikra . These MLLM models can be roughly divided into two categories: Both InstructBLIP and MiniGPT-4 adopt Q-former to bridge the features between vision and text modality, using just 32 tokens to efficiently depict image representations. While LLaVA-1.5 and Shikra simply leverage linear projection layers to align the features of two modalities, with 256 or even 576 image tokens as MLLM input. All of these MLLM models apply a well-pretrained model as their vision encoder, such as CLIP and EVA , as well as a pretrained language model like LLaMA or Vicuna . Note that all of models used in our paper are 7B models.

Baselines. Since our work targets on the decoding approaches of MLLMs, we choose four decoding methods as the baseline methods, including three common strategies greedy decoding, Nucleus sampling, Beam search decoding and one method DoLa that is designed for mitigating LLMs’ hallucination issues. Greedy decoding selects tokens step by step, greedily choosing the one with the highest probability in the language model logits. Improved on greedy decoding, Beam search decoding maintains a set of beams to enlarge the candidate range and select the best on in beams finally. Different from the aforementioned two methods, nucleus sampling concentrates concentrates on the predominant probability mass at each time step, maintaining a small subset of the vocabulary, typically ranging between one and a thousand candidates. DoLa , designed for hallucination reduction in LLMs, contrasts the logits of the mature layer with those of pre-mature layers, using the increment as the final output logits. We adopt the default settings of all of these baseline methods, where we unify $N_{beam}=5$ for both Beam search and our OPERA, and set $p=0.9$ for nucleus sampling. For DoLa, we use “0,2,4,6,8,10,12,14” as the indexes of candidate pre-mature layers and “32” as the index of the mature layer for DoLa.

Implementation details. Basically, OPERA is established on Beam search where $N_{beam}=5$ by default. We empirically select $\sigma=50$ as the scaling factor in Eq. (5), to ensure the attention values on knowledge aggregation patterns could be larger than 1 while the values on weaker attention areas could be smaller than 1. It aims to get the larger multiplication result on knowledge aggregation pattern. For the number $N_{can}$ of candidates, it is a configurable hyper-parameter like $N_{can}$ and we set $N_{can}=5$ by default. Too large $N_{can}$ will consume lots of time during decoding. Besides, we unify $\alpha=1$, $\beta=5$ and $r=15$ for all of MLLM models.

2 Quantitative Results

In this section, we evaluate OPERA’s performance of mitigating hallucinations on both long descriptions and simplified VQA answers.

CHAIR evaluation on hallucinations. The Caption Hallucination Assessment with Image Relevance (CHAIR) metric is a specifically crafted evaluation tool designed to assess object hallucination issues in image captioning task. More precisely, CHAIR quantifies the degree of object hallucination in a given image description by calculating the ratio of all objects mentioned in the description that are not present in the ground-truth label set. It comprises two distinct assessment dimensions, including CHAIRS that calculates on sentence-level and CHAIRI that calculates on image-level. Denoted as $C_{S}$ and $C_{I}$, these two variants can be formulated as the average results of

where the integration of CHAIRS and CHAIRI enables a thorough and detailed analysis of object hallucination issues in image captioning.

We conduct CHAIR evaluation on MSCOCO dataset , which contains more than 300,000 images and 80 objects with annotations. Specifically, we randomly select 500 images in the validate set of COCO 2014 and query different MLLM models with the prompt “Please describe this image in detail.” to get their descriptions. Considering the length of sequences can greatly affect the values of CHAIR , we restrict two types of max new tokens to generate descriptions for fair evaluation.

As shown in Table 1 and Table 2, our OPERA obviously surpasses all of baselines decoding methods in both terms of $C_{S}$ and $C_{I}$. Especially on Shikra, our method achieves $\sim$35% improvement on DoLa. The superior performances of OPERA are consistent between long description generation and short description generation.

GPT-4 assisted evaluation. CHAIR is a strong metric to evaluate the object-existence-level hallucination, while it fails to identify other kinds of hallucination, such as the attribute, location, and relation hallucination of objects. HalluBench is an advanced benchmark, which use the detailed object-level description in the VG dataset as ground-truth, and relay on the advanced GPT-4 to judge the hallucination in the description. In practice, the detailed objects-level description are gathered as a disordered comprehensive description about the image, and the GPT-4 is carefully prompted to judge the hallucination in the MLLM generated descriptions, sentence by sentence. Similar to Section 4.2, the MLLMs are prompted with the instruction “Please describe this image in detail.” and the max new tokens is set to 512. Details are shared in Section 4.4.

From Figure 6, we observe that our OPERA generally achieves much less hallucinated sentences or words for describing each image, e.g., $\sim$30.4% surpassing greedy decoding on the ratio of hallucinated sentences (HSR), and $\sim$15.4% surpassing DoLa at the ratio of hallucinated words (HWR). It indicates that OPERA does help the model partially overcome the hallucination issue caused by its bias or over-trusting problems. We also notice that OPERA somehow slightly reduce the length of MLLM’s output sequence, it is probably attributed by the reducing of those additional hallucinated contents.

GPT-4V assisted evaluation. We further resort to GPT-4Vision, a strong multi-modal assistant that can easily handle the input from vision, language, and voice modality. Typically, we randomly sample 50 images from MSCOCO’s validate set and ask different MLLM models to describe these images. For fair comparison, we following and compare the answers obtained from two decoding methods at the same time, i.e., providing the image and both the answers to GPT-4V and prompting it to give a judgement from 0-10 respectively. The prompt emphasizes mitigating the impact of the sequential order fed to GPT-4V and, additionally, paying special attention to the objects mentioned in answers but not appear in the provided image. This includes instances where the objects are represented in an incorrect form, such as wrong colors, positions, or relationships. Details are shared in Section 4.5.

As showcased in Table 3, our OPERA achieves up to 27.5% improvements compared with Beam search decoding, while keeping the detailedness of answers. Since GPT-4V’s abilities of perception and reasoning are very closed to human beings, the GPT-4V evaluation results somehow reflect the strong performance of reducing hallucinations from the perspective of human’s feeling.

POPE evaluation on hallucinations. The Polling-based Object Probing Evaluation (POPE) is a recently introduced method designed to assess hallucination issues in MLLMs. Similar to CHAIR, POPE focuses on evaluating object hallucination, utilizing an essay question format to prompt the model like “Is There a