Evaluating Object Hallucination in Large Vision-Language Models

Introduction

Large language models (LLMs) Zhao et al. 2023 have shown remarkable abilities to solve various complex tasks by following human instructions in a zero-shot manner. The success of LLMs drives the researchers to devise more powerful multimodal models based on the superior capacity of LLMs, to enhance the understanding of visual semantics Alayrac et al. 2022; Li et al. 2023b. As an exemplified work, GPT-4 OpenAI 2023 has exhibited the exciting performance of LLMs on multimodal tasks and scenarios.

Following this line of research, a surge of studies Zhu et al. 2023; Gao et al. 2023; Li et al. 2023a have been proposed to enhance the vision-language pre-trained model (VLPM) Gan et al. 2022 by incorporating powerful LLMs Touvron et al. 2023; Chiang et al. 2023, which are called large vision-language model (LVLM). Typically, existing work reuses the visual encoder in VLPMs to handle image data, while replacing the original language encoder with LLMs. After vision-language pre-training Alayrac et al. 2022; Li et al. 2022b and visual instruction tuning Liu et al. 2023, LVLMs can fulfill complex tasks according to human instructions, demonstrating strong capacities in solving various vision-language tasks, e.g., image captioning Ordonez et al. 2011; Hodosh et al. 2015; Sharma et al. 2018; Agrawal et al. 2019 and visual question answering Antol et al. 2015a; Zhang et al. 2016; Goyal et al. 2017.

Despite the success of LVLMs, previous work has revealed that their main components, i.e., LLMs and VLPMs, both suffer from hallucination. Especially, LLMs tend to hallucinate unintended text Huang et al. 2021; Bang et al. 2023, and VLPMs might generate nonexistent objects in the image Biten et al. 2022 (termed as object hallucination). It is generally believed that the hallucination would degrade the model performance and greatly harm the user experiences in real-world applications MacLeod et al. 2017; Ji et al. 2022. Therefore, it is natural to ask the question: does hallucination still exist in LVLMs? In this paper, we systematically evaluate the issue of object hallucination in existing LVLMs, which refers to generating contents that are inconsistent with ground-truth objects in the given image.

To conduct our study, we first use the CHAIR (Caption Hallucination Assessment with Image Relevance) metric Rohrbach et al. 2018, and examine the hallucination degree of several representative LVLMs on the MSCOCO dataset. Our preliminary experiments (Table 1) show that most of LVLMs severely suffer from object hallucination, and are even more prone to hallucinate than small vision-language models. Besides, we find that the existing object hallucination evaluation method may not be best suited for LVLMs and further propose a Polling-based Object Probing Evaluation (POPE) method. The basic idea is to convert the evaluation of hallucination into a binary classification task by prompting LVLMs with simple Yes-or-No short questions about the probing objects (e.g., Is there a car in the image?). We show that such a method is more stable and flexible. Besides, by using different object sampling strategies, we validate that existing LVLMs are prone to hallucinate objects which frequently appear or co-occur in the visual instruction dataset.

Our main contributions are as follows: (1) We conduct an empirical study on object hallucination for several representative LVLMs and find that they are highly affected by object hallucination. (2) We discuss the potential reasons behind this promblem, e.g., LVLMs tend to generate frequently appearing or co-occurring objects in the instruction corpora. (3) We propose an object hallucination evaluation approach called POPE, which is more stable and can be easily extended to unannotated datasets.

Background

Since LLMs have been shown to be general task solvers in a zero-shot/few-shot manner, a number of studies are devoted to improving VLPM by integrating powerful LLMs for more accurate language understanding and generation Zhu et al. 2023; Liu et al. 2023; Dai et al. 2023a. In this paper, we refer to the enhanced VLPMs with the integration of LLMs as Large Vision-Language Models (LVLM).

Generally speaking, an LVLM consists of a vision encoder, a language encoder (i.e., an LLM), and a cross-modal alignment network. The training of LVLMs is generally composed of three major steps. First, a vision encoder and a language encoder are pre-trained on large-scale unimodal data (i.e., image and text data, respectively). Second, these two encoders are aligned through image-text alignment pre-training, which enables the LLM to generate a meaningful caption for a given image. Third, the aligned model is further fine-tuned on image-text instructions, so that it can generate satisfactory answers w.r.t. to a natural language question regarding a specific image. Note that in the second and third steps, we can optionally fine-tune different components instead of performing full-parameter fine-tuning.

Once the visual encoder and the LLM are well aligned, the derived LVLM can demonstrate a superior visual understanding ability. It can not only grasp the visual semantics of objects in the image, but also deeply understand the linguistic semantics for these objects by leveraging the parametric knowledge in the LLM. Further, the LVLM can perform complex reasoning over the related concepts about these objects, thus achieving an improved performance on a variety of multimodal tasks, e.g., visual question answering (VQA).

2 Object Hallucination

Although LVLMs are powerful in solving vision-language tasks, they also suffer from the issue of object hallucination as VLPMs. In the literature of computer vision field Rohrbach et al. 2018; Biten et al. 2022, object hallucination refers that the model generating descriptions or captions that contain objects which are inconsistent with or even absent from the target image. In general, object hallucination can be defined at different semantic levels. The most straightforward way is to define it over the object level, while more fine-grained definitions might be concerned with the attributes or characteristics of objects. In this work, we focus on coarse-grained object hallucinations in the model-generated captions and leave fine-grained object hallucinations such as the number, attributes, and positions of the object for future work. We present an example of object hallucination in Figure 1, where the hallucinated object “meat bowl”,“bottle”, “beverage”, “condiment” are generated by the underlying LVLMs.

The hallucination phenomenon hinders the safe use of LVLMs in real-world deployment, as it may result in unexpected consequences caused by these hallucinated objects MacLeod et al. 2017. For example, due to an incorrect understanding of the external environment, an autonomous driving system would make wrong decisions when encountering unexpected events, which might lead to serious safety issues. In order to mitigate these issues, this work aims to study how object hallucination exists in LVLMs from an evaluation perspective.

Object Hallucination in LVLMs

In this section, we evaluate the object hallucination problem in popular LVLMs using an existing method. We first introduce the evaluation settings and then analyze the experimental results.

Caption Hallucination Assessment with Image Relevance (CHAIR) Rohrbach et al. 2018 is a popular metric for evaluating object hallucination in image captioning tasks. Given the ground truth objects in the image, CHAIR calculates the proportion of objects that appear in the caption but not the image. Existing work commonly adopts its two variants, i.e., $\mbox{CHAIR}_{I}$ and $\mbox{CHAIR}_{S}$, which evaluate the hallucination degree at the object instance level and sentence level respectively. They can be formulated as:

We select five recently released LVLMs, i.e., mPLUG-Owl Ye et al. 2023, LLaVA Liu et al. 2023, Multimodal-GPT Gong et al. 2023, MiniGPT-4 Zhu et al. 2023 and InstructBLIP Dai et al. 2023a and prompt them with following instructions to generate captions about images in MSCOCO Lin et al. 2014:

$\bullet$ $I_{1}$: Generate a short caption of the image.

$\bullet$ $I_{2}$: Provide a brief description of the given image.

Then, we calculate CHAIR on these captions. We leave more details about the introduction to the dataset and evaluated models in Appendix A.

2 Evaluation Results

As the evaluation results illustrated in Table 1, most instruction-tuned LVLMs suffer from the object hallucination problem, even more serious than small models, e.g., LLaVA (32.7) v.s. OSCARbase (13.0) on CHAIRS using Instruction 1. It indicates that object hallucination is an important problem for LVLMs and deserves to be concerned about. As a comparison, InstructBLIP hallucinates less than other LVLMs. A possible reason is that its visual instructions are collected from a wide variety of publicly available datasets, which are relatively short. In contrast, other LVLMs mostly employ the visual instructions generated by unimodal LLMs Liu et al. 2023. Such synthetic visual instructions are generally longer and more informative, but may involve unexpected descriptive information (hallucination inherent from LLMs) that is inconsistent with the image, which could mislead LVLMs.

Disadvantages of CHAIR.

As Table 1 shows, the evaluation results can be affected by other factors, e.g., instruction designs and the length of captions. Specifically, although the adopted two instructions have similar semantic meanings, LVLMs prompted by Instruction 2 can even result in doubled values of CHAIR metrics compared with those prompted by Instruction 1, and the performance order of some LVLMs also changes (e.g., CHAIRI values of LLaVA and MultiModal-GPT). It indicates the instability of the CHAIR metric when different instructions are employed. Besides, as CHAIR requires to examine whether the mentioned objects are hallucinated in the generated caption, it needs complex human-crafted parsing rules to perform exact matching, which has not been adapted to the special generation styles of LVLMs and may lead to misclassification errors.

Thus, it is necessary to consider a more suitable method that can stably and conveniently evaluate the object hallucination problem in LVLMs.

Influence of Instruction Data on Object Hallucination

Considering their impressive performance on complex vision-language tasks Chen et al. 2023; Bai et al. 2023; Li et al. 2023a, it is counter-intuitive that the hallucination problem of LVLMs is so severe. Since smaller VLPMs suffer less from object hallucination, it is possible that the visual instruction-tuning process of LVLMs exacerbates object hallucination. In this section, we investigate the influence of the visual instruction data. We first make two basic hypotheses in Section 4.1 and then conduct qualitative and quantitative analysis to verify them in Section 4.2 and Section 4.3.

As the visual instruction datasets of these LVLMs are mostly constructed based on MSCOCO Lin et al. 2014, they generally share a similar unbalanced object distribution where top frequent objects occupy a major part of the dataset. After being fine-tuned on them, LVLMs may also be prone to generate (or hallucinate) frequently appearing objects in MSCOCO. Additionally, the presence of frequently co-occurring object groups (e.g., laptop, mouse and keyboard) may also contribute to object hallucination. LVLMs can be elicited by the existing objects in the image to hallucinate other objects that frequently co-occur with them. Therefore, we hypothesize that (1) LVLMs are prone to hallucinate frequently appearing objects in the visual instruction datasets; (2) LVLMs are prone to hallucinate objects that frequently co-occur with ground-truth objects in the image. We conduct qualitative and quantitative analyses in the following parts to verify them.

2 Qualitative Analysis

We first qualitatively analyze the correlation between the appearance frequency and hallucination. For the first hypothesis, we plot a bar chart between the top ten frequently appearing objects in MSCOCO and their hallucination times in the validation set of MSCOCO; for the second hypothesis, we select the top ten frequently co-occurring objects with “dining table” and also plot a bar chart to show their hallucination times across images that really contain “dining table”. We show the results of MiniGPT-4, LLaVA, MultiModal-GPT and mPLUG-Owl in Figure 2. Obviously, with the decreasing of the occurrence frequency of objects (from right to left), there is a notable decrease in the hallucination times for all four LVLMs. It reveals that the frequently appearing and co-occurring objects in the visual instruction dataset are indeed more likely to be hallucinated by LVLMs. To better support our results, we also list the full statistics of all 80 COCO objects in Appendix B.

3 Quantitative Analysis

To further consolidate the above findings, we employ the top-$k$ hit ratio (HR@$k$) to measure the consistency between the appearance frequency and hallucination times of objects, which is defined as:

where HRA and HRC quantify the correlations between hallucination times and appearing and co-occurring frequency respectively. $n$ is the total number of images, ${\mbox{Hallucinated}}(i)$ denotes the number of hallucinated objects in the $i$-th example, ${\mbox{Hit@}k}(i)$ denotes the number of top-$k$ frequently appearing MSCOCO objects in ${\mbox{Hallucinated}}(i)$, and ${\mbox{Hit@}}k(i,o)$ denotes the number of top-$k$ frequently co-occurring objects with the probing object $o$ in ${\mbox{Hallucinated}}(i)$. Therefore, HR@$k$ can reflect the proportion of top-$k$ frequently appearing or co-occurring objects in all hallucinated objects.

We present the HRA and HRC(dining table) of top 30 objects in Table 2 and leave HRC@(chair) and HRC@(car) in Appendix C. The HRA@10 and HRC@10(dining table) of all LVLMs are near 0.5 and 0.6, respectively. It indicates that, on average, approximately half of the hallucinated objects in each image belong to the top 10 frequently appearing COCO objects, while more than half are among the top 10 frequently co-occurring objects with the objects already present in the image. When we broaden our observation to the top 30 objects, this proportion continues to increase. These findings further verify that LVLMs mostly hallucinate common objects in the visual instruction data and inspire us to design three sampling strategies in our evaluation pipeline.

POPE

In this section, we devise Polling-based Object Probing Evaluation (POPE), a simple yet effective approach for evaluating hallucination in LVLMs. We first provide an overview of POPE, and then evaluate the representative LVLMs with POPE. Finally, we discuss the stability and scalability of our method, and also analyze the impact of hallucination on VQA task.

In the empirical results of Section 3, we have revealed the severity of the object hallucination problem in LVLMs and highlighted the limitations of the existing evaluation method, e.g., sensitive to instructions and biased to short captions. Besides, existing methods mostly rely on parsing the generated captions to extract the predicted objects, which usually require human-crafted complex rules and are still inevitable to omit or misclassify objects.

Therefore, we consider devising a more suitable method for the stable, fair and flexible object hallucination evaluation of LVLMs, namely polling-based object probing evaluation (POPE). Specifically, POPE formulates the evaluation of object hallucination as a binary classification task that prompts LVLMs to output “Yes” or “No”, e.g., “Is there a chair in the image?”. In this way, by sampling objects that LVLMs are prone to hallucinate, we can construct a set of hard questions to poll LVLMs. As standard answers to these questions are just “Yes” or “No”, we can easily identify them without complex parsing rules, and avoid the influence of instruction designs and caption length, thus guaranteeing stability, fairness and flexibility.

Given an image caption dataset, POPE focuses on constructing a set of triples, each of which consists of an image, multiple questions and their answers (“Yes” or “No”). The formulated definition of a triple can be described as:

where $x$ denotes the image, $q(o_{i})$ is the question probing $o_{i}$ based on a template “Is there a/an