ViM: Out-Of-Distribution with Virtual-logit Matching

Introduction

Considering most deep image classification models are trained in the closed-world setting, the out-of-distribution (OOD) issue arises and deteriorates customer experience when the models are deployed in production, facing inputs coming from the open world . For instance, a model may wrongly but confidently classify an image of crab into the clapping class, even though no crab-related concepts appear in the training set. OOD detection is to decide whether an input belongs to the training distribution. OOD detection complements classification and finds its application in fields such as autonomous driving , medical analysis and industrial inspection . A comprehensive review of OOD and related topics including open set recognition, novelty detection and anomaly detection can be found in .

Built upon the success of prior arts, we design a novel scoring function termed the Virtual-logit Matching (ViM) score, which is the softmax score of a constructed virtual OOD class whose logit is jointly determined by the feature and the existing logits. To be specific, the scoring function first extracts the residual of the feature against a principal subspace, and then converts it to a valid logit by matching its mean over training samples to the average maximum logits. Finally, the softmax probability of the devised OOD class is the OOD score. From the construction of ViM, we can see intuitively that the smaller the original logits and the greater the residual, the more likely it is to be OOD.

Different from the aforementioned methods, another line of works tailors the features learned by the network to better identify ID and OOD by imposing dedicated regularization losses or by exposing generated or real collected OOD samples . As they all require the re-training of the network, we briefly mention them here and will not delve into the details.

Recently, OOD detection in large-scale semantic space has attracted increasing attention , advancing OOD detection methods toward real-world applications. However, the current shortage of clean and realistic OOD datasets for large-scale ID datasets becomes an impediment to the field. Previous OOD datasets were curated from public datasets which were collected with a predefined tag list, such as iNaturalist, Texture, and ImageNet-21k (Tab. 1). This may lead to a biased performance comparison, specifically, the hackability of small coverage as described in Sec. 5. To avoid this risk, we build a new OOD benchmark for ImageNet-1K models, OpenImage-O, from OpenImage dataset with natural class distribution. It contains 17,632 manually filtered images, and is $7.8\times$ larger than the recent ImageNet-O dataset.

We extensively evaluate our method on various models using ImageNet-1K as the ID dataset. The model architectures range from the classical ResNet-50 , to the recent BiT , and to the latest ViT-B16 , RepVGG , DeiT and Swin Transformer . From the results on four OOD datasets, including OpenImage-O, ImageNet-O, Texture, and iNaturalist, we found that model selection affected the performance of many baseline methods, while our method performs stably well. Specially, our method achieved an average AUROC of 90.91% using the BiT model, which greatly surpasses the best baseline whose average AUROC is 86.62%.

Our contributions are threefold. (1) We proposed a novel OOD detection method ViM, that works well for a large range of models and datasets, owing to the effective fusion of information from both features and logits. The method is lightweight and fast, requiring neither extra OOD data nor re-training. (2) We conducted comprehensive experiments and ablation studies on the ImageNet-1K dataset, including CNNs and vision transformers. (3) We curated a new OOD dataset for ImageNet-1K called OpenImage-O, which is very diverse and contains complex scenes. We believe it will facilitate research on large-scale OOD detection.

Related Work

Network/Loss Design

Many works redesign the training loss to be OOD-aware or add regularization terms to push part ID/OOD features. DeVries et al. augment the network with a confidence estimation branch that uses misclassified in-distribution examples as a proxy for out-of-distribution examples. MOS modifies the loss to use the pre-defined group structure so that the minimum group-wise “else” class probability can indicate the OOD-ness. Zaeemzadeh et al. forces the ID samples to embed into a union of 1-dimensional subspaces during training and computes the minimum angular distance from the feature to the class-wise subspaces. Generalized ODIN uses a dividend/divisor structure to encode the prior knowledge of decomposing the confidence of class probability. Different from these methods, our method does not require model retraining, thus not only is it easier to apply, but the ID classification accuracy is also preserved.

OOD Data Exposure

Outlier Exposure utilizes an auxiliary OOD dataset to improve OOD detection. Dhamija et al. regularize samples from extra background classes to have uniform logits and to have small feature norms. Lee et al. use GAN to generate OOD samples that lie near the ID samples and push the prediction of OOD samples to the uniform distribution. Several methods, including MCD , NGC and UDG , can utilize external unlabeled noisy data to enhance the OOD detection performances. Different from these methods, our method does not require additional OOD data and thus avoids biases towards the introduced OOD samples .

Motivation: The Missing Info in Logits

For a series of OOD detection methods that are based on logits or softmax probabilities, we find that their performances are limited. In Fig. 1, feature-based OOD scores such as Mahalanobis and Residual are good at detecting OOD in ImageNet-O, while all methods that are based on logit/probability lag behind. This is not an accident, as is again shown in Fig. 2. The AUROC of the state-of-the-art probability-based method KL Matching is still lower than straightforwardly designed OOD scores in feature space on Texture dataset. This motivates us to study the influence of the lost information going from features to logits.

Geometrically, each logit $l_{i}$ is the inner product between the feature ${\bm{x}}^{\prime}$ and the class vector ${\bm{w}}_{i}$ (the $i$-th column of ${\bm{W}}$). Later when generalizing logits to virtual logits, we will replace ${\bm{w}}_{i}$ with a subspace, and replace the inner product with a projection. The bias term is safely omitted in the new coordinate system. In the remaining part of the paper, we assume the feature space uses the new coordinate system. Logits contain class-dependent information, yet there is class-agnostic information in feature space that is not recoverable from logits. We study two cases (null space and principal space) and discuss the two OOD scores (NuSA and Residual) that rely on them, respectively.

A feature ${\bm{x}}$ can be decomposed into ${\bm{x}}={\bm{x}}^{W^{\perp}}+{\bm{x}}^{W}$, where $W$ is the column space of ${\bm{W}}$, ${\bm{x}}^{W^{\perp}}$ and ${\bm{x}}^{W}$ are projections of ${\bm{x}}$ to $W^{\perp}$ and $W$, respectively. $W^{\perp}$ is the null space of ${\bm{W}}^{T}$, and we have ${\bm{W}}^{T}{\bm{x}}^{W^{\perp}}={\bm{0}}$. The component ${\bm{x}}^{W^{\perp}}$ does not affect classification, but it influences OOD detection. It is demonstrated in that one can perturb an image intensely yet constrain the difference between the features in $W^{\perp}$. The resulting outlier images are not like any of the ID images but retains high confidence in classification. Taking advantage of this, they define an ID score NuSA (null space analysis) as

Intuitively, NuSA uses the angle ($=\arccos({\text{NuSA}}({\bm{x}}))$) between ${\bm{x}}$ and $W$ to indicate the OOD-ness. From Fig. 2 we can see that the simple angle information clearly distinguishes OOD examples in Texture with an AUROC 95.50%, surpassing methods based on logits and the competitive method KL Matching based on softmax probability.

OOD Score Based on Principal Space

It is generally assumed that features lie in low-dimensional manifolds . For simplicity, we use linear subspace (in the new coordinate system) passing through the origin ${\bm{o}}$ as the model. We define the principal space as the $D$-dimensional subspace $P$ spanned by eigenvectors of the largest $D$ eigenvalues of the matrix ${\bm{X}}^{T}{\bm{X}}$, where ${\bm{X}}$ is the ID data matrix. Features that deviate from the principal space are likely to be OOD examples. We can define

to capture the deviation of features from the principal space. Here ${\bm{x}}={\bm{x}}^{P}+{\bm{x}}^{P^{\perp}}$ and ${\bm{x}}^{P^{\perp}}$ is the projection of ${\bm{x}}$ to ${P^{\perp}}$. The residual score is similar to the reconstruction error in except that they employ nonlinear manifold learning for dimension reduction. Note that after the projection onto logits, this deviation is corrupted since the matrix ${\bm{W}}^{T}$ projects to a lower dimensional space than the feature space. Fig. 2 shows that Residual score improves over the NuSA score on both datasets, making the performance contrast between feature-based methods with logit/probability-based methods more striking.

Fusing Class-dependent and Class-agnostic Information

In contrast to methods on logit/probability, both the NuSA and the Residual do not consider information that is specific to individual ID classes, namely they are class-agnostic. As a consequence, these scores ignore the feature similarity to each ID class, and are ignorant about which class the input resembles most. This gives an explanation of their worse performance on the iNaturalist OOD benchmark, as iNaturalist samples need to distinguish subtle differences between fine-grained classes. We hypothesize that unifying the information from feature space and the logits could improve the detection performance on a broader type of OOD samples. Such a solution is presented in Sec. 4 using the concept of virtual logit.

Virtual-logit Matching

To unify the class-agnostic and class-dependent information for OOD detection, we propose an OOD score by Virtual-logit Matching, abbreviated as ViM. The pipeline is illustrated in Fig. 3, where there are three steps, operating at the feature, the logit, and the probability, respectively. To be specific, for feature ${\bm{x}}$, (1) extract the residual ${\bm{x}}^{P^{\perp}}$ of ${\bm{x}}$ against the principal subspace $P$; (2) convert the norm $\|{\bm{x}}^{P^{\perp}}\|$ to a virtual logit by rescaling; and (3) output the softmax probability of the virtual logit as the ViM score. Below we give more details. Recall the notations: $C$ is the number of classes, $N$ is the feature dimension, and ${\bm{W}}$ and ${\bm{b}}$ are the classification weight and bias, respectively.

Firstly we offset the feature space by a vector ${\bm{o}}=-({\bm{W}}^{T})^{+}{\bm{b}}$ so that it is bias-free in the computation of logits as Eq. 1. The principal subspace $P$ is defined by the training set ${\bm{X}}$, where rows are features in the new coordinate system with origin ${\bm{o}}$. Suppose the eigendecomposition on the matrix ${\bm{X}}^{T}{\bm{X}}$ is

Virtual-logit Matching

is the norm of the residual rescaled by a per-model constant $\alpha$. The norm $\|{\bm{x}}^{P^{\perp}}\|$ cannot be used as a new logit directly since the latter softmax will normalize over the exponential of logits and thus is very sensitive to the scale of logits. If the residual is very small compared to the largest logit, then after the softmax the residual will be buried in the noise of logits. To match the scales of the virtual logit, we compute the average norm of the virtual logit on the training set and also the mean of the maximum logit on the training set, then

where ${\bm{x}}_{1},{\bm{x}}_{2},\ldots,{\bm{x}}_{K}$ are uniformly sampled $K$ training examples, and $l^{i}_{j}$ is the $j$-th logit of ${\bm{x}}_{i}$. In this way, on average, the scale of the virtual logit is the same as the maximum of the original logits.

The ViM Score

We append the virtual logit to the original logits and compute the softmax. The probability corresponding to the virtual logit is defined as ViM. Mathematically, let the $i$-th logit of ${\bm{x}}$ be $l_{i}$, and then the score is

This equation reveals that two factors affect the ViM score: if its original logits are larger, then it is less of an OOD example; while if the norm of residual is larger, it is more likely to be OOD. The computational overhead is comparable to the last fully-connected layer (mapping from feature to logit) in the classification network, which is small.

Connection to Existing Methods

Note that applying a strictly increasing function to the scores does not affect the OOD evaluation. Apply the function $t(x)=-\ln\left(\frac{1}{x}-1\right)$ to the ViM score, then we have an equivalent expression

The first term is the virtual logit in Eq. 5 while the second term is the energy score . ViM completes the energy method by feeding extra residual information from features. The performance is much superior to energy and residual.

OpenImage-O Dataset

We build a new OOD dataset called OpenImage-O for the ID dataset ImageNet-1K. It is manually annotated, comes with a naturally diverse distribution, and has a large scale with 17,632 images. It is built to overcome several shortcomings of existing OOD benchmarks. OpenImage-O is selected image-by-image from the test set of OpenImage-V3, including 125,436 images collected from Flickr without a predefined list of class names or tags, leading to natural class statistics and avoiding an initial design bias.

Some previous works on large-scale OOD detection select a portion of other datasets solely based on class labels. While class-level annotation costs less, the resulting dataset might be much noisier than expected. For example, the Places and the SUN dataset selected by have a large portion of images that are indistinguishable from ID samples. Another example is the Texture , in which the bubbly texture overlaps with the bubble class in ImageNet. Thus creating OOD datasets by querying tags is not reliable and per-image human inspection is needed for the confirmation of validity.

Hackability of Small Coverage

If the OOD dataset has a central topic such as the Texture, featuring a less diverse distribution, then it might be easy to be “hacked”. In Tab. 2, the gap between the highest and the average AUROC over nine methods for BiT are: OpenImage-O 5.61, iNaturalist 6.06, Texture 10.52, and ImageNet-O 14.39. Having larger gaps implies that the dataset is easier to improve.

Construction Process of OpenImage-O

We construct the OpenImage-O based on the OpenImage-v3 dataset . For every image in its testing set, we let human labelers to determine whether it is an OOD sample. To assist labeling, we simplified the task as distinguishing the image from the top-10 categories predicted by an ImageNet-1K classification model, i.e., the image is OOD if it does not belong to any of the 10 categories. Category labels as well as the most similar image to the test image in each category, measured by cosine similarity in the feature space, were presented for visualization. To further improve the annotation quality, we design several schemes: (1) Labelers can choose “Difficult”, if they cannot decide whether the image belongs to any of the 10 categories; (2) Each image was labeled by at least two labelers independently, and we took the set of OOD images having consensus from the two; (3) Random inspection was performed to guarantee the quality.

Experiment

In this section, we compare our algorithm with state-of-the-art OOD detection algorithms. Following the prior work on large-scale OOD detection, we choose ImageNet-1K as the ID dataset. We benchmark the algorithms using both the CNN-based and the transformer-based models. Detailed experimental settings are as follows.

Four OOD datasets (Tab. 1) are used to comprehensively benchmark the algorithms. OpenImage-O is our newly collected large-scale OOD dataset. Texture consists of natural textural images and we removed four categories (bubbly, honeycombed, cobwebbed, spiralled) that overlapped with ImageNet. iNaturalist is a fine-grained species classification dataset. We use the subset from . Images in ImageNet-O are adversarially filtered so that they can fool OOD detectors.

Evaluation Metrics

Two commonly used metrics are reported. The AUROC is a threshold-free metric that computes the area under the receiver operating characteristic curve. Higher value indicates better detection performance. FPR95 is short for FPR@TPR95, which is the false positive rate when the true positive rate is 95%. The smaller FPR95 the better. We report both their numbers in percentage.

Experiment Settings

BiT (Big Transfer) is a variant of ResNet-v2, which employs group normalization and weight standardization. The BiT-S model series is pre-trained on ImageNet-1K, and we take the officially released checkpoint of BiT-S-R101$\times$1 for experiments. ViT (Vision Transformer) is a transformer-based image classification model which treats images as sequences of patches. We use the officially released ViT-B/16 model, which is pre-trained on ImageNet-21K and fine-tuned on ImageNet-1K. Since the compared algorithms do not require re-training, the ID accuracies are not affected. Results on more model architectures, including CNN-based RepVGG , ResNet-50d , and transformer based Swin and DeiT , are listed in Sec. 6.3. Their pre-trained weights are obtained from the timm repo . When estimating the principal space, $K=200,000$ images are randomly sampled from the training set. For features spaces with dimension $N>1500$, we set the dimension of principal space to $D=1000$, and set $D=512$ otherwise.

Baseline Methods

We compare ViM with eight baselines that do not require fine-tuning. They are MSP , Energy , ODIN , MaxLogit , KL Matching , Residual, ReAct and Mahalanobis . For Mahalanobis, we followed the setting in , which uses only the final feature instead of an ensemble of multiple layers . For ReAct, we use the Energy+ReAct setting with rectification percentile $p=99$. The Residual is defined in Eq. 3.

1 Results on BiT

We present the results of the BiT model at the first half of Tab. 2. The best AUROC is shown in bold and the second and third place ones are shown with underlines.

On three datasets, including OpenImage-O, Texture, and ImageNet-O, ViM achieves the largest AUROC and the smallest FPR95. On average ViM has 90.91% AUROC, which surpasses the second place by 4.29%. The average FPR95 is also the lowest among them. In particular, regarding Eq. 8, an interpretation of ViM in terms of the Residual score and the Energy score, the results show that ViM is significantly better than the two methods on all datasets. This indicates that ViM non-trivially combined the OOD information in Residual and in Energy. However, on iNaturalist, ViM is only on the third place. We hypothesize that its moderate performance on iNaturalist relates to how much information is contained in the residual, because iNaturalist has the smallest average residual norm among four OOD datasets (iNaturalist 4.65, OpenImage-O 5.04, ImageNet-O 5.16, and Texture 8.16).

Effect of Information Source

For OOD detection performances on BiT model, Tab. 2 shows an interesting pattern regarding the information source. If feature variations in the null space are absent, such as in methods that rely on logits and softmax, performances on Texture and ImageNet-O are restricted. For example, on the Texture dataset, the best performing method that relies on logit and softmax is KL Matching, which has 86.92% AUROC and is far behind ViM, Mahalanobis, and Residual, which operate on the feature space. In contrast, if the class-dependent information is dropped, such as in the Residual method, performances in iNaturalist and OpenImage-O are also limited. The proposed ViM score, however, is competent regardless of dataset types.

2 Results on ViT

has discussed the benefit of large-scale pre-trained transformers on OOD tasks. However, their experiments are conducted on CIFAR100/10 and only two baseline methods are compared. We provide a comprehensive OOD evaluation on ImageNet-1K over a wide range of methods in the second half of Tab. 2.

The two best-performing methods for the ViT model are ViM and Mahalanobis. Their AUROCs are close on all four datasets. However, Mahalanobis needs to compute the class-wise Mahalanobis distance, which makes its computation costly. In contrast, our method is lightweight and fast. Four methods, ReAct, Energy, MaxLogit, and ODIN, are the second best ones, and the remaining three methods have relatively low AUROCs.

Difference between ViT and BiT

Since the ViT model is pre-trained on the ImageNet-21K dataset, the semantics it has seen is much larger than the BiT model. The OOD performance is relatively saturated. Although on most OOD datasets ViT is significantly better than BiT, we observe that ViT performs less competitively on the Texture dataset. We hypothesize that it is related to the observation in that higher layers of ViT maintain spatial location information more faithfully than ResNets. ViT has high responses for local patches. However, textural images with similar local patches but not revealing the whole object are regarded as OOD of ImageNet (see example images in Fig. 2).

3 Results on More Model Architectures

We show more results on a variety of model architectures. In particular, we choose two CNN-based models RepVGG and ResNet-50d and two transformer-based models Swin Transformer and DeiT . Their average AUROCs and average FPR95s over the four OOD datasets are listed in Tab. 3. It is shown that ViM is robust to model architecture changes. The detailed experiment setting and results are in the supplementary materials.

4 The Effect of Hyperparameter

In the feature of each class is represented by a 1-dimensional subspace, so a natural choice for the dimension $D$ of principal space is the number of classes $C$. For models like ViT whose feature dimension $N$ may be less than the number of classes $C$, we empirically suggest taking a number in the range $[N/3,2N/3]$. We show in Fig. 4 that our method is robust to the selection of dimensions. However, if the application permits, one can adjust this parameter according to a hold-out OOD dataset. In our experiments, we set $D=1000$ for BiT and $D=512$ for ViT.

The Matching Parameter α𝛼\alpha

The matching parameter controls the relative importance of the trade-off between different OOD features. Since OOD distribution is unknown, we suggest keeping them to be of equal importance. This is how $\alpha$ is defined in Eq. 6. It is easy to tune the parameter to fit some types of OOD datasets, but it is hard to improve all datasets at the same time. We show the result of perturbing the matching parameter by multiplying a factor in Fig. 5. If the multiple is larger, then information from the feature space is given more weight. Otherwise, information from logits is given more importance. Overall the best choice is no perturbation, suggesting that the defined $\alpha$ is a good choice.

5 The Effect of Grouping

In addition, we also compare with MOS , which exploits grouping structure in large-scale semantic spaces. Two methods are added to the comparison. (1) MaxGroup is the group version of MSP, which first obtains the group-wise probability by summing over the constituent classes, and then takes the maximum group probability as the ID score. (2) ViM+Group also takes the maximum group probability as the ID score, except that the probabilities are taken from the $(C+1)$ dimensional vector, with an extra ViM virtual class participating in the softmax normalization.

MaxGroup and ViM+Group are evaluated on the pre-trained weights of BiT, while MOS needs to fine-tune the model using group-based learning. Results are shown in Tab. 4. We observe that (1) the average AUROC of MaxGroup improves over the vanilla MSP from 77.25% to 79.23%, showing the usefulness of group information; and (2) both our original ViM and the group version of ViM are better than MOS on three of four datasets by large margins.

6 Limitation of ViM

As we have noticed in Sec. 6.1, ViM shows less performance gains on OOD datasets that have small residuals, such as iNaturalist. Besides, the property that ViM does not need training is a double-edged sword. It means that ViM is limited by the feature quality of the original network.

Conclusion

In this paper, we present a novel OOD detection method: the Virtual-logit Matching (ViM) score. It combines the information from both the feature space and the logits, which provides the class-agnostic information and the class-dependent information, respectively. Extensive experiments on the large-scale OOD benchmarks show the effectiveness and robustness of the method. Especially, we tested ViM on both CNN-based models and transformer-based models, showing its robustness across model architectures. To facilitate the evaluation of large-scale OOD detection, we create the OpenImage-O dataset for ImageNet-1K, which is of high-quality and large-scale.

Acknowledgement

This work was supported in part by Innovation and Technology Commission of the Hong Kong Special Administrative Region, China (Enterprise Support Scheme under the Innovation and Technology Fund B/E030/18). Haoqi Wang was also supported by the Technology Leaders of Tomorrow (TLT) Programme of HKSTP InnoAcademy.

Appendix A Detailed Information of Models (Sec. 6)

In the experiment, we benchmarked a collection of deep classification models. Their detailed information, including the specification, the architecture, the pre-train information, and the top-1 accuracy, is listed in Tab. 5. To summarize, half of them are CNN-based, and half are transformer-based. Vision Transformer and Swin Transformer are pre-trained on ImageNet-21K before training on the ImageNet-1K.

Appendix B Detailed Results of Four Models (Sec. 6.3)

In Sec. 6.3 we gave the average AUROC and FPR95 for RepVGG, ResNet-50d, Swin Transformer and DeiT. We provide the detailed AUROC and FPR95 on OpenImage-O, Texture, iNaturalist, and ImageNet-O in Tab. 6.

Appendix C Details on OpenImage-O (Sec. 5)

An illustrative software interface for labelers is shown in Fig. 6. For each candidate OOD image to be labeled, we find the top 10 classes in ImageNet-1K predicted by a classification model. Then we gather the most similar images in those top 10 classes by cosine similarity in the feature space. Next, we patch them as well as their labels with the corresponding OpenImage samples, and let the labelers distinguish whether the OpenImage sample belongs to any of the top 10 categories. We also set a choice called difficult, so that labelers can put the undistinguishable hard samples into the difficult category. To reduce annotation noises, each image is labeled twice from different group of labelers. Then we take the set of OOD images having consensus from the two groups, resulting in an OOD dataset with 17,632 unique images. In the end, a random inspection process is performed to guarantee the quality of the OOD dataset.

The OpenImage-O follows a natural image distribution as both the source dataset and the labeling process do not involve any filtration based on pre-defined list of labels. To get a sense of its distribution, we use the BiT model to find the most similar ID class in ImageNet for each OOD image. Then the histogram is illustrated in Fig. 7. It shows that the coverage of OpenImage-O is broader compared to the other three OOD datasets.

Appendix D Details on Grouping (Sec. 6.5)

MOS is trained using the officially released code and its default parameter setting. For all experiments in Sec. 6.5, the grouping strategy follows the taxonomy grouping defined in .

The grouping strategy is less effective for the ViT model, as seen from results in Tab. 7. Comparing MSP with its group version, MaxGroup, we can see that the improvement on AUROC is very small, while FPRs become even worse. Examining ViM with its group variant ViM+Group, we can see that their difference is very small, and the original version of ViM is slightly better than ViM+Group.

Appendix E Details on Baselines (Sec. 6)

On the BiT model, when including lower level features, the performance of Mahalanobis degrades a lot. The average AUROC on the four OOD datasets is 56%, which is much worse than the baseline MSP. Similar results is also found in [18, Table 1]. In this paper, we implement the Mahalanobis score using the feature vector before the final classification fc layer, as in . The precision matrix and the class-wise average vector are estimated using 200,000 random training samples. The ground-truth class label is used during computation.

KL Matching

We estimate the class-wise average probability using 200,000 random training samples. Following the practice of , the predicted class is used instead of ground-truth labels. We would like to note that the hyperparameter selection for OOD methods should not base on the ID set that is used for computing FPR95 and AUROC (in our case, its the validation set of ImageNet), because once the OOD method overfits the validation set, the evaluation result can be higher than the actual performance.

ReAct

For ReAct, we use the Energy+ReAct setting, which is the most effective settings in . In the original paper, they recommended the 90-th percentile of activations estimated on the ID data for the clipping threshold. However, for BiT and ViT, we found that the rectification percentile $p=99$ works much better than 90. So we report results using $p=99$.

Appendix F OOD Examples Detected by KL Matching and Residual (Sec. 3)

In Sec. 3, we showed that feature-based OOD scores (e.g. Residual) and logit/softmax-based OOD scores (e.g. KL Matching) have different performances on the Texture OOD dataset. Here we visualize the OOD examples found by the two methods in Fig. 8.

Appendix G Running Time of Four Methods (Sec. 6.2)

From Tab. 2 and Tab. 6, it is clear that the four most competitive methods are ViM, Mahalanobis, KL Matching, and Residual. Our ViM is the fastest among all four methods. We show their inference time on the four datasets in Tab. 8.

References