Learning Bounds for Open-Set Learning

Introduction

Supervised learning has achieved dramatic successes in many applications such as object detection (Simonyan & Zisserman, 2015), speech recognition (Graves & Jaitly, 2014) and natural language processing (Collobert & Weston, 2008). These successes are partly rooted in the closed-set assumption that training and test samples share a same label space. Under this assumption, the standard supervised learning is also regarded as closed-set learning (CSL) (Geng et al., 2018; Yang et al., 2020).

However, the closed-set assumption is not realistic during the testing phase (i.e., there are no labels in the samples) since it is not known whether the classes of test samples are from the label space of training samples. Test samples may come from some classes (unknown classes) that are not necessarily seen during training. These unknown classes can emerge unexpectedly and drastically weaken the performance of existing closed-set algorithms (de O. Cardoso et al., 2017; Dhamija et al., 2018; Perera et al., 2020).

To solve supervised learning without closed-set assumption, Scheirer et al. (2013) proposed a new problem setting, open-set learning (OSL), in which the test samples can come from any classes, even unknown classes. An open-set classifier should classify samples from known classes into correct known classes while recognizing samples from unknown classes into unknown classes.

Remarkable advances have been achieved in open-set learning. The key challenge of OSL algorithms is to recognize the unknown classes accurately. To address this challenge, different strategies have been proposed such as open-space risk (Scheirer et al., 2013) and extreme value theory (Jain et al., 2014; Rudd et al., 2018). Further, to adapt deep networks support to OSL, Bendale & Boult (2016), Ge et al. (2017) proposed OpenMax and G-OpenMax, respectively.

While many OSL algorithms can be roughly interpreted as minimizing the open-space risk or using the extreme value theory, several disconnections still form non-negligible gaps between the theories and algorithms (Geng et al., 2018; Boult et al., 2019). Very little theoretical groundwork has been undertaken to reveal the generalization ability of OSL from the perspective of learning theory.

This work aims to bridge the gap between the theory and algorithm for OSL from the perspective of learning theory. In particular, our theory answers an important question: under some assumptions, given training samples with size $n$, then there exists an OSL algorithm such that the estimation error is close to $O_{p}(1/\sqrt{n})$. This result reveals OSL problem can achieve an order of estimation error that is the same as CSL (Shalev-Shwartz & Ben-David, 2014).

Since the test samples contain unknown classes, the distribution of test samples is intrinsically different from that of training samples. Based on this fact, we aim to establish the OSL theory from transfer learning (Luo et al., 2020a; Niu et al., 2020; Dong et al., 2020b; Lu et al., 2015; Dong et al., 2019, 2020a; Pan & Yang, 2010; Dong et al., 2021; Liu et al., 2019; Wang et al., 2020), which learns knowledge for a given domain from a different, but relative domain. Using the transfer learning theory, we focus on constructing a suitable auxiliary domain, which contains the information of unknown classes. The construction of auxiliary domain depends on covariate shift (Santurkar et al., 2018). Transferring information from the auxiliary domain, we construct the generalization bound for OSL by using the transfer learning bound developed by Ben-David et al. (2006), Mansour et al. (2009), Fang et al. (2020b), Zhong et al. (2020; 2021), Luo et al. (2020b).

Guided by our theory, we then devise an algorithm for OSL to bring the proposed OSL theory into reality. The novel algorithm auxiliary open-set risk (AOSR) is a neural network-based algorithm. AOSR mainly utilizes the instance-weighting strategy to align training samples and auxiliary samples generated by an auxiliary domain. Then, minimizing the auxiliary risk developed by our theory, AOSR can learn how to recognize unknown classes.

The contributions of this paper are summarized as follows.

$\bullet$ We provide the theoretical analysis for open-set learning based on transfer learning and PAC theory. This is the first work to investigate the generalization error bound for open-set learning.

$\bullet$ Our theory answers an important question: under some assumptions, there exists an OSL algorithm such that the order of the estimation error is close to $O_{p}(1/\sqrt{n})$, if given training samples with size $n$.

$\bullet$ We conduct experiments on toy and benchmark datasets. Experiments support our theoretical results and show that our theoretical guided algorithm AOSR can achieve competitive performance compared with several popular baselines.

Related Works

Open-Set Learning Theory. One of the pioneering theoretical works in this field was conducted by Scheirer et al. (2013; 2014). They proposed the open-space risk, which means that when a sample is far from the training samples, there is an increased risk that the sample is from unknown classes. By minimizing the open-space risk, samples from unknown classes can be recognized. Jain et al. (2014), Rudd et al. (2018) consider the extreme value theory to solve the OSL problem. Extreme value theory is a branch of statistics analyzing the distribution of samples of abnormally high or low values. Liu et al. (2018) first proposed the PAC guarantees for open-set detection. Unfortunately, the test samples are required to be used in the training phase. Fang et al. (2020b) considered the open-set domain adaptation (OSDA) problem (Luo et al., 2020b; Busto et al., 2020) and proposed the first estimation for the generalization error of OSDA by constructing a special term, open-set difference. However, similar to Liu et al. (2018), test samples are needed during the training phase.

Open-Set Algorithm. We can roughly separate OSL algorithms into two different categories: shadow algorithms (e.g., support vector machine (SVM)) and deep learning-based algorithms. In shadow algorithms, Scheirer et al. (2013; 2014) proposed the OSL algorithms based on SVM. Jain et al. (2014), Rudd et al. (2018) proposed OSL algorithms based on extreme value theory. Recently, deep-based algorithms have been developed dramatically. OpenMax as the first deep-based algorithm was proposed by Bendale & Boult (2016), to replace SoftMax in deep networks. Later, Ge et al. (2017) combined the generative adversarial networks (GAN) with OpenMax and proposed G-OpenMax. Counterfactual image generation proposed by Neal et al. (2018) is the first OSL algorithm to uses the data augmentation technique by generating the unknown classes so that the decision boundaries between unknown and known classes can be figured out. Oza & Patel (2019) used class conditioned auto-encoders to solve OSL problem, and modeled reconstruction errors using the extreme value theory to find the threshold for identifying known/unknown classes.

Theoretical Analysis of OSL

In this section, we introduce the basic notations used in this paper and then provide theoretical analysis for open-set learning. All proofs can be found in Appendices B-E.

Here we introduce the definition of open-set learning (OSL).

Known classes are a subset of $\mathcal{Y}$. We define the label space of known classes as $\mathcal{Y}_{k}$. Then, the unknown classes are from the space $\mathcal{Y}/\ \mathcal{Y}_{k}$. The open-set learning problem is defined as follows.

Given independent and identically distributed (i.i.d.) samples $S=\{(\mathbf{x}^{i},\mathbf{y}^{i})\}_{i=1}^{n}$ drawn from $P_{X,Y|Y\in\mathcal{Y}_{k}}$. The aim of open-set learning is to train a classifier $f$ using $S$ such that $f$ can classify 1) the sample from known classes into correct known classes; 2) the sample from unknown classes into unknown classes.

where $\alpha$ is the weight estimating the importance of unknown classes. When $\alpha=P(Y=\mathbf{y}_{C+1})$, it is easy to check that

Similarly, given a different joint distribution $Q_{X,Y}$, we can define $R_{Q,k}({\bm{h}}),R_{Q,u}({\bm{h}})$ and $R_{Q}^{\alpha}({\bm{h}})$.

Based on $\alpha$-risk, we define almost agnostic probably approximate correct (PAC) for OSL.

Theorems 5 and 6 imply there exists almost agnostic PAC learnable $\mathcal{H}$ for open-set learning under mild assumptions.

2 Transfer Between Domains

Since there are no samples regarding the unknown classes, we cannot directly analyze the partial risk for unknown classes only using samples $S$ from known classes. To analyze the partial risk for unknown classes, we introduce an auxiliary domain $Q_{X,Y}$, which is used to transfer the information from unknown classes.

A domain $Q_{X,Y}$ defined over $\mathcal{X}\times\mathcal{Y}$ is called the auxiliary domain for $P_{X,Y}$, if $Q_{X|Y\in\mathcal{Y}_{k}}=P_{X|Y\in\mathcal{Y}_{k}},Q_{Y|X}=P_{Y|X}$ and $P_{X}\ll Q_{X}$.

It is clear that $P_{X,Y}$ and $Q_{X,Y}$ are same if we restrict both of them in the support set of known classes.

Since we do not have any information about samples from unknown classes in the training set, it is unknown whether $Q_{X|Y=\mathbf{y}_{C+1}}=P_{X|Y=\mathbf{y}_{C+1}}$. In Section 3.3, we will introduce how to construct $Q_{X,Y}$ such that $Q_{X|Y=\mathbf{y}_{C+1}}$ is a uniform distribution. Namely, any sample drawn from $Q_{X|Y=\mathbf{y}_{C+1}}$ has the same probability.

Then, it is interesting to know the discrepancy between $R_{P}^{\alpha}({\bm{h}})$ and $R_{Q}^{\alpha}({\bm{h}})$ given the same hypothesis ${\bm{h}}$. Before doing this, the disparity discrepancy between distributions need to be introduced.

Using the disparity discrepancy, we can show that

is the combined risk for the unknown classes, $R_{P}^{\alpha}({\bm{h}})$ is the $\alpha$-risk for $P_{X,Y}$ and $R_{Q}^{\alpha}({\bm{h}})$ is the $\alpha$-risk for $Q_{X,Y}$.

Theorem 1 implies there exists a gap between $R_{P}^{\alpha}({\bm{h}})$ and $R_{Q}^{\alpha}({\bm{h}})$. The gap is related to domain discrepancy for unknown classes between $P_{X,Y}$ and $Q_{X,Y}$. To further eliminate the gap between $R_{P}^{\alpha}({\bm{h}})$ and $R_{Q}^{\alpha}({\bm{h}})$, additional conditions about the hypothesis space $\mathcal{H}$ are indispensable.

where $\phi$ is a function defined over $\mathcal{Y}$ and defined as follows $\phi(\mathbf{y})=\mathbf{y}_{C+1}$, if $\mathbf{y}=\mathbf{y}_{C+1}$; otherwise, $\phi(\mathbf{y})=\mathbf{y}_{1}$.

Assumption 1 implies that the hypothesis space $\mathcal{H}$ is complexity enough so that the unknown classes can be classified perfectly by many hypothesis functions. The assumption can be regarded as the open-set version of realization assumption (Shalev-Shwartz & Ben-David, 2014; Mohri et al., 2012). Realization assumption is a basic concept in learning theory.

3 Construction of Ideal Auxiliary Domain

As mentioned above, the auxiliary domain plays an important role to address the open-set learning problem from a transfer learning perspective. Thus, in this subsection, we first show how to construct an ideal auxiliary domain and then demonstrate how to estimate the ideal auxiliary domain via finite samples. Given an auxiliary distribution $U$ such that $P_{X|Y\in\mathcal{Y}_{k}}\ll U$, we denote $r(\mathbf{x})$ as the density ratio between $P_{X|Y\in\mathcal{Y}_{k}}$ and $U$, i.e., for any $U$-measurable set $A$,

and denote ${Q}_{U}^{0,\beta}$ as the marginal distribution defined over $\mathcal{X}$, i.e., for any $U$-measurable set $A$,

and $\beta>0$ is a parameter to tune the density of ${Q}_{U}^{0,\beta}$ for unknown classes. Then we define the ideal auxiliary domain.

Given the distribution $P_{X,Y}$ defined in Problem 1 and an auxiliary distribution $U$ defined over $\mathcal{X}$ such that $P_{X|Y\in\mathcal{Y}_{k}}\ll U$, then the ideal auxiliary domain regarding to $P_{X,Y}$ is

where ${Q}_{U}^{0,\beta}$ is defined in Eq. (5).

In Definition 5, the probability value of distribution $Q_{U}^{0,\beta}$ in space $\mathcal{X}/\ {\rm supp}~{}r$ is a constant $\beta$. In detail, if $P_{X,Y}$ has no overlap between known and unknown classes, any sample from $Q_{X|Y=\mathbf{y}_{C+1}}$ shares same probability (see Figure 1). In addition, an auxiliary distribution $U$ satisfying $P_{X|Y\in\mathcal{Y}_{k}}\ll U$ is needed. The samples drawn from $U$ can be generated by a gaussian distribution or uniform distribution with suitable support set.

To simple, we provide a mild assumption as follows.

Distributions $P_{X|Y\in\mathcal{Y}_{k}}$ and $U$ introduced in Definition 5 are continuous distributions with density functions $p(\mathbf{x})$ and $q(\mathbf{x})$, respectively.

The assumption that $P_{X|Y\in\mathcal{Y}_{k}}$ and $U$ are continuous can be replaced by a weaker assumption: $P_{X|Y\in\mathcal{Y}_{k}}$, $U\ll\mu$, where $\mu$ is a measure defined over $\mathcal{X}$. With the weaker assumption, all theorems still hold.

Note that the density ratio $r=p/q$ required in ${Q}_{U}^{0,\beta}$ is unknown. To compute the density ratio $r$ using $S$ and $T$, the density ratio estimation methods are indispensable. Considering the property of statistical convergence, we use kernelized variant of unconstrained least-squares importance fitting (KuLSIF) (Kanamori et al., 2012) to estimate the density ratio in the theoretical part: given RKHS space $\mathcal{H}_{K}$,

where $\lambda$ is the regularization parameter. Then, we assume $\widehat{w}$ is the solution of Eq. (7).

After instance re-weighting, we regard the following measure

as the approximation of $Q_{U}^{0,\beta}$, where $\gamma$ is defined in Eq. (6),

and $\tau>0$ is the threshold to select whether a sample $\mathbf{x}\in T$ is from unknown classes or known classes.

4 Empirical Estimation for IAD Risk

In this subsection, we first set the ideal auxiliary domain ${Q}_{U}^{0,\beta}\cdot P_{Y|X}$ as $Q_{X,Y}$, then we analyze the IAD risk $R^{\alpha}_{Q}({\bm{h}})$ from an approximate view, where \alpha=1-1/\big{(}1+\beta U(r=0)\big{)}. In detail, the IAD risk $R^{\alpha}_{Q}({\bm{h}})$ can be written as follows

Then, we use $\widehat{Q}_{T}^{\tau,\beta}$ (see Eq. (8)) to construct auxiliary risk to approximate the IAD risk.

Given samples $S$ with size $n$ drawn from $P_{X,Y|Y\in\mathcal{Y}_{k}}$ and $T$ with size $m$ drawn from $U$, i.i.d., then the auxiliary risk for a hypothesis function ${\bm{h}}$ is

here $\widehat{Q}_{T}^{\tau,\beta}$ is defined in Eq. (8) and $\gamma$ is defined in Eq. (6).

Theorem 3 implies that $(1-\alpha)\widehat{R}_{S,T}^{\tau,\beta}({\bm{h}})$ can approximate $R^{\alpha}_{Q}({\bm{h}})$ uniformly.

where $0<\delta<1$ is any constant, then for any $0\leq\alpha<1$,

where $O_{p}$ denotes the probabilistic order, $\gamma=1-\alpha,\beta=\frac{\alpha}{\gamma U(r=0)}$, $\widehat{R}_{S,T}^{\tau,\beta}({\bm{h}})$ is defined in Eq. (10), and $R^{\alpha}_{Q}({\bm{h}})$ is the IAD risk defined in Eq. (9).

Note that $U(0<r\leq 2\tau)\rightarrow 0$, if $\tau\rightarrow 0$, and Theorem 3 has indicated that if we omit the term $(1-\alpha)c\beta U(0<p/q\leq 2\tau)$ and set $m\geq n$, the gap between $(1-\alpha)\widehat{R}_{S,T}^{\tau,\beta}({\bm{h}})$ and $R^{\alpha}_{Q}({\bm{h}})$ is close to $O_{p}(1/\sqrt{n})$ by choosing a small $\delta$.

5 Main Theoretical Results

In this subsection, we analyze the relationship between $R^{\alpha}_{P}({\bm{h}})$ and $\widehat{R}_{S,T}^{\tau,\beta}({\bm{h}})$ based on Theorems 1, 2 and 3.

Given the same conditions and assumptions in Theorems 1 and 3, then for any $0\leq\alpha<1$, ${\bm{h}}\in\mathcal{H}$,

Theorem 4 indicates that the gap between $(1-\alpha)\widehat{R}_{S,T}^{\tau,\beta}({\bm{h}})$ and $R^{\alpha}_{P}({\bm{h}})$ is controlled by four special terms. The combined risk $\Lambda$ and domain discrepancy for unknown classes can be regarded as constants. The other two terms could be small enough, if $n,m\rightarrow+\infty$ and $\tau$ is a small value.

Given the same conditions and assumptions in Theorems 2 and 3, for any $0\leq\alpha<1$, if we assume $\widehat{\bm{h}}\in\operatorname*{arg\,min}_{{\bm{h}}\in\mathcal{H}}\widehat{R}_{S,T}^{\tau,\beta}({\bm{h}}),$ then $|R_{P}^{\alpha}(\widehat{\bm{h}})-\min_{{\bm{h}}\in\mathcal{H}}R^{\alpha}_{P}({\bm{h}})|$ has an upper bound

where $\lambda_{n,m}$ is defined in Theorem 3, $\gamma=1-\alpha,\beta=\frac{\alpha}{\gamma U(r=0)}$, $\widehat{R}_{S,T}^{\tau,\beta}({\bm{h}})$ is defined in Eq. (10) and $O_{p}$ is the probabilistic order (independent of $c,\beta,\tau$ and $\alpha$).

If we select a small $\tau$ to make $U(0<r\leq 2\tau)$ small enough and set $m\geq n$, then under some assumptions, the following optimization problem

is almost classifier-consistent The learned classifier by the algorithm is infinite-samples consistent to $\operatorname*{arg\,min}_{{\bm{h}}\in\mathcal{H}}R^{\alpha}_{P}({\bm{h}})$. with estimation error close to $O_{p}(1/\sqrt{n})$. Additionally, the weight estimation in Theorem 5 is crucial. To weaken the effect of weight estimation in area ${\rm supp}~{}P_{X|Y\in\mathcal{Y}_{k}}$, we introduce a proxy for $\widehat{R}_{S,T}^{\tau,\beta}({\bm{h}})$.

Given samples $S$ with size $n$ drawn from $P_{X,Y|Y\in\mathcal{Y}_{k}}$ and $T$ with size $m$ drawn from $U$, i.i.d., then the auxiliary risk for a hypothesis function ${\bm{h}}$ is

where $\gamma^{\prime}=1/U(r=0)$, $\widehat{R}_{S}({\bm{h}})$ is defined in Definition 6,

Then, a result similar to Theorem 5 for auxiliary risk $\widetilde{R}_{S,T}^{\tau,\beta}$ is given as follows.

Given the same conditions and assumptions in Theorem 5, for any $0\leq\alpha<1$, if we assume $\widetilde{\bm{h}}\in\operatorname*{arg\,min}_{{\bm{h}}\in\mathcal{H}}\widetilde{R}_{S,T}^{\tau,\beta}({\bm{h}})$, then $|R_{P}^{\alpha}(\widetilde{\bm{h}})-\min_{{\bm{h}}\in\mathcal{H}}R^{\alpha}_{P}({\bm{h}})|$ has an upper bound

where $\lambda_{n,m}$, $\beta$ are introduced in Theorem 5, $\gamma^{\prime}$ and $\widetilde{R}_{S,T}^{\tau,\beta}({\bm{h}})$ are defined in Definition 7, and $O_{p}$ is the probabilistic order (independent of $c,\beta,\tau,\gamma^{\prime}$ and $\alpha$).

A Principle Guided OSL Algorithm

Inspired by Theorem 6, we focus on the following problem

where $\mu$ is a positive parameter, $\widehat{R}_{S,T,u}^{\tau,\beta}({\bm{h}})$ is defined in Eq. (12), ${\bm{h}}_{\bm{\Theta}}$ is a hypothesis function based on a neural network, and ${\bm{\Theta}}$ is parameters of the neural network. To optimize ${\bm{h}}_{\bm{\Theta}}$ to solve the minimization problem defined in Eq. (13), we have the following five steps.

Step 1 (Feature Encoding). Train the samples $S$ to get a closed-set classifier ${\bm{h}}_{{\bm{\Theta}}_{0}}$, and designate the output of second to the last layer (without softmax) ${\bm{l}}$ of ${\bm{h}}_{{\bm{\Theta}}_{0}}$ as the encoded feature vector, i.e., $X_{\textrm{encoder}}={\bm{l}}(X)$. The new encoded feature space is denoted as $\mathcal{X}_{\textrm{encoder}}$.

Step 2 (Initialize the Auxiliary Domain). Randomly generate samples $T$ from space $\mathcal{X}_{\textrm{encoder}}$. By default, we generate $T$ by uniform distribution and set the size $m$ is $3n$. We update the samples $S=\{({\bm{l}}(\mathbf{x}),\mathbf{y}):(\mathbf{x},\mathbf{y})\in S\}$.

Step 3 (Construct the Auxiliary Domain). Estimate the weights $\widehat{w}$ with samples $S$ and $T$ as the input. The higher the weight is, the more likely a generated sample belongs to the known classes. The parameters selection details are shown as follows.

Weight estimation algorithm: In the theoretical part, KuLSIF is selected to estimate weights. Kernel mean matching (KMM) (Gretton et al., 2012) is also an alternative solution (Cortes et al., 2008). However, in practice, KuLSIF and KMM have time complexity $O((m+n)^{2})$ (Kanamori et al., 2012), which is not suitable for large datasets. The kernel bandwidth selection also impacts the overall performance (Liu et al., 2020). Thus, we recommend using the outlier sample score (with range $$) given by isolation forest (iForest) (Liu et al., 2008) as the sample weights, which has time complexity${O}((n+m)\log(n+m))$. Close to$1$ means known classes while close to means unknown classes.

The $\tau$ is a threshold to split the generated samples $T$ into known and unknown samples. Considering we are using iForest, based on the predicted sample score $[s_{1},...,s_{m}]$ (descending order), we set $\tau=s_{[t*m]}$, where $t\in(0,1)$ is the proportion that the generated samples selected as unknown samples. We set $t=10\%$ as default.

The $\beta$ and $\mu$ control jointly the importance of correctly classified unknown samples. We set $\mu$ as a dynamical parameter depending on $\beta$: $\mu=\frac{n\beta}{n^{\prime}+0.0001}$, where $n^{\prime}$ is number of samples in training samples actually predicted as unknown. For example, if $\beta=0.05$, $n$ is $1000$, there are $10$ samples in training samples are predicted as unknown, then $\mu\approx 5$.

Step 4 ($\mathbf{Softmax}_{C+1}$). Initialize an open-set learning neural network with samples $S$ and $T$ as the input and $C+1$ Softmax (Qin et al., 2019) nodes as the output.

Step 5 (Open-set Learning). Train the $\mathbf{Softmax}_{C+1}$ neural network with the cost function defined in Eq. (13) with both $S$ and $T$.

Experiments and Results

First, we implement AOSR on toy dataset with different sample size to reveal the relationship between sample size $n$ and error ($O(1/\sqrt{n})$). Then, we evaluate the efficacy of AOSR on benchmark datasets.

In this paper, we verify the efficacy of algorithm AOSR on double-moon dataset and several real world datasets:

$\bullet$ Double-moon dataset (toy). The double-moon dataset consists of two different clusters. Samples from different clusters are regarded as known samples with different label. Samples from other region are regarded as unknown samples drawn from uniform distribution, i.i.d. The ratio between the sizes of known and unknown samples is $1$.

$\bullet$ Following the set up in Yoshihashi et al. (2019), we use MNIST (LeCun & Cortes, 2010) as the training samples and use Omniglot (Ager, 2008), MNIST-Noise, and Noise (Liu et al., 2021) datasets as unknown classes. Omniglot contains alphabet characters. Noise is synthesized by sampling each pixel value from a uniform distribution on $$. MNIST-Noise is synthesized by adding noise on MNIST test samples. Each dataset has$10,000$ test samples.

$\bullet$ Following Yoshihashi et al. (2019), we use CIFAR-$10$ (Krizhevsky & Hinton, 2009) as training samples and collect unknown samples from ImageNet and LSUN. We resized/cropped them so that they would be the same size as the known samples. Hence, we generate four datasets ImageNet-crop, ImageNet-resize, LSUN-crop and LSUN-resize as unknown classes. Each dataset contains $10,000$ test samples.

$\bullet$ Following Yoshihashi et al. (2019), Chen et al. (2021), Sun et al. (2020), we use MNIST (LeCun & Cortes, 2010), SVHN (Netzer et al., 2011) and CIFAR-$10$ (Krizhevsky & Hinton, 2009) to construct different OSL tasks. For MNIST, SVHN and CIFAR-$10$, each dataset is randomly divided into $6$ known classes and $4$ unknown classes. In addition, we construct CIFAR+$10$ and CIFAR+$50$ by randomly selection $6$ known classes and $10$ or $50$ unknown classes from CIFAR-$100$ (Krizhevsky & Hinton, 2009).

2 Open-set Learning Demonstration

Here we break down the entire learning process and demonstrate the inter-media process of each step on the toy dataset. This experiment is aiming to provide an visualization aid on understanding the open-set learning process.

To start with, we plot the double-moon dataset in Figure 2 (a). The objective of closed-set learning is to build a classifier that can split the samples with different labels. To achieve this goal, we build a simple neural network with sparse categorical cross-entropy as the loss function.

The closed-set learning result is shown in Figure 2 (b). In this case, the closed-set classifier splits the samples with different labels well. However, the closed-set classifier does not consider the boundary of support set for training domain, that is, any new samples that does not located in the support set, the closed-set classifier still gives a known label.

Figure 2 (c) is the open-set learning result. To recognize the unknown samples, the open-set classifier should delineate a boundary between the known and unknown classes. To achieve this goal, we use $\mathbf{Softmax}_{C+1}$ as the final output and Eq. (13) as the cost function. The AOSR will push the neural network to give label $\mathbf{y}_{C+1}$ on unknown samples.

3 Experimental Setup

$\bullet$ AOSR has several hyper-parameters: $\beta$, $t$, $\mu$ and $m$. For all tasks, we set $m=3n,~{}t=10\%$ as default. $\mu$ is a dynamic parameter depending on $\beta$. $\beta$ is selected from $0.01$ to $2.5$. Details on the selection of parameters are available at github.com/Anjin-Liu/Openset_Learning_AOSR.

$\bullet$ For datasets MNIST, Omnilot, MNIST-Noise, Noise, we use the same setting of Yoshihashi et al. (2019) and Sun et al. (2020) to extract the features. Same as Yoshihashi et al. (2019), DHRNet-$92$ is used as the backbone for CIFAR-$10$, ImageNet and LSUN datasets. For different tasks MNIST, SVHN, CIFAR-$10$, CIFAR+$10$ and CIFAR+$50$, the backbone is the re-designed VGGNet used by Yoshihashi et al. (2019) and Sun et al. (2020).

$\bullet$ We select baseline algorithms as follows: SoftMax, OpenMax (Bendale & Boult, 2016), Counterfactual (Neal et al., 2018), CROSR (Yoshihashi et al., 2019), C2AE (Oza & Patel, 2019), and CGDL (Sun et al., 2020).

4 Evaluation

Following Yoshihashi et al. (2019), the macro-average F1 scores are used to evaluate OSL. The area under the receiver operating characteristic (AUROC) (Neal et al., 2018) is also frequently used (Neal et al., 2018; Chen et al., 2020). Note that AUROC used in (Neal et al., 2018; Chen et al., 2020) is suitable for global threshold-based OSL algorithms that recognize unknown samples by a fix threshold (Neal et al., 2018). However, AOSR recognizes unknown samples based on the score of hypothesis function, thus, AOSR uses different thresholds for different samples. This implies that AUROC used in (Neal et al., 2018; Chen et al., 2020) may be not suitable for our algorithm. In this paper, we use macro-average F1 scores to evaluate our algorithm.

5 Experimental Evaluation and Result Analysis

Experiment results on double-moon dataset are summarized in Figure 2 (d). We implement double-moon dataset with varying size $n$ select $n$ from $*100$ . We also generate $n$ test samples. For a different sample size, we run $100$ times and report the mean accuracy and standard error in Figure 2 (d). Based on Figure 2 (d), the accuracy increases as the increase of training sample size $n$ increases. When $n\rightarrow 15,000$, the accuracy approximates at $100\%$. In particular, the green curve $0.5/\sqrt{n}$ and the yellow curve $8/\sqrt{n}$ jointly control the curve of accuracy, implying the error of AOSR is controlled by $O(1/\sqrt{n})$.

Experiment results on real datasets are summarized in Tables 1, 2 and 3. For all tasks, we run AOSR $5$ times and report the mean results by using F1 score (Powers, 2020). In general, AOSR shows the promising performance when compared to baseline algorithms. The effectiveness of AOSR indicates that our theory is effective and practical.

Parameter analysis for $\beta$ and $t$ is given in Figure 2 (e), (f). We run AOSR with varying values of $\beta,t$ on MNIST tasks. From Figure 2 (e), we observe that 1) when $\beta$ increases from $0.01$ to $0.64$, the F1 scores for Noise and MNIST-Noise decrease; 2) as increasing $\beta$ from $0.01$ to $0.16$, the F1 score for Omniglot increases. When $\beta>1.6$, the performance for Omniglot dramatically dropped to baseline. Additionally, according to Figure 2 (f), we find that by changing $t$ in the range of $[0.05,0.30]$, AOSR achieve stable performance.

Ablation study on datasets MNIST, Omnilot, MNIST-Noise and Noise is shown in Table 4. By adjusting different components of AOSR, Table 4 indicates that each component of AOSR is important and necessary. Note that if we replace iForest by KMM in AOSR, the performance ($0.907$) is close to AOSR ($0.910$). This implies that KMM may be a good choice, if we omit the time complexity of KMM.

Discussion

Relation with Generative Models. Algorithms based on generative models are the mainstream for OSL. CGDL (Sun et al., 2020), C2AE (Oza & Patel, 2019) and Counterfactual (Neal et al., 2018) are the representative works based on generative models. AOSR can be regarded as the weight-based generative model, but is very different from the mainstream generative model-based algorithms (feature map-based generative model (Sun et al., 2020; Oza & Patel, 2019; Neal et al., 2018)). Form the theoretical perspective, it is necessary to develop theory to guarantee the generalization ability of feature map-based generative models. Here we propose an interesting and important problem: how to develop generalization theory for feature map-based generative models under open-set assumption ?

Relation with PU Learning. Positive-unlabeled learning (PU learning) (Niu et al., 2016) is a special binary classification task, which assumes only unlabeled samples and positive samples (i.e., samples with positive labels) are available. Our theory is deeply related to PU learning. If we regard the known samples $S$ and the auxiliary samples as the positive samples and the unlabeled samples, respectively. Then, our theory degenerates into the PU learning theory.

Remaining Problems in OSL Theory. We list several interesting and important problems for OSL theory as follows. 1. How to construct weaker assumption to replace assumption 1 for achieving similar results ? 2. Without assumption 1, what will happen ? 3. Is it possible for OSL to achieve agnostic PAC learnability and achieve fast learning rate $O_{p}(1/{n}^{a})$, for $a>0.5$ ? 4. Is it possible to construct OSL learning theory by stability theory (Bousquet & Elisseeff, 2002) ?

Conclusion and Future Work

This paper mainly focuses on the learning theory for open-set learning. The generalization error bounds proved in our work provide the first almost-PAC-style guarantee on open-set learning. Based on our theory, a principle guided algorithm AOSR is proposed. Experiments on real datasets indicate that AOSR achieves competitive performance when compared with baselines. In future, we will focus on developing more powerful OSL algorithms based on our theory and dynamic weight technique (Fang et al., 2020a). With the dynamic weight , we can update the weight for each epoch and make a better integration between instance-weighting and deep learning.

Acknowledgments

The work introduced in this paper was supported by the Australian Research Council (ARC) under FL190100149.

References