Conditional Probability Models for Deep Image Compression

Introduction

Image compression refers to the task of representing images using as little storage (i.e., bits) as possible. While in lossless image compression the compression rate is limited by the requirement that the original image should be perfectly reconstructible, in lossy image compression, a greater reduction in storage is enabled by allowing for some distortion in the reconstructed image. This results in a so-called rate-distortion trade-off, where a balance is found between the bitrate $R$ and the distortion $d$ by minimizing $d+\beta R$, where $\beta>0$ balances the two competing objectives. Recently, deep neural networks (DNNs) trained as image auto-encoders for this task led to promising results, achieving better performance than many traditional techniques for image compression . Another advantage of DNN-based learned compression systems is their adaptability to specific target domains such as areal images or stereo images, enabling even higher compression rates on these domains. A key challenge in training such systems is to optimize the bitrate $R$ of the latent image representation in the auto-encoder. To encode the latent representation using a finite number of bits, it needs to be discretized into symbols (i.e., mapped to a stream of elements from some finite set of values). Since discretization is non-differentiable, this presents challenges for gradient-based optimization methods and many techniques have been proposed to address them. After discretization, information theory tells us that the correct measure for bitrate $R$ is the entropy $H$ of the resulting symbols. Thus the challenge, and the focus of this paper, is how to model $H$ such that we can navigate the trade-off $d+\beta H$ during optimization of the auto-encoder.

Our proposed method is based on leveraging context models, which were previously used as techniques to improve coding rates for already-trained models , directly as an entropy term in the optimization. We concurrently train the auto-encoder and the context model with respect to each other, where the context model learns a convolutional probabilistic model of the image representation in the auto-encoder, while the auto-encoder uses it for entropy estimation to navigate the rate-distortion trade-off. Furthermore, we generalize our formulation to spatially-aware networks, which use an importance map to spatially attend the bitrate representation to the most important regions in the compressed representation. The proposed techniques lead to a simple image compression systemhttps://github.com/fab-jul/imgcomp-cvpr, which achieves state-of-the-art performance when measured with the popular multi-scale structural similarity index (MS-SSIM) distortion metric , while being straightforward to implement with standard deep-learning toolboxes.

Related work

Full-resolution image compression using DNNs has attracted considerable attention recently. DNN architectures commonly used for image compression are auto-encoders and recurrent neural networks (RNNs) . The networks are typically trained to minimize the mean-squared error (MSE) between original and decompressed image , or using perceptual metrics such as MS-SSIM . Other notable techniques involve progressive encoding/decoding strategies , adversarial training , multi-scale image decompositions , and generalized divisive normalization (GDN) layers .

Context models and entropy estimation—the focus of the present paper—have a long history in the context of engineered compression methods, both lossless and lossy . Most of the recent DNN-based lossy image compression approaches have also employed such techniques in some form. uses a binary context model for adaptive binary arithmetic coding . The works of use learned context models for improved coding performance on their trained models when using adaptive arithmetic coding. use non-adaptive arithmetic coding but estimate the entropy term with an independence assumption on the symbols.

Also related is the work of van den Oord et al. , who proposed PixelRNN and PixelCNN, powerful RNN- and CNN-based context models for modeling the distribution of natural images in a lossless setting, which can be used for (learned) lossless image compression as well as image generation.

Proposed method

where $H$ denotes the cost of encoding $\hat{\mathbf{z}}$ to bits, i.e., the entropy of $\hat{\mathbf{z}}$. Our system is realized by modeling $E$ and $D$ as convolutional neural networks (CNNs) (more specifically, as the encoder and decoder, respectively, of a convolutional auto-encoder) and minimizing (1) over the training set $\mathcal{X}$, where a large/small $\beta$ draws the system towards low/high average entropy $H$. In the next sections, we will discuss how we quantize $\mathbf{z}$ and estimate the entropy $H(\hat{\mathbf{z}})$. We note that as $E,D$ are CNNs, $\hat{\mathbf{z}}$ will be a 3D feature map, but for simplicity of exposition we will denote it as a vector with equally many elements. Thus, $\hat{z}_{i}$ refers to the $i$-th element of the feature map, in raster scan order (row by column by channel).

but rely on (differentiable) soft quantization

to compute gradients during the backward pass. This combines the benefit of where the quantization is restricted to a finite set of learned centers $\mathcal{C}$ (instead of the fixed (non-learned) integer grid as in ) and the simplicity of , where a differentiable approximation of quantization is only used in the backward pass, avoiding the need to choose an annealing strategy (i.e., a schedule for $\sigma$) as in to drive the soft quantization (3) to hard assignments (2) during training. In TensorFlow, this is implemented as

We note that for forward pass computations, $\bar{z}_{i}=\hat{z}_{i}$, and thus we will continue writing $\hat{z}_{i}$ for the latent representation.

2 Entropy estimation

To model the entropy $H(\hat{\mathbf{z}})$ we build on the approach of PixelRNN and factorize the distribution $p(\hat{\mathbf{z}})$ as a product of conditional distributions

where the 3D feature volume $\hat{\mathbf{z}}$ is indexed in raster scan order. We then use a neural network $P(\hat{\mathbf{z}})$, which we refer to as a context model, to estimate each term $p(\hat{z}_{i}|\hat{z}_{i-1},\ldots,\hat{z}_{1})$:

where $P_{i,l}$ specifies for every 3D location $i$ in $\hat{\mathbf{z}}$ the probabilites of each symbol in $\mathcal{C}$ with $l=1,\ldots,L$. We refer to the resulting approximate distribution as $q(\hat{\mathbf{z}}):=\prod_{i=1}^{m}P_{i,I(\hat{z}_{i})}(\hat{\mathbf{z}})$, where $I(\hat{z}_{i})$ denotes the index of $\hat{z}_{i}$ in $\mathcal{C}$.

Since the conditional distributions $p(\hat{z}_{i}|\hat{z}_{i-1},\ldots,\hat{z}_{1})$ only depend on previous values $\hat{z}_{i-1},\ldots,\hat{z}_{1}$, this imposes a causality constraint on the network $P$: While $P$ may compute $P_{i,l}$ in parallel for $i=1,\ldots,m,l=1,\ldots,L$, it needs to make sure that each such term only depends on previous values $\hat{z}_{i-1},\ldots,\hat{z}_{1}$.

The authors of PixelCNN study the use of 2D-CNNs as causal conditional models over 2D images in a lossless setting, i.e., treating the RGB pixels as symbols. They show that the causality constraint can be efficiently enforced using masked filters in the convolution. Intuitively, the idea is as follows: If for each layer the causality condition is satisfied with respect to the spatial coordinates of the layer before, then by induction the causality condition will hold between the output layer and the input. Satisfying the causality condition for each layer can be achieved with proper masking of its weight tensor, and thus the entire network can be made causal only through the masking of its weights. Thus, the entire set of probabilities $P_{i,l}$ for all (2D) spatial locations $i$ and symbol values $l$ can be computed in parallel with a fully convolutional network, as opposed to modeling each term $p(\hat{z}_{i}|\hat{z}_{i-1},\cdots,\hat{z}_{1})$ separately.

In our case, $\hat{\mathbf{z}}$ is a 3D symbol volume, with as much as $K=64$ channels. We therefore generalize the approach of PixelCNN to 3D convolutions, using the same idea of masking the filters properly in every layer of the network. This enables us to model $P$ efficiently, with a light-weightWe use a 4-layer network, compared to 15 layers in . 3D-CNN which slides over $\hat{z}$, while properly respecting the causality constraint. We refer to the supplementary material for more details.

As in , we learn $P$ by training it for maximum likelihood, or equivalently (see ) by training $P_{i,:}$ to classify the index $I(\hat{z}_{i})$ of $\hat{z}_{i}$ in $\mathcal{C}$ with a cross entropy loss:

Using the well-known property of cross entropy as the coding cost when using the wrong distribution $q(\hat{\mathbf{z}})$ instead of the true distribution $p(\hat{\mathbf{z}})$, we can also view the $CE$ loss as an estimate of $H(\hat{\mathbf{z}})$ since we learn $P$ such that $P=q\approx p$. That is, we can compute

Therefore, when training the auto-encoder we can indirectly minimize $H(\hat{\mathbf{z}})$ through the cross entropy $CE$. We refer to argument in the expectation of (7),

as the coding cost of the latent image representation, since this reflects the coding cost incurred when using $P$ as a context model with an adaptive arithmetic encoder . From the application perspective, minimizing the coding cost is actually more important than the (unknown) true entropy, since it reflects the bitrate obtained in practice.

3 Concurrent optimization

Given an auto-encoder $(E,D)$, we can train $P$ to model the dependencies of the entries of $\hat{\mathbf{z}}$ as described in the previous section by minimizing (7). On the other hand, using the model $P$, we can obtain an estimate of $H(\hat{\mathbf{z}})$ as in (12) and use this estimate to adjust $(E,D)$ such that $d(\mathbf{x},D(Q(E(\mathbf{x}))))+\beta H(\hat{\mathbf{z}})$ is reduced, thereby navigating the rate distortion trade-off. Therefore, it is natural to concurrently learn $P$ (with respect to its own loss), and $(E,D)$ (with respect to the rate distortion trade-off) during training, such that all models which the losses depend on are continuously updated.

4 Importance map for spatial bit-allocation

Recall that since $E$ and $D$ are CNNs, $\hat{\mathbf{z}}$ is a 3D feature-map. For example, if $E$ has three stride-2 convolution layers and the bottleneck has $K$ channels, the dimensions of $\hat{\mathbf{z}}$ will be $\frac{W}{8}\times\frac{H}{8}\times K$. A consequence of this formulation is that we are using equally many symbols in $\hat{\mathbf{z}}$ for each spatial location of the input image $\mathbf{x}$. It is known, however, that in practice there is great variability in the information content across spatial locations (e.g., the uniform area of blue sky vs. the fine-grained structure of the leaves of a tree).

This can in principle be accounted for automatically in the trade-off between the entropy and the distortion, where the network would learn to output more predictable (i.e., low entropy) symbols for the low information regions, while making room for the use of high entropy symbols for the more complex regions. More precisely, the formulation in (7) already allows for variable bit allocation for different spatial regions through the context model $P$.

where $y_{i,j}$ denotes the value of $\mathbf{y}$ at spatial location $(i,j)$. The transition value for $k\leq y_{i,j}\leq k+1$ is such that the mask smoothly transitions from 0 to 1 for non-integer values of $\mathbf{y}$.

We then mask $\mathbf{z}$ by pointwise multiplication with the binarization of $\mathbf{m}$, i.e., $\mathbf{z}\leftarrow\mathbf{z}\odot\lceil\mathbf{m}\rceil$. Since the ceiling operator $\lceil\cdot\rceil$ is not differentiable, as done by , we use identity for the backward pass.

With this modification, we have simply changed the architecture of $E$ slightly such that it can easily “zero out” portions of columns $\mathbf{z}_{i,j,:}$ of $\mathbf{z}$ (the rest of the network stays the same, so that (2) still holds for example). As suggested by , the so-obtained structure in $\mathbf{z}$ presents an alternative coding strategy: Instead of losslessly encoding the entire symbol volume $\hat{\mathbf{z}}$, we could first (separately) encode the mask $\lceil\mathbf{m}\rceil$, and then for each column $\hat{z}_{i,j,:}$ only encode the first $\lceil m_{i,j}\rceil+1$ symbols, since the remaining ones are the constant $Q(0)$, which we refer to as the zero symbol.

Work uses binary symbols (i.e., $\mathcal{C}=\{0,1\}$) and assumes independence between the symbols and a uniform prior during training, i.e., costing each 1 bit to encode. The importance map is thus their principal tool for controlling the bitrate, since they thereby avoid encoding all the bits in the representation. In contrast, we stick to the formulation in (5) where the dependencies between the symbols are modeled during training. We then use the importance map as an architectural constraint and use their suggested coding strategy to obtain an alternative estimate for the entropy $H(\hat{\mathbf{z}})$, as follows.

We observe that we can recover $\lceil\mathbf{m}\rceil$ from $\hat{\mathbf{z}}$ by counting the number of consecutive zero symbols at the end of each column $\hat{\mathbf{z}}_{i,j,:}$.If $\mathbf{z}$ contained zeros before it was masked, we might overestimate the number of entries in $\lceil\mathbf{m}\rceil$. However, we can redefine those entries of $\mathbf{m}$ as and this will give the same result after masking. $\lceil\mathbf{m}\rceil$ is therefore a function of the masked $\hat{\mathbf{z}}$, i.e., $\lceil\mathbf{m}\rceil=g(\hat{\mathbf{z}})$ for $g$ recovering $\lceil\mathbf{m}\rceil$ as described, which means that we have for the conditional entropy $H(\lceil\mathbf{m}\rceil|\hat{\mathbf{z}})=0$. Now, we have

If we treat the entropy of the mask, $H(\lceil\mathbf{m}\rceil)$, as constant during optimization of the auto-encoder, we can then indirectly minimize $H(\hat{\mathbf{z}})$ through $H(\hat{\mathbf{z}}|\mathbf{m})$.

To estimate $H(\hat{\mathbf{z}}|\mathbf{m})$, we use the same factorization of $p$ as in (5), but since the mask $\lceil\mathbf{m}\rceil$ is known we have $p(\hat{z}_{i}=c_{0})=1$ deterministic for the 3D locations $i$ in $\hat{\mathbf{z}}$ where the mask is zero. The $\log$s of the corresponding terms in (9) then evaluate to . The remaining terms, we can model with the same context model $P_{i,l}(\hat{\mathbf{z}})$, which results in

where $m_{i}$ denotes the $i$-th element of $\mathbf{m}$ (in the same raster scan order as $\hat{z}$).

Similar to the coding cost (13), we refer to the argument in the expectation in (18),

as the masked coding cost of $\hat{\mathbf{z}}$.

While the entropy estimate (18) is almost estimating the same quantity as (7) (only differing by $H(\lceil\mathbf{m}\rceil)$), it has the benefit of being weighted by $m_{i}$. Therefore, the encoder $E$ has an obvious path to control the entropy of $\hat{\mathbf{z}}$, by simply increasing/decreasing the value of $\mathbf{y}$ for some spatial location of $\mathbf{x}$ and thus obtaining fewer/more zero entries in $\mathbf{m}$.

When the context model $P(\hat{\mathbf{z}})$ is trained, however, we still train it with respect to the formulation in (8), so it does not have direct access to the mask $\mathbf{m}$ and needs to learn the dependencies on the entire masked symbol volume $\hat{\mathbf{z}}$. This means that when encoding an image, we can stick to standard adaptive arithmetic coding over the entire bottleneck, without needing to resort to a two-step coding process as in , where the mask is first encoded and then the remaining symbols. We emphasize that this approach hinges critically on the context model $P$ and the encoder $E$ being trained concurrently as this allows the encoder to learn a meaningful (in terms of coding cost) mask with respect to $P$ (see the next section).

In our experiments we observe that during training, the two entropy losses (7) and (18) converge to almost the same value, with the latter being around $\approx 3.5\%$ smaller due to $H(\lceil\mathbf{m}\rceil)$ being ignored.

While the importance map is not crucial for optimal rate-distortion performance, if the channel depth $K$ is adjusted carefully, we found that we could more easily control the entropy of $\hat{\mathbf{z}}$ through $\beta$ when using a fixed $K$, since the network can easily learn to ignore some of the channels via the importance map. Furthermore, in the supplementary material we show that by using multiple importance maps for a single network, one can obtain a single model that supports multiple compression rates.

5 Putting the pieces together

We made an effort to carefully describe our formulation and its motivation in detail. While the description is lengthy, when putting the resulting pieces together we get a quite straightforward pipeline for learned image compression, as follows.

Given the set of training images $\mathcal{X}$, we initialize (fully convolutional) CNNs $E$, $D$, and $P$, as well as the centers $\mathcal{C}$ of the quantizer $Q$. Then, we train over minibatches $\mathcal{X}_{B}=\{\mathbf{x}^{(1)},\cdots,\mathbf{x}^{(B)}\}$ of crops from $\mathcal{X}$. At each iteration, we take one gradient step for the auto-encoder $(E,D)$ and the quantizer $Q$, with respect to the rate-distortion trade-off

which is obtained by combining (1) with the estimate (18) & (19) and taking the batch sample average. Furthermore, we take a gradient step for the context model $P$ with respect to its objective (see (7) & (13))

To compute these two batch losses, we need to perform the following computation for each $\mathbf{x}\in\mathcal{X}_{B}$:

Obtain compressed (latent) representation $\mathbf{z}$ and importance map $\mathbf{y}$ from the encoder: $(\mathbf{z},\mathbf{y})=E(\mathbf{x})$

Expand importance map $\mathbf{y}$ to mask $\mathbf{m}$ via (14)

Mask $\mathbf{z}$, i.e., $\mathbf{z}\leftarrow\mathbf{z}\odot\lceil\mathbf{m}\rceil$

Quantize $\hat{\mathbf{z}}=Q(\mathbf{z})$

Compute the context $P(\hat{\mathbf{z}})$

Decode $\hat{\mathbf{x}}=D(\hat{\mathbf{z}})$,

which can be computed in parallel over the minibatch on a GPU since all the models are fully convolutional.

6 Relationship to previous methods

We are not the first to use context models for adaptive arithmetic coding to improve the performance in learned deep image compression. Work uses a PixelRNN-like architecture to train a recurrent network as a context model for an RNN-based compression auto-encoder. Li et al. extract cuboid patches around each symbol in a binary feature map, and feed them to a convolutional context model. Both these methods, however, only learn the context model after training their system, as a post-processing step to boost coding performance.

In contrast, our method directly incorporates the context model as the entropy term for the rate-distortion term (1) of the auto-encoder, and trains the two concurrently. This is done at little overhead during training, since we adopt a 3D-CNN for the context model, using PixelCNN-inspired masking of the weights of each layer to ensure causality in the context model. Adopting the same approach to the context models deployed by or would be non-trivial since they are not designed for fast feed-forward computation. In particular, while the context model of is also convolutional, its causality is enforced through masking the inputs to the network, as opposed to our masking of the weights of the networks. This means their context model needs to be run separately with a proper input cuboid for each symbol in the volume (i.e., not fully convolutionally).

Experiments

Our auto-encoder has a similar architecture as but with more layers, and is described in Fig. 2. We adapt the number of channels $K$ in the latent representation for different models. For the context model $P$, we use a simple $4$-layer 3D-CNN as described in Fig. 3.

Distortion measure

Following , we use the multi-scale structural similarity index (MS-SSIM) as measure of distortion $d(\mathbf{x},\hat{\mathbf{x}})=100\cdot(1-\text{MS-SSIM}(\mathbf{x},\hat{\mathbf{x}}))$ for our models. MS-SSIM reportedly correlates better with human perception of distortion than mean squared error (MSE). We train and test all our models using MS-SSIM.

Training

We use the Adam optimizer with a minibatch size of 30 to train seven models. Each model is trained to maximize MS-SSIM directly. As a baseline, we used a learning rate (LR) of $4\cdot 10^{-3}$ for each model, but found it beneficial to vary it slightly for different models. We set $\sigma=1$ in the smooth approximation (3) used for gradient backpropagation through $Q$. To make the model more predictably land at a certain bitrate $t$ when optimizing (1), we found it helpful to clip the rate term (i.e., replace the entropy term $\beta H$ with $\max(t,\beta H)$), such that the entropy term is “switched off” when it is below $t$. We found this did not hurt performance. We decay the learning rate by a factor 10 every two epochs. To obtain models for different bitrates, we adapt the target bitrate $t$ and the number of channels $K$, while using a moderately large $\beta=10$. We use a small regularization on the weights and note that we achieve very stable training. We trained our models for 6 epochs, which took around 24h per model on a single GPU. For $P$, we use a LR of $10^{-4}$ and the same decay schedule.

Datasets

We train on the the ImageNet dataset from the Large Scale Visual Recognition Challenge 2012 (ILSVRC2012) . As a preprocessing step, we take random $160\times 160$ crops, and randomly flip them. We set aside 100 images from ImageNet as a testing set, ImageNetTest. Furthermore, we test our method on the widely used Kodak dataset. To asses performance on high-quality full-resolution images, we also test on the datasets B100 and Urban100 , commonly used in super-resolution.

Other codecs

We compare to JPEG, using libjpeghttp://libjpeg.sourceforge.net/, and JPEG2000, using the Kakadu implementationhttp://kakadusoftware.com/. We also compare to the lesser known BPGhttps://bellard.org/bpg/, which is based on HEVC, the state-of-the-art in video compression, and which outperforms JPEG and JPEG2000. We use BPG in the non-default 4:4:4 chroma format, following .

Comparison

Like , we proceed as follows to compare to other methods. For each dataset, we compress each image using all our models. This yields a set of (bpp, MS-SSIM) points for each image, which we interpolate to get a curve for each image. We fix a grid of bpp values, and average the curves for each image at each bpp grid value (ignoring those images whose bpp range does not include the grid value, i.e., we do not extrapolate). We do this for our method, BPG, JPEG, and JPEG2000. Due to code being unavailable for the related works in general, we digitize the Kodak curve from Rippel & Bourdev , who have carefully collected the curves from the respective works. With this, we also show the results of Rippel & Bourdev , Johnston et al. , Ballé et al. , and Theis et al. . To validate that our estimated MS-SSIM is correctly implemented, we independently generated the BPG curves for Kodak and verified that they matched the one from .

Results

Fig. 1 shows a comparison of the aforementioned methods for Kodak. Our method outperforms BPG, JPEG, and JPEG2000, as well as the neural network based approaches of Johnston et al. , Ballé et al. , and Theis et al. . Furthermore, we achieve performance comparable to that of Rippel & Bourdev . This holds for all bpps we tested, from 0.3 bpp to 0.9 bpp. We note that while Rippel & Bourdev and Johnston et al. also train to maximize (MS-)SSIM, the other methods minimize MSE.

In each of the other testing sets, we also outperform BPG, JPEG, and JPEG2000 over the reported bitrates, as shown in Fig. 4.

In Fig. 5, we compare our approach to BPG, JPEG, and JPEG2000 visually, using very strong compression on kodim21 from Kodak. It can be seen that the output of our network is pleasant to look at. Soft structures like the clouds are very well preserved. BPG appears to handle high frequencies better (see, e.g., the fence) but loses structure in the clouds and in the sea. Like JPEG2000, it produces block artifacts. JPEG breaks down at this rate. We refer to the supplementary material for further visual examples.

Ablation study: Context model

In order to show the effectiveness of the context model, we performed the following ablation study. We trained the auto-encoder without entropy loss, i.e., $\beta=0$ in (20), using $L=6$ centers and $K=16$ channels. On Kodak, this model yields an average MS-SSIM of 0.982, at an average rate of 0.646 bpp (calculated assuming that we need $\log_{2}(L)=2.59$ bits per symbol). We then trained three different context models for this auto-encoder, while keeping the auto-encoder fixed: A zeroth order context model which uses a histogram to estimate the probability of each of the $L$ symbols; a first order (one-step prediction) context model, which uses a conditional histogram to estimate the probability of each of the $L$ symbols given the previous symbol (scanning $\hat{\mathbf{z}}$ in raster order); and $P$, i.e., our proposed context model. The resulting average rates are shown in Table 1. Our context model reduces the rate by 10 %, even though the auto-encoder was optimized using a uniform prior (see supplementary material for a detailed comparison of Table 1 and Fig. 1).

Importance map

As described in detail in Section 3.4, we use an importance map to dynamically alter the number of channels used at different spatial locations to encode an image. To visualize how this helps, we trained two auto-encoders $M$ and $M^{\prime}$, where $M$ uses an importance map and at most $K=32$ channels to compress an image, and $M^{\prime}$ compresses without importance map and with $K=16$ channels (this yields a rate for $M^{\prime}$ similar to that of $M$). In Fig. 6, we show an image from ImageNetTest along with the same image compressed to 0.463 bpp by $M$ and compressed to 0.504 bpp by $M^{\prime}$. Furthermore, Fig. 6 shows the importance map produced by $M$, as well as ordered visualizations of all channels of the latent representation for both $M$ and $M^{\prime}$. Note how for $M$, channels with larger index are sparser, showing how the model can spatially adapt the number of channels. $M^{\prime}$ uses all channels similarly.

Discussion

Our experiments showed that combining a convolutional auto-encoder with a lightweight 3D-CNN as context model and training the two networks concurrently leads to a highly effective image compression system. Not only were we able to clearly outperform state-of-the-art engineered compression methods including BPG and JPEG2000 in terms of MS-SSIM, but we also obtained performance competitive with the current state-of-the-art learned compression method from . In particular, our method outperforms BPG and JPEG2000 in MS-SSIM across four different testing sets (ImageNetTest, Kodak, B100, Urban100), and does so significantly, i.e., the proposed method generalizes well. We emphasize that our method relies on elementary techniques both in terms of the architecture (standard convolutional auto-encoder with importance map, convolutional context model) and training procedure (minimize the rate-distortion trade-off and the negative log-likelihood for the context model), while uses highly specialized techniques such as a pyramidal decomposition architecture, adaptive codelength regularization, and multiscale adversarial training.

The ablation study for the context model showed that our 3D-CNN-based context model is significantly more powerful than the first order (histogram) and second order (one-step prediction) baseline context models. Further, our experiments suggest that the importance map learns to condensate the image information in a reduced number of channels of the latent representation without relying on explicit supervision. Notably, the importance map is learned as a part of the image compression auto-encoder concurrently with the auto-encoder and the context model, without introducing any optimization difficulties. In contrast, in the importance map is computed using a separate network, learned together with the auto-encoder, while the context model is learned separately.

Conclusions

In this paper, we proposed the first method for learning a lossy image compression auto-encoder concurrently with a lightweight context model by incorporating it into an entropy loss for the optimization of the auto-encoder, leading to performance competitive with the current state-of-the-art in deep image compression .

Future works could explore heavier and more powerful context models, as those employed in . This could further improve compression performance and allow for sampling of natural images in a “lossy” manner, by sampling $\hat{\mathbf{z}}$ according to the context model and then decoding.

This work was supported by ETH Zürich and by NVIDIA through a GPU grant.

References

Conditional Probability Models for Deep Image Compression – Suppl. Material

As mentioned in Section 3.2, we rely on masked 3D convolutions to enforce the causality constraint in our probability classifier $P$. In a 2D-CNN, standard 2D convolutions are used in filter banks, as shown in Fig. 7 on the left: A $W\times H\times C_{\text{in}}$-dimensional tensor is mapped to a $W^{\prime}\times H^{\prime}\times C_{\text{out}}$-dimensional tensor using $C_{\text{out}}$ banks of $C_{\text{in}}$ 2D filters, i.e., filters can be represented as $f_{W}\times f_{H}\times C_{\text{in}}\times C_{\text{out}}$-dimensional tensors. Note that all $C_{\text{in}}$ channels are used together, which violates causality: When we encode, we proceed channel by channel.

Using 3D convolutions, a depth dimension $D$ is introduced. In a 3D-CNN, $W\times H\times D\times C_{\text{in}}$-dimensional tensors are mapped to $W^{\prime}\times H^{\prime}\times D^{\prime}\times C_{\text{out}}$-dimensional tensors, with $f_{W}\times f_{H}\times f_{D}\times C_{\text{in}}\times C_{\text{out}}$-dimensional filters. Thus, a 3D-CNN slides over the depth dimension, as shown in Fig. 7 on the right. We use such a 3D-CNN for $P$, where we use as input our $W\times H\times K$-dimensional feature map $\hat{\mathbf{z}}$, using $D=K,C_{\text{in}}=1$ for the first layer.

To explain how we mask the filters in $P$, consider the 2D case in Fig. 8. We want to encode all values $\hat{z}_{i}$ by iterating in raster scan order and by computing $p(\hat{z}_{i}|\hat{z}_{i-1},\dots,\hat{z}_{1})$. We simplify this by instead of relying on all previously encoded symbols, we use some $c\times c$-context around $\hat{z}_{i}$ (black square in Fig. 8). To satisfy the causality constraint, this context may only contain values above $\hat{z}_{i}$ or in the same row to the left of $\hat{z}_{i}$ (gray cells). By using the filter shown in Fig. 8 in the top right for the first layer of a CNN and the filter shown in Fig. 8 in the bottom right for subsequent filters, we can build a 2D-CNN with a $c\times c$ receptive field that forms such a context. We build our 3D-CNN $P$ by generalizing this idea to 3D, where we construct the mask for the filter of the first layer as shown in pseudo-code Algorithm 1. The mask for the subsequent layers is constructed analoguously by replacing “$<$” in line 7 with “$\leq$”. We use filter size $f_{W}=f_{H}=f_{D}=3$.

With this approach, we obtain a 3D-CNN $P$ which operates on $f_{H}\times f_{W}\times f_{D}$-dimensional blocks. We can use $P$ to encode $\hat{\mathbf{z}}$ by iterating over $\hat{\mathbf{z}}$ in such blocks, exhausting first axis $w$, then axis $h$, and finally axis $d$ (like in Algorithm 1). For each such block, $P$ yields the probability distribution of the central symbol given the symbols in the block. Due to the construction of the masks, this probability distribution only depends on previously encoded symbols.

Multiple compression rates

It is quite straightforward to obtain multiple operating points in a single network with our framework: We can simply share the network but use multiple importance maps. We did a simple experiment where we trained an autoencoder with 5 different importance maps. In each iteration, a random importance map $i$ was picked, and the target entropy was set to $i/5\cdot t$. While not tuned for performance, this already yielded a model competitive with BPG. The following shows the output of the model for $i=1,3,5$ (from left to right):

On the benefit of 3DCNN and joint training

We note that the points from Table 1 (where we trained different entropy models non-jointly as a post-training step) are not directly comparable with the curve in Fig. 1. This is because these points are obtained by taking the mean of the MS-SSIM and bpp values over the Kodak images for a single model. In contrast, the curve in Fig. 1 is obtained by following the approach of , constructing a MS-SSIM vs. bpp curve per-image via interpolation (see Comparison in Section 4). In Fig. 9, we show the black curve from Fig. 1, as well as the mean (MS-SSIM, bpp) points achieved by the underlying models ($\circ$). We also show the points from Tab. 1 ($+$). We can see that our masked 3DCNN with joint training gives a significant improvement over the separately trained 3DCNN, i.e., a 22% reduction in bpp when comparing mean points (the red point is estimated).

Non-realistic images

In Fig. 10, we compare our approach to BPG on an image from the Manga109http://www.manga109.org/ dataset. We can see that our approach preserves text well enough to still be legible, but it is not as crip as BPG (left zoom). On the other hand, our approach manages to preserve the fine texture on the face better than BPG (right zoom).

Visual examples

The following pages show the first four images of each of our validation sets compressed to low bitrate, together with outputs from BPG, JPEG2000 and JPEG compressed to similar bitrates. We ignored all header information for all considered methods when computing the bitrate (here and throughout the paper). We note that the only header our approach requires is the size of the image and an identifier, e.g., $\beta$, specifying the model.

Overall, our images look pleasant to the eye. We see cases of over-blurring in our outputs, where BPG manages to keep high frequencies due to its more local approach. An example is the fences in front of the windows in Fig. 14, top, or the text in Fig. 15, top. On the other hand, BPG tends to discard low-contrast high frequencies where our approach keeps them in the output, like in the door in Fig. 11, top, or in the hair in Fig. 12, bottom. This may be explained by BPG being optimized for MSE as opposed to our approach being optimized for MS-SSIM.

JPEG looks extremely blocky for most images due to the very low bitrate.