Tagger: Deep Unsupervised Perceptual Grouping

Introduction

Humans naturally perceive the world as being structured into different objects, their properties and relation to each other. This phenomenon which we refer to as perceptual grouping is also known as amodal perception in psychology. It occurs effortlessly and includes a segmentation of the visual input, such as that shown in in Figure 1. This grouping also applies analogously to other modalities, for example in solving the cocktail party problem (audio) or when separating the sensation of a grasped object from the sensation of fingers touching each other (tactile). Even more abstract features such as object class, color, position, and velocity are naturally grouped together with the inputs to form coherent objects. This rich structure is crucial for many real-world tasks such manipulating objects or driving a car, where awareness of different objects and their features is required.

In this paper, we introduce a framework for learning efficient iterative inference of such perceptual grouping which we call iTerative Amortized Grouping (TAG). This framework entails a mechanism for iteratively splitting the inputs and internal representations into several different groups. We make no assumptions about the structure of this segmentation and rather train the model end-to-end to discover which are the relevant features and how to perform the splitting.

By using an auxiliary denoising task we focus directly on amortizing the posterior inference of the object features and their grouping. Because our framework does not make any assumptions about the structure of the data, it is completely domain agnostic and applicable to any type of data. The TAG framework works completely unsupervised, but can also be combined with supervised learning for classification or segmentation.

Another class of recently proposed mechanisms for addressing complex structured inputs is attention . These methods simplify the problem of perception by learning to restrict processing to a part of the input. In contrast, TAG simply structures the input without directing the focus or discarding irrelevant information. These two systems are not mutually exclusive and could complement each other: the group structure can help in deciding what exactly to focus on, which in turn may help simplify the task at hand.

We apply our framework to two artificial datasets: a simple binary one with multiple shapes and one with overlapping textured MNIST digits on a textured background. We find that our method learns intuitively appealing groupings that support denoising and classification. Our results for the 2-digit classification are significantly better than a strong ConvNet baseline despite the use of a fully connected network. The improvements for semi-supervised learning with 1,000 labels are even greater, suggesting that grouping can help learning by increasing the sample efficiency.

Iterative Amortized Grouping (TAG)

Our goal is to enable neural networks to split inputs and internal representations into coherent groups that can be processed separately. We hypothesize that processing the whole input in one clump is often difficult due to unwanted interference. However, if we allow the network to separately process groups, it can make use of invariant distributed features without the risk of ambiguities. We thus define a group to be a collection of inputs and internal representations that are processed together (largely) independently of the other groups.

The “correct” grouping is often dynamic, ambiguous and task dependent. For example, when driving along a road, it is useful to group all the buildings together. To find a specific house, however, it is important to separate the buildings, and to enter one, they need to be subdivided even further. Rather than treating segmentation as a separate task, we provided a mechanism for grouping as a tool for our system. We make no assumptions about the correspondence between objects and groups. If the network can process several objects in one group without unwanted interference, then the network is free to do so.

Processing of the input is split into $K$ different groups, but it is left up to the network to learn how to best use this ability in a given problem, such as classification. To make the task of instance segmentation easy, we keep the groups symmetric in the sense that each group is processed by the same underlying model. We introduce latent binary variables $g_{k,j}$ to encode if input element $x_{j}$ is assigned to group $k$.More formally, we introduce discrete random variables $G_{j}$ for each input element indexed by $j$. As a shorthand for $p(G_{j}=k)$ we write $p(g_{k,j})$ and denote the discrete-valued vector of elements $g_{k,j}$ by $\bm{g}_{k}$.

We want our model to reason not only about the group assignments but also about the representation of each group. This amounts to inference over two sets of variables: the latent group assignments and the individual group representations; A formulation very similar to mixture models for which exact inference is typically intractable. For these models it is a common approach to approximate the inference in an iterative manner by alternating between (re-)estimation of these two sets (e.g., EM-like methods ). The intuition is that given the grouping, inferring the object features becomes easy, and vice versa. We employ a similar strategy by allowing our network to iteratively refine its estimates of the group assignments as well as the object representations. If the model can improve the estimates in each step, then it will converge to a final solution.

1 Definition of the TAG mechanism

where the summation is over iterations $i$. From here on we mostly omit $i$ from the equations for readability. Since this cost function does not require any class labels or intended grouping information, training can be completely unsupervised, though additional terms for supervised tasks can be added too.

Internally, the network maintains $K$ versions of its representations indexed by $k$. This can also be thought of as running $K$ separate copies of the same network, where each network only sees a subset of the inputs and outputs $\bm{z}_{k}=q(\bm{x}|\bm{g}_{k})$ (the expected value of the input for that group), and $m_{k}=q(\bm{g}_{k})$ (the group assignment probabilities). Each $\bm{z}_{k}$ and $\bm{m}_{k}$ has the same dimensionality as the input, and they are updated over iterations. Each group $k$ makes its own prediction about the original input based on $\bm{z}_{k}$. In the binary case we use $q(x_{j}\mid g_{k,j})=\operatorname{sigmoid}(z_{k,j})$, and in the continuous case we take $z_{k,j}$ to represent the mean of a Gaussian distribution with variance $v$. We assumed the variance of the Gaussian distribution to be constant over iterations and groups but learned it from the data. It would be easy to add a more accurate estimate of the variance.

The final prediction of the network is defined as:

The group assignment probabilities $q(g_{k,j})=\bm{m}_{k}$ are forced to be non-negative and sum up to one over $k$:

Because $\bm{z}_{k}$ are continuous variables, their likelihood is a function over all possible values of $\bm{z}_{k}$, and not all of this information can be easily represented. Typically, the relevant information is found close to the current estimate $\bm{z}_{k}$; therefore we use $\delta\bm{z}_{k}$, which is proportional to the gradient of the negative log likelihood. Essentially, it represents the remaining modeling error:

The derivation of the analogous term in the binary case is presented in Appendix A.5.

Note that we normalize $L(m_{k,j})$ over $k$ such that it sums up to one for each value of $j$. This amounts to providing each group information about how likely each input element belongs to them rather than some other group. In other words, this is equivalent to likelihood ratio rather than the raw likelihood. Intuitively, the term $L(\bm{m}_{k})$ describes how well each group reconstructs the individual input elements relative to the other groups.

The denoising task encourages the network to iteratively group its inputs into coherent groups that can be modeled efficiently. The trained network can be useful for a real-world denoising application, but typically, the idea is to encourage the network to learn interesting internal representations. Therefore, it is not $q(\bm{x})$ but rather $\bm{m}_{k}$, $\bm{z}_{k}$ and the internal representations of the parametric mapping that we are typically concerned with.

where we model the group reconstruction $q_{i}(\bm{x}|\bm{g_{k}})$ as a Gaussian with mean $\bm{z}_{k}^{i}$ and variance $v$, and the group assignment posterior $q_{i}(\bm{g_{k}})$ as a categorical distribution $\bm{m}_{k}$.

The trainable part of the TAG framework is given by a parametric mapping that operates independently on each group $k$ and is used to compute both $\bm{z}_{k}^{i}$ and $\bm{m}_{k}^{i}$ (which is afterwards normalized using an elementwise softmax over the groups). This parametric mapping is usually implemented by a neural network and the whole system is trained end-to-end using standard backpropagation through time.

The input to the network for the next iteration consists of the vectors $\bm{z}_{k}^{i}$ and $\bm{m}_{k}^{i}$ along with two additional quantities: The remaining modelling error $\bm{\delta z}_{k}^{i}$ and the group assignment likelihood ratio $L(\bm{m}_{k}^{i})$ which carry information about how the estimates can be improved:

2 The Tagger: Combining TAG and Ladder Network

𝑖1i+1. Note the optional class prediction output $\bm{y}^{i}_{g}$ for classification tasks. See supplementary material for details. We chose the Ladder network as the parametric mapping because its structure reflects the computations required for posterior inference in hierarchical latent variable models. This means that the network should be well equipped to handle the hierarchical structure one might expect to find in many domains. We call this Ladder network wrapped in the TAG framework Tagger. This is illustrated in Figure 3 and the corresponding pseudocode can be found in Algorithm 1.

We mostly used the specifications of the Ladder network as described by Rasmus et al. , but there are some minor modifications we made to fit it to the TAG framework. We found that the model becomes more stable during iterations when we added a sigmoid function to the gating variable $v$ [22, Equation 2] used in all the decoder layers with continuous outputs. None of the noise sources or denoising costs were in use (i.e., $\lambda_{l}=0$ for all $l$ in Eq. 3 of Ref. ), but Ladder’s classification cost ($C_{c}$ in Ref. ) was added to the Tagger’s cost (Equation 1) for the semi-supervised tasks.

All four inputs ($\bm{z}^{i}_{k}$, $\bm{m}^{i}_{k}$, $\delta\bm{z}^{i}_{k}$, and $L(\bm{m}^{i}_{k})$) were concatenated and projected to a hidden representation that served as the input layer of the Ladder Network. Subsequently, the values for the next iteration were simply read from the reconstruction ($\hat{\bm{x}}$ in Ref. ) and projected linearly into $\bm{z}^{i+1}_{k}$ and via softmax to $\bm{m}^{i+1}_{k}$ to enforce the conditions in Equation 3. For the binary case, we used a logistic sigmoid activation for $\bm{z}^{i+1}_{k}$.

Experiments and results

We explore the properties and evaluate the performance of Tagger both in fully unsupervised settings and in semi-supervised tasks in two datasetsThe datasets and a Theano reference implementation of Tagger are available at http://github.com/CuriousAI/tagger. Although both datasets consist of images and grouping is intuitively similar to image segmentation, there is no prior in the Tagger model for images: our results (unlike the ConvNet baseline) generalize even if we permute all the pixels .

We use the simple Shapes dataset to examine the basic properties of our system. It consists of 60,000 (train) + 10,000 (test) binary images of size 20x20. Each image contains three randomly chosen shapes ($\bigtriangleup$, $\bigtriangledown$, or $\square$) composed together at random positions with possible overlap.

We generated a two-object supervised dataset (TextureMNIST2) by sequentially stacking two textured 28x28 MNIST-digits, shifted two pixels left and up, and right and down, respectively, on top of a background texture. The textures for the digits and background are different randomly shifted samples from a bank of 20 sinusoidal textures with different frequencies and orientations. Some examples from this dataset are presented in the column of Figure 5. We use a 50k training set, 10k validation set, and 10k test set to report the results. The dataset is assumed to be difficult due to the heavy overlap of the objects in addition to the clutter due to the textures. We also use a textured single-digit version (TextureMNIST1) without a shift to isolate the impact of texturing from multiple objects.

1 Training and evaluation

We train Tagger in an unsupervised manner by only showing the network the raw input example $\bm{x}$, not ground truth masks or any class labels, using 4 groups and 3 iterations. We average the cost over iterations and use ADAM for optimization. On the Shapes dataset we trained for 100 epochs with a bit-flip probability of 0.2, and on the TextureMNIST dataset for 200 epochs with a corruption-noise standard deviation of 0.2. The models reported in this paper took approximately 3 and 11 hours in wall clock time on a single Nvidia Titan X GPU for Shapes and TextureMNIST2 datasets respectively.

To understand how model size, length of the iterative inference, and the number of groups affect the modeling performance, we evaluate the trained models using two metrics: First, the denoising cost on the validation set, and second we evaluate the segmentation into objects using the adjusted mutual information (AMI) score and ignore the background and overlap regions in the Shapes dataset (consistent with Greff et al. ). Evaluations of the AMI score and classification results in semi-supervised tasks were performed using uncorrupted input. The system has no restrictions regarding the number of groups and iterations used for training and evaluation. The results improved in terms of both denoising cost and AMI score when iterating further, so we used 5 iterations for testing. Even if the system was trained with 4 groups and 3 shapes per training example, we could test the evaluation with, for example, 2 groups and 3 shapes, or 4 groups and 4 shapes.

2 Unsupervised Perceptual Grouping

Table 1 shows the median performance of Tagger on the Shapes dataset over 20 seeds. Tagger is able to achieve very fast convergences, as shown in 1(a). Through iterations, the network improves its denoising performances by grouping different objects into different groups. Comparing to Greff et al. , Tagger performs significantly better in terms of AMI score (see 1(b)).

Figure 4 and Figure 5 qualitatively show the learned unsupervised groupings for the Shapes and textured MNIST datasets. Tagger uses its TAG mechanism slightly differently for the two datasets. For Shapes, $\bm{z}_{g}$ represents filled-in objects and masks $\bm{m}_{g}$ show which part of the object is actually visible. For textured MNIST, $\bm{z}_{g}$ represents the textures and masks $\bm{m}_{g}$ texture segments. In the case of the same digit or two identical shapes, Tagger can segment them into separate groups, and hence, it performs instance segmentation. We used 4 groups for training even though there are only 3 objects in the Shapes dataset and 3 segments in the TexturedMNIST2 dataset. The excess group is left empty by the trained system but its presence seems to speed up the learning process.

The hand-picked examples A-C in Figure 4 illustrate the robustness of the system when the number of objects changes in the evaluation dataset or when evaluation is performed using fewer groups.

Example $E$ is particularly interesting; $E_{1}$ shows how the normal evaluation looks like but $E_{2}$ demonstrates how we can remove the topmost digit from the scene and let the system fill in digit below and the background. We do this by setting the corresponding group assignment probabilities $\bm{m}_{g}$ to a large negative number just before the final softmax over groups in the last iteration.

To solve the textured two-digit MNIST task, the system has to combine texture cues with high-level shape information. The system first infers the background texture and mask which are finalized on the first iteration. Then the second iteration typically fixes the texture used for topmost digit, while subsequent iterations clarify the occluded digit and its texture. This demonstrates the need for iterative inference of the grouping.

3 Classification

We investigate the role of grouping for the task of classification. We evaluate the Tagger against four baseline models on the textured MNIST task. As our first baseline we use a fully connected network (FC) with ReLU activations and batch normalization after each layer. Our second baseline is a ConvNet (Conv) based on Model C from Springenberg et al. , which has close to state-of-the-art results on CIFAR-10. We removed dropout, added batch normalization after each layer and replaced the final pooling by a fully connected layer to improve its performance for the task. Furthermore, we compare with a fully connected Ladder (FC Ladder) network.

All models use a softmax output and are trained with 50,000 samples to minimize the categorical cross entropy error. In case there are two different digits in the image (most examples in the TextureMNIST2 dataset), the target is $p=0.5$ for both classes. We evaluate the models based on classification errors. For the two-digit case, we score the network based on the two highest predicted classes (top 2).

For Tagger, we first train the system in an unsupervised phase for 150 epochs and then add two fresh randomly initialized layers on top and continue training the entire system end to end using the sum of unsupervised and supervised cost terms for 50 epochs. Furthermore, the topmost layer has a per-group softmax activation that includes an added ’no class’ neuron for groups that do not contain any digit. The final classification is then performed by summing the softmax output over all groups for the true 10 classes and renormalizing this sum to add up to one.

The final results are summarized in Table 2. As shown in this table, Tagger performs significantly better than all the fully connected baseline models on both variants, but the improvement is more pronounced for the two-digit case. This result is expected because for cases with multi-object overlap, grouping becomes more important. It, moreover, confirms the hypothesis that grouping can help classification and is particularly beneficial for complex inputs. Remarkably, Tagger, despite being fully connected, is on par with the convolutional baseline for the TexturedMNIST1 dataset and even outperforms it in the two-digit case. We hypothesize that one reason for this result is that grouping allows for the construction of efficient invariant features already in the low layers without losing information about the assignment of features to objects. Convolutional networks solve this problem to some degree by grouping features locally through the use of receptive fields, but that strategy is expensive and can break down in cases of heavy overlap.

4 Semi-Supervised Learning

Training TAG does not rely on labels and is therefore directly usable in a semi-supervised context. For semi-supervised learning, the Ladder is arguably one of the strongest baselines with SOTA results on 1,000 MNIST and 60,000 permutation invariant MNIST classification. We follow the common practice of using 1,000 labeled samples and 49,000 unlabeled samples for training Tagger and the Ladder baselines. For completeness, we also report results of the convolutional (ConvNet) and fully-connected (FC) baselines trained fully supervised on only 1,000 samples.

To put these results in context, we performed informal tests with five human subjects. The task turned out to be quite difficult and the subjects needed to have regular breaks to be able to maintain focus. The subjects improved significantly over training for a few days but there were also significant individual differences. The best performing subjects scored around 10 % error for TextureMNIST1 and 30 % error for TextureMNIST2. For the latter task, the test subject took over 30 seconds per sample.

Related work

Attention models have recently become very popular, and similar to perceptual grouping they help in dealing with complex structured inputs. These approaches are not, however, mutually exclusive and can benefit from each other. Overt attention models control a window (fovea) to focus on relevant parts of the inputs. Two of their limitations are that they are mostly tailored to the visual domain and are usually only suited to objects that are roughly the same shape as the window. But their ability to limit the field of view can help to reduce the complexity of the target problem and thus also help segmentation. Soft attention mechanisms on the other hand use some form of top-down feedback to suppress inputs that are irrelevant for a given task. These mechanisms have recently gained popularity, first in machine translation and then for many other problems such as image caption generation . Because they re-weigh all the inputs based on their relevance, they could benefit from a perceptual grouping process that can refine the precise boundaries of attention.

Our work is primarily built upon a line of research based on the concept that the brain uses synchronization of neuronal firing to bind object representations together. This view was introduced by von der Malsburg and has inspired many early works on oscillations in neural networks (see the survey for a summary). Simulating the oscillations explicitly is costly and does not mesh well with modern neural network architectures (but see ). Rather, complex values have been used to model oscillating activations using the phase as soft tags for synchronization . In our model, we further abstract them by using discretized synchronization slots (our groups). It is most similar to the models of Wersing et al. , Hyvärinen & Perkiö and Greff et al. . However, our work is the first to combine this with denoising autoencoders in an end-to-end trainable fashion.

Another closely related line of research has focused on multi-causal modeling of the inputs. Many of the works in that area build upon Restricted Boltzmann Machines. Each input is modeled as a mixture model with a separate latent variable for each object. Because exact inference is intractable, these models approximate the posterior with some form of expectation maximization or sampling procedure. Our assumptions are very similar to these approaches, but we allow the model to learn the amortized inference directly (more in line with Goodfellow et al. ).

Since recurrent neural networks (RNNs) are general purpose computers, they can in principle implement arbitrary computable types of temporary variable binding , unsupervised segmentation , and internal and external attention . For example, an RNN with fast weights can rapidly associate or bind the patterns to which the RNN currently attends. Similar approaches even allow for metalearning , that is, learning a learning algorithm. Hochreiter et al. , for example, learned fast online learning algorithms for the class of all quadratic functions of two variables. Unsupervised segmentation could therefore in principle be learned by any RNN as a by-product of data compression or any other given task.

The recurrent architecture most similar to the Tagger is the Neural Abstraction Pyramid (NAP; ) – a convolutional neural network augmented with lateral connections which help resolve local ambiguities and feedback connections that allow incorporation of high-level information. In early pioneering work the NAP was trained for iterative image binarization and iterative image denoising , much akin to the setup we use. Being recurrent, the NAP layers too, could in principle learn a perceptual grouping as a byproduct. That does not, however, imply that every RNN will, through learning, easily discover and implement this tool. The main improvement that our framework adds is an explicit mechanism for the network to split the input into multiple representations and thus quickly and efficiently learn a grouping mechanism. We believe this special case of computation to be important enough for many real-world tasks to justify this added complexity.

Future Work

So far we’ve assumed the groups to represent independent objects or events. However, this assumption is unrealistic in many cases. Assuming only conditional independence would be considerably more reasonable, and could be implemented by allowing all groups to share the same top-layer of their Ladder network.

The TAG framework assumes just one level of (global) groups, which does not reflect the hierarchical structure of the world. Therefore, another important future extension is to rather use a hierarchy of local groupings, by using our model as a component of a bigger system. This could be achieved by collapsing the groups of a Tagger network by summing them together at some hidden layer. That way this abstract representation could serve as input for another tagger with new groupings at this higher level. We hypothesize that a hierarchical Tagger could also represent relations between objects, because they are simply the couplings that remain from the assumption of independent objects.

Movement is a strong segmentation cue and a simple temporal extensions of the TAG framework could be to allow information to flow forward in time between higher layers, not just via the inputs. Iteration would then occur in time alongside the changing inputs. We believe that these extensions will make it possible to scale the approach to video.

Conclusion

In this paper, we have argued that the ability to group input elements and internal representations is a powerful tool that can improve a system’s ability to handle complex multi-object inputs. We have introduced the TAG framework, which enables a network to directly learn the grouping and the corresponding amortized iterative inference in a unsupervised manner. The resulting iterative inference is very efficient and converges within five iterations. We have demonstrated the benefits of this mechanism for a heavily cluttered classification task, in which our fully connected Tagger even significantly outperformed a state-of-the-art convolutional network. More impressively, we have shown that our mechanism can greatly improve semi-supervised learning, exceeding conventional Ladder networks by a large margin. Our method makes minimal assumptions about the data and can be applied to any modality. With TAG, we have barely scratched the surface of a comprehensive integrated grouping mechanism, but we already see significant advantages. We believe grouping to be crucial to human perception and are convinced that it will help to scale neural networks to even more complex tasks in the future.

The authors wish to acknowledge useful discussions with Theofanis Karaletsos, Jaakko Särelä, Tapani Raiko, and Søren Kaae Sønderby. And further acknowledge Rinu Boney, Timo Haanpää and the rest of the Curious AI Company team for their support, computational infrastructure, and human testing. This research was supported by the EU project “INPUT” (H2020-ICT-2015 grant no. 687795).

References

Appendix A Supplementary Material

A.2 Input

In its basic form (without supervision) Tagger receives as input only a datapoint $\bm{x}$. It corresponds to either a binary vector or a real-valued vector and is then corrupted with either bitflip or Gaussian noise. The training objective is the removal of this noise.

In the case of binary inputs we use bitflip noise for corruption:

where $\oplus$ denotes componentwise XOR, and $\mathcal{B}(\beta)$ is Bernoulli distributed noise with probability $\beta$. In our experiments on the Shapes dataset we use $\beta=0.2$.

If the inputs are real-valued, we corrupt it using Gaussian noise:

where $\sigma$ is the standard deviation of the input noise. We used $\sigma_{input}=0.2$.

A.3 Group Assignments

Within the TAG framework the group assignment is represented by the $K$ vectors $\bm{m}_{k}$ which contain one entry for each input element or pixel. These entries $m_{k,j}=q(g_{k,j})$ of $\bm{m}_{k}$ represent the discreet probability distribution over $K$ groups for each input $x_{j}$. They therefore sum up to one:

Similar to expectation maximization, the group assignment is initialized randomly, but such that Equation 7 holds. So we first sample an auxiliary $m^{\prime}_{k,j}$ from a standard Gaussian distribution and then normalize it using a softmax:

A.4 Predicted Inputs

Tagger maintains an input reconstruction $\bm{z}_{k}$ for each group $k$.

In the binary case we use a sigmoid activation function on $\bm{z}_{k}$ and interpret it directly as the probability

For the continuous case we interpret $\bm{z}_{k}$ as the means of an isotropic Gaussian with learned variance $v$:

Using the additivity of Gaussian distributions we directly get:

For simplicity we initialize all $\bm{z}_{k}$ to the expectation of the data for all $k$. In our experiments these values are $0.5$ for the TextureMNIST datasets and $0.26$ for the Shapes dataset.

A.5 Modeling Error

More precisely for a single iteration (omitting the index $i$) we have::

Note that since the network will multiply its inputs with weights, we can always omit any constant multipliers.

Let us denote the corruption bit-flip probability by $\beta$ and define

A.6 Ladder Modifications

We mostly used the specifications of the Ladder network as described by Rasmus et al. , but there are some minor modifications we made to fit it to the TAG framework. We found that the model becomes more stable during iterations when we added a sigmoid function to the gating variable $v$ [22, Equation 2] used in all the decoder layers with continuous outputs. None of the noise sources or denoising costs were in use (i.e., $\lambda_{l}=0$ for all $l$ in Eq. 3 of Ref. ), but Ladder’s classification cost ($C_{c}$ in Ref. ) was added to the Tagger’s cost for the semi-supervised tasks.

All four inputs ($\bm{z}^{i}_{k}$, $\bm{m}^{i}_{k}$, $\delta\bm{z}^{i}_{k}$, and $L(\bm{m}^{i}_{k})$) were concatenated and projected to a hidden representation that served as the input layer of the Ladder Network. Subsequently, the values for the next iteration were simply read from the reconstruction ($\hat{\bm{x}}$ in Ref. ) and projected linearly into $\bm{z}^{i+1}_{k}$ and via softmax to $\bm{m}^{i+1}_{k}$ to enforce the conditions in Equation 7. For the binary case, we used a logistic sigmoid activation for $\bm{z}^{i+1}_{k}$.

A.7 Pseudocode

In this section we put it all together and provide the pseudocode for running Tagger both on binary (Algorithm 3) and real-valued inputs (Algorithm 2). The provided code shows the steps needed to run for $T$ iterations on a single example $\bm{x}$ using $G$ groups. Here we use three activation functions: $f(x)=\max(x,0)$ is the rectified linear function, $g(x)=\frac{1}{1+e^{-x}}$ is the logistic sigmoid, and $\text{softmax}(x)_{g}=\frac{e^{x_{g}}}{\sum_{h=1}^{G}e^{x_{h}}}$ is a softmax operation over the groups. All three include a batch-normalization operation, which we omitted for clarity. Only the forward pass for a single example is shown, but derivatives of the cost $C$ wrt. parameters $v$, $W_{h}$, $W_{u}$ and $\Theta$ are computed using regular backpropagation through time. For training we use ADAM with a batch-size of $100$.