Independently Controllable Factors

Introduction

Whether in static or dynamic environments, decision making for real world problems is often confronted with the hard challenge of finding a “good” representation of the problem. In the context of supervised or semi-supervised learning, it has been argued (Bengio, 2009) that good representations separate out underlying explanatory factors, which may be causes of the observed data. In such problems, feature learning often involves mechanisms such as autoencoders (Hinton and Salakhutdinov, 2006; Vincent et al., 2008), which find latent features that explain the observed data. In interactive environments, the temporal dependency between successive observations creates a new opportunity to notice causal structure in data which may not be apparent using only observational studies. The need to experiment in order to discover causal relationships has already been well explored in psychology (e.g. Gopnik and Wellman (2012)). In reinforcement learning, several approaches explore mechanisms that push the internal representations of learned models to be “good” in the sense that they provide better control (see §4), and control is a particularly important causal relationship between an agent and elements of its environment.

We propose and explore a more direct mechanism for representation learning, which explicitly links an agent’s control over its environment with its internal feature representations. Specifically, we hypothesize that some of the factors explaining variations in the data correspond to aspects of the world that can be controlled by the agent. For example, an object that could be pushed around or picked up independently of others is an independently controllable aspect of the environment. Our approach therefore aims to jointly discover a set of features (functions of the environment state) and policies (which change the state) such that each policy controls the associated feature while leaving the other features unchanged as much as possible. In §2 and §3 we explain this mechanism and show experimental results for the simplest instantiation of this new principle. In §5 we discuss how this principle could be applied more generally, and what are the research challenges that emerge.

Independently controllable features

To make the above intuitions concrete, assume that there are factors of variation underlying the observations coming from an interactive environment that are independently controllable. That is, a controllable factor of variation is one for which there exists a policy which will modify that factor only, and not the others. For example, the object associated with a set of pixels could be acted on independently from other objects, which would explain variations in its pose and scale when we move it around while leaving the others generally unchanged. The object position in this case is a factor of variation. What poses a challenge for discovering and mapping such factors into computed features is the fact that the factors are not explicitly observed. Our goal is for the agent to autonomously discover such factors – which we call independently controllable features – along with policies that control them. While these may seem like strong assumptions about the nature of the environment, we argue that these assumptions are similar to regularizers, and are meant to make a difficult learning problem (that of learning good representations which disentangle underlying factors) better constrained.

There are many possible ways to express the preference for learning independently controllable features as an objective. §2.2 proposes such an objective for a simple scenario. §3.1 illustrates the effect of this objective when all the features of the environment are simple and controllable by the agent. Moreover, in §3.2, we show that by itself the objective we propose is strong enough to recover underlying factors of variation without additional reconstruction loss. In §2.3, we aim to generalize such an objective for a continuous representation of factors and policies. In §3.3, we present our experiments on the Mazebase domain (Sukhbaatar et al., 2015). §3.3.1 shows that, using these continuous embeddings, we are able to disentangle the latent space and the controllable factors. In §3.3.2, we show how the learnt representations can be used for planning and for inferring the sequence of actions performed between two states.

It is common in the case of a vanilla autoencoder to assume that $n\ll d$. This causes $f$ and $g$ to perform dimensionality reduction of $X$, i.e. compression, since there is a dimension bottleneck through which information about the input data must pass. Often, this bottleneck forces the optimization procedure to uncover principal factors of variation of the data on which they are trained. However, this does not necessarily imply that the different components of the vector $h=f(x)$ are individually meaningful. In fact, note that for any bijective transformation $T$, we could obtain the same reconstruction error by replacing $f$ by $T\circ f$ and $g$ by $g\circ T^{-1}$, so we should not expect any form of disentangling of the factors of variation unless some additional constraints or penalties are imposed on $h$. This motivates the approach we are about to present. Specifically, we have a preference for policies that can separately influence one of the coordinates of $h$, and we want to express a preference for learning representations that make such policies possible.

Note that there may be several other ways to discover and disentangle underlying factors of variation. Many deep generative models, including variational autoencoders (Kingma and Welling, 2014) and other descendants of the Helmholtz machine (Dayan et al., 1995), generative adversarial networks (Goodfellow et al., 2014) or non-linear versions of ICA (Dinh et al., 2014; Hyvarinen and Morioka, 2016) attempt to disentangle the underlying factors of variation by assuming that their joint distribution (marginalizing out the observed $x$) factorizes, i.e., that they are marginally independent. Here we explore another direction, trying to exploit the ability of a learning agent to act in the world in order impose a further constraint on the representation.

2 Disentangling independently controllable factors in the simplest case

In order to quantify the change in $f_{k}$ when actions are taken according to $\pi_{k}$, we define the selectivity of a feature as:

where $s$,$s^{\prime}$ are successive raw state representations (e.g. pixels), $a$ is the action, and $\mathcal{P}_{ss^{\prime}}^{a}$ is the environment transition distribution from $s$ to $s^{\prime}$ under action $a$. The normalization factor in the denominator of the above equation ensures that the selectivity of $f_{k}$ is maximal when only that single feature $f_{k}$ changes as a result of some action.

By having an objective that maximizes selectivity and minimizes the autoencoder objective, we can ensure that the features learned can both capture the main factors of variation in the data and recover independently controllable factors. Hence, we define the following objective, which can be minimized jointly on $\pi_{k}$, $f$ and $g$, via stochastic gradient descent:

Note that many variations of this objective are possible. For example it is also possible to have directed selectivity: by using $\max\{0,f_{k}^{\prime}-f_{k}\}$ (denoted $|f_{k}^{\prime}-f_{k}|_{+}$) or simply $f_{k}^{\prime}-f_{k}$ instead of the absolute value $|f_{k}^{\prime}-f_{k}|$ in the numerator of (1), the policies must learn to increase the learned latent feature rather than simply change it. This may be useful if the policy to gradually increase a feature is distinct from the policy that decreases it. Using log-selectivity, $\log sel$, or this sharpened form, $\log(sel/(1-sel))$, may also lead to easier optimization.

The learning algorithm we propose is summarized in Algorithm 1, where $W_{f}$ and $W_{g}$ are the parameters of $f,g$ and $\theta_{k}$ the parameters of $\pi_{k}$.

The gradients on lines 3 and 4 are computed exactly via backpropagation. In our experiments, the gradient on line 6 is also computed by backpropagation and sampling of the expectation, while the gradient on line 7 is computed with the REINFORCE (Glynn, 1987; Williams, 1992) estimator:

where $b(s)$ is a baseline function, which can for example be chosen to be the mean reward or an estimate of the value of the state.

3 From enumerated factors to continuous embeddings of individual factors

In this scenario, one strategy to determine whether some selected attribute variation $A(h^{\prime}-h,\phi)$ evolves independently from other attributes variations is to compare its value (in expectation over the policy actions) to the values obtained with other $\phi^{\prime}$ factors. We thus compute the following selectivity that acts as an intrinsic reward signal, generalizing (1):

where $h^{\prime}=f(s^{\prime})$. We approximate the expectation over $\phi^{\prime}$ by sampling a fixed number of factor embeddings. This model is then trained by jointly minimizing the autoencoder reconstruction cost $\mathcal{L}_{ae}$ and the disentanglement objective $\mathcal{L}_{sel}$ as depicted in Figure 1.

Ideally $A(h^{\prime}-h,\phi)$ could be an arbitrary function, e.g. a neural network, but such function may be harder to optimize. Instead, we observe that in the discrete case mentioned previously, using $A(h^{\prime}-h,\phi)$ to select attribute $k$ is equivalent to $\phi^{\top}|h^{\prime}-h|$ where $\phi$ is a one-hot vector at index $k$. One simple step towards continuous embeddings is to relax this constraint, and let $\phi$ be a function of $h$ and random vector $z$, drawn from uniform distribution, and compute $A$ as $A(h^{\prime}-h,\phi)=\phi(h,z)^{\top}|h^{\prime}-h|$. However, in most of our experiments, we used a gaussian kernel: $A(h^{\prime}-h,\phi)=\mbox{exp}({-||h^{\prime}-h-\phi||^{2}/(2\sigma^{2})})$ because of the better numerical stability it provides.

Unlike in the finite case, we are not sampling uniformly over policies $\pi_{k}$, as we now let a neural network choose $\phi$’s probability distribution. This could lead to exploration issues. We demonstrate that simple strategies allow for a network to learn simple distributions in the experiments of §3.3.

Experimental results

In order to validate that our method learns independently controllable features, we perform several experiments. First, in the most basic gridworld-like setting, an agent is allowed to move around in four directions. This basic domain allows us to verify whether in the discrete case, the learning process disentangles the underlying features and recovers the ground truth properties of the environment.

Then, we show results of our continuous factors embeddings method applied to MazeBase (Sukhbaatar et al., 2015), as well as how we can use the learned representations to tackle policy inference or planning problems.

Our first experiment is performed on a gridworld-like setting, illustrated in Figure 2(a): the agent sees a $2\times 2$ square on a $12\times 12$ pixel grid, and has 4 actions that move it up, down, left or right. By interacting with the environment, an autoencoder We use the following architecture: $f$ has two $16\times 3\times 3$ ReLU convolutional layers with stride 2, followed by a fully connected ReLU layer of 32 units, and a $\tanh$ layer of $n=4$ features; $g$ is the transpose architecture of $f$; $\pi_{k}$ is a softmax policy over 4 actions, computed from the output of the ReLU fully connected layer. We use Adam (Kingma and Ba, 2014) to perform gradient descent. with directed selectivity (objective (1) without absolute value in the numerator) learns latent features that map to the $(x,y)$ position of the square (see Figure 2(b,c)), without ever having explicit access to these values, and while reconstructing its input properly. In contrast, a plain autoencoder also reconstructs properly but without learning the two latent $(x,y)$ features explicitly.

Note that in this setting, the learning process is robust to a stochastic version of the environment – where with probability $p$ either no action is taken ($s=s^{\prime}$) or a random action is taken. We have successfully trained models recovering $\pm x$ and $\pm y$ with up to $p=0.5$, using the same architecture but a smaller learning rate.

2 Selectivity as an only objective

We also find experimentally that training discrete independently controllable features without training the autoencoder objective correctly recovers ground truth features and their associated control policies. Albeit slower than when jointly training an autoencoder, this shows that the objective we propose is strong enough to provide a learning signal for discovering a disentangled latent representation.

We train such a model on a gridworld MNIST environment, where instead of a $2\times 2$ square there are two MNIST digits . The two digits can be moved on the grid via 4 directional actions (so there are 8 actions total), the first digit is always odd and the second digit always even, so they are distiguishable. In Figure 3 we plot each latent feature $f_{k}$ as a curve, as a function of each ground truth. For example we see that the black feature recovers $+x_{1}$, the horizontal position of the first digit, or that the purple feature recovers $-y_{2}$, the vertical position of the second digit.

3 Experiments on MazeBase

We use MazeBase (Sukhbaatar et al., 2015) to assess the performance of our continuous embeddings approach on a more complex and well-known environment. MazeBase contains 10 different 2D games in which an agent has to solve a specific task (going to a certain location on the board, activate switches, move a block to a specific place…). We do not aim to solve the game, and only deal with one-step policies.

In this setting, the agent (a red circle) can move in a small environment ($64\times 64$ pixels) and perform the actions down, left, right, up, and, to complexify the disentanglement task, we add the redundant action up as well as the action down+left. The agent can go anywhere except on the orange blocks.

In Figure 2, we show that the learned representation is such that for each underlying factor of variation, the learned representation clusters $dh$ vectors such that it is possible to decompose the variation between two arbitrary state representations as a sum of small variations along a trajectory (Figure 5).

After jointly training the reconstruction and selectivity losses, our algorithm disentangles four directed factors of variations as seen in Figure 2: $\pm x$-position and $\pm y$-position of the agent. For visualization purposes, in the rest of the section, we chose the bottleneck of the autoencoder to be of size $K=2$.

The disentanglement appears clearly as the latent features corresponding to the $x$ and $y$ position are orthogonal in the latent space. Moreover, we notice that our algorithm assigns both actions up (white and pink dots in Figure 2.a) to the same feature. It also does not create a signifant mode for the feature corresponding to the action down+left (light blue dots in Figure 2.a) as this feature is already explained by features down and left.

3.2 Towards planning and policy inference

This disentangled structure could be used to address many challenging issues in reinforcement learning. We give two examples in figure 5:

Model-based predictions: Given an initial state, $s_{0}$, and an action sequence $a_{\{0:T-1\}}$, we want to predict the resulting state $s_{T}$.

A simplified deterministic policy inference problem: Given an initial state $s_{start}$ and a terminal state $s_{goal}$, we aim to find a suitable action sequence $a_{\{0:T-1\}}$ such that $s_{goal}$ can be reached from $s_{start}$ by following it.

Because of the $tanh$ activation on the last layer of $\phi(h,z)$, the different factors of variation $dh=h^{\prime}-h$ are placed on the vertices of a hypercube of dimension $K$, and we can think of the the policy inference problem as finding a path in that simpler space, where the starting point is $h_{start}$ and the goal is $h_{goal}$. We believe this could prove to be a much easier problem to solve.

However, this disentangled representation alone cannot solve completely these two issues in an arbitrary environment. Indeed, the only factors we are able to disentangle are the factors directly controllable by the agent, thus, we are not able to account for the ambiant dynamics or other agents’ influence.

Related work

There is a large body of work on learning features in RL focusing on indirectly learning good internal representations. In Jaderberg et al. (2016), agents learn off-policy to control their pixel inputs, forcing them to learn features that help control the environment (at the pixel level). Oh et al. (2015) propose models that learn to predict the future, conditioned on action sequences, which push the agent to capture temporal features. Many more works go in this direction, such as (deep) successor feature representations (Dayan, 1993; Kulkarni et al., 2016) or the options framework (Sutton et al., 1999; Precup, 2000) when used in conjunction with neural networks (Bacon et al., 2016).

Our approach is similar in spirit to the Horde architecture (Sutton, 2011). In that scenario, agents learn policies that maximize specific inputs, whereas we learn policies that control simultaneously learned features of the input. The predictions for all these policies then become features for the agent. Our objective is defined specifically in the context of autoencoders but can be generalized to other representation-learning frameworks. Unlike recent work on the predictron (David Silver, 2017), our approach is not focused on solving a planning task, and the goal is simply to learn how agents control their environment.

Conclusion and discussion: Scaling to general environments, controllability and the binding problem

We have introduced a novel type of clue aiming at learning representations which disentangle the underlying factors of variation. The main assumption is that some of those factors correspond to independently controllable aspects of the environment. This leads to training frameworks in which one learns jointly a set of exploratory policies and corresponding features of the learned representation which disentangle those controlled aspects. This is only a first step towards training agents which learn to control their environments at the same time as learning good representations of it.

We focused on the simpler setups in which the environment is made of a static set of objects. In this case, if the objective posited in §2.2 is learned correctly, we can assume that feature $k$ of the representation can unambiguously refer to some controllable property of some specific object in the environment. For example, the agent’s world might contain only a red circle and a green rectangle, which are only affected by the actions of the agent (they do not move on their own) and we only change the positions and colours of these objects from one trial to the next. Hence, a specific feature $f_{k}$ can learn to unambiguously refer to the position or the colour of one of these two objects.

In reality, environments are stochastic, and the set of objects in a given scene is drawn from some distribution. The number of objects may vary and their types may be different. It then becomes less obvious how feature $k$ could refer in a clear way to some feature of one of the objects in a particular scene. If we have instances of objects of different types, some addressing or naming scheme is required to refer to the particular objects (instances) present in the scene, so as to match the policy with a particular attribute of a particular object to selectively modify. While our proposed distributed alternative (§2.3) is an attempt to address this, a fundamental representational problem remains.

This is connected to the binding problem in neuro-cognitive science: how to represent a set of objects, each having different attributes, so that we do not confuse, for example, the set $\{$red circle, blue square$\}$ with $\{$red square, blue circle$\}$. The binding problem has received some attention in the representation learning literature (Minin et al., 2012; Greff et al., 2016), but still remains mostly unsolved. Jointly considering this problem and learning controllable features may prove fruitful.

These ideas may also lead to interesting ways of performing exploration. The RL exploration process could be driven by a notion of controllability, predicting the interestingness of objects in a scene and choosing features and associated policies with which to attempt controlling them – such ideas have only been briefly explored in the literature (e.g. Ratitch and Precup (2003)). How do humans choose with which object to play? We are attracted to objects for which we do not yet know if and how we can control them, and such a process may be critical to learn how the world works.

References