Towards quantifying information flows: relative entropy in deep neural networks and the renormalization group

Introduction

In recent years, a number of works have pointed to similarities between deep neural networks and the renormalization group (RG) beny2013deep ; mehta2014exact ; Lin_2017 ; Iso_2018 ; Koch_Janusz_2018 ; Funai_2020 ; De_Mello_Koch_2020 ; Lenggenhager_2020 ; roDLRG ; Roberts:2021fes . This connection was originally made in the context of lattice models, where decimation RG bears a superficial resemblance to certain feedforward neural network architectures. Structurally, both systems involve a coarse-graining procedure that extracts relevant information by marginalizing over hidden degrees of freedom. In the Wilsonian approach to RG, ultra-violet (UV) degrees of freedom are integrated out above some energy scale, resulting in an effective field theory that allows to make accurate predictions about the infra-red (IR). In Bayesian language, this simply corresponds to marginalizing over the high-energy information that low-energy observers cannot access. Similarly, the idea behind deep neural networks is the extraction of increasingly abstractHere, we mean “abstract” in the usual sense for such hierarchical systems; e.g., relative to a stream of pixel values, higher-level features and objects (such as shapes or faces) are considered more abstract. An outstanding question in machine learning related to this work is to quantify abstraction in deep networks; see for example KOZMA ; simpson2015abstract . features from the input data—e.g., characterizing the equivalence class of “cats” from a stream of pixel values.

where $H^{\prime}(\mathbf{x}^{\prime})$ is the coarse-grained Hamiltonian defined in terms of the remaining neurons, which play the role of the IR degrees of freedom. However, this is merely a structural analogy that holds for any (Bayesian) hierarchical model (including deep or stacked RBMs, as one can see by iteratively applying (1) at subsequent layers). Contrary to some suggestions in the literature, this formal analogy with RG does not suffice to explain how deep neural networks learn. During the learning process, the couplings in the Hamiltonian are dynamically updated, whereas these are fixed under RG via the constraint (1). Nonetheless, the analogy may be helpful for understanding the behavior and properties of these networks. In particular, one interesting direction is to examine the flow of cumulants in successive layers, which can induce higher-order interactions that encode the correlations between marginalized degrees of freedom Lenggenhager_2020 ; roDLRG ; Roberts:2021fes .

It is therefore interesting to examine this structural analogy in more detail, in particular with an eye towards understanding how “information” is processed at subsequent RG steps/network layers. The fact that information is lost under RG is well-known: intuitively, RG may be thought of as a coarse-graining procedure, in which fine-grained information gets traced out or marginalized over at low energies. In relativistic quantum field theory, this corresponds to the fact that high-energy degrees of freedom decouple, and the total number of degrees of freedom decreases as one moves into the IR. This has been quantified by Zamolodchikov’s famous $c$-theorem and various extensions Zamo ; CARDY1988 ; JACK1990 ; Casini:2006es ; Casini:2015woa ; Casini:2004bw ; Komargodski:2011vj ; Taylor:2016kic , which define a quantity which decreases monotonically under RG. The $c$-function can be straightforwardly related to entanglement entropy in certain cases (see for example Casini:2006es ; Casini:2015woa ). Alternatively, it may be related to relative entropy Casini:2016udt , which we define momentarily and which plays an important role in our analysis.

For neural networks however, there is so far no general quantification of hierarchical abstraction in terms of information-theoretic language. An ultimate goal would be to not only quantify the flow of information through subsequent layers, but also to find measures to qualify the learning process. While some progress in this direction has been achieved (see for example poole2016exponential ; schoenholz2017deep ; xiao2018dynamical ; chen2018dynamical ; gilboa2019dynamical ; BahriRev ; Roberts:2021fes ; KOZMA ; simpson2015abstract ; Lin_2017 ; VJ and references therein), this goal has not yet been universally achieved. We expect that further insight from physics may be of use in this context.

In this paper, we pursue a more modest goal of quantifying the information flow in these two systems, using the relative entropy or Kullback-Leibler (KL) divergence,

which quantifies the extent to which the target distribution $q$ differs from some reference distribution $p$. The KL divergence is non-negative,Proofs of non-negativity implicitly assume that both distributions are normalized with respect to the same measure. As we will discuss in sections 2 and 3, the changing dimensionality of the Hilbert space must be taken into account in order to avoid a non-monotonic, potentially negative result. and equals zero only when the two distributions are identical. Note that insofar as the KL divergence is asymmetric and fails to satisfy the triangle inequality, it is not a metric distance;It is, however, intimately related to the Fisher information metric; see for example Erdmenger:2020vmo for a recent exploration in the context of field theory. nonetheless, it provides a measure of the difference in information content of $q$ relative to $p$. In the present context for example, $p$ corresponds to the UV theory at equilibrium, or to the first (input) layer of a deep neural network; $q$ then respectively represents the theory at some point along the RG flow to the IR, or the distribution of neurons on an arbitrary hidden layer.We will give an argument in section 2 below as to why it is incorrect to invert the identifications such that $p$ instead represents the IR. We then wish to understand how the relative entropy behaves as a function of depth – both along the RG flow, and when moving to deeper and deeper layers of the network – as a means of quantifying information in such systems.

Specifically, we compute the KL divergence explicitly for the 1- and 2-dimensional Ising models under decimation RG on the one hand, and a simple feedforward random networkThis class of deep neural network will be reviewed in section 3. on the other, and examine the similarities between the two. The results in both cases are qualitatively identical: the KL divergence rapidly and monotonically increases before asymptoting to some value that depends on either the coupling constants (in the Ising model) or the parameters characterizing the weights and biases (in the neural network). For the Ising models, the only work of which we are aware that studies a similar (though not identical) KL divergence under decimation RG is Fowler’s 2020 thesis Fowler:2020rkl ; we will comment on the differences between our analyses in section 2. For the neural networks, as far as we are aware, this is the first such explicit computation to have appeared in the literature, though the use of various entropic quantities in machine learning has a deep and diverse history. For example, the entropy of a given layer, as well as the closely-related mutual information, have recently been evaluated via replica methods in Gabri__2019 . Entropy has also been proposed as a training mechanism in the context of adversarial learning in sperl2020optimizing ; see also Musso_2021 ; musso2021entropic . Additionally, estimates of $f$-divergences (of which the KL divergence is perhaps the canonical example) are relevant for generative adversarial networks, see e.g., nowozin2016fgan . Mutual information in particular – which may be defined as the KL divergence of $p(x)p(y)$ relative to $p(x,y)$ – has a wide utility ranging from the information bottleneck tishby2015deep to various maximization (training) methods foggo2020maximum ; MMI ; ravanelli2019learning ; hjelm2019learning ; Shen_2019_ICCV . For a non-exhaustive sampling of other works on estimating or bounding entropic quantities in deep learning, see Belghazi ; Molavipour_2020 ; gao2018estimating ; pooleBounds ; Paninski .

Interestingly, while we shall give some further arguments below as to why the monotonicity may be expected,We note that the monotonicity of relative entropy has been proven for perturbed 2d CFTs in Casini:2016udt . we find the KL divergence to be insensitive to the phase behavior of the system. The 2d Ising model, for example, exhibits a continuous phase transition from ordered to disordered at some finite value of the couplings, and the behavior of the KL divergence shows no apparent change as we smoothly dial the couplings through this point. Similarly, as we will review in section 3, recent work by poole2016exponential ; schoenholz2017deep showed that the feedforward neural networks under study also exhibit such a phase transition, which appears to control the depth to which a given network can be trained (see also xiao2018dynamical ; chen2018dynamical ; gilboa2019dynamical , for extensions of this work to more complicated architectures, or roCrit ; BahriRev for reviews). The basic intuition is that the critical point is characterized by a divergent correlation length, which allows information about the input data to propagate all the way through a network of theoretically arbitrary depth. However, quantifying precisely what this means in information-theoretic language is challenging, and indeed one original motivation for this work was to determine whether the Kullback-Leibler divergence could provide a complementary or alternative characterization of information propagation in this context.

Before entering into the heart of our analysis, let us mention that the relation between relative entropy and RG flow in the context of quantum field theory has been explored in a different setting in Balasubramanian:2014bfa . There, relative entropy was used to define a proximity notion between probability distributions associated to different QFTs. Its second derivative with respect to the RG scale was related to the Zamolodchikov metric.

This paper is organized into two main sections: in section 2, we compute the relative entropy (KL divergence) for both the 1d and 2d classical Ising models under decimation RG. Specifically, we fix the reference distribution $p$ to be the initial system, and take the target distribution $q$ to be the system under sequential RG steps. In section 3, we first introduce the feedforward, random neural networks of the same type studied in poole2016exponential ; schoenholz2017deep , and briefly summarize the notion of criticality in these networks for the sake of completeness. We then compute the KL divergence as a function of depth by fixing the reference distribution to be that of the neurons comprising the input layer, and move the target distribution through all subsequent (hidden) layers. We conclude with some discussion in section 4. For the sake of completeness, we have included two short appendices: a review of the real-space decimation procedure for the 1d Ising model in appendix A, and some details about the evaluation of the KL divergence in our deep neural networks via Monte Carlo integration in appendix B.

Relative entropy of spin models under decimation

In this section, we will examine the relative entropy or Kullback-Leibler (KL) divergence between different points along the RG flow of the 1d and 2d Ising models. Specifically, we will perform the RG flow in real-space via the standard decimation procedure, and track the KL divergence of the coarse-grained or infra-red (IR) distribution of spin states at successive steps relative to the initial fine-grained or ultra-violet (UV) distribution.

The decimation RG procedure applied to the Ising models has been well studied since the introduction of the “block spin” concept by Kadanoff in 1966 Kadanoff_blockspin . There are many excellent articles and reviews we could cite for this topic, but we will limit ourselves to those we actively referenced in the course of this work. These are Wilson’s seminal work on renormalization and critical phenomena Wilson_RG , the early pedagogical review of renormalization by Maris and Kadanoff Maris_Kadanoff_RG , the statistical mechanics text by Pathria Pathria , and Vvedensky’s course notes Vvedensky .We are grateful to Dimitri Vvedensky for his correspondence and course notes. In addition, we will of course refer to Onsager’s exact solution of the 2d Ising model Onsager .

We begin with a general discussion of calculating this relative entropy for a general spin system. Let the spins be labeled by some index $i$ as $\sigma_{i}=\pm 1$ and let $H(\sigma_{i})$ be the Hamiltonian of the system. The partition function is $Z=\sum e^{-H(\sigma_{i})}$, where the sum is over the entire Hilbert space $\mathcal{H}$ of spin states of the system (i.e., $\sum_{\{\sigma\}}\equiv\prod_{i}\sum_{\sigma_{i}=\pm 1}$), and we have absorbed the inverse temperature $\beta$ into the couplings.In our numerical experiments below, we will effectively set this to 1 along with Boltzmann’s constant. The state of the system at any given time follows a Boltzmann distribution,

As a probability distribution function, $p$ is an element of all positive semi-definite normalized real functions on $\mathcal{H}$.

Decimation is then defined by some map $\sigma_{j}^{\prime}(\sigma_{i})$ from $\mathcal{H}$ to a subset $\mathcal{H}^{\prime}\subset\mathcal{H}$. Formally, $\mathcal{H}^{\prime}$ should be strictly smaller than $\mathcal{H}$ in the sense that the difference is nonempty $\mathcal{H}\backslash\mathcal{H}^{\prime}\neq\emptyset$. In Wilsonian language, this corresponds to reducing the energy scale at which one probes the system, such that the fine-grained degrees of freedom – in this case, the elements in $\mathcal{H}\backslash\mathcal{H}^{\prime}$ of $H(\sigma_{i})$ – have been marginalized over in the effective Hamiltonian $H^{\prime}(\sigma_{j}^{\prime})$ on $\mathcal{H}^{\prime}$. A key fact of the renormalization group is that it preserves the partition function, i.e.,

where the identification in the penultimate step follows from the fact that the IR distribution $p^{\prime}(\sigma^{\prime})$ is obtained by marginalizing or tracing over $\sigma_{i}\in\mathcal{H}\backslash\mathcal{H}^{\prime}$ in the UV distribution $p(\sigma_{i})$:

A crucial consequence of this structure, underlying both our analysis below and our ability to do physics in general, is that the expectation value of an IR observable $\mathcal{O}^{\prime}$ with respect to $p^{\prime}$ is the same as that with respect to $p$:

Note that this does not work in reverse: it is incorrect to compute $\langle\mathcal{O}\rangle_{p^{\prime}}$ because a UV observable $\mathcal{O}$ will generically depend on fine-grained information about which the IR distribution $p^{\prime}$ is ignorant.

We can then formally define the entropy of the distribution after decimation relative to the distribution before decimation as a slight modification of (2):

where $|\mathcal{H}\backslash\mathcal{H}^{\prime}|$ is the dimension of the complement of $\mathcal{H}^{\prime}$ in $\mathcal{H}$. The reason why this extra term must be added is due to the fact that, while $p$ is properly normalized with respect to $\mathcal{H}$, the distribution $p^{\prime}(\sigma_{j}^{\prime})$ is properly normalized only on $\mathcal{H}^{\prime}$. If we think of the spins $\sigma_{j}^{\prime}$ as being a map from $\mathcal{H}$ to $\mathcal{H}^{\prime}$, then this map is many-to-one and, therefore, thought of as a distribution on $\mathcal{H}$, the integral of $p^{\prime}$ is not unity, but rather the dimension of the complement of $\mathcal{H}^{\prime}$:

In other words, p^{\prime}\bigl{(}\sigma_{j}^{\prime}(\sigma_{i})\bigr{)} must be divided by $|\mathcal{H}\backslash\mathcal{H}^{\prime}|$ to be a normalized distribution over all of $\mathcal{H}$, which leads to the extra term in the relative entropy (7). A similar normalization issue will arise when we consider the relative entropy in deep neural networks in section 3; see the discussion around (91) therein.

Due to the fact that the decimation map preserves the partition function, the expression for the relative entropy simplifies to

where on the second line, we have used the observation (6) that $\langle H^{\prime}\rangle_{p}=\langle H^{\prime}\rangle_{p^{\prime}}$.

Let us now apply the formalism above to the 1d classical Ising model with $N$ spins:

where we impose the periodic boundary condition $\sigma_{N+1}=\sigma_{1}$. As usual, we expect the choice of boundary condition to be irrelevant in the thermodynamic limit $N\rightarrow\infty$.

To minimize technical complications, we shall perform the simplest decimation procedure by which we sum over the spins at the even lattice sites:

i.e., $\mathcal{H}^{\prime}=\text{odd spins}$ and $\mathcal{H}\backslash\mathcal{H}^{\prime}=\text{even spins}$, with dimensions $|\mathcal{H}^{\prime}|=|\mathcal{H}\backslash\mathcal{H}^{\prime}|=2^{\frac{N}{2}}$. After some standard algebra (see appendix A), one arrives at the result for the new Hamiltonian

Similarly, let $K^{(n)}$ and $K_{0}^{(n)}$ denote the result of iterating this recursion relation $n$ times:

where $2K^{(n)}$ can be written as $m$ nested $\ln\cosh$’s acting on $2K^{(n-m)}$ for $m=1,\ldots,n$.

Next, we must compute the expectation values of the Hamiltonians with respect to their respective Boltzmann distributions. This is straightforward to obtain from the reduced free energy per site $f\coloneqq\tfrac{\beta}{N}F=-N^{-1}\ln Z$, which one can find in any decent statistical mechanics textbook:

As discussed in the introduction, the partition function $Z$ must be preserved under the decimation procedure. While this is not obvious at the level of the iterated recursion relations (14), we have proven it explicitly in appendix A.1.

The expectation value of $H$ with respect to $p$ is then

For convenience, let us define the normalized relative entropy $s(p||p^{\prime})$ to be the relative entropy divided by the total entropy of the initial (reference/UV) system in the infinite-temperature (i.e., $K\rightarrow 0$) limit, which in the present case is simply

Thus, after a single decimation step, we have

If we then iterate this procedure $n$ times, the dimensions of the Hilbert spaces become

and the normalized relative entropy after $n$ RG steps is

where $K_{0}^{(n)}$ and $K^{(n)}$ are the couplings after $n$ decimation steps, given in (14). Note that $K_{0}$ drops out of the final expression; this is to be expected, since $K_{0}$ simply represents an arbitrary choice for the zero-point of the free energy. The plot of this normalized relative entropy as a function of the decimation step for various values of the nearest-neighbor interaction coupling $K$ is shown in the left panel of fig. 1.

To provide some physical insight into this result, recall that the extra $\ln|\mathcal{H}\backslash\mathcal{H}^{\prime}|$ term in the definition of the relative entropy (9) – which gives rise to the $1\!-\!\tfrac{1}{2^{n}}$ in (21) – is purely due to the fact that the decimation procedure reduces the dimensionality of the system by half. Had we not included this term, we would instead obtain the result shown in the right panel of fig. 1. The bottom curve, corresponding to weak coupling $K\ll 1$, is then quite easy to interpret: at weak coupling, the spins hardly communicate with one another, and therefore the effect of decimation is to simply halve the number of degrees of freedom – i.e., halve the information content – at each step. Obviously, we cannot lose any more information than was present in the initial configuration to begin with, which in the small-$K$ limit is simply the total entropy of $N$ independent spins, $S_{0}=N\ln 2$. This is why the bottom curve on the right plot in fig. 1 is bounded below by $-1$, as well as our reason for defining the normalized relative entropy above.

Now, as we increase the coupling, the spins become (classically) correlated, and hence the total information content of the system consists in part of information stored in these correlations. The stronger the coupling, the more information can be stored in long-range correlations, and hence the more robust it will be against decimation. This is why the top curve on the right plot in fig. 1 is approximately 0: the monotonic reduction in the number of degrees of freedom does not noticeably alter the information content, since as $K\rightarrow\infty$ the correlations become so strong that the system behaves like a single collective mode.

We therefore observe a clear connection between the relative entropy without the extra normalization factor $\ln|\mathcal{H}\backslash\mathcal{H}^{\prime}|$ and the celebrated $c$-theorem, in which the eponymous $c$-function measures the number of degrees of freedom Zamo ; CARDY1988 ; JACK1990 ; Casini:2006es ; Casini:2015woa ; Casini:2004bw ; Taylor:2016kic . The fact that it decreases monotonically under RG from the UV to the IR precisely and rigorously demonstrates the intuitive idea that the number of degrees of freedom decreases as we flow to low energies.Here, we refer to the $c$-theorem in the broader sense that there exists a function measuring the number of degrees of freedom that decreases along the RG flow. We are not claiming a direct relationship with the Zamolodchikov $c$-function in perturbed CFTs. Indeed, the monotonic increase in the relative entropy observed in the left plot of fig. 1 is obtained only when accounting for this reduction in dimensionality.

In fact, it is easy to prove that the normalized relative entropy must be monotonically increasing. First, observe that repeated application of (13) yields

Substituting this into (21), we have an expression for the normalized relative entropy in terms of the coupling $K$ at every decimation step:

Hence, the difference between successive decimation steps satisfies the relation

This analysis is in line with the result of Casini:2016udt based on modular Hamiltonians in perturbed 2d CFTs.

We can go slightly further in our discussion of weak- vs. strong-coupling above and derive some approximate expressions for the relative entropy in these limits. When $K\ll 1$, the recursion relations (13) read

Plugging these into the normalized relative entropy, we find

where $s^{(0)}=0$. Thus, after the first decimation step, the normalized relative entropy remains approximately constant, and simply scales quadratically with the strength of the nearest-neighbor interaction.

Strongly-coupled limit

In the strongly-coupled limit, when $K\gg 1$, the recursion relations (13) instead read

whence the normalized relative entropy becomes

at which point $K^{(n)}\approx 0$, such that the large-$K$ approximation is no longer possible.

Asymptote of the relative entropy

Let us now ask about the asymptotic value of $s^{(\infty)}$ itself. We know that it must behave like (29) for small $K$, and that it must go to unity as $K\rightarrow\infty$ due to the normalization, cf. (18). However, while we are unable to evaluate the asymptote analytically in the large-$K$ limit, we can obtain a surprisingly good fit with the following ansatz:

where $f(K)$ is function to be determined. In the limit $K\rightarrow\infty$, the only restriction on $f(K)$ is that it must not go to zero faster than $\frac{1}{K^{2}}$, so that

Meanwhile, to have the correct small-$K$ limit (29), the behavior of $f(K)$ as $K\rightarrow 0$ must be

We then compare various choices of $f(K)$ satisfying these two limits to the data obtained above. By inspection of fig. 1, $n\!=\!10$ is a sufficient approximation to $n\!=\!\infty$ for the present purpose. After some experimentation, we obtain the following function fit:

This is plotted in fig. 2, which shows remarkably good agreement with the numerical results.

Comparison with previous work

After completing this work, we became aware of the thesis by Fowler Fowler:2020rkl , in which a quantity called the “KL density” (KL divergence per spin) is computed for the 1d and 2d Ising models under decimation RG. In that case however, what the author refers to as the KL divergence is in fact the mutual information (MI) between the joint distribution of spins for the initial system and the product distribution of spins at subsequent decimation steps. Both quantities – the KL divergence we compute here, and the MI in figs. 8.1 and 8.2 of Fowler:2020rkl – increase monotonically to an asymptotic value which itself increases with increasing coupling constant $K$. In fig. 1 (left plot), we have normalized by the entropy of $N$ spins, cf. (18), so that the $K\rightarrow\infty$ asymptote approaches unity, in contrast to Fowler’s $\ln 2\approx 0.69$.

An important difference between these quantities is that the relative entropy is not only monotonically increasing as a function of decimation step but, at any fixed decimation step, is also monotonically increasing as a function of $K$. Therefore, it displays no qualitative difference in behaviour as we move towards the unstable fixed point at $K\rightarrow\infty$. In contrast, the mutual information in Fowler:2020rkl takes more and more decimation steps for the curves to rise as we increase $K$; that is, stronger coupling delays the increase in the mutual information.

Conceptually, another difference is the relation to the notion of information loss under RG that we discussed in the introduction. While the KL divergence provides a direct measurement of the total information content as a function of RG step, the MI probes the information stored in correlations between the product distributions. A priori, this may increase, decrease, or remain unchanged depending on the system at hand, so the relation between RG and the monotonicity we observed for the KL divergence above does not necessarily hold for a generic hierarchical system. In fact, for the neural networks we study in section 3, this mutual information is identically zero for all layers, since the distribution on each layer factorizes. Thus the KL divergence (2) is a more useful quantity for our purposes, since it allows us to compare the similarities between these different systems.

2 2d classical Ising model

While the one-dimensional Ising model provides a clean proof of concept – as well as the most obvious parallel to the feedforward networks we shall consider in section 3 – the lack of a non-trivial critical point means that we cannot investigate whether the relative entropy is sensitive to the phase structure of the theory. We therefore move to the two-dimensional classical Ising model, which exhibits a well-known phase transition at finite temperature.

For simplicity, we shall consider the isotropic case on a square lattice of $N$ spins, so that the Hamiltonian is

where the sum runs over nearest-neighbor pairs, and we impose the usual periodic boundary conditions, which are again inconsequential in the thermodynamic limit $N\rightarrow\infty$.

We consider the standard “checker board” decimation procedure whereby we marginalize over the spins whose lattice coordinates sum to an odd number, i.e., every-other spin in both the horizontal and vertical directions. Unfortunately, the Hamiltonian is not closed under this decimation procedure (that is, more complicated interaction terms are generated at each decimation step).That real-space RG is generally problematic has been known since the work of Griffiths and Pearce GP ; Griffiths ; see also Sokal:1994un ; vanEnter:1994cb . Physically, the intuition is that in momentum space, the RG should remove only those modes which lie above some UV cut-off, corresponding to the inverse lattice spacing. If however we marginalize over, say, every-other spin, we have removed not only the nearest-neighbor correlations at the cut-off scale, but also some of the long-range correlation between distant marginalized spins. For example, at the first decimation step, a next-to-nearest neighbor interaction term as well as a four-point “square plaquette” interaction term are generated; the associated couplings are denoted $L^{\prime}$ and $M^{\prime}$ in Pathria’s text, respectively Pathria , so that after one decimation step, the Hamiltonian becomes

where n.n.n. stands for next-to-nearest-neighbors and sq. stands for square plaquette. The new parameters are related to the initial (UV) coupling $K$ by

Let us make a few comments at this point. First, we cannot solve this set of coupled equations analytically. Additionally, we observe that there is no finite solution for the critical value $K_{c}$, since the only solutions to $K^{\prime}=K$ are $K=0$ and $K=\infty$. Therefore, even trying to solve these equations numerically would be ineffectual. Second, since longer-range interactions represented by the $L^{\prime}$ and $M^{\prime}$ terms are generated after decimation, one should have introduced such terms in the initial Hamiltonian (38) for consistency. However, more and more complicated interactions will be generated under each subsequent decimation step, and therefore no finite truncation of these interactions is strictly consistent. Nevertheless, we expect that there should be a regime in which higher-order interactions are relatively weak, and the dynamics are primarily governed by the nearest-neighbor term. Furthermore, this hierarchy of interaction scales should be preserved under RG.If this were not the case, then there would be no way to make sense of the pure 2d classical Ising model at all, since higher-order terms would become increasingly important at low energies. With this in mind, several approximations have been proposed to obtain useful recursion relations from (40).

The approach originally proposed by Wilson in 1975 Wilson_RG is to ignore all interaction terms except for $K$ and $L$, introduce an $L$ interaction term in the Hamiltonian $H$ to begin with, and assume that $K,L\ll 1$ so that the recursion relations may be expanded. Another approach proposed by Maris and Kadanoff Maris_Kadanoff_RG is to ignore $M^{\prime}$ and to define an “effective” nearest-neighbor coupling after decimation to be the sum of $K^{\prime}$ and $L^{\prime}$, since having a next-to-nearest-neighbor interaction is somewhat analogous to having a slightly stronger nearest-neighbor interaction. There is a slightly more detailed energetic argument behind this idea in the original paper by Maris and Kadanoff, to which we refer the interested reader. In this approach, the recursion relations read

where $K^{\prime}$ here is the effective $K^{\prime}$, being the sum of $K^{\prime}$ and $L^{\prime}$ in (40).

Both of these methods modify the recursion relation (40b) slightly so as to produce a fixed point at some finite value of $K$, called the critical coupling $K_{c}$. Without modification, the only fixed points (i.e., solutions to $K^{\prime}=K$) are $K=0$ and $K=\infty$, with $K=0$ being stable and $K=\infty$ being unstable. In the Maris-Kadanoff approach there is a third fixed point at

which is now the unstable fixed point, while $K=0$ and $K=\infty$ are now both stable.

In the approach of Wilson, the story is slightly more complicated. We will summarize the main result here and refer the reader to Wilson_RG ; Pathria for the details. There is a fixed point in the $(K,L)$-plane at $K^{*}=\frac{1}{3}$ and $L^{*}=\frac{1}{9}$ and one critical line of points that flow to this fixed point under renormalization. This line is extrapolated linearly to the $K$-axis, where $L=0$, and the intersection is the critical value $K_{c}$, which gives

Meanwhile, the exact solution for the critical coupling due to Onsager is Onsager

To somewhat tie these various approaches together, one could consider a one-parameter extension of the Maris-Kadanoff approach in which we replace (41) with

where $\alpha$ is a positive real number. The original recursion relation (40) corresponds to the value $\alpha=1$, while completely ignoring $L^{\prime}$ and $M^{\prime}$. Again, this has no non-trivial fixed points besides $K=0$ and $K=\infty$. The modification due to Maris and Kadanoff in (41) corresponds to $\alpha=\frac{3}{2}$, which gives a non-trivial fixed point at (42). The value $\alpha^{*}$ of $\alpha$ that would produce the exact critical coupling (44) is slightly greater than this:

All of these approaches suffer at large coupling. This is a fundamental drawback of real space decimation RG when applied to the 2d Ising model; see footnote 11, which offers some intuition for the fact that problems are more pronounced when $K$ is large.

However the model is exactly solvable by the methods first developed by Onsager Onsager , so we know $K_{c}$ exactly (44), and we know that this fixed point is repulsive on both sides: if $K<K_{c}$, then the system flows towards the $K=0$ free fixed point and if $K>K_{c}$, then the system flows towards the $K=\infty$ strongly-coupled fixed point. Therefore, in the region $0\leq K\leq K_{c}$, it is justified to ignore all the complicated additional interaction terms that are generated at each decimation step. Above $K_{c}$ however, $K$ tends to grow after each decimation step, which means that the additional interactions (e.g., the $M$-type interaction), will also tend to grow in magnitude. Ignoring these interactions will thus result in an increasingly bad approximation at larger and larger $K$. This will become patently clear when we plot the normalized relative entropy for the 2d Ising model at strong coupling below (see fig. 4): if we push $K$ above the critical value, the normalized relative entropy begins to decrease, and eventually becomes negative as $K\rightarrow 1$, signalling an obvious breakdown of the approximation.

Now, to calculate the relative entropy, we need the expectation value of the Hamiltonian. As before, we start with the reduced free energy per site, which for the isotropic model with no external magnetic field, reads

Then, the expectation value of the original Hamiltonian is given by

After some algebra, one can massage this into the form

Meanwhile, the expectation value of the Hamiltonian after one decimation step is

Therefore, the normalized relative entropy after $n$ decimation steps is given by

As it should, $K_{0}$ once again drops out of this expression. The plot of this normalized relative entropy as a function of the decimation step for various values of the nearest-neighbor interaction coupling constant $K$ within the $\alpha$-extended Maris-Kadanoff approach (45) is shown in fig. 3 and 4. Here, we have chosen $\alpha=\alpha_{*}$ in (46) so that the value of the critical coupling $K_{c}$ matches the exact result given in (44).

As expected from our analysis above, we observe a qualitative difference in behavior for $K$ above vs. below the critical value $K_{c}$. For the small values of $K$ in fig. 3, the behavior is the same that we observed in the 1d Ising model: the relative entropy monotonically increases to some asymptotic value that depends on the coupling strength. For the large values of $K$ in fig. 4 however, the breakdown of the approximation already becomes apparent when $K$ is only slightly above the critical point: in the $K=0.5$ curve, the relative entropy slightly overshoots its asymptote before approaching it from above. This behavior becomes more and more pronounced as we increase $K$ even further, until eventually the approximation is so poor that the normalized relative entropy becomes negative. We emphasize that negativity in the normalized relative entropy is indeed pathological, as the change in dimensionality responsible for the (non-pathological) negativity in the unnormalized relative entropy has already been taken into account. See also Fowler:2020rkl for further discussion of this behaviour in the context of the closely related KL divergence considered therein.

Since the approximation breaks down above $K_{c}$, we can only reliably study the asymptotic value of the normalized relative entropy for $K\leq K_{c}$. In the small-$K$ limit, one finds

We take $s^{(\infty)}$ to be well-approximated by $s^{(12)}$, and plot this approximation on top of the numerical results in fig. 5, showing good agreement for small values of $K$.

Having completed our analysis of the relative entropy in these simple spin models, let us now turn our attention to deep neural networks, to see whether the analogy with RG – with subsequent layers playing the role of successive RG steps – extends to the behavior of the KL divergence in these systems.

Relative entropy in deep neural nets

where $\mathcal{D}z=(2\pi)^{-1/2}e^{-z^{2}/2}$ is the standard Gaussian measure.

Now, as mentioned in the introduction, the critical point – in the 2d phase space parametrized by $\sigma_{w}^{2}$, $\sigma_{b}^{2}$ – is characterized by a divergence in some correlation length. To find this, poole2016exponential consider the two-point correlator (i.e., the covariance matrix) between two different inputs to the same network, denoted $x_{a}=\{x_{i,a},\ldots,x_{N,a}\}$ (that is, $y_{a}^{0}=x_{a}$). The correlation between pre-activations is then

where the standard normal variables $z_{1}$, $z_{2}$ are related to $z_{a}$, $z_{b}$ as

Interestingly, poole2016exponential showed that both (56) and the correlation $\rho$ exhibit fixed points at particular points in the 2d phase space parametrized by $\sigma_{w}^{2}$, $\sigma_{b}^{2}$. The former is defined by

where the asterisks denote the evaluation of $z_{a},z_{b}$ at $\rho=\rho^{*}$, $q=q^{*}$. It is easy to see that this expression admits a fixed point at $\rho^{*}=1$, corresponding to perfect correlation. One can then probe the stability of this fixed point by considering

In physics, such a critical point at $\chi_{1}=1$ is characterized by a divergent correlation length. This is worked-out in schoenholz2017deep , and denoted $\xi_{c}$:

One can numerically evaluate this for a particular point $(\sigma_{w}^{2},\sigma_{b}^{2})$ in phase space by first finding the value of $q^{*}$ via (60), and then using this to obtain the corresponding value of $\rho^{*}$ via (61), which we overlay in fig. 6. Precisely at $\rho^{*}=1$, however, the argument of the logarithm reduces to $\chi_{1}=1$, and thus the correlation length diverges at the critical point as expected.

This is an important result that goes a long way to explaining the trainability of networks of different depths, to wit: such networks are not trainable if their depth exceeds the correlation length. This implies that initializing networks near criticality will enable one to train substantially deeper networks than would otherwise be possible; see for example xiao2018dynamical , in which this was used to train a network with an unprecedented 10,000 layers. This idea is illustrated in fig. 5 of schoenholz2017deep , which we have reproduced for our custom networks in fig. 6. Even for our relatively crude experiments, one sees a clear fall-off in trainability on either side of the critical point. Curiously, as in schoenholz2017deep ; xiao2018dynamical , the fall-off does not correspond to the correlation length itself, but to some $\sigma_{w}^{2}$-dependent function thereof. We suspect that this is due to finite-width effects, which give subleading corrections to the large-$N$ “mean field” analysis reviewed above Roberts:2021fes ; GJ2021 .

To warm up, let $p(z)$, $q(z)$ denote the normal distributions, with potentially different means and standard deviations, of some continuous random variable $z$, i.e.,

and similarly for $q$. The relative entropy between these distributions – with $q$ interpreted as the approximation to or description of the reference distribution $p$ – is

The remaining expectation value is readily evaluated:

Substituting this and (66) into (65), we obtain

Note that when $q=p$, the last term reduces to 1/2 and combines with the second term to yield the entropy (66), in which case $S(p||q)=0$, as expected.

The above result is straightforwardly generalized to the multidimensional case

The first term in (65) is just the entropy of a multivariate Gaussian, and can be computed in generality:

where $|\cdot|$ denotes the determinant (this is easily obtained via the so-called “trace trick”). As for the second term, since both $\Sigma_{q}$ and $\Sigma_{p}$ are diagonal, the expectation value in (67) becomes

where $[\Sigma]_{ii}$ is the diagonal element of $\Sigma$. In going to the second line, we have used the factorization of the multivariate Gaussian (i.e., diagonality of $\Sigma_{p}^{-1}$), and the fact that the integrals over all $z_{j}\neq z_{i}$ evaluate to unity, leaving a sum of terms of the form (68). (Note also that the trace acts on everything to the right, not just $\Sigma_{q}^{-1}$, we are simply suppressing a surplus of brackets). Thus in place of (69), we have

The KL divergence for the multivariate case, with diagonal covariance matrices, is therefore

Note that when $q=p$, the last term reduces to $N/2$, so that we again recover $D(p||p)=0$.

Taking $\phi(z)=\tanh(z)$, and using the factorization $p(\mathbf{z})=\prod_{i}p(z_{i})$ (i.e., diagonality of $\Sigma_{p}$ as before), the remaining expectation values become

which unfortunately must be evaluated numerically; we have denoted them by $f_{j}^{0}$ and $f_{j,k}^{0}$ for compactness. With these in hand, (73) becomes

where the identification in the last line has been introduced for shorthand. Collecting results, we obtain

Now, since all the recursion is contained in the functions $f_{j}^{0}$, $f_{j,k}^{0}$, we can easily write – at least formally – the expression for arbitrary $m$:

and the recursive functions to be computed numerically are

2 Implementation and results

Since our focus in this work is a theoretical analysis of the KL divergence rather than state-of-the-art machine learning performance, we will employ a basic feedforward random network trained on the standard MNIST dataset deng2012mnist as well as CIFAR-10 (converted to greyscale), with no optimizers or other bells and whistles.The code to reproduce all results is available at https://github.com/ro-jefferson/entropy_dnn. The key feature is the random initialization and relatively large width (784 for the $28\times 28$ pixels comprising a given MNIST image, or 1024 for CIFAR-10 after converting to greyscale), for which the bulk layers remain approximately Gaussian.

and so on, where we have refrained from substituting in $\tanh$ in all but the first case for compactness. Note however that from $m\!=\!3$ onwards, there is no new $z$-dependence: we have already summed-over all free indices in the weight matrices that couple to $\mathbf{z}^{0}$ by the time we reach $m\!=\!2$. (This is easy to see visually by sketching out the architecture of the network: in order to compute the activation for a neuron in layer 2, we must sum over all the neurons in layer 1 (one index), and then sum over all the neurons in layer 0 for each of them (two indices)). Indeed, it is computationally very inefficient to re-compute the activations $\phi(\mathbf{z}^{m})$ of a given layer every time we step through the recursive tower. Rather, given sufficient RAM, the following procedure provides a speed-up of about 6 orders of magnitude in our experiments on a humble desktop machine.

Starting at $m\!=\!1$, compute $n_{\textrm{\tiny{MC}}}$ samples of the vector of activations $\phi(\mathbf{z}^{0})=\{\phi(z_{i}^{0})\}$, drawn from the reference layer (see appendix B).

At $m\!=\!2$, multiply each element of this vector by the appropriate weight, and sum over $j$ to obtain $n_{\textrm{\tiny{MC}}}$ samples of $W_{ij}^{1}\phi(z_{j}^{0})$. After adding the bias $b_{i}^{1}$, taking the $\tanh$, and repeating for all $i$, we obtain $n_{\textrm{\tiny{MC}}}$ samples of the vector $\phi(\mathbf{z}^{1})\equiv\{\phi(z_{i}^{1})\}=\{\tanh(W_{ij}^{1}\phi(z_{j}^{0})+b_{i}^{1})\}$.

Iterate the previous step to obtain $n_{\textrm{\tiny{MC}}}$ samples of $\phi(\mathbf{z}^{m-1})$ for $m\geq 3$.

Use the $n_{\textrm{\tiny{MC}}}$ samples of $\phi(\mathbf{z}^{m-1})$ from each layer $m$ to evaluate $f_{j}^{m-1}$, $f_{j,k}^{m-1}$ via Monte Carlo integration (see appendix B).

Substitute these into (87) to obtain the contribution $\langle Q^{m}\rangle$.

The results for a network with 30 layers (so as to lie well-within the trainable regime determined in fig. 6) and five different points in parameter space are shown in fig. 7. Several comments are in order. First, we observe a close parallel with the behavior of the KL divergence in both the 1d and 2d Ising models under real-space RG, cf. figs. 1 and 3. The KL divergence sharply and briefly increases, and then asymptotes to a value which depends on the network’s location in phase space; here, this is controlled by $\sigma_{w}^{2}$ (for fixed $\sigma_{b}^{2}$), while in the Ising models the asymptote is controlled by the coupling strength. In both cases, the quantity of information ceases to evolve once the KL divergence reaches its asymptotic value.

This raises several questions. As long as the network is within the trainable regime in fig. 6, we observed no marked difference in classification accuracy between networks with only a few layers and those lying well into the asymptotic region. While one might be tempted to interpret this as evidence that the additional layers are unnecessary, this may simply be a coincidence: the fact that the total information content no longer changes does not imply that the information is no longer being processed. The KL divergence is a relatively coarse-grained probe, the inputs to which are, in this case, ultimately just the parameters of the Gaussian characterizing each layer. Even in these relatively small networks with around 22,000 neurons, there are many, many ways to rearrange the individual weights (the fine-grained state of the network) without changing the Gaussian (the coarse-grained state). Therefore, it seems that more fine-grained probes will be necessary if one wishes to usefully quantify the information content of a given layer.

A related question is the value of the asymptote itself. At least in the 1d Ising model, we could understand this in terms of the total information in the fine-grained (UV) system, where – absent the normalization factor – the limit of zero coupling leaves us with $n\ln 2$ bits for a system of $n$ spins. For these Gaussian networks, the analogous quantity is the entropy of the initial layer, $\tfrac{1}{2}\ln|2\pi e\Sigma_{p(\mathbf{z}^{0})}|$, which does not correspond to the asymptotic value for any choice of parameters we explored. Furthermore, as discussed in the introduction, there is a structure vs. dynamics distinction that must be kept in mind when drawing parallels between deep neural networks and the renormalization group. As shown in fig. 7, we observe no qualitative difference in the value of the KL asymptote before vs. after training. On the one hand, we shouldn’t be too surprised by this insofar as the change in the KL divergence ultimately corresponds to a Gaussian changing its width. Said differently, RG captures the structural relationship between subsequent layers in the form of effective couplings that arise under coarse-graining, which depend on the weights (see roDLRG for a simple example). During training, the numerical values of the weights will change, but the form of the hierarchical relationship (that is, the formal expression for the effective couplings) will not. Nonetheless, it may be interesting to quantitatively disentangle the contributions to the entropy from the structure vs. the training dynamics; this would however require more exhaustive numerics to elucidate the role of initialization, since this sets the baseline curve against which the trained curve may be compared in fig. 7.

Another notable feature is the apparent insensitivity of the relative entropy to the phase structure of the system. The critical point for the networks in fig. 7 lies at approximately $(\sigma_{b}^{2},\sigma_{w}^{2})=(0.05,1.76)$, which is between the blue and orange curves in the figure. One sees no qualitative difference in the behavior of the KL divergence as a function of depth between the two phases. For MNIST, we observe that the value of the asymptote seems to grow with the width of the weight distribution, but this does not appear to hold for CIFAR-10. In both cases however, one can fit the KL divergence to a function of the form

where $x$ is the depth, and $a,b$ must be determined numerically. The results for the after-training values in the case of MNIST are shown in fig. 8, where the corresponding parameters are given in table 1. While it is tempting to speculate on the $\sigma_{w}$-dependence of the asymptotic value $a$ (e.g., one obtains a decent fit with $a=\alpha\tanh(\beta\sigma_{w}^{2})$ for some constants $\alpha,\beta$), we have not generated sufficient data to do so.

We must also comment on the issue of normalization we encountered in the Ising models, which surfaces again in the present case. Recall that since the dimensionality of the spin chain was reduced at each step, and that it was necessary to account for this to obtain a monotonically increasing result. Similarly, we found that if the width of the network is monotonically reduced at subsequent layers, the relative entropy can decrease, and even become negative! While this seems to contradict the well-known proofs that the KL divergence is non-negative, the loop-hole is the assumption that both distributions are normalized with respect to the same measure. That is, consider the negative of the KL divergence between the one-dimensional distributions $p(x)$ and $q(x)$:

Discussion

In this work, we have taken a step towards quantifying information flow in lattice RG and deep neural networks, taking insight from both sides. Our main contribution is an explicit computation of the relative entropy or Kullback-Leibler divergence for both the 1d and 2d Ising models under decimation RG, as well as a simple feedforward random neural network of arbitrary depth. For the MNIST and CIFAR-10 datasets under study, the behaviour is qualitatively identical in all cases:At least where the expressions are valid; as discussed in section 2, the approximation for the 2d Ising model breaks-down at sufficiently large coupling, resulting in an unphysical, non-monotonic decrease. we observe a relatively steep increase to some asymptotic value that depends on the choice of parameters (couplings and network initializations, respectively). Stronger coupling in the spin system, or broader Gaussians for the weight matrices, results in larger values for the asymptote. For the Ising models, we are able to understand this in terms of the maximum information content in the system in the infinite-temperature limit, in which the spins decouple into $N$ non-interacting degrees of freedom, for an entropy of $N\ln 2$. Reducing the temperature – i.e., turning on interactions – cannot increase the entropy, so this represents the maximum amount of information we can possibly lose under RG.

A related open question in machine learning is in disentangling compression and generalization in deep networks. To that end, one direction for future work is to investigate whether the asymptotic behavior imposes a fundamental bound on generalizability. That is, the process of deep learning – as distinct from the structure of the network – is very much like an RG in the sense that it involves a loss of information. Borrowing physics terminology, the whole point is to remove irrelevant (UV) information, and preserve only the relevant (IR) information necessary to compute the observables (e.g., perform the classification task) at hand. Once the KL divergence reaches its asymptotic value however, the total information content in the system no longer changes under subsequent layers / decimation steps—the RG is essentially completed. Intuitively, once too much information about the training data is removed, one might expect that the network will be unable to generalize; for example, some of the correlations in the training data may be common to the entire category of 5’s or cats. Provided one could reliably disentangle the contributions to the relative entropy from the structure of the network vs. from the data, it would be interesting to explore whether training should be halted before the asymptotic value is reached. See Shen_2019_ICCV ; Gabri__2019 for some entropic work in this vein, as well as xiao2020disentangling and references therein for connections to gradient descent and the neural tangent kernel.

More generally, as mentioned in the introduction, one can define the mutual information (MI) in terms of the KL divergence, and hence the asymptotic behavior may have implications for various algorithms that maximize MI, such as information flow maximization Shen_2019_ICCV , or in bounding the MI in hidden representations in the information bottleneck foggo2020maximum ; tishby2015deep .

While an initial motivation for this work was to explore the notion of criticality through an entropic lens, we found that the KL divergence appears insensitive to the phase structure of both systems, beyond a monotonic increase in the asymptotic value as one moves further and further in the chaotic direction of parameter space. We believe the reason for this is that relative entropy is simply not well adapted to probing the structure of correlations that characterizes these phases. While poole2016exponential ; schoenholz2017deep and related works examined the two-point function of individual neurons, which exhibits the characteristic divergence in the correlation length we expect from physics, the KL divergence is only sensitive to the collective behavior of the entire system/layer. In the Ising models, this amounted to an expectation value of the Hamiltonian, which is determined by the coupling constants $K,K_{0}$; in the neural network, since we worked in the large-$N$ or mean-field limit, each layer was described by a Gaussian, which is entirely characterized by its second cumulant (i.e., for $\mu\!=\!0$, just $\sigma^{2}$). Thus, while the value of the asymptote itself may have some implication for network initialization or the study of lattice RG, the relative entropy does not appear to guide us in identifying the critical point itself. Nonetheless, given the impressive advantages in trainability observed for networks initialized near criticality, it may be of practical interest to better understand the “flow of information” in these systems in more precise, information-theoretic terms.

At a meta level, we present this work as a contribution to the rapidly growing intersection of physics and machine learning, whereby techniques from theoretical physics and information theory have been increasingly applied to the study of deep neural networks. For a small sample of other work on neural networks drawing from quantum and statistical field theory, in addition to those on criticality mentioned above, see for example BahriRev ; Halverson_2021 ; bondesan2021hintons ; Bachtis:2021xoh ; Roberts:2021fes ; GJ2021 and references therein.

Acknowledgments

We thank James Giammona for many stimulating discussions, and for feedback on a draft of this manuscript. We also thank Dimitri Vvedensky for his course notes on the renormalization group. J.E. and K.T.G. acknowledge financial support from the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under Germany’s Excellence Strategy through the Würzburg-Dresden Cluster of Excellence on Complexity and Topology in Quantum Matter ct.qmat (EXC 2147, project id 390858490). K.T.G. also acknowledges the support of the Hallwachs-Röntgen Postdoc Program of ct.qmat.

Appendix A Decimation RG

To make this paper self-contained for the benefit of our machine learning readers, we will here review the real-space decimation RG procedure for the 1d classical Ising model. This is entirely standard and can be found in many textbooks; we shall here follow Pathria . In section A.1, we have also included an explicit proof that this decimation procedure preserves the partition function.

For generality, let us include the external magnetic field $h$ (though this will be set to zero in the main text):

with the periodic identification of boundaries $\sigma_{N+1}=\sigma_{1}$. For reasons which will shortly become apparent, we will also introduce a parameter $K_{0}$, so that the Hamiltonian becomes

where $K_{0}=0$, $K_{1}=\beta J$, and $K_{2}=\beta h$; note that here we have also absorbed the inverse temperature into the couplings for notational convenience, so that the partition function for this model is given by

where $\sum_{\{\sigma_{i}\}}\coloneqq\prod_{i=1}^{N}\sum_{\sigma_{i}=\pm 1}$.

Now, each step of the RG flow corresponds to a decimation procedure in which we sum over half the degrees of freedom. Hence, taking $N$ even for simplicity, we wish to evaluate the sum over $\sigma_{i}$ in (95) for which $i$ is even. To do so, we first re-express the exponential of the Hamiltonian (94) in the following form:

whereupon the sum over all even spins becomes a sum over $\sigma_{2j}=\pm 1$, which yields

where on the second line we have defined $\sigma_{j}^{\prime}=\sigma_{2j-1}\implies\sigma_{j+1}^{\prime}=\sigma_{2j+1}$. The partition function – that is, the sum over these remaining spins – can then be written

The key step is then to express this partition function in the form (95), i.e., we define the renormalized couplings $K_{0}^{\prime}$, $K_{1}^{\prime}$, and $K_{2}^{\prime}$ such that

In order for this to hold, we require that for all possible spin configurations,

Configurations then fall into three equivalence classes: $\sigma_{j}^{\prime}=\sigma_{j+1}^{\prime}=\pm 1$, and $\sigma_{j}^{\prime}=-\sigma_{j+1}=\pm 1$ (these last being equivalent in $Z$). Substituting these into (100) and solving for the renormalized coefficients then leads to the recursion relations

Note that even if $K_{0}$ were set to zero, a non-zero $K_{0}^{\prime}$ would appear as a consequence of the change in normalization. Setting the external magnetic field $h$ (i.e., $K_{2}$) to zero, we obtain the recursion relations (13).

Though the preservation of the partition function follows immediately from the Bayesian marginalization procedure discussed in the introduction, we can also show this explicitly for the real-space decimation prescription in the 1d Ising model. Recall from (15) that the logarithm of the partition function is given by

Let $Z^{(n)}$ be the partition function after $n$ decimations:

Now, plug in the expression for $K_{0}^{(n)}$ in (14):

This is not at all obvious from the recursion relations (14), but it is in fact true and we can prove it using induction. The identity is trivial for $n=0$. To prove the inductive step, we first find the following nontrivial identity:

Note that plugging $n=0$ into (107) gives precisely (105) with $n=1$. The inductive step is then straightforward:

where the second line follows from the inductive hypothesis that (105) hold up to $n$, and the third line follows from (107). Thus, we have proven that $Z$ remains unchanged under decimation.

Appendix B Monte Carlo integrals

where the volume of the integration region is

Given the high-dimensional ($d=784,1024$) nature of the problem at hand, MC methods are inordinately more efficient than standard numerical integration, e.g., on some regular grid. Instead, the basic idea is to sample $n_{\textrm{\tiny{MC}}}$ points $\mathbf{x}_{i}$, ${i\in\{1,\ldots,n_{\textrm{\tiny{MC}}}\}}$ uniformly over $\Omega$. The law of large numbers then ensures that in the $n_{\textrm{\tiny{MC}}}\rightarrow\infty$ limit, the integral will be given by the expectation value of the function $f(\mathbf{x})$ in the sample ensemble, i.e.,

However, this naïve algorithm is problematic in the present case, since the volume of the integration region is infinite, and the average value of the integrand is zero. This issue can be overcome via importance sampling, whereby, rather than drawing samples $\mathbf{x}_{i}$ uniformly, we draw them from a Gaussian $p(\mathbf{x}_{i})$ with the same mean and standard deviation as the Gaussian integral under consideration. That is, our integrands take the form (writing the expressions in 1d for simplicity)

where $g(x_{i})$ is some non-Gaussian factor. If we then draw samples from $p(x_{i})=f(x_{i})/g(x_{i})$, then we have simply

Intuitively, the idea is that we weight each sample by an amount proportional to how often it appears in the ensemble, hence the name.

Similarly, to obtain $f_{j,k}^{0}$ (86), we simply plug in all possible products of $j,k$, taking care not to mix different samples when computing the averages:

The computation of the MC integrals (114), (115) is the same for all higher layers $m>1$, except that we no longer need to perform any sampling: the integrals are always evaluated with respect to the reference layer, so we simply construct each sample of $\phi_{s}(z_{i}^{m})$ from a sample of the previous layer $\phi_{s}(\mathbf{z}^{m-1})$, i.e.,

and feed the collection of samples into the MC integrators to obtain $f_{j}^{m-1}$, $f_{j,k}^{m-1}$.

The code for the deep neural network analysis in section 3 and the computation described in this appendix is available here: https://github.com/ro-jefferson/entropy_dnn.

References