Independent mechanism analysis, a new concept?

Introduction

One of the goals of unsupervised learning is to uncover properties of the data generating process, such as latent structures giving rise to the observed data. Identifiability formalises this desideratum: under suitable assumptions, a model learnt from observations should match the ground truth, up to well-defined ambiguities. Within representation learning, identifiability has been studied mostly in the context of independent component analysis (ICA) , which assumes that the observed data $\mathbf{x}$ results from mixing unobserved independent random variables $s_{i}$ referred to as sources. The aim is to recover the sources based on the observed mixtures alone, also termed blind source separation (BSS). A major obstacle to BSS is that, in the nonlinear case, independent component estimation does not necessarily correspond to recovering the true sources: it is possible to give counterexamples where the observations are transformed into components $y_{i}$ which are independent, yet still mixed with respect to the true sources $s_{i}$ . In other words, nonlinear ICA is not identifiable.

In order to achieve identifiability, a growing body of research postulates additional supervision or structure in the data generating process, often in the form of auxiliary variables . In the present work, we investigate a different route to identifiability by drawing inspiration from the field of causal inference which has provided useful insights for a number of machine learning tasks, including semi-supervised , transfer , reinforcement , and unsupervised learning. To this end, we interpret the ICA mixing as a causal process and apply the principle of independent causal mechanisms (ICM) which postulates that the generative process consists of independent modules which do not share information . In this context, “independent” does not refer to statistical independence of random variables, but rather to the notion that the distributions and functions composing the generative process are chosen independently by Nature . While a formalisation of ICM in terms of algorithmic (Kolmogorov) complexity exists, it is not computable, and hence applying ICM in practice requires assessing such non-statistical independence with suitable domain specific criteria . The goal of our work is thus to constrain the nonlinear ICA problem, in particular the mixing function, via suitable ICM measures, thereby ruling out common counterexamples to identifiability which intuitively violate the ICM principle.

Motivating example. To build intuition, we turn to a famous example of ICA and BSS: the cocktail party problem, illustrated in Fig. 1 (Left). Here, a number of conversations are happening in parallel, and the task is to recover the individual voices $s_{i}$ from the recorded mixtures $x_{i}$. The mixing or recording process $\mathbf{f}$ is primarily determined by the room acoustics and the locations at which microphones are placed. Moreover, each speaker influences the recording through their positioning in the room, and we may think of this influence as $\nicefrac{{\partial\mathbf{f}}}{{\partial s_{i}}}$. Our independence postulate then amounts to stating that the speakers’ positions are not fine-tuned to the room acoustics and microphone placement, or to each other, i.e., the contributions $\nicefrac{{\partial\mathbf{f}}}{{\partial s_{i}}}$ should be independent (in a non-statistical sense).For additional intuition and possible violations in the context of the cocktail party problem, see § B.4.

Our approach. We formalise this notion of independence between the contributions $\nicefrac{{\partial\mathbf{f}}}{{\partial s_{i}}}$ of each source to the mixing process (i.e., the columns of the Jacobian matrix $\mathbf{J}_{\mathbf{f}}$ of partial derivatives) as an orthogonality condition, see Fig. 1 (Right). Specifically, the absolute value of the determinant $|\mathbf{J}_{\mathbf{f}}|$, which describes the local change in infinitesimal volume induced by mixing the sources, should factorise or decompose as the product of the norms of its columns. This can be seen as a decoupling of the local influence of each partial derivative in the pushforward operation (mixing function) mapping the source distribution to the observed one, and gives rise to a novel framework which we term independent mechanism analysis (IMA). IMA can be understood as a refinement of the ICM principle that applies the idea of independence of mechanisms at the level of the mixing function.

Contributions. The structure and contributions of this paper can be summarised as follows:

we review well-known obstacles to identifiability of nonlinear ICA (§ 2.1), as well as existing ICM criteria (§ 2.2), and show that the latter do not sufficiently constrain nonlinear ICA (§ 3);

we propose a more suitable ICM criterion for unsupervised representation learning which gives rise to a new framework that we term independent mechanism analysis (IMA) (§ 4); we provide geometric and information-theoretic interpretations of IMA (§ 4.1), introduce an IMA contrast function which is invariant to the inherent ambiguities of nonlinear ICA (§ 4.2), and show that it rules out a large class of counterexamples and is consistent with existing identifiability results (§ 4.3);

we experimentally validate our theoretical claims and propose a regularised maximum-likelihood learning approach based on the IMA constrast which outperforms the unregularised baseline (§ 5); additionally, we introduce a method to learn nonlinear ICA solutions with triangular Jacobian and a metric to assess BSS which can be of independent interest for the nonlinear ICA community.

Background and preliminaries

Our work builds on and connects related literature from the fields of independent component analysis (§ 2.1) and causal inference (§ 2.2). We review the most important concepts below.

Assume the following data-generating process for independent component analysis (ICA)

If the true model belongs to the model class $\mathcal{M}$, then $\sim$-identifiability ensures that any model in $\mathcal{M}$ learnt from (infinite amounts of) data will be $\sim$-equivalent to the true one. An example is linear ICA which is identifiable up to permutation and rescaling of the sources on the subspace $\mathcal{M}_{\textsc{lin}}$ of pairs of (i) invertible matrices (constraint on $\mathcal{F}$) and (ii) factorizing densities for which at most one $s_{i}$ is Gaussian (constraint on $\mathcal{P}$) , see Appendix A for a more detailed account.

In the nonlinear case (i.e., without constraints on $\mathcal{F}$), identifiability is much more challenging. If $s_{i}$ and $s_{j}$ are independent, then so are $h_{i}(s_{i})$ and $h_{j}(s_{j})$ for any functions $h_{i}$ and $h_{j}$. In addition to permutation-ambiguity, such element-wise $\mathbf{h}(\mathbf{s})=(h_{1}(s_{1}),...,h_{n}(s_{n}))$ can therefore not be resolved either. We thus define the desired form of identifiability for nonlinear BSS as follows.

The equivalence relation $\sim_{\textsc{bss}}$ on $\mathcal{F}\times\mathcal{P}$ defined as in Defn. 2.1 is given by

where $\mathbf{P}$ is a permutation and $\mathbf{h}(\mathbf{s})=(h_{1}(s_{1}),...,h_{n}(s_{n}))$ is an invertible, element-wise function.

A fundamental obstacle—and a crucial difference to the linear problem—is that in the nonlinear case, different mixtures of $s_{i}$ and $s_{j}$ can be independent, i.e., solving ICA is not equivalent to solving BSS. A prominent example of this is given by the Darmois construction .

Let $\mathbf{R}\in O(n)$ be an orthogonal matrix, and denote by $\mathbf{F}_{\mathbf{s}}(\mathbf{s})=(F_{s_{1}}(s_{1}),...,F_{s_{n}}(s_{n}))$ and $\bm{\Phi}(\mathbf{z})=(\Phi(z_{1}),...,\Phi(z_{n}))$ the element-wise CDFs of a smooth, factorised density $p_{\mathbf{s}}$ and of a Gaussian, respectively. Then the “rotated-Gaussian” MPA $\mathbf{a}^{\mathbf{R}}(p_{\mathbf{s}})$ is

2 Causal inference and the principle of independent causal mechanisms (ICM)

Rather than relying only on additional assumptions on $\mathcal{P}$ (e.g., via auxiliary variables), we seek to further constrain (1) by also placing assumptions on the set $\mathcal{F}$ of mixing functions $\mathbf{f}$. To this end, we draw inspiration from the field of causal inference . Of central importance to our approach is the Principle of Independent Causal Mechanisms (ICM) .

The causal generative process of a system’s variables is composed of autonomous modules that do not inform or influence each other.

These “modules” are typically thought of as the conditional distributions of each variable given its direct causes. Intuitively, the principle then states that these causal conditionals correspond to independent mechanisms of nature which do not share information. Crucially, here “independent” does not refer to statistical independence of random variables, but rather to independence of the underlying distributions as algorithmic objects. For a bivariate system comprising a cause $\mathbf{c}$ and an effect $\mathbf{e}$, this idea reduces to an independence of cause and mechanism, see Fig. 2(c). One way to formalise ICM uses Kolmogorov complexity $K(\cdot)$ as a measure of algorithmic information .

However, since Kolmogorov complexity is is not computable, using ICM in practice requires assessing Principle 2.6 with other suitable proxy criteria .“This can be seen as an algorithmic analog of replacing the empirically undecidable question of statistical independence with practical independence tests that are based on assumptions on the underlying distribution” . Allowing for deterministic relations between cause (sources) and effect (observations), the criterion which is most closely related to the ICA setting in (1) is information-geometric causal inference (IGCI) .For a similar criterion which assumes linearity and its relation to linear ICA, see § B.1. IGCI assumes a nonlinear relation $\mathbf{e}=\mathbf{f}(\mathbf{c})$ and formulates a notion of independence between the cause distribution $p_{\mathbf{c}}$ and the deterministic mechanism $\mathbf{f}$ (which we think of as a degenerate conditional $p_{\mathbf{e}|\mathbf{c}}$) via the following condition (in practice, assumed to hold approximately),

where $(\mathbf{J}_{\mathbf{f}}(\mathbf{c}))_{ij}=\nicefrac{{\partial f_{i}}}{{\partial c_{j}}}(\mathbf{c})$ is the Jacobian matrix and $|\cdot|$ the absolute value of the determinant. $C_{\textsc{igci}}$ can be understood as the covariance between $p_{\mathbf{c}}$ and $\log\left|\mathbf{J}_{\mathbf{f}}\right|$ (viewed as r.v.s on the unit cube w.r.t. the Lebesgue measure), so that $C_{\textsc{igci}}=0$ rules out a form of fine-tuning between $p_{\mathbf{c}}$ and $|\mathbf{J}_{\mathbf{f}}|$. As its name suggests, IGCI can, from an information-geometric perspective, also be seen as an orthogonality condition between cause and mechanism in the space of probability distributions , see § B.2, particularly eq. 19 for further details.

Existing ICM measures are insufficient for nonlinear ICA

Our aim is to use the ICM Principle 2.6 to further constrain the space of models $\mathcal{M}\subseteq\mathcal{F}\times\mathcal{P}$ and rule out common counterexamples to identifiability such as those presented in § 2.1. Intuitively, both the Darmois construction (4) and the rotated Gaussian MPA (5) give rise to “non-generic” solutions which should violate ICM: the former, $(\mathbf{f}^{\text{D}},p_{\mathbf{u}})$, due the triangular Jacobian of $\mathbf{f}^{\text{D}}$ (see Remark 2.4), meaning that each observation $x_{i}=f^{\text{D}}_{i}(\mathbf{y}_{1:i})$ only depends on a subset of the inferred independent components $\mathbf{y}_{1:i}$, and the latter, $(\mathbf{f}\circ\mathbf{a}^{\mathbf{R}}(p_{\mathbf{s}}),p_{\mathbf{s}})$, due to the dependence of $\mathbf{f}\circ\mathbf{a}^{\mathbf{R}}(p_{\mathbf{s}})$ on $p_{\mathbf{s}}$ (5).

However, the ICM criteria described in § 2.2 were developed for the task of cause-effect inference where both variables are observed. In contrast, in this work, we consider an unsupervised representation learning task where only the effects (mixtures $\mathbf{x}$) are observed, but the causes (sources $\mathbf{s}$) are not. It turns out that this renders existing ICM criteria insufficient for BSS: they can easily be satisfied by spurious solutions which are not equivalent to the true one. We can show this for IGCI. Denote by $\mathcal{M}_{\textsc{igci}}=\{(\mathbf{f},p_{\mathbf{s}})\in\mathcal{F}\times\mathcal{P}:C_{\textsc{igci}}(\mathbf{f},p_{\mathbf{s}})=0\}\subset\mathcal{F}\times\mathcal{P}$ the class of nonlinear ICA models satisfying IGCI (6). Then the following negative result holds.

(1) is not $\sim_{\textsc{bss}}$-identifiable on $\mathcal{M}_{\textsc{igci}}$.

IGCI (6) is satisfied when $p_{\mathbf{s}}$ is uniform. However, the Darmois construction (4) yields uniform sources, see Fig. 2(a). This means that $(\mathbf{f}^{\text{D}}\circ\mathbf{a}^{\mathbf{R}}(p_{\mathbf{u}}),p_{\mathbf{u}})\in\mathcal{M}_{\textsc{igci}}$, so IGCI can be satisfied by solutions which do not separate the sources in the sense of Defn. 2.2, see footnote 2 and .∎

As illustrated in Fig. 2(c), condition (6) and other similar criteria enforce a notion of “genericity” or “decoupling” of the mechanism w.r.t. the observed input distribution.In fact, many ICM criteria can be phrased as special cases of a unifying group-invariance framework . They thus rely on the fact that the cause (source) distribution is informative, and are generally not invariant to reparametrisation of the cause variables. In the (nonlinear) ICA setting, on the other hand, the learnt source distribution may be fairly uninformative. This poses a challenge for existing ICM criteria since any mechanism is generic w.r.t. an uninformative (uniform) input distribution.

Independent mechanism analysis (IMA)

As argued in § 3, enforcing independence between the input distribution and the mechanism (Fig. 2(c)), as existing ICM criteria do, is insufficient for ruling out spurious solutions to nonlinear ICA. We therefore propose a new ICM-inspired framework which is more suitable for BSS and which we term independent mechanism analysis (IMA).The title of the present work is thus a reverence to Pierre Comon’s seminal 1994 paper . All proofs are provided in Appendix C.

As motivated using the cocktail party example in § 1 and Fig. 1 (Left), our main idea is to enforce a notion of independence between the contributions or influences of the different sources $s_{i}$ on the observations $\mathbf{x}=\mathbf{f}(\mathbf{s})$ as illustrated in Fig. 2(d)—as opposed to between the source distribution and mixing function, cf. Fig. 2(c). These contributions or influences are captured by the vectors of partial derivatives $\nicefrac{{\partial\mathbf{f}}}{{\partial s_{i}}}$. IMA can thus be understood as a refinement of ICM at the level of the mixing $\mathbf{f}$: in addition to statistically independent components $s_{i}$, we look for a mixing with contributions $\nicefrac{{\partial\mathbf{f}}}{{\partial s_{i}}}$ which are independent, in a non-statistical sense which we formalise as follows.

The mechanisms by which each source $s_{i}$ influences the observed distribution, as captured by the partial derivatives $\nicefrac{{\partial\mathbf{f}}}{{\partial s_{i}}}$, are independent of each other in the sense that for all $\mathbf{s}$:

Geometric interpretation. Geometrically, the IMA principle can be understood as an orthogonality condition, as illustrated for $n=2$ in Fig. 1 (Right). First, the vectors of partial derivatives $\nicefrac{{\partial\mathbf{f}}}{{\partial s_{i}}}$, for which the IMA principle postulates independence, are the columns of $\mathbf{J}_{\mathbf{f}}$. $|\mathbf{J}_{\mathbf{f}}|$ thus measures the volume of the $n-$dimensional parallelepiped spanned by these columns, as shown on the right. The product of their norms, on the other hand, corresponds to the volume of an $n$-dimensional box, or rectangular parallelepiped with side lengths $\lVert\nicefrac{{\partial\mathbf{f}}}{{\partial s_{i}}}\rVert$, as shown on the left. The two volumes are equal if and only if all columns $\nicefrac{{\partial\mathbf{f}}}{{\partial s_{i}}}$ of $\mathbf{J}_{\mathbf{f}}$ are orthogonal. Note that (7) is trivially satisfied for $n=1$, i.e., if there is no mixing, further highlighting its difference from ICM for causal discovery.

Information-geometric interpretation and comparison to IGCI. The additive contribution of the sources’ influences $\nicefrac{{\partial\mathbf{f}}}{{\partial s_{i}}}$ in (7) suggests their local decoupling at the level of the mechanism $\mathbf{f}$. Note that IGCI (6), on the other hand, postulates a different type of decoupling: one between $\log|\mathbf{J}_{\mathbf{f}}|$ and $p_{\mathbf{s}}$. There, dependence between cause and mechanism can be conceived as a fine tuning between the derivative of the mechanism and the input density. The IMA principle leads to a complementary, non-statistical measure of independence between the influences $\nicefrac{{\partial\mathbf{f}}}{{\partial s_{i}}}$ of the individual sources on the vector of observations. Both the IGCI and IMA postulates have an information-geometric interpretation related to the influence of (“non-statistically”) independent modules on the observations: both lead to an additive decomposition of a KL-divergence between the effect distribution and a reference distribution. For IGCI, independent modules correspond to the cause distribution and the mechanism mapping the cause to the effect (see (19) in § B.2). For IMA, on the other hand, these are the influences of each source component on the observations in an interventional setting (under soft interventions on individual sources), as measured by the KL-divergences between the original and intervened distributions. See § B.3, and especially (22), for a more detailed account.

We finally remark that while recent work based on the ICM principle has mostly used the term “mechanism” to refer to causal Markov kernels $p(X_{i}|PA_{i})$ or structural equations , we employ it in line with the broader use of this concept in the philosophical literature.See Table 1 in for a long list of definitions from the literature. To highlight just two examples, states that “Causal processes, causal interactions, and causal laws provide the mechanisms by which the world works; to understand why certain things happen, we need to see how they are produced by these mechanisms”; and states that “Mechanisms are events that alter relations among some specified set of elements”. Following this perspective, we argue that a causal mechanism can more generally denote any process that describes the way in which causes influence their effects: the partial derivative $\nicefrac{{\partial\mathbf{f}}}{{\partial s_{i}}}$ thus reflects a causal mechanism in the sense that it describes the infinitesimal changes in the observations $\mathbf{x}$, when an infinitesimal perturbation is applied to $s_{i}$.

2 Definition and useful properties of the IMA contrast

We now introduce a contrast function based on the IMA principle (7) and show that it possesses several desirable properties in the context of nonlinear ICA. First, we define a local contrast as the difference between the two integrands of (7) for a particular value of the sources $\mathbf{s}$.

The local IMA contrast $c_{\textsc{ima}}(\mathbf{f},\mathbf{s})$ of $\mathbf{f}$ at a point $\mathbf{s}$ is given by

This corresponds to the left KL measure of diagonality for $\sqrt{\mathbf{J}_{\mathbf{f}}(\mathbf{s})^{\top}\mathbf{J}_{\mathbf{f}}(\mathbf{s})}$.

The local IMA contrast $c_{\textsc{ima}}(\mathbf{f},\mathbf{s})$ quantifies the extent to which the IMA principle is violated at a given point $\mathbf{s}$. We summarise some of its properties in the following proposition.

The local IMA contrast $c_{\textsc{ima}}(\mathbf{f},\mathbf{s})$ defined in (8) satisfies:

$c_{\textsc{ima}}(\mathbf{f},\mathbf{s})\geq 0$, with equality if and only if all columns $\nicefrac{{\partial\mathbf{f}}}{{\partial s_{i}}}(\mathbf{s})$ of $\mathbf{J}_{\mathbf{f}}(\mathbf{s})$ are orthogonal.

$c_{\textsc{ima}}(\mathbf{f},\mathbf{s})$ is invariant to left multiplication of $\mathbf{J}_{\mathbf{f}}(\mathbf{s})$ by an orthogonal matrix and to right multiplication by permutation and diagonal matrices.

Property (i) formalises the geometric interpretation of IMA as an orthogonality condition on the columns of the Jacobian from § 4.1, and property (ii) intuitively states that changes of orthonormal basis and permutations or rescalings of the columns of $\mathbf{J}_{\mathbf{f}}$ do not affect their orthogonality. Next, we define a global IMA contrast w.r.t. a source distribution $p_{\mathbf{s}}$ as the expected local IMA contrast.

The global IMA contrast $C_{\textsc{ima}}(\mathbf{f},p_{\mathbf{s}})$ of $\mathbf{f}$ w.r.t. $p_{\mathbf{s}}$ is given by

The global IMA contrast $C_{\textsc{ima}}(\mathbf{f},p_{\mathbf{s}})$ thus quantifies the extent to which the IMA principle is violated for a particular solution $(\mathbf{f},p_{\mathbf{s}})$ to the nonlinear ICA problem. We summarise its properties as follows.

The global IMA contrast $C_{\textsc{ima}}(\mathbf{f},p_{\mathbf{s}})$ from (9) satisfies:

Property (i) is the distribution-level analogue to (i) of Prop. 4.4 and only allows for orthogonality violations on sets of measure zero w.r.t. $p_{\mathbf{s}}$. This means that $C_{\textsc{ima}}$ can only be zero if $\mathbf{f}$ is an orthogonal coordinate transformation almost everywhere , see Fig. 3 for an example. We particularly stress property (ii), as it precisely matches the inherent indeterminacy of nonlinear ICA: $C_{\textsc{ima}}$ is blind to reparametrisation of the sources by permutation and element wise transformation.

We now show that, under suitable assumptions on the generative model (1), a large class of spurious solutions—such as those based on the Darmois construction (4) or measure preserving automorphisms such as $\mathbf{a}^{\mathbf{R}}$ from (5) as described in § 2.1—exhibit nonzero IMA contrast. Denote the class of nonlinear ICA models satisfying (7) (IMA) by ${\mathcal{M}_{\textsc{ima}}=\{(\mathbf{f},p_{\mathbf{s}})\in\mathcal{F}\times\mathcal{P}:C_{\textsc{ima}}(\mathbf{f},p_{\mathbf{s}})=0\}\subset\mathcal{F}\times\mathcal{P}}$. Our first main theoretical result is that, under mild assumptions on the observations, Darmois solutions will have strictly positive $C_{\textsc{ima}}$, making them distinguishable from those in $\mathcal{M}_{\textsc{ima}}$.

Assume the data generating process in (1) and assume that $x_{i}\not\!\perp\!\!\!\perp x_{j}$ for some $i\neq j$. Then any Darmois solution $(\mathbf{f}^{\text{D}},p_{\mathbf{u}})$ based on $\mathbf{g}^{\text{D}}$ as defined in (4) satisfies $C_{\textsc{ima}}(\mathbf{f}^{\text{D}},p_{\mathbf{u}})>0$. Thus a solution satisfying $C_{\textsc{ima}}(\mathbf{f},p_{\mathbf{s}})=0$ can be distinguished from $(\mathbf{f}^{\text{D}},p_{\mathbf{u}})$ based on the contrast $C_{\textsc{ima}}$.

The proof is based on the fact that the Jacobian of $\mathbf{g}^{\text{D}}$ is triangular (see Remark 2.4) and on the specific form of (4). A specific example of a mixing process satisfying the IMA assumption is the case where $\mathbf{f}$ is a conformal (angle-preserving) map.

Under assumptions of Thm. 4.7, if additionally $\mathbf{f}$ is a conformal map, then $(\mathbf{f},p_{\mathbf{s}})\in\mathcal{M}_{\textsc{ima}}$ for any $p_{\mathbf{s}}\in\mathcal{P}$ due to Prop. 4.6 (i), see Defn. 4.8. Based on Thm. 4.7, $(\mathbf{f},p_{\mathbf{s}})$ is thus distinguishable from Darmois solutions $(\mathbf{f}^{\text{D}},p_{\mathbf{u}})$.

This is consistent with a result that proves identifiability of conformal maps for $n=2$ and conjectures it in general .Note that Corollary 4.9 holds for any dimensionality $n$. However, conformal maps are only a small subset of all maps for which $C_{\textsc{ima}}=0$, as is apparent from the more flexible condition of Prop. 4.6 (i), compared to the stricter Defn. 4.8.

Consider the non-conformal transformation from polar to Cartesian coordinates (see Fig. 3), defined as $(x,y)=\mathbf{f}(r,\theta):=(r\cos(\theta),r\sin(\theta))$ with independent sources $\mathbf{s}=(r,\theta)$, with $r\sim U(0,R)$ and $\theta\sim U(0,2\pi)$.For different $p_{\mathbf{s}}$, $(x,y)$ can be made to have independent Gaussian components (, II.B), and $C_{\textsc{ima}}$-identifiability is lost; this shows that the assumption of Thm. 4.7 that $x_{i}\not\!\perp\!\!\!\perp x_{j}$ for some $i\neq j$ is crucial. Then, $C_{\textsc{ima}}(\mathbf{f},p_{\mathbf{s}})=0$ and $C_{\textsc{ima}}(\mathbf{f}^{\text{D}},p_{\mathbf{u}})>0$ for any Darmois solution $(\mathbf{f}^{\text{D}},p_{\mathbf{u}})$ —see Appendix D for details.

Finally, for the case in which the true mixing is linear, we obtain the following result.

In a “blind” setting, we may not know a priori whether the true mixing is linear or not, and thus choose to learn a nonlinear unmixing. Corollary 4.11 shows that, in this case, Darmois solutions are still distinguishable from the true mixing via $C_{\textsc{ima}}$. Note that unlike in Corollary 4.9, the assumption that $x_{i}\not\!\perp\!\!\!\perp x_{j}$ for some $i\neq j$ is not required for Corollary 4.11. In fact, due to Theorem 11 of , it follows from the assumed linear ICA model with non-Gaussian sources, and the fact that the mixing matrix is not the product of a diagonal and a permutation matrix (see also Appendix A).

Having shown that the IMA principle allows to distinguish a class of models (including, but not limited to conformal maps) from Darmois solutions, we next turn to a second well-known counterexample to identifiability: the “rotated-Gaussian” MPA $\mathbf{a}^{\mathbf{R}}(p_{\mathbf{s}})$ (5) from Defn. 2.5. Our second main theoretical result is that, under suitable assumptions, this class of MPAs can also be ruled out for “non-trivial” $\mathbf{R}$.

Let $(\mathbf{f},p_{\mathbf{s}})\in\mathcal{M}_{\textsc{ima}}$ and assume that $\mathbf{f}$ is a conformal map. Given $\mathbf{R}\in O(n)$, assume additionally that $\exists$ at least one non-Gaussian $s_{i}$ whose associated canonical basis vector $\mathbf{e}_{i}$ is not transformed by $\mathbf{R}^{-1}=\mathbf{R}^{\top}$ into another canonical basis vector $\mathbf{e}_{j}$. Then $C_{\textsc{ima}}(\mathbf{f}\circ\mathbf{a}^{\mathbf{R}}(p_{\mathbf{s}}),p_{\mathbf{s}})>0$.

Thm. 4.12 states that for conformal maps, applying the $\mathbf{a}^{\mathbf{R}}(p_{\mathbf{s}})$ transformation at the level of the sources leads to an increase in $C_{\textsc{ima}}$ except for very specific rotations $\mathbf{R}$ that are “fine-tuned” to $p_{\mathbf{s}}$ in the sense that they permute all non-Gaussian sources $s_{i}$ with another $s_{j}$. Interestingly, as for the linear case, non-Gaussianity again plays an important role in the proof of Thm. 4.12.

Experiments

Our theoretical results from § 4 suggest that $C_{\textsc{ima}}$ is a promising contrast function for nonlinear blind source separation. We test this empirically by evaluating the $C_{\textsc{ima}}$ of spurious nonlinear ICA solutions (§ 5.1), and using it as a learning objective to recover the true solution (§ 5.2).

We sample the ground truth sources from a uniform distribution in $^{n}$; the reconstructed sources are also mapped to the uniform hypercube as a reference measure via the CDF transform. Unless otherwise specified, the ground truth mixing $\mathbf{f}$ is a Möbius transformation (i.e., a conformal map) with randomly sampled parameters, thereby satisfying Principle 4.1. In all of our experiments, we use JAX and Distrax . For additional technical details, equations and plots see Appendix E. The code to reproduce our experiments is available at this link.

Learning the Darmois construction. To learn the Darmois construction from data, we use normalising flows, see . Since Darmois solutions have triangular Jacobian (Remark 2.4), we use an architecture based on residual flows which we constrain such that the Jacobian of the full model is triangular. This yields an expressive model which we train effectively via maximum likelihood.

$C_{\textsc{ima}}$ of Darmois solutions. To check whether Darmois solutions (learnt from finite data) can be distinguished from the true one, as predicted by Thm. 4.7, we generate $1000$ random mixing functions for $n=2$, compute the $C_{\textsc{ima}}$ values of learnt solutions, and find that all values are indeed significantly larger than zero, see Fig. 4 (a). The same holds for higher dimensions, see Fig. 4 (b) for results with $50$ random mixings for $n\in\{2,3,5,10\}$: with higher dimensionality, both the mean and variance of the $C_{\textsc{ima}}$ distribution for the learnt Darmois solutions generally attain higher values.the latter possibly due to the increased difficulty of the learning task for larger $n$ We confirmed these findings for mappings which are not conformal, while still satisfying (7), in § E.5.

$C_{\textsc{ima}}$ of MPAs. We also investigate the effect on $C_{\textsc{ima}}$ of applying an MPA $\mathbf{a}^{\mathbf{R}}(\cdot)$ from (5) to the true solution or a learnt Darmois solution. Results for $n=2$ dim. for different rotation matrices $\mathbf{R}$ (parametrised by the angle $\theta$) are shown in Fig. 4 (c). As expected, the behavior is periodic in $\theta$, and vanishes for the true solution (blue) at multiples of $\nicefrac{{\pi}}{{2}}$, i.e., when $\mathbf{R}$ is a permutation matrix, as predicted by Thm. 4.12. For the learnt Darmois solution (red, dashed) $C_{\textsc{ima}}$ remains larger than zero.

$C_{\textsc{ima}}$ values for random MLPs. Lastly, we study the behavior of spurious solutions based on the Darmois construction under deviations from our assumption of $C_{\textsc{ima}}=0$ for the true mixing function. To this end, we use invertible MLPs with orthogonal weight initalisation and leaky_tanh activations as mixing functions; the more layers $L$ are added to the mixing MLP, the larger a deviation from our assumptions is expected. We compare the true mixing and learnt Darmois solutions over $20$ realisations for each $L\in\{2,3,4\}$, $n=5$. Results are shown in figure Fig. 4 (d): the $C_{\textsc{ima}}$ of the mixing MLPs grows with $L$; still, the one of the Darmois solution is typically higher.

Summary. We verify that spurious solutions can be distinguished from the true one based on $C_{\textsc{ima}}$.

Results. In Fig. 4 (Top), we show an example of the distortion induced by different spurious solutions for $n=2$, and contrast it with a solution learnt using our proposed objective (rightmost plot). Visually, we find that the $C_{\textsc{ima}}$-regularised solution (with $\lambda=1$) recovers the true sources most faithfully. Quantitative results for 50 learnt models for each $\lambda\in\{0.0,0.5,1.0\}$ and $n\in\{5,7\}$ are summarised in Fig. 5 (see Appendix E for additional plots) . As indicated by the KL divergence values (left), most trained models achieve a good fit to the data across all values of $\lambda$.models with $n=7$ have high outlier KL values, seemingly less pronounced for nonzero values of $\lambda$ We observe that using $C_{\textsc{ima}}$ (i.e., $\lambda>0$) is beneficial for BSS, both in terms of our nonlinear Amari distance (center, lower is better) and MCC (right, higher is better), though we do not observe a substantial difference between $\lambda=0.5$ and $\lambda=1$.In § E.5, we also show that our method is superior to a linear ICA baseline, FastICA .

Summary: $C_{\textsc{ima}}$ can be a useful learning signal to recover the true solution.

Discussion

Assumptions on the mixing function. Instead of relying on weak supervision in the form of auxiliary variables , our IMA approach places additional constraints on the functional form of the mixing process. In a similar vein, the minimal nonlinear distortion principle proposes to favor solutions that are as close to linear as possible. Another example is the post-nonlinear model , which assumes an element-wise nonlinearity applied after a linear mixing. IMA is different in that it still allows for strongly nonlinear mixings (see, e.g., Fig. 3) provided that the columns of their Jacobians are (close to) orthogonal. In the related field of disentanglement , a line of work that focuses on image generation with adversarial networks similarly proposes to constrain the “generator” function via regularisation of its Jacobian or Hessian , though mostly from an empirically-driven, rather than from an identifiability perspective as in the present work.

Towards identifiability with $C_{\textsc{ima}}$. The IMA principle rules out a large class of spurious solutions to nonlinear ICA. While we do not present a full identifiability result, our experiments show that $C_{\textsc{ima}}$ can be used to recover the BSS equivalence class, suggesting that identifiability might indeed hold, possibly under additional assumptions—e.g., for conformal maps .

IMA and independence of cause and mechanism. While inspired by measures of independence of cause and mechanism as traditionally used for cause-effect inference , we view the IMA principle as addressing a different question, in the sense that they evaluate independence between different elements of the causal model. Any nonlinear ICA solution that satisfies the IMA Principle 4.1 can be turned into one with uniform reconstructed sources—thus satisfying IGCI as argued in § 3—through composition with an element-wise transformation which, according to Prop. 4.6 (ii), leaves the $C_{\textsc{ima}}$ value unchanged. Both IGCI (6) and IMA (7) can therefore be fulfilled simultaneosly, while the former on its own is inconsequential for BSS as shown in Prop. 3.1.

BSS through algorithmic information. Algorithmic information theory has previously been proposed as a unifying framework for identifiable approaches to linear BSS , in the sense that commonly-used contrast functions could, under suitable assumptions, be interpreted as proxies for the total complexity of the mixing and the reconstructed sources. However, to the best of our knowledge, the problem of specifying suitable proxies for the complexity of nonlinear mixing functions has not yet been addressed. We conjecture that our framework could be linked to this view, based on the additional assumption of algorithmic independence of causal mechanisms , thus potentially representing an approach to nonlinear BSS by minimisation of algorithmic complexity.

ICA for causal inference & causality for ICA. Past advances in ICA have inspired novel causal discovery methods . The present work constitutes, to the best of our knowledge, the first effort to use ideas from causality (specifically ICM) for BSS. An application of the IMA principle to causal discovery or causal representation learning is an interesting direction for future work.

Conclusion. We introduce IMA, a path to nonlinear BSS inspired by concepts from causality. We postulate that the influences of different sources on the observed distribution should be approximately independent, and formalise this as an orthogonality condition on the columns of the Jacobian. We prove that this constraint is generally violated by well-known spurious nonlinear ICA solutions, and propose a regularised maximum likelihood approach which we empirically demonstrate to be effective in recovering the true solution. Our IMA principle holds exactly for orthogonal coordinate transformations, and is thus of potential interest for learning spatial representations , robot dynamics , or physics problems where orthogonal reference frames are common .

Acknowledgements

The authors thank Aapo Hyvärinen, Adrián Javaloy Bornás, Dominik Janzing, Giambattista Parascandolo, Giancarlo Fissore, Nasim Rahaman, Patrick Burauel, Patrik Reizinger, Paul Rubenstein, Shubhangi Ghosh, and the anonymous reviewers for helpful comments and discussions.

Funding Transparency Statement

This work was supported by the German Federal Ministry of Education and Research (BMBF): Tübingen AI Center, FKZ: 01IS18039B; and by the Machine Learning Cluster of Excellence, EXC number 2064/1 - Project number 390727645.

References

Checklist

Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope? [Yes]

Did you describe the limitations of your work? [Yes] See § 6, where we discuss limitations of our theory (e.g. lines 354-357) and open questions.

Did you discuss any potential negative societal impacts of your work? [N/A] Our work is mainly theoretical, and we believe it does not bear immediate negative societal impacts.

Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]

If you are including theoretical results…

Did you state the full set of assumptions of all theoretical results? [Yes] We formally define the problem setting in § 2 and § 3, and transparently state and discuss our assumptions in § 4.

Did you include complete proofs of all theoretical results? [Yes] Due to space constraints, full proofs and detailed explanations are mainly reported in appendix C and appendix D; the proof of Prop. 3.1 is given in the main text.

Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)? [Yes] The code, data, and the configuration files are included in the supplemental material.

Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] All details, including hyperparameters, seed for random number generators, etc. are specified in the configuration files which will be included in the supplemental.

Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)? [Yes] We visualized the distribution of the considered quantities via histograms and violin plots, see e.g. Figure 4 and Figure 5.

Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] They are specified appendix E.

If you are using existing assets (e.g., code, data, models) or curating/releasing new assets…

If your work uses existing assets, did you cite the creators? [Yes] We use the Python libraries JAX and Distrax and cited the creators in the article.

Did you mention the license of the assets? [Yes] Both packages have Apache License 2.0; we report this in the appendices.

Did you include any new assets either in the supplemental material or as a URL? [Yes] Implementations of our proposed methods and metrics will be provided in the supplemental.

Did you discuss whether and how consent was obtained from people whose data you’re using/curating? [N/A]

Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]

If you used crowdsourcing or conducted research with human subjects…

Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]

Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]

Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]

Overview

Appendix A contains further elaboration on the notion of identifiability as used in the present work, as well as connections to linear ICA.

Appendix B contains additional discussion of existing ICM criteria and their relation to IMA.

Appendix C presents the full proofs for all theoretical results from the main paper.

Appendix D contains a worked out computation of the value of $C_{\textsc{ima}}$ for the mapping from radial to Cartesian coordinates.

Appendix E contains experimental details and additional results.

Appendix F contains additional background on conformal maps and Möbius transformations

Appendix A Additional background on identifiability and linear ICA

In this Appendix, we provide additional background on the notion of identifiability and illustrate it using the example of linear ICA.

Traditionally, identifiability for a class of models $p_{\theta}$ for observed data $\mathbf{x}$ parametrised by $\theta\in\Theta$ is expressed as the condition that there needs to be a one-to-one mapping between the space of models and the space of parameters, i.e., the model class $p_{\theta}$ is said to be identifiable if

However, the equality on the RHS of (10) is a very strong condition which makes this type of (strong or unique) identifiability impractical for many settings. For example, in the case of (linear or nonlinear) ICA, the ordering of the sources cannot be determined, so strong identifiability in the sense of (10) is infeasible.

The equality in parameter space on the RHS of the implication in (10) is therefore sometimes replaced by an equivalence relation $\sim$ , as is also the case for our Defn. 2.1. An equivalence relation $\sim$ on a set $A$ is a binary relation between pairs of elements of $A$ which satisfies the following three properties:

Reflexivity: $a\sim a$, $\forall a\in A$.

Symmetry: $a\sim b\implies b\sim a$, $\forall a,b\in A$.

Transitivity: $(a\sim b)\land(b\sim c)\implies a\sim c$.

An equivalence relation on a set $A$ imposes a partition into disjoint subsets. Each such subset corresponds to an equivalence class, i.e., the collection of all elements which are $\sim$-related to each other; for example, $[a]=\{b\in A:a\sim b\}$ denotes the equivalence class containing the element $a$.

A trivial example of an equivalence relation is equality ($=$). More useful examples in the context of ICA are equivalence up to permutation, rescaling, or scalar transformation.

Defining an appropriate equivalence class for the problem at hand therefore allows us to specify exactly the type of indeterminancies which cannot be resolved and up to which the true generative process can be recovered. As argued in § 2, for nonlinear ICA, the desired notion of identifiability—in the sense of the strongest feasible type of identifiable that is possible without further (parametric) assumptions—is captured by $\sim_{\textsc{bss}}$ from Defn. 2.2. We give another example for linear ICA in § A.2.

Since the generative process of nonlinear ICA (1) is determined by the choice of mixing function and source distribution, the space $\Theta$ from (10), in this case, corresponds to the product space of the space of mixing functions $\mathcal{F}$ and source distributions $\mathcal{P}$. Moreover, the pushforward density $\mathbf{f}_{*}p_{\mathbf{s}}$ in Defn. 2.1 corresponds to the density of the observed mixtures $p_{\mathbf{x}}$, or $p_{\theta}(\mathbf{x})$ in (10).

We deliberately choose to define identifiability and to express the observed distribution in terms of the source distribution and the mixing function—as opposed to in terms of the observed distribution and the unmixing function as in some prior work —because this is aligned with the causal direction of data generation, and thus more consistent with the causal perspective at nonlinear BSS taken in the present work. We also believe that, in this framework, separate constraints on the space of mixing functions $\mathcal{F}$ and source distributions $\mathcal{P}$ are expressed more naturally.

Next, we illustrate the above ideas for the well-studied case of linear ICA.

A.2 Identifiability of linear ICA

Linear ICA corresponds to the setting in which a linear mixing is applied to independent sources, i.e.,

Additionally, we can assume w.l.o.g. that the mixing matrix is orthogonal ($\mathbf{A}\mathbf{A}^{\top}=\mathbf{I}$), because we can always whiten $\mathbf{x}$ first through an invertible linear transformation and obtain an orthogonal mixing , as explained in more detail in § A.3.

Now suppose that the reconstructed sources

Thm. A.1 shows that the two ambiguities deemed unresolvable (scale and ordering of the sources) are, in fact, the only ambiguities, as long as at most one of the $s_{i}$ is Gaussian. That is, linear ICA is identifiable up to rescaling and permutation of the sources, i.e., linearly transforming the observations $\mathbf{x}$ into independent components is equivalent to separating the sources.

More formally, in terms of an equivalence relation, if we take $\mathcal{F}^{\prime}$ from (2) as the space of invertible $n\times n$ matrices and $\mathcal{P}^{\prime}$ as the space of source distributions with at most one Gaussian marginal, then linear ICA is $\sim_{\textsc{lin}}$-identifiable on $\mathcal{F}^{\prime}\times\mathcal{P}^{\prime}$ where the equivalence relation $\sim_{\textsc{lin}}$ on $\mathcal{F}^{\prime}$ is defined as

Two other deviations from a Gaussian i.i.d. setting lead to identifiability: nonstationarity and time correlation . A general information-geometric framework links these three different routes to identifiability .

A.3 Whitening in the context of linear ICA

For completeness, we give a brief account of the role of whitening in linear ICA, which was mentioned in A.2 and which again plays a role in B.1. The following exposition is partly based on , §7.4.2.

A zero-mean random vector, say $\mathbf{y}$, is said to be white if its components are uncorrelated and their variances equal unity. In other words, the covariance matrix of $\mathbf{y}$ is equal to the identity matrix:

It is always possible to whiten a zero-mean random vector $\mathbf{x}$ through a linear operation,

As an example, a popular method for whitening uses the eigenvalue decomposition (EVD) of the covariance matrix,

Whitening is only half ICA. Assume a linear ICA model,

Note that whitening does not solve linear ICA, since uncorrelatedness is weaker than independence. To see this, consider any orthogonal transformation $\mathbf{U}$ of $\mathbf{z}$:

Due to the orthogonality of $\mathbf{U},$ we have

so, $\mathbf{y}$ is white as well. Thus, we cannot tell if the independent components are given by $\mathbf{z}$ or $\mathbf{y}$ using the whiteness property alone. Since $\mathbf{y}$ could be any orthogonal transformation of $\mathbf{z},$ whitening gives the independent components only up to an orthogonal transformation.

Whitening thus “solves half of the problem of ICA”. Because whitening is a very simple and standard procedure—much simpler than any ICA algorithm—it is a good idea to reduce the complexity of the problem this way. The remaining half of the parameters has to be estimated by some other method.

Appendix B Existing ICM criteria and their relationship to ICA and IMA

We now provide additional discussion of the ICM principle and its connection to ICA and IMA. First, we introduce a linear ICM criterion and discuss its relation with linear ICA in § B.1.

As mentioned in § 2.2, besides IGCI, another existing ICM criterion that is closely related to ICA due to also assuming a deterministic relation between cause $\mathbf{c}$ and effect $\mathbf{e}$ is the trace method . The trace method assumes a linear relationship,

and formulates ICM as an “independence” between the covariance matrix $\bm{\Sigma}$ of $\mathbf{c}$ and the mechanism $\mathbf{A}$ (which, as for IGCI, we can again think of as a degenerate conditional $p_{\mathbf{e}|\mathbf{c}}$) via the condition

where $\tau(\cdot)$ denotes the renormalized trace. Intuitively, this condition (17) rules out a fine-tuning of $\mathbf{A}$ to the eigenvectors of $\bm{\Sigma}$ which would violate the assumption of no shared information between the cause distribution (specifically, its covariance structure) and the mechanism.

As with IGCI and nonlinear ICA, it can be seen by comparing (16) and (11) that the trace method assumes the same generative model as linear ICA (where the cause $\mathbf{c}$ corresponds to the independent sources $\mathbf{s}$ and the effect to the observed mixtures $\mathbf{x}$). While the focus of the present work is on nonlinear ICA, we briefly discuss the usefulness of the trace method as a constraint for achieving identifiability in a linear ICA setting.

As is clear from (17), the trace condition is trivially satisfied if the covariance matrix of the sources (causes) is the identity, $\bm{\Sigma}=\mathbf{I}$. However, as explained in § A.3, in the context of linear ICA this can easily be achieved by whitening the data. As with IGCI, the trace method was developed for cause-effect inference where both variables are observed, and thus relies on the observed cause distribution being informative. This renders is unsuitable (on its own) to constrain the unsupervised representation learning problem of linear ICA problem where the sources are unobserved.

Note, however, that this is qualitatively different from the IGCI argument presented in § 3, as whitening on its own does not necessarily lead to independent variables, but only uncorrelated ones, and thus does not solve linear ICA—unlike the Darmois construction in the case of nonlinear ICA which also yields independent components.

B.2 Information geometric interpretation of the ICM principle

There is a well-established connection between IGCI and the trace method . At the heart of this derivation lies an information-geometric interpretation of the ICM principle for probability distributions, which we sketch in this section. First, we need to review some basic concepts.

An interesting property of the KL divergence is its invariance to reparametrisation. Consider an invertible transformation $h$, mapping random variables $X$ and $Y$ to $h(X)$ and $h(Y)$, respectively (the domains and codomains being arbitrary spaces, e.g., discrete or Euclidean of arbitrary dimension). Then the KL divergence between $P_{X}$ and $P_{Y}$ is preserved by the pushforward operation implemented by $h$, such that

Consider a deterministic causal relationship of the form $Y:=f(X)$, and denote by $P_{X}$ and $P_{Y}$ the marginal distributions of the cause $X$ and the effect $Y$, respectively. The “irregularity” of each distribution can be quantified by evaluating their divergence to a reference set $\mathcal{E}$ of “regular” distributions,Here “regular” is only meant in an intuitive sense, not implying any further mathematical notion. If $\mathcal{E}$ is the set of Gaussians, for instance, the distance from $\mathcal{E}$ measures non-Gaussianity.

Let us assume that these infima are reached at a unique point, their projections onto $\mathcal{E}$:

As elaborated in [46, §4], the choice of $\mathcal{E}$ is context-dependent. For example, in the context of the trace method , $X$ and $Y$ are assumed to be $n$-dimensional multivariate Gaussian random vectors, and $\mathcal{E}$ is taken as the set of multivariate isotropic Gaussian distributions. In contrast, when IGCI is applied in contexts where the considered mechanism is a deterministic non-linear diffeomorphism, the reference distributions are typically uniform distributions .

Overall, it can be shown that the independence postulate underlying these approaches leads to the following decomposition of the irregularity of $P_{Y}$ (see [46, Thm. 2]):

where $\vec{P}_{Y}$ denotes the distribution of $f(U_{X})$, i.e., the hypothetical distribution of the effect that would be obtained if the cause $X$ were replaced by the random variable $U_{X}$ (which corresponds to the closest regularly distributed random variable to $X$).

Since applying the bijection $f^{-1}$ preserves the KL divergences, see (18), we can obtain the equivalent relation

This relation can be interpreted as an orthogonality principle in information space by considering the KL divergences as a generalization of the squared Euclidean norm for the difference vectors $\overrightarrow{P_{Y}U_{Y}}$, $\overrightarrow{P_{Y}\vec{P_{Y}}}$ and $\overrightarrow{\vec{P_{Y}}U_{Y}}$. It can thus be viewed as a Pythagorean theorem in the space of distributions, see Fig. 6 for an illustration.

The orthogonality principle (19) thus captures a decomposition of the irregularity $D_{KL}(P_{Y}\|U_{Y})$ of $P_{Y}$ on the LHS into the sum of two irregularities on the RHS: the irregularity $D_{KL}(P_{X}\|U_{X})$ of $P_{X}$, and the term $D_{KL}(\vec{P}_{Y}\|U_{Y})$ which measures the irregularity of the mechanism $f$ indirectly, via the “irregularity” of the distribution resulting from applying $f$ to a regular distribution $U_{Y}$.

Overall, the decomposition (19) links the postulate of independence between the cause distribution, on the one hand, and the mechanism, on the other hand, to an orthogonality of their irregularities in information space (namely the statistical manifold of information geometry). As proposed in , this can be intuitively interpreted as a geometric form of independence if we assume that Nature chooses such irregularities independently of each other, and “isotropically” in a high-dimensional subspace of irregularities.

While, to date, we are not aware of similar results in the context of information geometry (i.e., on the statistical manifold), this intuition is supported by concentration of measure results in Euclidean spaces. Indeed, in high-dimensions, it is likely that two vectors are close to orthogonal if they are chosen independently according to a uniform prior .

We will take inspiration of the decomposition (19) to justify IMA in the following section.

B.3 Decoupling of the influences in IMA and comparison with IGCI

In contrast to § B.2, in this section we will, for notational consistency with the main paper, assume that all distributions have a density with respect to the Lebesgue measure, and thus consider, with a slight abuse of notation, that the KL divergence is a distance between two densities on the relevant support, such that

In line with the information-geometric interpretation of IGCI presented in § B.2, we also consider an interpretation of IMA in information space. We consider the KL-divergence between the observed density $p_{\mathbf{x}}$ of $\mathbf{x}=\mathbf{f}(\mathbf{s})$ and an interventional distribution $p_{\widehat{\mathbf{x}}}$ of $\widehat{\mathbf{x}}=\widehat{\mathbf{f}}(\mathbf{s})$, resulting from a soft intervention that replaces the mixing function $\mathbf{f}$ with another mixing $\widehat{\mathbf{f}}$. We take $D_{KL}(p_{\mathbf{x}}\|p_{\widehat{\mathbf{x}}})$ as a measure of the causal effect of the soft intervention (or perturbation) that turns $\mathbf{f}$ into $\widehat{\mathbf{f}}$—similarly to how $D_{KL}(P_{Y}\|U_{Y})$ is used as a measure of the irregularity of the effect distribution in the context of IGCI (§ B.2).

As we will show, under suitable assumptions, the functional form imposed on $\mathbf{f}$ by the IMA Principle 4.1 can lead to a decomposition of the causal effect of an intervention on the mechanism into a sum of terms, corresponding to the causal effects of separate soft interventions on the mechanisms associated to each source. In contrast, IGCI decomposes irregularities of the effect distribution into two terms, one irregularity of the cause and one irregularity of the mechanism.

Assume $\mathbf{f}$ satisfies the IMA principle. We consider interventions performed through the element-wise transformation $\bm{\sigma}$ such that

This can be seen as a composition of $n$ soft interventions $\{\bm{\sigma}_{j}\}$ on each individual source component $j$, implemented through univariate smooth diffeomorphisms ${\sigma}_{j}$, such that

and $\bm{\sigma}=\bm{\sigma}_{n}\circ\cdots\circ\bm{\sigma}_{1}$ (in arbitrary order, since the individual $\bm{\sigma}_{j}$ commute). This soft intervention can be seen as turning the random variable $\mathbf{s}$ into $\widehat{\mathbf{s}}$, yielding the intervened observations $\widehat{\mathbf{x}}=\mathbf{f}(\widehat{\mathbf{s}})$. Alternatively, the intervention on $\mathbf{x}$ can be implemented by replacing $\mathbf{f}$ by $\widehat{\mathbf{f}}=\mathbf{f}\circ\bm{\sigma}$—i.e., $\widehat{\mathbf{x}}=\widehat{\mathbf{f}}(\mathbf{s})$. Notably, since $\mathbf{f}$ satisfies the IMA principle, so does $\widehat{\mathbf{f}}$ (due to Prop. 4.6, (ii), since $\bm{\sigma}$ is an element-wise nonlinearity). Moreover, the partial derivatives of the intervened function are given by

The classical change of variable formula for bijection $\mathbf{f}$ yields the expression of the pushforward density of $\mathbf{x}$ as

Let us now compute the KL divergence between the intervened and observed distribution,

Expressing the density of the observed variables as a pushforward of the density of the sources, and without additional assumptions on $\mathbf{f}$ and $\widehat{\mathbf{f}}$ besides smoothness and invertibility, we get,

We now consider a factorization of $\mathbf{s}$ over a directed acyclic graph (DAG), such that

where $\mbox{pa}(s_{j})$ denotes the components associated to the parents of node $j$ in the DAG. Because $\bm{\sigma}$ is an element-wise transformation the factorization will be the same for $p_{\widehat{\mathbf{s}}}$.

If we now additionally assume that $\mathbf{f}$ and $\widehat{\mathbf{f}}$ satisfy the IMA postulate, we get

By reparameterizing the integral in terms of the source coordinates, we get (using $\widehat{\mathbf{f}}^{-1}=\bm{\sigma}^{-1}\circ{\mathbf{f}}^{-1}$)

such that the $KL$ divergence can be written as a sum of $n$ terms, each associated to the intervention on a mechanism $\frac{\partial\mathbf{f}}{\partial s_{i}}$. Positivity of these terms would suggest that we can interpret each of them as quantifying the individual contribution of a soft intervention $\bm{\sigma}_{j}$ applied to the original sources.

In the following, we propose a justification for the positivity of these terms in a restricted setting where only the $m$ leaf nodes of the graph are intervened on (with $1\leq m\leq n$).A leaf node in a DAG is one that does not have any descendants. In the special case of independent sources, all nodes are leaves and $m=n$.

Under this assumption, we consider (without loss of generality) an ordering of the nodes such that the $m$ first nodes are the leaf nodes in the DAG. Then we argue that the terms of the right-hand side of (21) associated to leaf nodes ($i\leq m$) are positive, as they correspond to the expectations of KL-divergences. Indeed, taking one of the first $m$ terms, denoted $i$, we have the factorization

where $\prod_{j\neq i}p_{j}(s_{j}|\mbox{pa}(s_{j}))$ does not depend on $s_{i}$ because node $i$ is a leaf node. Moreover, as non-leaf nodes are not intervened on, the transformation $\bm{\sigma}$ does not modify the value of any parent variables in these factorizations. As a consequence, the integral can be computed as an iterated integral with respect to $s_{i}$ and $\mathbf{s}_{-i}\,$, where $\mathbf{s}_{-i}$ denotes the vector including all source variables but $s_{i}$, such that

Similarly, we get the expression of the pushfoward distribution $\widehat{\mathbf{f}}_{*}p_{i}(\,.\,|\bm{\sigma}^{-1}(\mbox{pa}(s_{i})))$ from $\mathcal{L}_{\bm{\sigma}^{-1}(\mathbf{s}_{-i})}$ to the curve $\mathbf{f}[\mathcal{L}_{\mathbf{s}_{-i}}]$ (using again the fact that parent variables are not intervened on, and thus left unchanged by $\bm{\sigma}$)

These terms appear when rewriting the $i$-th term (for a leaf variable) in (21) as a curvilinear integral:

The inner integral term can thus be interpreted as the KL divergence between two pushforward measures defined on $\mathbf{f}_{*}[\mathcal{L}_{\mathbf{s}_{-i}}]$ by $\mathbf{f}$ and $\widehat{\mathbf{f}}$, that we can denote by

To conclude, this implies that the causal effect of the soft intervention $\mathbf{f}\rightarrow\widehat{\mathbf{f}}$ can be decomposed as the following sum of $m$ positive terms associated to interventions on each leaf variable, plus an additional term for the remaining non-leaf variables, which further simplifies (in comparison to (21)) due to the assumption that those variables are unintervened.

This expression suggests that the KL-divergences appearing in the first $m$ terms each reflect the causal effect of an intervention on the mechanism at the level of one single source coordinate $i$, turning $\frac{\partial{\mathbf{f}}}{\partial s_{i}}$ into $\frac{\partial\widehat{\mathbf{f}}}{\partial s_{i}}$. When the sources are jointly independent, we have $m=n$ and the right hand side of (22) contains only positive terms. An interesting direction for future work would be to analyse the remaining term in the case of non unconditionally independent sources.

In contrast to the decomposition (19) in the context of IGCI, the IMA decomposition (22) involves $m$ (expectations of) KL-divergence terms instead of two, each related to the intervention on the part of the mechanism $\frac{\partial\mathbf{f}}{\partial s_{i}}$ that reflects the influence of a single source.

B.4 Independence of cause and mechanism and IMA

We now discuss an example in which a formalisation of the principle of independence of cause and mechanism is violated, and one in which the IMA principle is violated.

In the context of the Trace method , used in causal discovery, a technical example of fine-tuning can be constructed by taking a vector of i.i.d. random variables with arbitrary (not diagonal) covariance matrix $\Sigma$ as the cause, and by constructing the mechanism as a whitening matrix, turning the cause variables into uncorrelated (effect) variables. By doing so, the singular values and singular vectors of the matrix (the mechanism) are fine-tuned to the input covariance matrix (a property of the cause distribution), and such fine-tuning can be quantified via the Trace method (see , Section 1).

B.4.2 Violations of the IMA principle

As mentioned in § 3, an example of a mixing function $\mathbf{f}$ which is non-generic according to the IMA principle is an autoregressive function, for example an autoregressive normalising flow , where the $k$-th component of the observations only depends on the $k$-th sources: intuitively, this would correspond to the unlikely cocktail-party setting where the $k$-th microphone only picks up the voices of the first speakers. More precisely, as we show in Lemma C.1, this leads to positive $C_{\textsc{ima}}$ value for such mixing.

A cocktail party (Fig. 1, left) may violate our IMA principle when the locations of several speakers and the room acoustics have been fine tuned to one another. This is for example the case in concert halls where the acoustics of the room have been fine-tuned to the position and configuration of multiple locations on the stage, where the sources (i.e., the voices of the actors or singers) are emitted—in order to make the listening experience as homogeneous as possible across the spectators (that is, the influence of each of the sources on the different listeners should not differ too much). This would lead to an increase in collinearity between the columns of the mixing’s Jacobian, thus violating the IMA principle.

Additionally, we recall that the ICM principle is often informally introduced by referencing the fine-tuning and non-generic viewpoints giving rise to certain visual illusions, such as the Beuchet chair (see , Section 2); in a similar vein, we can imagine that violations of the IMA principle in the cocktail party setting may be related to illusions in binaural hearing such as for example the Franssen effect, where the listener is tricked into incorrectly localizing a sound .

Appendix C Proofs

We now provide the proofs of all our theoretical results from the main paper.

Before giving the proof, it is useful to rewrite the local IMA constrast (8) as follows:

where the quantity in (23) is called the left KL measure of diagonality of the matrix $\mathbf{J}^{\top}_{\mathbf{f}}(\mathbf{s})\mathbf{J}_{\mathbf{f}}(\mathbf{s})$ (see Remark 4.3):

From (23), it can be seen that $c_{\textsc{ima}}(\mathbf{f},\mathbf{s})$ is a function of $\mathbf{J}_{\mathbf{f}}(\mathbf{s})$ only through $\mathbf{J}^{\top}_{\mathbf{f}}(\mathbf{s})\mathbf{J}_{\mathbf{f}}(\mathbf{s})$.

For ease of exposition, we denote the value of the Jacobian of $\mathbf{f}$ evaluated at the point $\mathbf{s}$ by $\mathbf{J}_{\mathbf{f}}(\mathbf{s})=\mathbf{W}$. The two properties can then be proved as follows:

This is a consequence of Hadamard’s inequality, applied to the expression on the RHS of (8), which states that, for a matrix $\mathbf{W}$ with columns $\mathbf{w}_{i}$, $\sum_{i=1}^{n}\log\left\lVert\mathbf{w}_{i}\right\rVert\geq\log|\mathbf{W}|$; equality in Hadamard’s inequality is achieved iff. the vectors $\mathbf{w}_{i}$ are orthogonal.

C.2 Proof of Prop. 4.6

From property (i) of Prop. 4.4, we know that $c_{\textsc{ima}}(\mathbf{f},\mathbf{s})\geq 0$. Hence, $C_{\textsc{ima}}(\mathbf{f},p(\mathbf{s}))\geq 0$ follows as a direct consequence of integrating the non-negative quantity $c_{\textsc{ima}}(\mathbf{f},\mathbf{s})$.

Equality is attained iff. $c_{\textsc{ima}}(\mathbf{f},\mathbf{s})=0$ almost surely w.r.t. $p_{\mathbf{s}}$, which according to property (i) of Prop. 4.4 occurs iff. the columns of $\mathbf{J}_{\mathbf{f}}(\mathbf{s})$ are orthogonal almost surely w.r.t. $p_{\mathbf{s}}$.

It remains to show that this is the case iff. $\mathbf{J}_{\mathbf{f}}(\mathbf{s})$ can be written as $\mathbf{O}(\mathbf{s})\mathbf{D}(\mathbf{s})$, with $\mathbf{O}(\mathbf{s})$ and $\mathbf{D}(\mathbf{s})$ orthogonal and diagonal matrices, respectively. (To avoid confusion, note that orthogonal columns need not have unit norm, whereas an orthogonal matrix $\mathbf{O}$ satisfies $\mathbf{O}\mathbf{O}^{\top}=\mathbf{I}$.)

The if is clear since right multiplication by a diagonal matrix merely re-scales the columns, and hence does not affect their orthogonality.

For the only if, let $\mathbf{J}_{\mathbf{f}}(\mathbf{s})$ be any matrix with orthogonal columns $\mathbf{j}_{i}(\mathbf{s})$, $\mathbf{j}_{i}(\mathbf{s})^{\top}\mathbf{j}_{j}(\mathbf{s})=0,\forall i\neq j$, and denote the column norms by $d_{i}(\mathbf{s})=||\mathbf{j}_{i}(\mathbf{s})||$. Further denote the normalised columns of $\mathbf{J}_{\mathbf{f}}(\mathbf{s})$ by $\mathbf{o}_{i}(\mathbf{s})=\mathbf{j}_{i}(\mathbf{s})/d_{i}(\mathbf{s})$ and let $\mathbf{O}(\mathbf{s})$ and $\mathbf{D}(\mathbf{s})$ be the orthogonal and diagonal matrices with columns $\mathbf{o}_{i}(\mathbf{s})$ and diagonal elements $d_{i}(\mathbf{s})$, respectively. Then $\mathbf{J}_{\mathbf{f}}(\mathbf{s})=\mathbf{O}(\mathbf{s})\mathbf{D}(\mathbf{s})$.

where, for the second equality, we have used the fact that

since $\mathbf{P}\circ\mathbf{h}$ is an invertible tranformation (see, e.g., ). It thus suffices to show that

where we have repeatedly used the chain rule for Jacobians, as well as that $\mathbf{P}^{-1}\mathbf{P}=\mathbf{I}$; that permutation is a linear operation, so $\mathbf{J}_{\mathbf{P}}(\mathbf{s})=\mathbf{P}$ for any $\mathbf{s}$; and that $\mathbf{h}$ (and thus $\mathbf{h}^{-1}$) is an element-wise transformation, so the Jacobian $\mathbf{J}_{\mathbf{h}^{-1}}$ is a diagonal matrix $\mathbf{D}(\mathbf{s})$.

The equality in (25) then follows from (26) by applying property (ii) of Prop. 4.4, according to which $c_{\textsc{ima}}$ is invariant to right multiplication of the Jacobian $\mathbf{J}_{\mathbf{f}}(\mathbf{s})$ by diagonal and permutation matrices.

Substituting (25) into the RHS of (24), we finally obtain

C.3 Remark on a similar condition to IMA, expressed in terms of the rows of the Jacobian

We remark that the condition imposed by the IMA Principle 4.1 needs to be expressed in terms of the columns of the Jacobian, and would not lead to a criterion with desirable properties for BSS if it were instead expressed in terms of its rows (which correspond to gradients of the $f_{i}(\mathbf{s})$). One way to justify this is that, for the same condition expressed on the rows of the Jacobian, that is

property (ii) of Prop. 4.4 would not hold (because invariance would hold w.r.t. right, not left, multiplication with a diagonal matrix). As a consequence, the resulting global contrast would not be blind to reparametrisation of the source variables by permutation and element-wise invertible transformations, thereby not being a good contrast in the context of blind source separation.

C.4 Proof of Thm. 4.7

Before proving the main theorem, we first introduce some additional details on the Jacobian of the Darmois construction which will be important for the proof.

Consider the Darmois construction for $n=2$,

In the general case, the Jacobian of the Darmois construction will be

where the components $c_{ji}(\mathbf{x}_{1:j})$ of $\mathbf{C}(\mathbf{x})$ for all $i<j$ are defined by

It is additionally useful to introduce the following lemmas.

A function $\mathbf{f}$ with triangular Jacobian has $C_{\textsc{ima}}(\mathbf{f},p_{\mathbf{s}})=0$ iff. its Jacobian is diagonal almost everywhere. Otherwise, $C_{\textsc{ima}}(\mathbf{f},p_{\mathbf{s}})>0$.

Let $\mathbf{f}$ have lower triangular Jacobian at $\mathbf{s}$, and denote $\mathbf{J}_{\mathbf{f}}(\mathbf{s})=\mathbf{W}$. Then we have

where $w_{ji}=[\mathbf{W}]_{ji}$. Since the logarithm is a strictly monotonically increasing function and since

with equality iff. $w_{ji}=0,\forall j\neq i$ (i.e., iff. $\mathbf{W}$ is a diagonal matrix), we must have $c_{\textsc{ima}}(\mathbf{f},\mathbf{s})=0$ iff. $\mathbf{W}$ is diagonal.

$C_{\textsc{ima}}(\mathbf{f},p_{\mathbf{s}})$ is therefore equal to zero iff. $\mathbf{f}$ has diagonal Jacobian almost everywhere, and it is strictly larger than zero otherwise. ∎

Let $\mathbf{f}$ be a smooth function with diagonal Jacobian everywhere.

Consider the function $f_{i}(\mathbf{s})$ for any $i\in\{1,...,n\}$. Suppose for a contradiction that $f_{i}$ depends on $s_{j}$ for some $j\neq i$. Then there must be at least one point $\mathbf{s}^{*}$ such that $\nicefrac{{\partial f_{i}}}{{\partial s_{j}}}(\mathbf{s}^{*})\neq 0$. However, this contradicts the assumption that $\mathbf{J}_{\mathbf{f}}$ is diagonal everywhere (since $\nicefrac{{\partial f_{i}}}{{\partial s_{j}}}$ is an off-diagonal element for $i\neq j$). Hence, $f_{i}$ can only depend on $s_{i}$ for all $i$, i.e., $\mathbf{f}$ is an element wise function. ∎

First, the Jacobian $\mathbf{J}_{\mathbf{g}^{\text{D}}}(\mathbf{x})$ of the Darmois construction $\mathbf{g}^{\text{D}}$ is lower triangular $\forall\mathbf{x}$, see (28).

Because CDFs are monotonic functions (strictly monotonically increasing given our assumptions on $\mathbf{f}$ and $p_{\mathbf{s}}$), $\mathbf{g}^{\text{D}}$ is invertible.

We can thus apply the inverse function theorem (with $\mathbf{f}^{\text{D}}=(\mathbf{g}^{\text{D}})^{-1}$) to write

Since the inverse of a lower triangular matrix is lower triangular, we conclude that $\mathbf{J}_{\mathbf{f}^{\text{D}}}(\mathbf{y})$ is lower triangular for all $\mathbf{y}=\mathbf{g}^{\text{D}}(\mathbf{x})$.

Now, according to Lemma C.1, we have $C_{\textsc{ima}}(\mathbf{f}^{\text{D}},p_{\mathbf{u}})>0$, unless $\mathbf{J}_{\mathbf{f}^{\text{D}}}$ is diagonal almost everywhere.

Suppose for a contradiction that $\mathbf{J}_{\mathbf{f}^{\text{D}}}$ is diagonal almost everywhere.

Since $\mathbf{f}$ and $p_{\mathbf{s}}$ are smooth by assumption, so is the push-forward $p_{\mathbf{x}}=\mathbf{f}_{*}p_{\mathbf{s}}$, and thus also $\mathbf{g}^{\text{D}}$ (CDF of a smooth density) and its inverse $\mathbf{f}^{\text{D}}$. Hence, the partial derivatives $\nicefrac{{\partial f^{\text{D}}_{i}}}{{\partial y_{j}}}$, i.e., the elements of $\mathbf{J}_{\mathbf{f}^{\text{D}}}$ are continuous.

Consider an off-diagonal element $\nicefrac{{\partial f^{\text{D}}_{i}}}{{\partial y_{j}}}$ for $i\neq j$. Since these are zero almost everywhere, and because continuous functions which are zero almost everywhere must be zero everywhere, we conclude that $\nicefrac{{\partial f^{\text{D}}_{i}}}{{\partial y_{j}}}=0$ everywhere for $i\neq j$, i.e., the Jacobian $\mathbf{J}_{\mathbf{f}^{\text{D}}}$ is diagonal everywhere.

Hence, we conclude from Lemma C.2 that $\mathbf{f}^{\text{D}}$ must be an element-wise function, $\mathbf{f}^{\text{D}}(\mathbf{y})=(f^{\text{D}}_{1}(y_{1}),...,f^{\text{D}}_{1}(y_{n}))$.

Since $\mathbf{y}$ has independent components by construction, it follows that $x_{i}=f^{\text{D}}_{i}(y_{i})$ and $x_{j}=f^{\text{D}}_{j}(y_{j})$ are independent for any $i\neq j$.

However, this constitutes a contradiction to the assumption that x_{i}\not\mathchoice{\mathrel{\hbox to0.0pt{\displaystyle\perp\hss}\mkern 2.0mu{\displaystyle\perp}}}{\mathrel{\hbox to0.0pt{\textstyle\perp\hss}\mkern 2.0mu{\textstyle\perp}}}{\mathrel{\hbox to0.0pt{\scriptstyle\perp\hss}\mkern 2.0mu{\scriptstyle\perp}}}{\mathrel{\hbox to0.0pt{\scriptscriptstyle\perp\hss}\mkern 2.0mu{\scriptscriptstyle\perp}}}x_{j} for some $x_{j}$.

We conclude that $\mathbf{J}_{\mathbf{f}^{\text{D}}}$ cannot be diagonal almost everywhere, and hence, by Lemma C.1, we must have $C_{\textsc{ima}}(\mathbf{f}^{\text{D}},p_{\mathbf{u}})>0$. ∎

C.5 Proof of Corollary 4.9

C.6 Proof of Corollary 4.11

Since, by assumption, the mixing matrix is non-trivial (i.e., not the product of a diagonal and permutation matrix), and at most one of the $s_{i}$ is Gaussian, according to Thm. A.1 there must be at least one pair $x_{i},x_{j}$, with $i\neq j$, such that $x_{i}\not\!\perp\!\!\!\perp x_{j}$.

We can then use the same argument as in the proof of Thm. 4.7 to show that the Darmois construction has nonzero $C_{\textsc{ima}}$, whereas the linear orthogonal transformation $\mathbf{A}$ has orthogonal Jacobian, and thus $C_{\textsc{ima}}=0$ by property (i) of Prop. 4.6. ∎

C.7 Proof of Thm. 4.12

For notational convenience, we denote $\bm{\sigma}=\bm{\Phi}^{-1}\circ\mathbf{F}_{\mathbf{s}}$ and write

Note that, since both $\mathbf{F}_{\mathbf{s}}$ and $\bm{\Phi}$ are element-wise transformations, so is $\bm{\sigma}$.

First, by using property (ii) of Prop. 4.6 (invariance of $C_{\textsc{ima}}$ to element-wise transformation), we obtain

with $\mathbf{z}=\bm{\sigma}(\mathbf{s})$ such that $p_{\mathbf{z}}$ is an isotropic Gaussian distribution.

Suppose for a contradiction that $C_{\textsc{ima}}(\mathbf{f}\circ\bm{\sigma}^{-1}\circ\mathbf{R},p_{\mathbf{z}})=0$.

According to property (i) of Prop. 4.6, this entails that the matrix

Since $(\mathbf{f},p_{\mathbf{s}})\in\mathcal{M}_{\textsc{ima}}$ by assumption, by property (i) of Prop. 4.6, the inner term on the RHS of (29),

is diagonal. Moreover, since $\bm{\sigma}$ is an element-wise transformation, $\mathbf{J}_{\bm{\sigma}^{-1}}(\mathbf{z})^{\top}$ and $\mathbf{J}_{\bm{\sigma}^{-1}}(\mathbf{z})$ are also diagonal. Taken together, this implies that

is diagonal (i.e., (29) is of the form $\mathbf{R}^{\top}\mathbf{D}(\mathbf{z})\mathbf{R}$ for some diagonal matrix $\mathbf{D}(\mathbf{z})$).

Without loss of generality, we assume the first component $s_{1}$ of $\mathbf{s}$ is non-Gaussian and satisfies the assumptions stated relative to $\mathbf{R}$ (axis not invariant nor sent to another canonical axis).

Now, since both the Gaussian CDF $\bm{\Phi}$ and the CDF $\mathbf{F}_{\mathbf{s}}$ are smooth (the latter by the assumption that of $p_{\mathbf{s}}$ is a smooth density), $\bm{\sigma}$ is a smooth function, and thus has continuous partial derivatives.

By continuity of the partial derivative, the first diagonal element $\frac{\partial\sigma_{1}^{-1}}{\partial z_{1}}$ of $\mathbf{J}_{\bm{\sigma}^{-1}}$ must be strictly monotonic in a neighborhood of some $z_{1}^{0}$ (otherwise $\sigma_{1}$ would be an affine transformation, which would contradict non-Gaussianity of $s_{1}$).

On the other hand, our assumptions relative to $\mathbf{R}$ entail that there are at least two non-vanishing coefficients in the first row of $\mathbf{R}$ (i.e., first column of $\mathbf{R}^{\top}$).In short, if this were not the case, this column would have a single non-vanishing coefficient, which would need to be one due to the unit norm of the rows of this orthogonal matrix. Such structure of the matrix $\mathbf{R}$ would entail that the associated canonical basis vector $\mathbf{e}_{1}$ is transformed by $\mathbf{R}^{-1}=\mathbf{R}^{\top}$ into a canonical basis vector $\mathbf{e}_{j}$ which contradicts the assumptions. Let us call $i\neq j$ such pair of coordinates, i.e., $r_{1j}\neq 0$ and $r_{1i}\neq 0$.

Now consider the off-diagonal term $(i,j)$ of (29), which we assumed (for a contradiction) must be zero almost surely w.r.t. $p_{\mathbf{z}}$. Since the term in (30) is diagonal, this off-diagonal term is given by:

where for the first equality we have used the fact that $\mathbf{f}$ is a conformal map with conformal factor $\lambda(\mathbf{s})$ (by assumption), and where the second equality must hold almost surely w.r.t. $p_{\mathbf{z}}$.

Since $\mathbf{f}$ is invertible, it has non vanishing Jacobian determinant. Hence, the conformal factor $\lambda$ must be a strictly positive function, so

Thus, for almost all $\mathbf{z}$, we must have:

Now consider the first term $\left(\frac{d\sigma_{1}^{-1}}{dz_{1}}(z_{1})\right)^{2}r_{1i}r_{1j}$ in the sum.

Recall that $r_{1i}r_{1j}\neq 0$, and that $\frac{d\sigma_{1}^{-1}}{dz_{1}}(z_{1})$ is strictly monotonic on a neighborhood of $z_{1}^{0}$.

As a consequence, $\left(\frac{d\sigma_{1}^{-1}}{dz_{1}}(z_{1})\right)^{2}r_{1i}r_{1j}$ is also strictly monotonic with respect to $z_{1}$ on a neighborhood of $z_{1}^{0}$ (where the other variables $(z_{2},...,z_{n})$ are left constant), while the other terms in the sum in (31) are left constant because $\bm{\sigma}$ is an element-wise transformation.

This leads to a contradiction as (31) (which should be satisfied for all $\mathbf{z}$) cannot stay constantly zero as $z_{1}$ varies within the neighbourhood of $z_{1}^{0}$.

Hence our assumption that $C_{\textsc{ima}}(\mathbf{f}\circ\mathbf{a}^{\mathbf{R}}(p_{\mathbf{s}}),p_{\mathbf{s}})=0$ cannot hold.

We conclude that $C_{\textsc{ima}}(\mathbf{f}\circ\mathbf{a}^{\mathbf{R}}(p_{\mathbf{s}}),p_{\mathbf{s}})>0$. ∎

Appendix D Worked out example

Consider the following example of a nonlinear ICA model which represents a change of basis from polar to Cartesian coordinates:

First, we consider the Jacobian of the true mixing $\mathbf{f}$ which is given by:

and its determinant and column norms are given by

In other words, the columns of $\mathbf{J}_{\mathbf{f}}(\mathbf{s})$ are orthogonal for all $\mathbf{s}$, so that $C_{\textsc{ima}}=0$ for the true solution.

First, we write the joint density of $(x_{1},x_{2})$ using the change of variable formula:

Next, we compute the marginal density $p(x_{1})$. Note that the observations $\mathbf{x}$ live on the disk of radius $R$, $\left\lVert\mathbf{x}\right\rVert\leq R$, so $p(x_{1},x_{2})=0$ whenever $x_{1}^{2}+x_{2}^{2}>R^{2}$.

Applying the change of variable $t=\frac{x_{2}}{x_{1}}$ with $dt=\frac{dx_{2}}{x_{1}}$, and using the integral $\int(1+t^{2})^{-\frac{1}{2}}dt=\operatorname{arcsinh}(t)+C$, as well as the fact that $\operatorname{arcsinh}$ is an odd function, we obtain

Next, we compute the conditional density $p(x_{2}|x_{1})$:

Finally, we compute the off-diagonal term in the general form of the inverse Jacobian for Damois-style solutions in (27):

Using the derivative $\frac{\partial}{\partial t}\operatorname{arcsinh}(t)=(t^{2}+1)^{-\frac{1}{2}}$ and repeatedly applying the chain rule, we obtain:

Again, recall that this only holds inside the disk of radius $R$, otherwise $c_{12}=0$ (as the CDF will be zero or one, irrespective of $x_{1}$).

The $C_{\textsc{ima}}$ for the Darmois solution thus takes the form:

where the strict inequality in the last step follows from the fact that the fraction inside the logarithm, and hence the entire integrand, is strictly positive within the disk of integration.

We have thus shown that for the example of an orthogonal coordinate transformation from polar to Cartesian coordinates, which is not a conformal map, the $C_{\textsc{ima}}$ os the true solution is zero and that of the Darmois construction is strictly greater than zero, hence the two can be distinguished based on the value of the $C_{\textsc{ima}}$ contrast.

Appendix E Experiments

The code for our experiments (enclosed in the supplemental material) is in Python; we use Jax , Distrax and Haiku to implement our models; the Jacobian and $C_{\textsc{ima}}$ computation and optimisation are performed with the automatic differentiation tools provided in Jax.

E.2 How to implement the Darmois construction

In the following, we describe how the Darmois construction can be implemented based on normalising flow models . The key idea is that the components $g_{i}^{\text{D}}$ of the Darmois construction (4) are conditional (cumulative) density functions corresponding to a given factorisation $p(\mathbf{x})=\prod_{i=1}^{n}p(x_{i}|\mathbf{x}_{1:i-1})$ of the likelihood. A flow model with triangular Jacobian can be used to maximise the likelihood of the observations under a change of variable respecting said factorisation, and learning to map the observed variables onto a given (factorised) base distribution. After training, and provided that the model is expressive enough, the CDF of each component of the reconstructed sources should match that of the base distribution. By further transforming each reconstructed variable through said CDF, we achieve a global mapping of the observations onto a Uniform distribution on the $n$-dimensional hypercube, with a triangular Jacobian, matching the transformation operated by the Darmois construction (see also see , section 2.2). Note that, for the purpose of computing the $C_{\textsc{ima}}$ of the Darmois construction, this final step can be omitted due to Prop. 4.6, (ii), stating that the contrast is blind to element-wise reparametrisations of the sources.

We remark that, while the possibility of using normalising flows to “learn” the Darmois construction is mentioned in , where a similar construction is mentioned in a theoretical argument to prove “universal approximation capacity for densities” for normalising flow models with triangular Jacobian, it has to the best of our knowledge not been tested empirically, since autoregressive modules with triangular Jacobian are typically used in combination with permutation, shuffling or linear layers which overall lead to architectures with a non-triangular Jacobian.

To obtain an expressive normalizing flow with triagular Jacobian, we modify the residual flow model .We describe how to implement a function with upper triangular Jacobian, but the reasoning can be extended to implement functions whose Jacobian is lower triangular. A residual flow is a residual network which is made invertible through spectral normalization. Each layer is given by

Here, $h_{i}$ is the number of hidden units dedicated to transforming $\mathbf{z}_{i}$ with the constraint $\sum_{i=1}^{n}h_{i}=h$. We perform an even split such that the $h_{i}$ and $h_{j}$ differ by at most 1 for $i,j\in\{1,...,n\}$. The weight matrices are restricted to be block triangular during optimization by setting the respective matrix elements to zero after each iteration of the optimizer. The model can simply be made and kept invertible using the same spectral normalization as is used for dense residual flows . We train our model to map onto a standard Normal base distribution.

E.3 Generating random MLP mixing functions

In order to generate random MLP mixing functions, we adopt the same initalisation as in : we initialise the square weight matrices to be orthogonal,Note that orthogonality of the weight matrices in a MLP does not guarantee satisfying Principle 4.1, due to the element-wise nonlinearities between the layers, which overall lead to a Jacobian whose columns are in general not orthogonal. and use the leaky_tanh invertible nonlinearity.

The modified maximum likelihood objective described in § 5.2 can be written as follows:while the objective in § 5.2 involves an expectation over $p_{\mathbf{x}}$, we consider the loss for a single point $\mathbf{x}$ here, $\mathcal{L}(\mathbf{g};\mathbf{x})$.

where $[\mathbf{J}^{-1}_{\mathbf{g}}(\mathbf{x})]_{i}$ represents the $i$-th column of the inverse of the Jacobian of $\mathbf{g}$ computed at $\mathbf{x}$.

We use the same model as the one described in § E.2, but without the constraint that the Jacobian should be triangular, and train with a Logistic base distribution.

Note that the computational efficiency of optimising objective (38) is cubic in the input size $n$, due to a number of operations (matrix inversion, Jacobian and determinant computation via automatic differentiation, etc.) which are $\mathcal{O}(n^{3})$. However, similarly to what already observed in , we found that for data of moderate dimensionality computing and optimising objective (38) with automatic differentiation is feasible. For example, training a residual flow with 64 layers for $10^{5}$ iterations takes roughly 5.3 hours for $n=2$, 5.7 hours for $n=5$, and 6.3 hours for $n=7$ on the same hardware (see section E.5). An interesting direction for future work would be to find computationally efficient ways of optimising (38).